A few days ago in a post titled Twitter & The Global Brain I blogged about the parallels between twitter and giant neural network. Now I want to flesh out that model and make it a little more tangible by describing an app that I call TweetStream that could potentially solve several of twitter’s current problems:

  1. Taming the torrent of information that blasts current twitter users.
  2. Monetizing content and rewarding participation in the twitter experience.
  3. Moving the global system towards a coordinated efficient information exchange framework through which global consciousness crise and be exercised.

The TweetStream App

Imagine an app that provided not only the chronological list of friends updates that are is currently provided by Seesmic and Tweetdeck, but also provides what I’ll call a “personalized tweet stream”. My personalized tweet stream would be composed of two parts, presented by the App in two separate column – my “Input” stream and my “Output” stream.

My Input stream would show me tweets extracted from the global twitter stream that an algorithm (described below) predicts will be of most interest to me.  My input stream is theoretically arbitrarily long, but tweets would be sorted so that those towards the top of my incoming list are the ones it expects me to be most interested in reading.

My Output stream would represent the list of tweets that my followers would see if they choose to view what I find most interesting in the global twitter stream at the moment – although do such a “direct view” of an individual’s tweet stream will be rare, for reasons given below.

If I’m not on-line and actively managing my output stream, my input stream will be copied directly to my output stream.  Anyone following me would therefore see what the algorithm thinks I would consider the most interesting content in the twitter stream at the moment.  TweetStream would serve as my digital agent, offering up to the world a source of information filtered through a ‘virtual me’ and therefore tailored to my interests.  Since I’m a fan of Steelers football and a mobile gadgets, the output ‘DeanPomerleau Stream’ that others might follow would likely contain a mix of stories about the latest Steelers sports news and information about the latest in mobile technology, and a few others stories of broader interest that I find interesting

When I’m on-line and actively engaged in managing my personal tweet stream, my interaction with the TweetStream app would entail reading my input stream, surfing the web to find interesting content, or generating new content myself (e.g. blogging) and then posting to my output stream the stuff I consider most interesting.

My output stream is analogous to sequence of tweets and retweets that people generate now on Twitter, except that rather than a single most recent post and a long tail of past posts, I have a set of 25-100 posts that I (with the assistance of my digital agent) consider the most interesting content currently flowing in the twitter stream.

The TweetStream Rank Algorithm

Central to the success of TweetStream will be its ranking algorithm that understands what type of content interests me. TweetStream will use this knowledge to extract and display a manageable amount of personalized content for me to enjoy out of the torrent of information flowing through the global twitter stream.

TweetStream’s personalize content ranking algorithm will learn my preferences by observing my viewing habits. There will be no need to explicitly search out interesting people to follow unless I want to.  When I sign up, I’ll select a few topics that interest me from a list (e.g. ‘Steelers football’ & ‘mobile gadgets’).  These will be used to seed my initial rank algorithm.  What selecting these topics will do is to automatically connect my input stream to the output stream of users who have are interesting in one or both of those topics.

Each of the users I’m connected to will have a weight associated with them, which reflects how closely our interests match each other.  Each time I read (and perhaps rank) a tweet from someone, the weight they are given by my ranking algorithm is increased, so content they generate in the future will be more likely to show up near the top of my incoming stream.  In addition, my ranking algorithm will increase the weight given to other users who have also shown interest in the tweet that I enjoyed (by reading it themselves), drawing me closer to other who may be passive content viewers (rather than generators) but who share my interests.  The closer someone is to the source of the original message that interests me (either in time or retweet depth), the more the algorithm will increase their weight – embodying the idea that I’m likely to enjoy information from the ‘thought leader’ on a topic more than retweets by one of his many followers.

In this way, TweetStream will leverage my viewing (and maybe ranking) history to create list of people for me to follow that is tailored to my interests. They more I use TweetStream, the better it will understand my interests, and the more effective it will be at delivering on my input stream the content I’ll enjoy.  And of course, I’m free to explicitly add or remove people from this automatic following list to personalize it even further.

From Rank to Input Stream

To generate the list of tweets that I see on my input stream, TweetStream will take a weighted sum of the output streams of the people I’m following.

Suppose for example, several of the people I’m following have the same tweet on their output stream right now, either because they read it and enjoyed it directly, or simply because their automatic ranking algorithm thinks they would enjoy it if they were on-line now.  The ranking algorithm will interpret this convergence of support for a tweet from several people with whom I share interests as an indicator I too will likely find it worth reading.  So TweetStream will be placed high on the list of tweets on my input stream.

Alternatively, suppose I follow has generated a tweet on their output stream, and they are the only one to have tweeted about it so far.  If they are someone who I value highly, and if they have placed a high score on this tweet, the strong endorsement of a single person for whom I have a high affinity for will be sufficient to ensure their tweet shows up on my input stream.  But if the person ‘cries wolf’ too often, perhaps by tweeting an ad or simply by tweeting content that doesn’t interest me or that I’ve already seen, my choice not to read their post (or to give it a low rating) will cause their weight will be decreased, so in the future I won’t be as likely to see their content. Instead, their post will have to be endorsed by others I trust if it is make it into my input stream.

At the opposite end of the spectrum, an important breaking news story (e.g. death of a leader, or terrorist attack) that isn’t directly aligned with previously expressed core set of interests could nonetheless make it onto my input stream if many users (each of whom I may be only very weakly connected) are reading about it and retweeting it.

In part 2 – what does this model buy the user, and what does it mean for the emergence of collective intelligence on the web?