A few days ago in a post titled Twitter & The Global Brain I blogged about the parallels between twitter and giant neural network. Now I want to flesh out that model and make it a little more tangible by describing an app that I call TweetStream that could potentially solve several of twitter’s current problems:
- Taming the torrent of information that blasts current twitter users.
- Monetizing content and rewarding participation in the twitter experience.
- Moving the global system towards a coordinated efficient information exchange framework through which global consciousness crise and be exercised.
The TweetStream App
Imagine an app that provided not only the chronological list of friends updates that are is currently provided by Seesmic and Tweetdeck, but also provides what I’ll call a “personalized tweet stream”. My personalized tweet stream would be composed of two parts, presented by the App in two separate column – my “Input” stream and my “Output” stream.
My Input stream would show me tweets extracted from the global twitter stream that an algorithm (described below) predicts will be of most interest to me. My input stream is theoretically arbitrarily long, but tweets would be sorted so that those towards the top of my incoming list are the ones it expects me to be most interested in reading.
My Output stream would represent the list of tweets that my followers would see if they choose to view what I find most interesting in the global twitter stream at the moment – although do such a “direct view” of an individual’s tweet stream will be rare, for reasons given below.
If I’m not on-line and actively managing my output stream, my input stream will be copied directly to my output stream. Anyone following me would therefore see what the algorithm thinks I would consider the most interesting content in the twitter stream at the moment. TweetStream would serve as my digital agent, offering up to the world a source of information filtered through a ‘virtual me’ and therefore tailored to my interests. Since I’m a fan of Steelers football and a mobile gadgets, the output ‘DeanPomerleau Stream’ that others might follow would likely contain a mix of stories about the latest Steelers sports news and information about the latest in mobile technology, and a few others stories of broader interest that I find interesting
When I’m on-line and actively engaged in managing my personal tweet stream, my interaction with the TweetStream app would entail reading my input stream, surfing the web to find interesting content, or generating new content myself (e.g. blogging) and then posting to my output stream the stuff I consider most interesting.
My output stream is analogous to sequence of tweets and retweets that people generate now on Twitter, except that rather than a single most recent post and a long tail of past posts, I have a set of 25-100 posts that I (with the assistance of my digital agent) consider the most interesting content currently flowing in the twitter stream.
The TweetStream Rank Algorithm
Central to the success of TweetStream will be its ranking algorithm that understands what type of content interests me. TweetStream will use this knowledge to extract and display a manageable amount of personalized content for me to enjoy out of the torrent of information flowing through the global twitter stream.
TweetStream’s personalize content ranking algorithm will learn my preferences by observing my viewing habits. There will be no need to explicitly search out interesting people to follow unless I want to. When I sign up, I’ll select a few topics that interest me from a list (e.g. ‘Steelers football’ & ‘mobile gadgets’). These will be used to seed my initial rank algorithm. What selecting these topics will do is to automatically connect my input stream to the output stream of users who have are interesting in one or both of those topics.
Each of the users I’m connected to will have a weight associated with them, which reflects how closely our interests match each other. Each time I read (and perhaps rank) a tweet from someone, the weight they are given by my ranking algorithm is increased, so content they generate in the future will be more likely to show up near the top of my incoming stream. In addition, my ranking algorithm will increase the weight given to other users who have also shown interest in the tweet that I enjoyed (by reading it themselves), drawing me closer to other who may be passive content viewers (rather than generators) but who share my interests. The closer someone is to the source of the original message that interests me (either in time or retweet depth), the more the algorithm will increase their weight – embodying the idea that I’m likely to enjoy information from the ‘thought leader’ on a topic more than retweets by one of his many followers.
In this way, TweetStream will leverage my viewing (and maybe ranking) history to create list of people for me to follow that is tailored to my interests. They more I use TweetStream, the better it will understand my interests, and the more effective it will be at delivering on my input stream the content I’ll enjoy. And of course, I’m free to explicitly add or remove people from this automatic following list to personalize it even further.
From Rank to Input Stream
To generate the list of tweets that I see on my input stream, TweetStream will take a weighted sum of the output streams of the people I’m following.
Suppose for example, several of the people I’m following have the same tweet on their output stream right now, either because they read it and enjoyed it directly, or simply because their automatic ranking algorithm thinks they would enjoy it if they were on-line now. The ranking algorithm will interpret this convergence of support for a tweet from several people with whom I share interests as an indicator I too will likely find it worth reading. So TweetStream will be placed high on the list of tweets on my input stream.
Alternatively, suppose I follow has generated a tweet on their output stream, and they are the only one to have tweeted about it so far. If they are someone who I value highly, and if they have placed a high score on this tweet, the strong endorsement of a single person for whom I have a high affinity for will be sufficient to ensure their tweet shows up on my input stream. But if the person ‘cries wolf’ too often, perhaps by tweeting an ad or simply by tweeting content that doesn’t interest me or that I’ve already seen, my choice not to read their post (or to give it a low rating) will cause their weight will be decreased, so in the future I won’t be as likely to see their content. Instead, their post will have to be endorsed by others I trust if it is make it into my input stream.
At the opposite end of the spectrum, an important breaking news story (e.g. death of a leader, or terrorist attack) that isn’t directly aligned with previously expressed core set of interests could nonetheless make it onto my input stream if many users (each of whom I may be only very weakly connected) are reading about it and retweeting it.
In part 2 – what does this model buy the user, and what does it mean for the emergence of collective intelligence on the web?
13 comments
Comments feed for this article
December 7, 2009 at 10:15 pm
TweetStream: An App to Drive the Global Brain – Pt2 « Thoughtful Cog Blog
[…] Cog Blog Just another WordPress.com weblog « TweetStream: An App to Drive the Global Brain – Pt 1 TweetStream: An App to Drive the Global Brain – Pt2 December 7, 2009 In the last […]
December 13, 2009 at 6:59 pm
Openworld
An approach towards a similar end could be to enhance Alltop (or something like it).
A Twitter user could create framework page and lay out sections for interesting topics. Each section could then “follow” Twitterers whom the user respected as thought leaders on the topic.
In each section, the top-ranked “stories” would be automatically generated. This could be done by having the Alltop-style page automatically comb through links in the Tweet streams for each topic, and flag the links most commonly cited by the topic thought leader. It would be ideal if an outliner-style capability could be included in the interface. This way, the headline/summary text for a popular link could be expanded to show the specific Tweets associated with the item.
Ideally, back in Twitter, the user could create “lists” for each of the topic areas covered on the associated web page. In this way, those who wished to follow a specific topic-based channel could subscribe directly to it – and be spared drowning in the flood of each lists’ discrete tweets.
What do you think?
Mark Frazier
Openworld
@openworld
December 13, 2009 at 7:26 pm
Dean Pomerleau
Mark,
Thanks for your comment! Your idea of an enhanced Alltop-like service that is semi-automatically generated by gleaning topic-centric content posted or cited by identified ‘thought leaders’ is exactly the kind of think I had in mind with this post.
Some kind of PageRank-like algorithm for identifying both good content and good contributors seems like a must if this kind of model is going to work. The big question in my mind is whether we need some kind of central organization (like Google) to fight to keep the spammers and SEO gamers from overrunning the system. Despite my admiration for Google, I hope a more Digg-like approach to social filtering of real-time content will emerge.
December 14, 2009 at 2:27 am
Ilya Grigorik
Dean, Mark, this is a really interesting idea.
The PostRank system we’ve built is somewhat similar to what you’ve described. Albeit, the big difference is that we treat each user as an anonymous agent – we don’t offer personalization or look within your network. Instead, PostRank aggregates activity from all the users from over a dozen of most popular networks and uses that as an overall signal for each piece of content. Drill into any feed on a topic and hover over the score, you’ll see what I mean.
Ex: http://www.postrank.com/topic/twitter
We’ve given the personalization angle a lot of thought in the past, but just haven’t had the time to get to it yet. I think the combination of ‘global’ and ‘personal network’ signals would be a really interesting combination.
Actually, we also have a collection of twitter bots for some topics (http://www.postrank.com/twitter), wonder if we can apply your model to those. Hmmm.
December 13, 2009 at 7:23 pm
Openworld
(replaces my previous post – some fixups made)
An approach towards a similar end could be to enhance Alltop (or something like it).
A Twitter user could create framework page and lay out sections for interesting topics. Each section could then “follow” Twitterers whom the user respected as thought leaders on the topic.
In each section, the top-ranked “stories” would be automatically generated. This could be done by having the Alltop-style page automatically comb through links in the Tweet streams for each topic, and flag the links most commonly cited by the topic thought leaders.
It would be ideal if an outliner-style capability could be included in the interface. This way, the headline/summary relating to a popular item could be expanded to show the specific Tweets associated with the item.
Below the toplisted items, each section could also feature individual Tweets (and/or their linked article) from the topic’s associated thought leaders. The section creator might also assign “authority” weights to the Twitterers associated with the topic, so Tweets from the most highly respected thought leaders would have higher visibility in the section than those of others.
Ideally, back in Twitter, the user would create “lists” for each of the topic areas covered on the associated Alltop-style web page.
Lists of the topics would enable followers of the user to subscribe directly to specific topic “feeds” — perhaps the top five or 10 summaries posted each day on the topic(s) of most interest. In this way, a follower could avoid drowning in floods of tweets.
What do you think of this approach to social filtering?
Mark Frazier
Openworld
@openworld
December 13, 2009 at 9:07 pm
Dean Pomerleau
Update – I came across a review of mytweetsense, now only an iPhone app but that looks promising for the kind of filtering we’re talking about. It got nice review on Techcrunch.
http://www.techcrunch.com/2009/08/27/twittersense-its-coming/
Anybody tried it, or its sister app, my6sense?
–Dean
December 14, 2009 at 2:48 am
Dean Pomerleau
Ilya,
PostRank.com looks promising. I like the idea of content aggregation by topic. But the topics look pretty high-level and generic, similar to services like alltop.com.
I’m hoping smart entrepreneurs are working right now a way to infer my personal interests by analyzing the content of my tweets, retweets, and the posts I read. Then I want it to find more stuff like that.
mytweetsense claims it does (or will do) something like that. Check out the video on this page:
http://www.techcrunch.com/2009/08/27/twittersense-its-coming/
I’m looking forward to more and more sophisticated services like this appearing.
–Dean
@deanpomerleau
December 14, 2009 at 3:42 am
Ilya Grigorik
Dean, topic relevancy is certainly a challenge, especially when you do the aggregation on feed level. One distinction between what we’re building and Alltop is that we’re cultivating a user generated classification (with all the up and downsides associated with that process). As a visitor you’re more than welcome to remove or add any feeds, or start a new and perhaps more targeted topic.
Next step is, as you pointed out, to add some personalization into the mix.
December 14, 2009 at 3:54 am
Openworld
Dean and Ilya,
Have you checked out @JohnReaves work?
There’s a great presentation from his firm on tag-influenced taxonomies.
Here’s a link to his group that I think you’ll like —
http://j.mp/6zz6Pj (see slides on prompting users to create folksonomy-influenced taxonomies)
In response to his work, I made comments to him on a structured way to tag Tweets (as well as tag other information chunks):
>>I like your tag-influenced taxonomies. Prompts also can help users tag deep patterna in micro/macro narratives.
>>fractal structure to tag micro/macro narratives: 1) attractor 2) stress 3) opportunity 4) strategy 5) test 6) decision
>>same storyline pattern – I think – can be found in good ads, articles/blog posts, talks, scientific essays, books, movies
>>[big leap] a storyline fractal may have helped consciousness evolve (comment #7 on @kk blog) http://j.mp/4p3jwE
Hope some of the above can help your apps roll out social filtering tools that enable tribes to take action on virtual or actual projects.
Best,
Mark
Openworld.com
@openworld
December 18, 2009 at 4:16 pm
SeH
Hi! Wondering how we might collaborate:
http://transalchemy.blogspot.com/2009/12/introduction-to-netention-semantic.html
http://transalchemy.blogspot.com/2009/12/twitter-operating-system.html
December 20, 2009 at 10:18 pm
Towards a web of activity streams realizing the synaptic web paradigm « web2society
[…] streams, and that could help you get connected to peers whom are interesting to you, see ‘TweetStream: An App to Drive the Global Brain – Pt 1’ and part 2. Although Dean applies these ideas specifically to Twitter, they are also valid in a […]
January 1, 2010 at 2:51 pm
The Sentient Web and an Autonomous Economy « Thoughtful Cog
[…] could be an alternative route to the Global Brain I previously envisioned as the end result of the TweetStream application. By whichever route get there (and there are likely others yet to be identified), the […]
April 7, 2010 at 8:11 pm
Barak Hachamov
Hi Dean,
my6sense is a leader and pioneer in a new service category ‘hyper-personalized streams’, which uses ‘Digital Intuition’ technology to tame the overwhelming amount of content that flows into your personal space.
my6sense surfaces your most relevant information at the right time and the right context – connecting you with what matters to you most. – it’s personal, it’s insightful, it’s effortless!
The my6sense iPhone app is the first and only intuitive Stream Reader that uses – ‘Digital Intuition’tchnology – to automatically and personally rank content from users’ information streams (Twitter, Facebook, news, RSS). Each user’s most valuable and relevant content is ‘magically’ flooded to the top, ensuring that he never misses out on important insights.
The app’s latest major feature, ‘mytweetsense’, filters the tweets which contain links and intuitively recognizes what is important to each user, presenting the most relevant content first. mytweetsense scours deep within the linked content, looking well beyond the 140 characters of the tweet, or the title of an article. As a result, the application has the power to uncover hidden gems of relevant content that may have otherwise gone unnoticed.
You can try the magic on iPhone: http://www.itunes.com/app/my6sense
Attention API
At DEMO Spring 2010 my6sense launch our ‘Attention API’ which enables third party developers and publishers to create hyper-personalized streams of content for their users, powered by ‘Digital Intuition’ technology.
The service is content/stream agnostic and can automatically rank information from any and all types of sources. The service requires zero end user intervention relying solely on a user’s natural consumption behavior.
It is highly sensitive to the finest nuances of each user’s interests, taking into consideration the content components; user environment (time and in the future, location) as well as community elements, providing the outlet’s users and/or audiences with accurately filtered relevant information at the top of their streams.
Please find some recent write-ups:
Mashable: Give Your Readers Personalized Content with My6sense
http://mashable.com/2010/03/22/my6sense-api/
Louis Gray: my6sense Intros Attention API for Hyper-Relevant Web
http://blog.louisgray.com/2010/03/my6sense-debuts-attention-api-for-devs.html
CNN: How can we cope with information overload?
http://www.cnn.com/2010/TECH/02/03/content.overload/index.html
Would love to share more info…
Barak Hachmov,
Founder & Visionary Geek
My6sense