Twitter shared a fascinating presentation with information on how they are handling analysis of it’s 100 Billion tweets. Much of the analysis is focused on numerous technical requirements but there are some really fascinating business requirements as well.
Those requirements were related to not just the storage of the masses of data, but the analysis of the data and it is in those requirements that I think Twitter is asking it’s most important questions:
- Which features get people hooked?
- Which features do successful users use often?
Goes to show, even with the mammoth task of just storing 100 Billion tweets in ways that are parseable, splitable, reusable, small in data size and hierarchical, they are still focused on some important behavior analysis.
The presentation was put together by Kevin Weil who is the Analytics Lead at Twitter.
Thanks also to both Mack Collier and Rob Kubarych for pointing me to additional material 
6 Comments
Great article! I’m glad I picked up Karl’s thread on twitter. It lead me to some fantastic stuff. I had never heard of hadoop, protocol buffers, and pig (oh my!). What powerful stuff we have at our finger tips today! Can it take us some place good for us though?
I came an interview with Jaron Lanier at http://www.guardian.co.uk/technology/video/2010/feb/20/jaron-lanier-web20 yesterday which made me stop and think. I don’t know what the result of that thinking will be yet. He explained how 25 years ago he and others had dreamed of the internet being a fountain of wealth and opportunity where people could openly give away the fruits of their brains and hearts. He explains how that vision has particularly not been realized in the domain of music in particular but in other domains also, and shares some important observations on human behavior, biology, and democracy.
Did all of this really exist before? Can we have web 2.0 without the mob and a mediocre mush? I think his ideas, http://www.guardian.co.uk/technology/2010/feb/21/my-bright-idea-jaron-lanier, are merit serious thought.
I know what Jaron Lanier is saying about mediocre mush, but twitter has a lot of premier cream. Just one example out of many, many, David E. Guggenheim. His twitter page is http://twitter.com/oceandoctor.
I’m a little confused about his saying web 2.0 is a failure, but the web on the whole is a success story? Makes me think he’s just talking about social networks. I think the point is that none of these tools are monolithic communities, but are platforms for group forming. Anyone on twitter is having a unique experience because they probably follow a unique group, so I think it’s erroneous to say any platform has failed, you just have to look at the community, culture and value that evolves.
I agree with you. There’s a lot of good things going on within twitter that are consistent with that dream of an open place for sharing hearts and minds. I would love it if he would comment here and clarify.
“Which features get people hooked?” Is this a positive question (i.e. a social responsibility question) or a negative corporate question seeking to grab users as play things?
Hey Karl,
Thanks for this post, I had no idea Twitter had this strategy cued up ready to roll out when they hit that magic number. That’s one hell of a project.
To me, not coming from a developer background, the most interesting part of all of this isn’t what features twitter users prefer to use over others but in fact what it is they’re actually saying. Measuring the sentiment, polarity and frequency of the commentary over the one *billion* tweets through history I bet you could pull out some amazing facts and figures.
Much like how Google reveals its most searched terms once a year, so too could Twitter embark on such an endeavour. Ok, they kind of already do this with their live-feed trend analysis – but long term? We could be looking at a conversational tapestry of modern history!
Very exciting indeed.
Cheers,
James