Dave Winer gave me the idea that clicked a bunch of pieces into place on my thoughts about Twitter’s need to scale. I want to put forth an idea that comes from my background in enterprise IT, where we had an application that wrote hundreds of thousands of short records a second to a database, and where we had processes in place for when the platform went down. Here are the problems, and then how I’d solve them with my enterprise IT hat on:
1.) Twitter, when it goes down, has nowhere to pass the traffic. This frustrates the customer base badly.
In my former wireless telecom world, we made a function that would permit calls to process while our database was offline. Call detail records would store up in a smaller database run by the same front end and middleware, and then when the databases were back up, we’d insert the records, and process everything accordingly.
2.) Twitter is essentially an app writing into a back end.
We need one layer of abstraction. One way to do this would be to use a client like Twhirl, and give it the availability to write tweets to two places: an RSS feed (so we could do more with the data- and I want that feature anyway, Loic. Okay?), and the second to an intermediate database somewhere on the Amazon S3 cloud.
When Twitter’s down, we run from Twhirl’s second pointer. When it comes back up, the database of new tweets gets reinserted.
If this is best accomplished by an XML feed, think about it: how much storage are 140 characters (okay, plus the meta data) for everyone you’re following. Make it a Twhirl-only feature for all I care.
Or, if Not Twhirl
I was thinking that we need SOME kind of front end, like a Firefox for this new kind of app. Something that resides in open source so that we can fork it, adjust it, adapt it, and work on the same code core, with the same baseline features, but with our own bells and whistles.
Enterprise Thinking
It’s strange that reading that one comment from Dave gave me a whole refresher course in how I used to work with fast-moving enterprise-grade data, and that it certainly has some parallels in what Twitter’s trying to do.
Take all these thoughts with love, Twitter and Twhirl. And please try and help us keep the flow alive.