The guys over at Assetbar, a new Google Reader competitor in early beta mode, have made an interesting proposal to host a proxy for Twitter’s database on their database system:
It so happens that our new distributed database technology is rather well suited for twitter-style high-volume reliable messaging. If there is sufficient community interest we could help solve downtime by putting together a “twitter-proxy” that keeps twitter users on twitter, but provides an additional layer of data accessibility in the ecosystem. Not compete, just help keep users happy.
According to Assetbar, the problem for Twitter is not Ruby on Rails or their hosting service, but the scalability of their database backend [Update: I just found some more info about Assetbar's technology backend]
Basically, every time Scoble fires off a message, the databases have to work overtime to write that message into everybody’s databases and then again to read it again for everybody who wants to retrieve the message. Adding overhead, Assetbar estimates 100.000 disk IO’s if Scoble, Arrington and Calacanis get into a discussion.
Now, I’m no database expert and I have to take Assetbar’s word at face value that this does indeed create problems for a database.
With a proxy set in between Twitter and the user, Assetbar’s database would provide an extra level of redundancy. If Twitter goes down, users would just be using their system exclusively for the time being and everything gets synchronized after the outages is over.
Here is how it would work:
- You enter your twitter credentials on the proxy site
- You can post your tweets to the proxy. If twitter is up, we’ll post there, too.
- We’ll get your friend list and GET and store their tweets in our db.
They are asking Twitter for the go-ahead to do this, which I think is a very fair move. Because Twitter’s API is quite good, they could probably implement this without asking them.
Why would Assetbar offer to do this? Asserbar is clearly not trying to compete with Twitter, as Louis Gray points out and there is no money in it for them directly. But if their database technology is really this good, they would probably see a sale or two because of this collaboration.
One thing I do wonder about, though, is if Twitter isn’t thinking along those same lines already. Twitter must have some redundancy built into the system… I would hope…
MySql write saturates at 400 writes a second, no matter how you cluster it, without some really arcane acceleration workarounds.
What makes these guys think that they have the answer? They are just as likely to trash themselves as help Twitter or its users.
Twitter is down now, again, even after the NTT move; they need to migrate to a new, non RDBMS data persistence architecture - painful, yes. Essential, yes.
Ah..now I see that assetbar is rethinking the database….good luck guys.
Twitter needs to do something. It has become almost ususable due to
slowness and downtime.
when will Google/Jaiku wake up?
Wow, Twitter scalability problems aren’t caused by RoR, what a shocker, NOT!! Glad to be getting that message out.
Their proposal will never work. That’s because you need something like CEP (Complex Event Processing — think Coral8 or StreamBase) to meet the throughput needs. Writing to a database and immediately querying doesn’t work in real-time.
http://www.texttechnologies.com/2008/02/09/scalable-twitter/ has a detailed write-up, supported by http://www.dbms2.com/2008/01/16/twitter-could-easily-be-made-reliable/
I am not a big twitter fan so don't use it much often but they seems to have solved their DB problem as back in 07, there uptime was 96% !
Good Idea! Twitter can be hard to live without lol