I’d heard that earlier in the Gilmor Gang : Blaine clarified that static resources (e.g. profile images) are no issue at all, and that the the bigger issue is with each person’s “home page”, where multiple streams are joined and ordered chronologically. When 1 of the connected people update, all related cached “home pages” are obsolete - hence they are generated painfully often. (so don’t refresh your own page so much!)
I’m not a scaling expert. Nor do I know anything about Twitter’s real problems. So treat this as noise maybe.
Anyways, ignoring other things unknown and purely focusing on this particular aspect… I thought it’d be interesting to consider what we might gain, if we’re just willing to give up certain things?
Would that be better? And what did we give up? The ability to properly browse your own Twitter homepage on a browser without JS & clients using API need a few more pulls and do merging / sorting themselves too.
Can we live with that?
Update: Slim pointed to an article in the comments, and “Part III” of that article, Eran had talked abt the same alternative:
Update: Then again, you have scoble who follows 30k other people making it impractical for the client to fetch those streams independently… I guess most systems would decide to place limits and “punt” the issue, e.g. you can have at most 5000 friends. And that’s something Twitter is unwilling to let go.