Formed in 2009, the Archive Team (not to be confused with the archive.org Archive-It Team) is a rogue archivist collective dedicated to saving copies of rapidly dying or deleted websites for the sake of history and digital heritage. The group is 100% composed of volunteers and interested parties, and has expanded into a large amount of related projects for saving online and digital history.
For Developers > Design Documents >
Sync
What To make this sync infrastructure scale to millions of users, we decided to leverage existing XMPP-based Google Talk servers to give us "push" semantics, rather than only depending on periodically polling for updates. This means when a change occurs on one Google Chrome client, a part of the infrastructure effectively sends a tiny XMPP message, like a chat message, to other actively connected clients telling them to sync. To put that gain into perspective, consider a 3 minute polling interval. 3 minutes is far from real time, or "immediately" as our goal was stated. But already, at the very least, every 3 minutes every client needs to ask the server if anything changed. Even with just one thousand users, we're already talking about a server having to handle a poll request every 0.18 seconds on average (or roughly 5.6 queries per second). And that's just when nothing is happening! Using XMPP pushes, the sync servers don't need to waste cycles for no reason. |