Polling, long polling and WebSockets, and sizing a tier for the connections it holds and the pushes it sends.
What you will learn
Count what polling costs when most answers are "nothing new".
Describe how long polling and connections that stay open let a server push news.
Size a tier of connection servers for the connections it holds and for the pushes it sends.
The cost of asking
HTTP is started by the client: the server answers requests, it does not send news on its own. The simplest way to learn about new messages is polling: the client asks every few seconds whether anything has changed.
Polling costs in proportion to the clients, not the news. Half a million connected phones that each ask every 10 seconds send 50,000 requests a second, even when nobody is chatting. Most answers are empty, yet each one passes the load balancer, an app server and often a store. Asking less often saves requests but delays the news.
Keeping the connection open
Long polling lets the client ask once and have the server hold the request open until there is news or a timeout passes; then the client asks again. Requests now follow the news more closely, though each answer still needs a new request.
A WebSocket goes further: the client opens one connection, and both sides may send messages over it whenever they like, for as long as it stays open. The server pushes news the moment it happens, and an idle client costs a held connection and little else.
Connection servers
Servers that hold connections are a tier of their own, sized by two figures, not one:
Connections held: every online user keeps one open, so a million users need enough servers to hold a million connections, with a spare.
Pushes sent: each message goes to each of its recipients. A message to a group of 20 is 20 pushes. A position update shared with 8 friends is 8 pushes. Count pushes a second, not messages.
The recipients of a message are usually held by other servers than the sender's. A bus between the connection servers, a publish-and-subscribe channel, carries each push to the server that holds the recipient, and it is sized for the pushes too.
When a connection server fails
A connection is held by one server. When that server fails, every user it held is disconnected, and their apps connect again, usually within seconds and all at once. That burst of new connections lands on whatever the apps call to come back: the login path, the history they reload. Size that path for the users of one server connecting again together, not only for the steady rate.
Connection servers hold every online user and push each message, through a bus, to the server holding each recipient.
Messages travel over the connections the users hold: each is pushed to its recipients through the bus, then stored. Everything else, such as loading a chat’s history, goes to the API servers. Raise the rate: the connection servers’ load is set by the connections they hold and barely moves, while the bus and the message store climb with the messages.
Phones: 1 × Users
Load balancer: 3 × Load balancer
Connection servers: 7 × Connection server
Bus: 3 × Cache
API servers: 6 × App server
Message store: 3 × NoSQL store
Phones sends requests to Load balancer
Load balancer sends requests to Connection servers
Load balancer sends requests to API servers
Connection servers sends requests to Bus
Connection servers sends requests to Message store
API servers sends requests to Message store
Key takeaways
Polling costs in proportion to connected clients, news or not.
Long polling and WebSockets let the server push news over a connection the client keeps open.
Size connection servers for both the connections they hold and the pushes they send, and the path a lost server's users connect again through.