Lesson Beyond 3• 5 min read

Connections that stay open

Polling, long polling and WebSockets, and sizing a tier for the connections it holds and the pushes it sends.

What you will learn

  • Count what polling costs when most answers are "nothing new".
  • Describe how long polling and connections that stay open let a server push news.
  • Size a tier of connection servers for the connections it holds and for the pushes it sends.

The cost of asking

HTTP is started by the client: the server answers requests, it does not send news on its own. The simplest way to learn about new messages is polling: the client asks every few seconds whether anything has changed.

Polling costs in proportion to the clients, not the news. Half a million connected phones that each ask every 10 seconds send 50,000 requests a second, even when nobody is chatting. Most answers are empty, yet each one passes the load balancer, an app server and often a store. Asking less often saves requests but delays the news.

Keeping the connection open

Long polling lets the client ask once and have the server hold the request open until there is news or a timeout passes; then the client asks again. Requests now follow the news more closely, though each answer still needs a new request.

A WebSocket goes further: the client opens one connection, and both sides may send messages over it whenever they like, for as long as it stays open. The server pushes news the moment it happens, and an idle client costs a held connection and little else.

Connection servers

Servers that hold connections are a tier of their own, sized by two figures, not one:

  • Connections held: every online user keeps one open, so a million users need enough servers to hold a million connections, with a spare.
  • Pushes sent: each message goes to each of its recipients. A message to a group of 20 is 20 pushes. A position update shared with 8 friends is 8 pushes. Count pushes a second, not messages.

The recipients of a message are usually held by other servers than the sender's. A bus between the connection servers, a publish-and-subscribe channel, carries each push to the server that holds the recipient, and it is sized for the pushes too.

When a connection server fails

A connection is held by one server. When that server fails, every user it held is disconnected, and their apps connect again, usually within seconds and all at once. That burst of new connections lands on whatever the apps call to come back: the login path, the history they reload. Size that path for the users of one server connecting again together, not only for the steady rate.

Phones1× instanceLoad balancer3× instancesConnection servers7× instancesBus3× instancesAPI servers6× instancesMessage store3× instancesPhones1× instanceLoad balancer3× instancesConnection servers7× instancesBus3× instancesAPI servers6× instancesMessage store3× instancesPhones1× instanceLoad balancer3× instancesConnection servers7× instancesBus3× instancesAPI servers6× instancesMessage store3× instances
Figure 1

Connection servers hold every online user and push each message, through a bus, to the server holding each recipient.

Messages travel over the connections the users hold: each is pushed to its recipients through the bus, then stored. Everything else, such as loading a chat’s history, goes to the API servers. Raise the rate: the connection servers’ load is set by the connections they hold and barely moves, while the bus and the message store climb with the messages.

  • Phones: 1 × Users
  • Load balancer: 3 × Load balancer
  • Connection servers: 7 × Connection server
  • Bus: 3 × Cache
  • API servers: 6 × App server
  • Message store: 3 × NoSQL store
  • Phones sends requests to Load balancer
  • Load balancer sends requests to Connection servers
  • Load balancer sends requests to API servers
  • Connection servers sends requests to Bus
  • Connection servers sends requests to Message store
  • API servers sends requests to Message store

Key takeaways

  • Polling costs in proportion to connected clients, news or not.
  • Long polling and WebSockets let the server push news over a connection the client keeps open.
  • Size connection servers for both the connections they hold and the pushes they send, and the path a lost server's users connect again through.
Sources4