Lesson 7.1• 9 min read

Queues, task queues and log-based message brokers

Point-to-point task queues (RabbitMQ, SQS) versus distributed append-only logs (Kafka, Pulsar).

What you will learn

  • Use Little's law to see why slow synchronous work starves a thread pool.
  • Take slow work off the request path with a queue and a quick acknowledgment.
  • Compare task queues with a log-based message broker such as Kafka, and scale Kafka consumers by partition.

The synchronous bottleneck and thread pool starvation

In a traditional synchronous request-response architecture, an incoming HTTP request blocks an operating system or runtime worker thread for the entire duration of downstream processing:

Tsync=tnetwork+tauth+ttranscode+tdisk_io+tnotificationT_{\text{sync}} = t_{\text{network}} + t_{\text{auth}} + t_{\text{transcode}} + t_{\text{disk\_io}} + t_{\text{notification}}what its symbols mean

Each term is the time one step takes: the network, authentication, transcoding, disk I/O and sending a notification.

If video transcoding or PDF rendering takes 3 to 15 seconds3\text{ to }15\text{ seconds}, a server with 200200 worker threads is fully saturated at just:

Max Concurrency=200 threads5 seconds=40 requests/sec\text{Max Concurrency} = \frac{200\text{ threads}}{5\text{ seconds}} = 40\text{ requests/sec}

By Little's law (L=λWL = \lambda Wwhat its symbols mean: the requests in progress equal the arrival rate times the time each takes), 200 busy threads at 5 seconds each can sustain at most 200/5=40200 / 5 = 40 requests per second. Beyond that, requests queue up waiting for a thread, latency climbs, and clients (or proxies in front) start timing out.


Asynchronous decoupling: fast acknowledgment, deferred work

Asynchronous workflows take expensive operations off the request path. With a message queue:

  1. The application publishes a job to the queue and tells the user the job's status straight away (for example, HTTP 202 Accepted with a status URL). The user is not blocked.
  2. A worker picks up the job, processes it in the background at its own pace, and signals when it is done.

The client can even make the result look immediate: a posted tweet appears on your own timeline at once, though delivering it to all your followers takes longer. The cost: queues add delay and complexity, so cheap calculations and real-time workflows may be better done synchronously.

Common choices differ: Redis is a simple broker but can lose messages; RabbitMQ is popular but uses the AMQP protocol and you run the nodes; Amazon SQS is hosted but can have high latency and may deliver a message twice. Task queues such as Celery go further: they run the task and return its result, and can schedule jobs.


Task queues versus log-based message brokers

Kafka's documentation says it can serve as a kind of external commit log for a distributed system; Designing Data-Intensive Applications calls brokers like it log-based message brokers.

Dimension Message / task queue (RabbitMQ, SQS, Celery) Log-based message broker (Kafka)
Storage A buffer from which messages are removed once processed An append-only log; events are not deleted when consumed
Consumption Competing workers take messages; each message goes to one worker Each consumer group tracks its own position (offset) in each partition
Ordering Usually best effort, especially across retries (SQS FIFO queues are the exception) Events with the same key go to the same partition and are read in exactly the order written
Replay Not normally possible once a message is consumed Rewind a consumer group's offset, for as long as the topic's retention keeps the events
Typical use Background jobs: emails, transcoding, scheduled tasks Event streams read by many consumers, change data capture, audit trails

Kafka is built for throughput: it relies on sequential disk writes and the operating system's page cache, and uses sendfile to send data from the page cache to the network without copying it through the application.


Partitioning and consumer group scaling in Kafka

A Kafka topic is divided into PPwhat its symbols mean partitions; by default an event's key is hashed to choose its partition:

Partition=hash(key) mod P\text{Partition} = \text{hash}(\text{key}) \bmod Pwhat its symbols mean
  • Ordering: Events with the same key (e.g. user_id_492) land on the same partition, and every consumer reads a partition's events in the same order (as long as the number of partitions does not change).
  • Consumer groups: Each partition is consumed by exactly one consumer within a consumer group at any given time, so a group can use at most PP consumers in parallel (Nconsumers≤PN_{\text{consumers}} \le Pwhat its symbols mean); extra consumers sit idle.

Task queue versus log-based message broker

Task queue model (RabbitMQ, SQS): Messages are transient units of work. Competing workers pull items from a shared buffer. Once acknowledged (ACK), the message is deleted permanently. Designed for fine-grained task distribution and arbitrary scale-out.
Message Buffer (2 Pending)
FIFO order
msg_101
video_transcode_1080p
ACKNOWLEDGED
msg_102
push_notification_order_paid
PROCESSING
msg_103
email_invoice_pdf_gen
PENDING
msg_104
fraud_score_evaluation
PENDING
Competing worker pool (consumer fleet)
Worker-A
Status: Idle (Polling)Processed: 14 tasks
Worker-B
Status: Processing msg_102Processed: 11 tasks
Worker-C
Status: Idle (Polling)Processed: 9 tasks
Clients1× instanceDNS1× instanceLoad balancer3× instancesAPI servers6× instancesLog-based messagebroker1× instanceTranscoding workers8× instancesObject storage1× instanceClients1× instanceDNS1× instanceLoad balancer3× instancesAPI servers6× instancesLog-based messagebroker1× instanceTranscoding workers8× instancesObject storage1× instance
Figure 1

Uploads answered at once, with the transcoding done by workers later.

Follow an upload from a client: an API server publishes a transcoding job to the broker and answers at once with 202 Accepted, so its thread is free again in milliseconds. Workers take jobs from the broker at their own pace and write the finished videos to object storage. The slow work no longer holds a request thread, so it cannot starve the API servers.

  • Clients: 1 × Users
  • DNS: 1 × DNS
  • Load balancer: 3 × Load balancer
  • API servers: 6 × App server
  • Log-based message broker: 1 × Queue
  • Transcoding workers: 8 × App server
  • Object storage: 1 × Object store
  • Clients sends requests to DNS
  • DNS sends requests to Load balancer
  • Load balancer sends requests to API servers
  • API servers sends requests to Log-based message broker
  • Log-based message broker sends requests to Transcoding workers
  • Transcoding workers sends requests to Object storage

Key takeaways

  • 200 threads holding requests for 5 seconds each serve at most 40 requests a second; past that, requests queue and time out.
  • Publish a job, answer at once (for example 202 Accepted) and let workers do the work; queues add delay and complexity, so keep cheap work synchronous.
  • A task queue removes each message once one worker processes it; a log-based broker keeps events so every consumer group reads at its own offset and can replay.
  • Kafka keeps order per partition, and a consumer group can use at most one consumer per partition.
Sources4