Use Little's law to see why slow synchronous work starves a thread pool.
Take slow work off the request path with a queue and a quick acknowledgment.
Compare task queues with a log-based message broker such as Kafka, and scale Kafka consumers by partition.
The synchronous bottleneck and thread pool starvation
In a traditional synchronous request-response architecture, an incoming HTTPrequest blocks an operating system or runtime worker thread for the entire duration of downstream processing:
Tsync=tnetwork+tauth+ttranscode+tdisk_io+tnotificationwhat its symbols mean
Each term is the time one step takes: the network, authentication, transcoding, diskI/O and sending a notification.
If video transcoding or PDF rendering takes 3 to 15 seconds, a server with 200 worker threads is fully saturated at just:
Max Concurrency=5 seconds200 threads=40 requests/sec
By Little's law (L=λWwhat its symbols mean: the requests in progress equal the arrival rate times the time each takes), 200 busy threads at 5 seconds each can sustain at most 200/5=40requests per second. Beyond that, requests queue up waiting for a thread, latency climbs, and clients (or proxies in front) start timing out.
Asynchronous decoupling: fast acknowledgment, deferred work
Asynchronousworkflows take expensive operations off the request path. With a message queue:
The application publishes a job to the queue and tells the user the job's status straight away (for example, HTTP 202 Accepted with a status URL). The user is not blocked.
A worker picks up the job, processes it in the background at its own pace, and signals when it is done.
The client can even make the result look immediate: a posted tweet appears on your own timeline at once, though delivering it to all your followers takes longer. The cost: queues add delay and complexity, so cheap calculations and real-time workflows may be better done synchronously.
Common choices differ: Redis is a simple broker but can lose messages; RabbitMQ is popular but uses the AMQP protocol and you run the nodes; Amazon SQS is hosted but can have high latency and may deliver a message twice. Task queues such as Celery go further: they run the task and return its result, and can schedule jobs.
Task queues versus log-based message brokers
Kafka's documentation says it can serve as a kind of external commit log for a distributed system; Designing Data-Intensive Applications calls brokers like it log-based message brokers.
Dimension
Message / task queue (RabbitMQ, SQS, Celery)
Log-based message broker (Kafka)
Storage
A buffer from which messages are removed once processed
An append-only log; events are not deleted when consumed
Consumption
Competing workers take messages; each message goes to one worker
Each consumer group tracks its own position (offset) in each partition
Ordering
Usually best effort, especially across retries (SQS FIFO queues are the exception)
Events with the same key go to the same partition and are read in exactly the order written
Replay
Not normally possible once a message is consumed
Rewind a consumer group's offset, for as long as the topic's retention keeps the events
Event streams read by many consumers, change data capture, audit trails
Kafka is built for throughput: it relies on sequential disk writes and the operating system's page cache, and uses sendfile to send data from the page cache to the network without copying it through the application.
Partitioning and consumer group scaling in Kafka
A Kafkatopic is divided into Pwhat its symbols meanpartitions; by default an event's key is hashed to choose its partition:
Partition=hash(key)modPwhat its symbols mean
Ordering: Events with the same key (e.g. user_id_492) land on the same partition, and every consumer reads a partition's events in the same order (as long as the number of partitions does not change).
Consumer groups: Each partition is consumed by exactly one consumer within a consumer group at any given time, so a group can use at most P consumers in parallel (Nconsumers≤Pwhat its symbols mean); extra consumers sit idle.
Task queue versus log-based message broker
Task queue model (RabbitMQ, SQS): Messages are transient units of work. Competing workers pull items from a shared buffer. Once acknowledged (ACK), the message is deleted permanently. Designed for fine-grained task distribution and arbitrary scale-out.
Uploads answered at once, with the transcoding done by workers later.
Follow an upload from a client: an API server publishes a transcoding job to the broker and answers at once with 202 Accepted, so its thread is free again in milliseconds. Workers take jobs from the broker at their own pace and write the finished videos to object storage. The slow work no longer holds a request thread, so it cannot starve the API servers.
Clients: 1 × Users
DNS: 1 × DNS
Load balancer: 3 × Load balancer
API servers: 6 × App server
Log-based message broker: 1 × Queue
Transcoding workers: 8 × App server
Object storage: 1 × Object store
Clients sends requests to DNS
DNS sends requests to Load balancer
Load balancer sends requests to API servers
API servers sends requests to Log-based message broker
Log-based message broker sends requests to Transcoding workers
Transcoding workers sends requests to Object storage
Key takeaways
200 threads holding requests for 5 seconds each serve at most 40 requests a second; past that, requests queue and time out.
Publish a job, answer at once (for example 202 Accepted) and let workers do the work; queues add delay and complexity, so keep cheap work synchronous.
A task queue removes each message once one worker processes it; a log-based broker keeps events so every consumer group reads at its own offset and can replay.
Kafka keeps order per partition, and a consumer group can use at most one consumer per partition.