Replication, failover and client routing
Smart clients, proxies and Redis Cluster’s redirects; asynchronous replication and partial resynchronization; Sentinel and Redis Cluster failover, and the writes they can lose.
What you will learn
- Compare smart clients, a proxy tier and Redis Cluster redirects for finding the right server.
- Explain Redis replication: asynchronous streams, replication IDs and partial resynchronization.
- Follow a Sentinel failover and a Redis Cluster failover, and say what writes can be lost.
How a client finds the right server
- Smart client. The client library knows the servers, or Redis Cluster's slot map, and connects straight to the right one. This needs no extra hop, but every client holds connections to every server.
- Proxy tier. Clients talk to a proxy (Twemproxy, Envoy and others) that hashes the key and forwards the request. Fewer connections reach the cache servers, at the cost of an extra hop and another tier to run.
- Redis Cluster redirects. A client may send a command to any node. A node that does not own the slot replies
-MOVED 3999 127.0.0.1:6381, and the client should update its slot map. During resharding, a node may reply-ASKinstead. That means "try the other node for this one request"; the client should not update its map yet.
Replication (Redis)
From Redis's replication documentation:
- "Redis uses by default asynchronous replication." The primary sends its write commands to replicas as a stream, without waiting for them.
- Each dataset history has a replication ID and a replication offset. A reconnecting replica sends both with
PSYNC. If the primary still has the missed part in its replication backlog, it sends only that part, a partial resync. Otherwise the replica needs a full copy. The defaultrepl-backlog-sizeis 1 MB inredis.conf; size it for the outages you want to survive. WAITcan wait for replicas to acknowledge a write, but "it does not turn a set of Redis instances into a CP system": acknowledged writes can still be lost in a failover.
Failover with Redis Sentinel (non-clustered Redis)
Sentinel monitors primaries and replicas and promotes a replica when the primary fails:
- A Sentinel that cannot reach the primary for
down-after-millisecondsmarks it subjectively down (+sdown). - When enough Sentinels agree, the quorum, the primary is objectively down (
+odown). But "the quorum is only used to detect the failure". To fail over, the Sentinels hold a leader election: one must be elected leader with "the vote of the majority of the Sentinel processes". - The leader picks a replica by
replica-priority, then the largest replication offset, then the smaller run ID, and promotes it. - Clients learn of the new primary from the
+switch-masterevent.
Writes during a partition. If the old primary is cut off together with some clients, those clients "may continue to write data to the old master". That data is "lost forever" when the old primary rejoins as a replica. min-replicas-to-write makes a primary refuse writes when it cannot reach enough replicas, which bounds the loss.
Failover in Redis Cluster
Redis Cluster does not use Sentinel. A node flags another it cannot reach for longer than NODE_TIMEOUT as PFAIL (possible failure), and when a majority of primaries agree, the node is flagged FAIL. A replica of the failed primary then starts an election, and the primaries vote; the replica that wins promotes itself and takes over the slots. The specification warns that, with asynchronous replication, "there is always a window of time" in which acknowledged writes can be lost.
Key takeaways
- A smart client saves a hop but connects to every server; a proxy saves connections but adds a hop and a tier; Redis Cluster nodes answer
MOVEDso the client updates its map. - Redis replicates asynchronously;
WAITwaits for replicas but does not make Redis CP, so acknowledged writes can still be lost in a failover. - Sentinels agree the primary is down, elect a leader, and promote the replica with the highest priority, then the largest replication offset.
- Clients cut off with an old primary may keep writing, and those writes are lost when it rejoins as a replica.
Sources6
- System Design Primer: Fail-over
- System Design Primer: Master-slave replication
- Redis Cluster specification: redirection and failover
- Redis: Replication
- Redis: High availability with Sentinel
- Adapted from The System Design Primer by Donne Martin, licensed under CC BY 4.0. Changes were made.