Lesson 14.4• 9 min read

Replication, failover and client routing

Smart clients, proxies and Redis Cluster’s redirects; asynchronous replication and partial resynchronization; Sentinel and Redis Cluster failover, and the writes they can lose.

What you will learn

  • Compare smart clients, a proxy tier and Redis Cluster redirects for finding the right server.
  • Explain Redis replication: asynchronous streams, replication IDs and partial resynchronization.
  • Follow a Sentinel failover and a Redis Cluster failover, and say what writes can be lost.

How a client finds the right server

  1. Smart client. The client library knows the servers, or Redis Cluster's slot map, and connects straight to the right one. This needs no extra hop, but every client holds connections to every server.
  2. Proxy tier. Clients talk to a proxy (Twemproxy, Envoy and others) that hashes the key and forwards the request. Fewer connections reach the cache servers, at the cost of an extra hop and another tier to run.
  3. Redis Cluster redirects. A client may send a command to any node. A node that does not own the slot replies -MOVED 3999 127.0.0.1:6381, and the client should update its slot map. During resharding, a node may reply -ASK instead. That means "try the other node for this one request"; the client should not update its map yet.

Replication (Redis)

From Redis's replication documentation:

  • "Redis uses by default asynchronous replication." The primary sends its write commands to replicas as a stream, without waiting for them.
  • Each dataset history has a replication ID and a replication offset. A reconnecting replica sends both with PSYNC. If the primary still has the missed part in its replication backlog, it sends only that part, a partial resync. Otherwise the replica needs a full copy. The default repl-backlog-size is 1 MB in redis.conf; size it for the outages you want to survive.
  • WAIT can wait for replicas to acknowledge a write, but "it does not turn a set of Redis instances into a CP system": acknowledged writes can still be lost in a failover.

Failover with Redis Sentinel (non-clustered Redis)

Sentinel monitors primaries and replicas and promotes a replica when the primary fails:

  1. A Sentinel that cannot reach the primary for down-after-milliseconds marks it subjectively down (+sdown).
  2. When enough Sentinels agree, the quorum, the primary is objectively down (+odown). But "the quorum is only used to detect the failure". To fail over, the Sentinels hold a leader election: one must be elected leader with "the vote of the majority of the Sentinel processes".
  3. The leader picks a replica by replica-priority, then the largest replication offset, then the smaller run ID, and promotes it.
  4. Clients learn of the new primary from the +switch-master event.

Writes during a partition. If the old primary is cut off together with some clients, those clients "may continue to write data to the old master". That data is "lost forever" when the old primary rejoins as a replica. min-replicas-to-write makes a primary refuse writes when it cannot reach enough replicas, which bounds the loss.


Failover in Redis Cluster

Redis Cluster does not use Sentinel. A node flags another it cannot reach for longer than NODE_TIMEOUT as PFAIL (possible failure), and when a majority of primaries agree, the node is flagged FAIL. A replica of the failed primary then starts an election, and the primaries vote; the replica that wins promotes itself and takes over the slots. The specification warns that, with asynchronous replication, "there is always a window of time" in which acknowledged writes can be lost.

Client routing, proxies and Redis Cluster failover
Compare smart clients, a proxy tier and Redis Cluster redirects; then fail a primary and watch a replica take over.
Smart client
Network path: 1 network hop: client to the node that owns the key
Advantages: No extra hop: the client library knows the servers, or the slot map, and sends each command straight to the node that owns the key
Costs: Every client keeps the map up to date and holds connections to every server
Client application
Key:
GET
CRC16 % 16384 = slot 2396 (shard A)
Test a failure:
Shards (a primary and a replica each)
Failover: healthy
Shard A (0–5460)
Primary A
10.0.1.10 (Online)
Replica A1
10.0.1.11 (Online)
Shard B (5461–10922)Target
Primary B
10.0.1.20 (Online)
Replica B1
10.0.1.21 (Online)
Shard C (10923–16383)
Primary C
10.0.1.30 (Online)
Replica C1
10.0.1.31 (Online)
Log
> Cluster of 3 primaries (A, B, C), each with one replica (A1, B1, C1). The 16,384 slots are split three ways.

Key takeaways

  • A smart client saves a hop but connects to every server; a proxy saves connections but adds a hop and a tier; Redis Cluster nodes answer MOVED so the client updates its map.
  • Redis replicates asynchronously; WAIT waits for replicas but does not make Redis CP, so acknowledged writes can still be lost in a failover.
  • Sentinels agree the primary is down, elect a leader, and promote the replica with the highest priority, then the largest replication offset.
  • Clients cut off with an old primary may keep writing, and those writes are lost when it rejoins as a replica.
Sources6