Log shipping, synchronous vs asynchronous replication, replication lag anomalies, and split-brain.
What you will learn
Explain how a primary's log feeds its read replicas, and why replicas do not add read capacity linearly.
Compare asynchronous, synchronous and semisynchronous replication by latency and data loss.
Fix read-your-writes and monotonic-read anomalies, and guard multi-leader writes against split-brain.
Relational database scaling fundamentals
Single-node relationaldatabase management systems (RDBMS) like PostgreSQL and MySQL are constrained by the physical limits of a single server: diskI/O operations per second (IOPS), memorybuffer pool capacity, CPU core count, and connection limits.
To scale beyond a single machine, we decouple reads and writes through data replication.
1. Single-leader (primary-replica) replication
In a single-leader topology, one node takes every write. Redis and the System Design Primer call that node the master, and the arrangement master-replica:
Primary (leader): Serves reads and accepts all writes (INSERT, UPDATE, DELETE). Each change is first recorded in a log: PostgreSQL's write-ahead log (WAL) is flushed to disk before the data files are changed; MySQLreplicas copy the binary log (binlog).
Read replicas (followers): Copy the primary's log and replay it, then serve read-only queries (SELECT). If the primary goes offline, the system can keep serving reads until a replica is promoted.
Adding replicas adds read capacity, but not linearly: every replica must replay every write, so a write-heavy primary can bog the replicas down, and more replicas mean more to replicate and more replication lag. On some systems the primary applies writes with many threads while a replica replays them with one.
Replication synchronization modes
Mode
What a commit waits for
Write response time
Data loss if the primary fails
Asynchronous (PostgreSQL's default)
Nothing beyond the primary
Lowest
Transactions not yet replicated are lost, in proportion to the lag
Synchronous (PostgreSQL)
A configured number of synchronous standbys (FIRST n or ANY n) to write the commit to their WAL; with remote_apply, to replay it so it is visible to reads there
Highest
None for committed transactions; a slow synchronous standby slows every commit
Semisynchronous (MySQL)
At least one replica to acknowledge it has received and logged the transaction
In between
None while a replica acknowledges; after a timeout the primary reverts to asynchronous
2. Replication lag anomalies & mitigations
Under asynchronous replication, network jitter or heavy batch updates cause replication lag (tlagwhat its symbols mean). This creates classic distributed consistency anomalies:
A. Read-your-writes
Anomaly: A user updates their profile avatar, reloads the page, but reads from a replica that has not received the update. The user sees their old avatar.
Solutions:
Route that user's reads to the primary for a short time (e.g. 5 seconds) after any write they make.
Remember the position of the user's last write in the replication log and only read from a replica that has applied it. In MySQL, record the write's global transaction identifier (GTID) and call WAIT_FOR_EXECUTED_GTID_SET on the replica first.
In PostgreSQL, synchronous_commit = remote_apply makes each commit wait until the synchronousstandbys have replayed it, so reads there see it.
B. Monotonic reads
Anomaly: A user reads from Replica A (little lag) and sees a new comment, then refreshes and hits Replica B (more lag), and the comment disappears: time appears to go backwards.
Solution: Send each user's reads to the same replica (for example, chosen by hashing user_id), so successive reads never come from a replica that is further behind.
3. Multi-leader (active-active) replication and split-brain
In multi-leader topologies, more than one node accepts writes, and the leaders coordinate with each other on writes.
Write conflicts: If two leaders accept conflicting updates to the same row at the same time, the conflict must be resolved later. A data store can only apply simple policies such as last-write-wins; the application, which knows what the data means, can merge versions instead (Amazon's Dynamo, for example, keeps conflicting versions, tracked with vector clocks, and lets the application reconcile them). Conflicts become more common as you add write nodes and as latency between them grows.
Trade-off: Most multi-leader systems are either loosely consistent (violating ACID) or respond to writes more slowly because the leaders synchronize.
Split-brain: If a network partition separates the leaders (or a failover goes wrong), two nodes can each believe they are the leader and accept writes that conflict.
Fencing tokens: One defense: the lock or leader-electionservice hands out a number that increases every time leadership changes. The storage layer remembers the highest number it has seen and rejects writes carrying an older one, so a deposed leader cannot overwrite newer data.
Active-passive heartbeat and promotion
Virtual IP (VIP) router192.168.1.100Routing to: Node A (Active)
Node A (primary instance)Active leader
Serving: 5,000 RPSHeartbeat: OK (10ms)
Node B (standby replica)Passive standby
Serving: 0 RPS (warm spare)Monitoring node A
Failover event:Normal operation: primary processes all 5,000 RPS. Standby consumes 0 user traffic, maintaining binlog sync.
Writes1× instanceReads1× instancePrimary2× instancesReplicas in the US3× instancesReplicas in Europe3× instancesWrites1× instanceReads1× instancePrimary2× instancesReplicas in the US3× instancesReplicas in Europe3× instances
Figure 1
A primary with a standby and two groups of read replicas.
Follow a write: it goes to the primary, which records it in its log; the replicas copy that log and replay it, and reads go to the replicas. Replicated asynchronously, a replica can miss a write until it arrives, which is replication lag. Adding replicas adds read capacity but not linearly, because every replica must replay every write.
Writes: 1 × Users
Reads: 1 × Users
Primary: 2 × Relational database
Replicas in the US: 3 × Read replica
Replicas in Europe: 3 × Read replica
Writes sends requests to Primary
Replicas in the US keeps a copy of Primary
Replicas in Europe keeps a copy of Primary
Reads sends requests to Replicas in the US
Reads sends requests to Replicas in Europe
Key takeaways
Replicas replay every write the primary makes, so a write-heavy primary slows them down and more replicas mean more lag.
Asynchronouscommits are fastest but can lose unreplicated transactions; synchronous commits lose none but wait for the configured standbys, so a slow one slows every commit; MySQL's semisynchronous sits between.
Route a user's reads to the primary just after their write, or to a replica that has applied it, for read-your-writes; pin each user to one replica for monotonic reads.
Multi-leader replication needs conflict resolution, and fencing tokens stop a deposed leader overwriting newer data after a split-brain.