One Writer, Many Readers
The most common replication strategy in databases is embarrassingly simple: one node accepts all writes (the leader), and it streams those changes to one or more followers that handle reads. PostgreSQL, MySQL, MongoDB, Redis, Kafka — they all use this pattern.
Why not let every node accept writes? Because distributed consensus on every write is expensive. A single leader avoids write conflicts entirely. There is exactly one source of truth for the order of operations. Followers just replay the leader's log.
Amazon RDS PostgreSQL can support up to 15 read replicas per primary instance. A single db.r6g.16xlarge leader with 15 followers can serve over 1 million read queries per second.
How Does Replication Actually Work?
The leader writes every change to a write-ahead log (WAL). Followers connect to the leader and stream this log in real time. Each follower applies the log entries in order, maintaining an identical copy of the data. This is log-based replication.
Figure 1: The leader accepts all writes and streams changes via WAL to followers. Followers serve read queries, scaling read throughput horizontally.
Synchronous vs. Asynchronous Replication
Synchronous replication: the leader waits for at least one follower to confirm it has written the change before acknowledging the client. If the leader crashes, the synchronized follower has the latest data. PostgreSQL calls this a "synchronous standby." The tradeoff: every write is slower by at least one network round trip.
Asynchronous replication: the leader acknowledges the client immediately after writing locally. Followers catch up in the background. Writes are fast, but if the leader crashes before the followers replicate, those writes are lost. MySQL's default replication is async.
Semi-synchronous: wait for one follower to ack, send async to the rest. This is the sweet spot most production systems use. MySQL's semi-sync replication and PostgreSQL's synchronous_commit = remote_write both implement this.
What Happens When the Leader Dies?
This is the hardest problem in leader-follower replication: failover. When the leader becomes unreachable, the system must promote a follower to become the new leader. This involves:
- Detecting the failure: usually via heartbeat timeouts. If the leader doesn't respond for N seconds, it's declared dead. But network partitions can cause false positives — the leader might still be alive and accepting writes.
- Choosing a new leader: the follower with the most up-to-date replication log wins. But with async replication, all followers might be behind the old leader.
- Reconfiguring the system: all clients must now send writes to the new leader. The old leader, if it comes back, must become a follower.
GitHub's 2018 outage lasted 24 hours because a 43-second network partition triggered a failover, but the new MySQL primary was 30 seconds behind the old one. Those 30 seconds of lost writes cascaded into inconsistent data across their entire system.
Replication Lag: The Silent Problem
With async replication, followers are always slightly behind the leader. Usually milliseconds. But under load — large transactions, network congestion, slow disks — lag can grow to seconds or minutes. This creates bizarre bugs:
- Read-your-own-writes violation: you update your profile, refresh the page (reads from a lagging follower), and your old profile appears. The fix: always read your own writes from the leader.
- Monotonic read violation: two consecutive reads hit different followers with different lag. The second read returns older data than the first. Time appears to go backward. The fix: pin a user's reads to the same follower (session stickiness).
- Causal ordering violation: user A writes a comment, user B replies. A follower that received B's reply but not A's comment shows a reply to a nonexistent comment.
How Kafka Uses Leader-Follower
Kafka partitions each use leader-follower replication. Each partition has one leader broker and N-1 follower brokers (called ISR — In-Sync Replicas). Producers write to the leader. The leader waits for all ISR members to replicate before acknowledging (when acks=all). If the leader broker dies, a follower from the ISR is elected as the new leader. Kafka's controller node handles this election in milliseconds.
When to Use Leader-Follower
Leader-follower works best when your workload is read-heavy (10:1 or higher read-to-write ratio). Add followers to scale reads. The leader handles the write load alone. If you need to scale writes, you need sharding (splitting data across multiple leaders) or active-active replication (multiple leaders accepting writes, with conflict resolution).