The primary stops answering at 02:14. A replica is healthy, the dashboard is red, and an engineer is deciding whether to promote it. Promote too early and the old primary may come back, still accepting writes. Promote too late and the site is down. The core lesson explained copies and sharding. This one covers what happens when they break and when they must move.
Interviewers spend the last fifteen minutes of a data question here. This is the second half of database sharding and replication explained.
- Database sharding and replication explained: primaries, replicas, shard keys and why
hash mod Nmoves most keys. - The CAP theorem for the consistency trade-offs, and consistent hashing for how keys find nodes.
Why can a user save a paste and then not see it? A read went to a replica that had not received the write yet. Reading from the primary for a short window after a write, or waiting for the replica to catch up, fixes it.
- Failover has two risks: losing writes the replica never received, and split-brain, where two primaries both accept writes. Fence the old primary first.
- Leaderless stores stay available through failures by asking several copies. With W + R > N a read overlaps the latest write, at the cost of latency and edge cases.
- Online resharding copies in the background, streams changes, verifies, then switches after a short write pause. Keep a rollback until you are sure.
- Each extra region trades write latency against staleness. State the number you give up.
Walk through a primary failure step by step, work out how many writes can be lost, choose quorum settings for a leaderless store, outline an online resharding with a rollback, and compare options for a second region.
What Happens After the Primary Dies?#
The primary crashes. Assume its replica was 200 milliseconds behind and the system takes 1,000 writes a second. How many acknowledged writes can the replica be missing?
About 200 (0.2 s x 1,000 writes per second), taking both figures as assumptions and the result as an order of magnitude. PostgreSQL’s documentation says that with asynchronous replication “some transactions might be lost” in the switch to a backup. The client was told “done”, yet the new primary never saw them.
The order matters, and step 2 is the one people skip. The same documentation warns that if the old primary restarts, “you must have a mechanism for informing the old primary that it is no longer the primary”, called STONITH, “to avoid situations where both systems think they are the primary, which will lead to confusion and ultimately data loss”. That state is split-brain.
A primary that is slow, not dead, looks the same as a dead one. The documentation notes that a third “witness server” can prevent some inappropriate failovers, though it adds complexity. Two nodes alone cannot tell “the other node died” from “the link between us died”.
Two ways to shrink the loss, each with a price:
- Synchronous standbys. Commits wait until a standby confirms, so an acknowledged write exists in two places. The minimum added wait is “the round-trip time between primary and standby”, per the same documentation.
- Quorum of standbys. PostgreSQL supports
ANY 2 (s1, s2, s3): a commit waits for any two of three. One standby can be down without stalling writes. The docs say to lower the number you wait for, or disable it, if you cannot keep that many standbys.
Whichever you choose, a replica is not a backup. A bad DELETE reaches it within moments.
Asynchronous failover picks availability and speed: the service comes back fast and may lose the last moments of writes. Synchronous standbys pick consistency: no acknowledged write is lost, and writes pause if too few standbys are reachable. Say which one the product needs. A chat message can tolerate the first, a payment cannot.
Can Replication Work Without a Leader?#
Some stores have no primary at all. Any copy accepts a write, and the system keeps copies close by asking several of them. Apache Cassandra’s documentation describes its consistency levels as “a version of Dynamo’s R + W > N consistency mechanism”. Here N is the number of copies, W the copies that must confirm a write, and R the copies a read asks.
The rule: if W + R > N, a quorum read overlaps the quorum that took the write, so it sees the latest acknowledged write. The docs give the example of a replication factor of 3 with QUORUM for both: “at least one of the replicas is guaranteed to participate in both the write and the read”.
N = 3 copies
W=2 R=2: 4>3 overlaps
W=1 R=1: 2<3 may read stale
W=3 R=1: 4>3 but writes need allSo the choice is a dial between speed, safety and availability. The same documentation says weaker levels such as ONE may be used to improve throughput when strong consistency is not required.
Leaderless systems add repair work. Cassandra’s docs list best-effort techniques for convergence: read repair on the read path and hinted handoff on the write path, where a coordinator stores a missed write and replays it later. Hints are best-effort, so a separate anti-entropy repair compares data to guarantee eventual consistency.
If you need transactions across rows or a single clear order of writes, a leader is simpler. A quorum is not a transaction, and two concurrent writes to one key still need a rule for which wins.
How Do You Move Data Without Stopping the Site?#
The shard count that was right last year is wrong now. hash mod N is out because it moves most keys. The usual answer is a fixed set of slots or ranges, so a move shifts whole pieces, plus an online procedure.
Vitess documents this as a workflow. Its MoveTables command creates the copy, lets you validate it with a data diff, then SwitchTraffic. For writes, “we temporarily make the source databases read-only” while the target catches up, and a lag limit can block the switch if replication is too far behind. By default a reverse stream runs back to the source, so ReverseTraffic is a quick rollback. Complete is destructive and drops the old tables.
MongoDB’s resharding follows the same shape. Its manual says the application must “tolerate a period of two seconds where the affected collection blocks writes”, that an operation takes at least 5 minutes, and that each recipient shard needs room for ((collection size + index size) x 2) / shard count. It also asks for I/O below 50% and CPU below 80%, because the copy competes with live traffic.
Assume 2 TB to copy at 100 MB/s, and start while the old shards are well below saturation, as the manual’s limits suggest:
2,000,000 MB / 100 MB/s
= 20,000 s ≈ 5.6 hours
(on top of live traffic)- Starting when the source is already saturated, so the copy slows real users or never catches up.
- Dropping the old copy before a data comparison and a rollback window.
- Resharding to a key that is still hot. A new key with the same skew moves the problem.
What Changes With More Than One Region?#
A second region adds a speed-of-light bill. A write confirmed by another region waits at least one round trip between them. A write confirmed locally returns fast, and the other region is behind. That is the same replication-lag trade as before, now measured in tens or hundreds of milliseconds (the exact figure depends on the pair of regions).
Three shapes, as design judgement rather than measured results:
| Shape | Writes | Reads | Costs |
|---|---|---|---|
| One writer region, read replicas elsewhere | far regions pay a round trip | local, may be stale | failover crosses regions |
| Each user has a home region | local for most users | local | a user who travels pays more |
| Leaderless across regions | local quorum | local quorum | conflicts need a rule |
Cassandra’s docs describe LOCAL_QUORUM as a “weaker but still useful guarantee”: reads see the latest write within the same datacenter, often enough because “clients homed to a single datacenter will read their own writes”.
Managed databases show the same trade in product form. Amazon Aurora Global Database replicates across regions asynchronously. Its docs put recovery time in “the order of minutes” and the recovery point, the data you can lose, typically “in seconds”, set by replication lag when the failure happens.
A planned switchover first waits for the secondary to sync, so its RPO is 0, and the docs give a typical read/write recovery under 30 seconds on qualifying engine versions.
An unplanned failover does not wait. The docs call write fencing of the old primary “best-effort”, warn of split-brain, and save a snapshot of the old volume so missing writes can be recovered.
Decide the loss you accept and the time to recover, then test the failover rather than assume it. Aurora’s docs advise a global writer endpoint with a DNS TTL of about 5 seconds, so clients notice the switch quickly.
Practice: Plan a Failover#
Assume a shop runs one primary and two asynchronous replicas, takes 500 writes a second at peak, and lags by 300 ms at peak. The primary loses power. Work out:
- How many acknowledged orders can the promoted replica be missing?
- What stops the old primary from taking writes when it returns?
- Would you change to synchronous standbys, and what does it cost?
Hint
Multiply the lag in seconds by the writes per second. Then think about which orders the business cannot afford to lose.
One reasonable answer
Lag: 0.3 s
Writes: 500 / s
Missing ≈ 0.3 x 500 = 150About 150 orders at most, assuming the figures hold. Fence the old primary (cut its network or storage, or shut it down) before promoting, so it cannot rejoin as a second primary. For orders and payments, require one synchronous standby or ANY 1 of two, and accept the round trip on each commit. Keep the rest asynchronous.
Expect these after the main design. Answer each with a trade-off:
| They ask | Answer in one line |
|---|---|
| Primary dies? | Fence it, promote the freshest replica |
| Lose any writes? | Async: the lag window. Sync: none |
| Split-brain? | Two primaries; prevent with fencing |
| Quorum math? | W + R > N, so reads overlap writes |
| Add a shard live? | Copy, stream, verify, switch, keep rollback |
| One hot shard? | Better key, then split that range |
| Cross-shard transaction? | Avoid it; else two-phase commit, slower |
| Second region? | Pick latency or freshness, say which |
| Back up a sharded DB? | Per shard, taken at one agreed point |
Check Yourself#
1.Why can a promoted replica be missing writes the client was told succeeded?
Under asynchronous replication, the primary acknowledges a commit before the replica has it. Whatever was in flight when the primary died is lost on promotion. Synchronous standbys close that gap at the price of a round trip per commit.
2.With N=5 copies, what is the smallest W if you read with R=2 and want reads to see the latest write?
W + R > N means W > 3, so W = 4. Reads at R=2 are fast, but writes now need four of five copies, so two failures block writes.
3.A reshard to a new key is planned for next quarter. What would make you delay it?
A source already near its capacity, no data comparison tooling, no rollback path, or a new key that has the same skew as the old one. Fix those first.
Say the failover order aloud in one breath: detect, fence, pick, promote, repoint. Then explain why fence comes before promote. Repeat tomorrow without looking at the diagram.
In One Picture#
Failover detect, FENCE, pick,
promote, repoint
Quorum W + R > N overlaps
Reshard copy, stream, verify,
switch, keep rollback
Regions latency or freshnessFence before you promote, overlap your quorums, and keep a way back.
- The core lesson: sharding and replication explained for copies, splits and shard keys.
- How keys find nodes: consistent hashing.
- The consistency side: CAP theorem explained.
