Module 04 · Day 4 · Kafka 4.3

Replication

Each partition lives on several brokers. One leads, the others copy. This module explains which records are safe, which records consumers can see, and what happens to both when a broker fails.

Try it: three copies of one partition

Green records are copied to every in-sync broker: they are safe, and consumers can read them. Amber records exist only on some brokers so far.

offset 0

What it is and why it exists

A single broker can fail, and its disk can be lost. So Kafka keeps each partition on several brokers. The number of copies is the replication factor. Each copy is a replica.

For each partition, one replica is the leader. Producers write to the leader and consumers normally read from it. The other replicas are followers. A follower does one thing: it fetches new records from the leader and appends them to its own log, in the same order.

Kafka tracks which followers are up to date. That set, with the leader, is the in-sync replicas, usually written ISR. A record is committed when every in-sync replica has it. Only committed records are shown to consumers, and only an in-sync replica may become the new leader. These two rules together mean that a consumer never sees a record that a later leader does not have.

The one-sentence version. Followers copy the leader, the in-sync set defines which records are committed, and a new leader is chosen from that set so that committed records survive.

offset 1

Mental model

Think of one writer and two people who copy the writer's notebook line by line. A line counts as official when all the copiers who are keeping up have it. If a copier falls far behind, the writer stops waiting for that person. If the writer leaves, one of the copiers who was keeping up takes over, and nothing official is lost.

Four numbers describe a partition at any moment.

Term	Meaning
Log end offset	Each replica has its own: the offset its next record will get.
In-sync replicas (ISR)	The leader plus the followers that have caught up within `replica.lag.time.max.ms`, 30 seconds.
High watermark	The lowest log end offset among the in-sync replicas. Everything below it is committed.
`min.insync.replicas`	The smallest in-sync set for which the leader still accepts writes with `acks=all`.

Use the demo at the top of the page. Write three records, let the followers copy, and watch the green area grow. Then make broker 3 slow and write more: the green area stops, because broker 3 is still in the in-sync set. Let 30 seconds pass: broker 3 is removed, and the green area jumps forward.

How many failures can you survive?

Try it: replication factor and the in-sync minimum

Replication factor min.insync.replicas

–brokers can fail and acks=all writes continue

–brokers can be lost for ever without losing a committed record

offset 2

How it works internally

How a follower copies

The follower sends a fetch request to the leader. The request says "give me records from offset N", where N is the follower's log end offset.
The leader answers with records, and with the current high watermark.
The follower appends the records and remembers the high watermark.
The next fetch request asks from a higher offset. This tells the leader that the follower has everything before it.
The leader moves the high watermark to the lowest position that all in-sync replicas have reached. Producers that wait with acks=all get their answer.

A follower learns a new high watermark one fetch later than the leader. This small delay matters when a consumer reads from a follower.

Leaving and joining the in-sync set

A follower is removed from the in-sync set when it has not caught up to the leader's log end for replica.lag.time.max.ms. The rule is about time, not about a number of records. A follower that is slow but steady stays in. A follower that is stopped is removed after 30 seconds, or at once if the broker shuts down cleanly.

The leader proposes each change to the controller, the part of Kafka that owns cluster metadata (module 05). A follower rejoins when it has caught up to the leader's log end offset.

Choosing a new leader

When a leader fails, the controller picks a new one from the in-sync set. In the lab this took about one second. Clients get an error on their next request, refresh their metadata, and continue with the new leader.

The first broker in a partition's replica list is the preferred leader. Kafka spreads preferred leaders evenly when it creates a topic. After failures, leadership drifts to fewer brokers. Kafka moves it back by itself (auto.leader.rebalance.enable), or you can ask for it with a tool.

Leader epochs: how logs are made equal again

Each time a partition gets a new leader, its leader epoch goes up by one. Every batch stores the epoch of the leader that wrote it. Each replica keeps a small file, leader-epoch-checkpoint, that lists each epoch and the offset where it started.

When a broker returns as a follower, it may hold records that the new leader never had. It asks the leader: "where did epoch N end for you?" If its own log is longer for that epoch, it cuts off the extra records and then copies from the leader. This is truncation.

Truncation of records above the high watermark is normal and harmless: those records were never committed. Truncation below the high watermark is data loss. It happens only after an unclean election.

Unclean leader election

Suppose every in-sync replica is down. The partition has no leader and is offline. You have two choices. Wait for an in-sync replica to return: no data is lost, but the partition is unavailable. Or let a replica that was not in sync become leader: the partition works again, but every record that replica did not have is gone. The second choice is an unclean leader election. It is off by default.

Step through it: an unclean election, as it happened in the lab

broker 2

broker 3

Eligible leader replicas

Kafka 4 adds a second list for each partition, the eligible leader replicas, shown as Elr in the describe output. When the in-sync set becomes smaller than min.insync.replicas, a replica that leaves the set is not forgotten. It moves to this list, because it is known to hold every committed record. The high watermark does not move while the in-sync set is below the minimum. If the in-sync set becomes empty, the controller may still elect a replica from this list safely.

You saw both effects in earlier labs. In module 02, a record written with acks=1 below the minimum was accepted but not visible. In this module's lab, a partition with no leader showed Isr: empty and Elr: 2: the controller knew that broker 2 was the safe choice and waited for it. unverified beyond what the lab showed

Racks and zones

If all replicas of a partition sit in one rack or one cloud zone, one failure takes all of them. Set broker.rack on each broker. Kafka then places the replicas of each partition in different racks when it creates a topic.

Moving replicas

To add a broker, remove one, or balance load, you move replicas with the reassignment tool. The new replica copies the whole partition, joins the in-sync set, and then the old replica is removed. The copy uses network and disk, so on a busy cluster you set a throttle to protect normal traffic.

offset 3

Every relevant config

The defaults below were read from the lab cluster with kafka-configs.sh --describe --all, where the tool marks each value as a default or an override.

Config	Where	Default	What it does	When to change it
replication factor	Set when a topic is created	from `default.replication.factor`	Number of copies of each partition.	Use 3 in production. Use 5 only for data that must survive two failures while writes continue.
`default.replication.factor`	Broker	1 (the lab sets 3)	Replication factor for topics created without one.	Set 3 on every production cluster.
`min.insync.replicas`	Topic or broker	1 (the lab sets 2)	Smallest in-sync set that accepts `acks=all` writes.	Set 2 with a replication factor of 3. With the default of 1, `acks=all` can mean "one broker".
`unclean.leader.election.enable`	Topic or broker	`false`	Allows a replica that was not in sync to become leader.	Turn on only for topics where being available matters more than losing records, such as metrics.
`replica.lag.time.max.ms`	Broker	30000	A follower that has not caught up for this long leaves the in-sync set.	Lower it to drop a slow follower sooner, so that `acks=all` latency recovers faster. Too low makes the set unstable.
`num.replica.fetchers`	Broker	1	Threads a follower uses to copy from each leader broker.	Raise it when followers cannot keep up on a busy cluster.
`replica.fetch.max.bytes`	Broker	1048576	Most data a follower takes from one partition in one fetch.	Raise it together with the largest allowed record size.
`replica.fetch.wait.max.ms`, `replica.fetch.min.bytes`	Broker	500, 1	How long the leader may hold a follower's fetch when there is no new data.	Rarely.
`replica.fetch.backoff.ms`	Broker	1000	Pause after a failed fetch.	Rarely.
`replica.socket.timeout.ms`	Broker	30000	Network timeout for follower fetches.	Rarely.
`replica.high.watermark.checkpoint.interval.ms`	Broker	5000	How often the high watermark is saved to disk.	Rarely.
`auto.leader.rebalance.enable`	Broker	`true`	Moves leadership back to preferred leaders in the background.	Leave it on.
`leader.imbalance.check.interval.seconds`	Broker	300	How often that check runs.	Rarely.
`broker.rack`	Broker	null	The rack or zone of this broker.	Always set it when brokers are in more than one rack or zone.
`replica.selector.class`	Broker	null	Lets consumers fetch from a follower in their own rack.	Set the rack-aware selector in multi-zone clusters to cut cross-zone traffic.
`controlled.shutdown.enable`	Broker	`true`	On a clean stop, the broker first hands its leaderships to other brokers.	Leave it on.
`acks`	Producer	`all`	See module 02. Replication protects a record only when the producer waits for it.	—

offset 4

Failure scenarios

A leader fails

What happens: The controller elects a new leader from the in-sync set. In the lab, about 1 second after the broker stopped.
You observe: Clients log "not leader" errors for a moment, refresh metadata, and continue. The describe output shows a smaller Isr.
Risk: None for committed records. Producers with acks=1 can lose the records the old leader had not yet shared.

Under-replicated partitions

What breaks: Nothing yet. The in-sync set is smaller than the replica list, so you have less protection than you paid for.
You observe: kafka-topics.sh --describe --under-replicated-partitions lists them. The broker metric "under-replicated partitions" is above 0.
Cause: A broker is down, or slow (disk, network, long pauses), or it cannot keep up with the write rate.
Fix: Treat a value above 0 that does not go away as an incident. Find the slow broker before a second one fails.

The in-sync set falls below the minimum

What breaks: Producers with acks=all get NOT_ENOUGH_REPLICAS. Consumers can still read what is already committed.
You observe: kafka-topics.sh --describe --under-min-isr-partitions lists them.
Fix: Restore the missing broker. This state is the system working as designed: it refuses writes it cannot protect.

A partition has no leader

What breaks: No reads and no writes for that partition.
You observe: Leader: none. In the lab: Isr: empty, Elr: 2. A producer times out with "Expiring 1 record(s) … The request has not been sent".
Choice: Bring back a broker from the Elr or last in-sync set: no loss. Or force an unclean election: the partition works again and some records are lost. Make this decision for each topic before the incident, not during it.

Unclean election loses committed records

What breaks: Records that producers were told are safe disappear. Consumers that already read them now hold offsets that point to different records.
You observe: The end offset goes backwards. In the lab it went from 20 to 10, and the returning broker logged: "Truncating risky2-0 to offset 10 below high watermark 20".
Fix: There is none after the fact. Prevent it: three replicas, a minimum of 2, replicas in different racks, and unclean election off.

One slow follower makes every write slow

What breaks: Latency for acks=all rises on partitions that the slow broker follows.
Cause: While the follower is still in the in-sync set, the leader waits for it. It stays in for up to 30 seconds after it stops keeping up, and a follower that is slow but steady may never be removed.
Fix: Find the slow broker by its follower fetch metrics. Fix it or remove it from service. This is why uneven hardware in one cluster is a problem.

A reassignment overloads the cluster

What breaks: Moving many large partitions at once uses the network and disks. Client latency rises and followers fall out of sync.
Fix: Move in small groups and set a throttle with the --throttle option of the reassignment tool. Run --verify at the end, which also removes the throttle.

offset 5

Hands-on lab

Every command below was run on 4 October 2026 against the lab cluster. This lab stops and starts brokers. Never stop two brokers of the lab cluster at the same time except where a step says so: two of three must run for the cluster to manage itself (module 05).

k() { docker exec -i kafka-1 /opt/kafka/bin/"$@"; }
B=kafka-1:19092
d() { k kafka-topics.sh --bootstrap-server $B --describe --topic $1 | tail -1; }

Step 1. Stop a leader

Create a topic whose replica list is 3, 1, 2. Broker 3 is the preferred leader. Give only --replica-assignment; the tool refuses it together with --partitions.

k kafka-topics.sh --bootstrap-server $B --create --topic ha2 --replica-assignment 3:1:2
seq 1 100 | k kafka-console-producer.sh --bootstrap-server $B --topic ha2
d ha2
docker exec kafka-1 cat /tmp/kraft-combined-logs/ha2-0/leader-epoch-checkpoint

docker stop kafka-3
d ha2

Topic: ha2  Partition: 0  Leader: 3  Replicas: 3,1,2  Isr: 3,1,2  Elr:  LastKnownElr:
0
1
0 0                ← epoch 0 started at offset 0

Topic: ha2  Partition: 0  Leader: 1  Replicas: 3,1,2  Isr: 1,2    Elr:  LastKnownElr:

Broker 1 became leader about one second after broker 3 stopped. The in-sync set is now 1 and 2.

Step 2. Write under the new leader and read the epoch file

seq 101 150 | k kafka-console-producer.sh --bootstrap-server $B --topic ha2
docker exec kafka-1 cat /tmp/kraft-combined-logs/ha2-0/leader-epoch-checkpoint

0
2
0 0
1 100              ← epoch 1 started at offset 100

The file now has two lines of data. Epoch 0, under broker 3, covers offsets 0 to 99. Epoch 1, under broker 1, starts at offset 100. A returning replica uses exactly this to find where its log may differ.

Step 3. Bring the broker back and restore the preferred leader

docker start kafka-3
# wait 14 seconds
d ha2
k kafka-leader-election.sh --bootstrap-server $B --election-type preferred --topic ha2 --partition 0
d ha2

Topic: ha2  Partition: 0  Leader: 1  Replicas: 3,1,2  Isr: 1,2,3
Successfully completed leader election (PREFERRED) for partitions ha2-0
Topic: ha2  Partition: 0  Leader: 3  Replicas: 3,1,2  Isr: 1,2,3

Broker 3 copied the 50 records it missed, rejoined the in-sync set, and then took leadership back. All 150 records are there.

Step 4. Lose data with an unclean election

This topic has two replicas, on brokers 2 and 3, and a minimum of 1. That is a weak setup, chosen so that the failure is easy to cause.

k kafka-topics.sh --bootstrap-server $B --create --topic risky2 --replica-assignment 2:3 \
  --config min.insync.replicas=1
seq 1 10 | k kafka-console-producer.sh --bootstrap-server $B --topic risky2

docker stop kafka-3            # broker 3 now misses everything that follows
# wait 8 seconds
seq 11 20 | k kafka-console-producer.sh --bootstrap-server $B --topic risky2
k kafka-get-offsets.sh --bootstrap-server $B --topic risky2

docker stop kafka-2; docker start kafka-3    # the only in-sync broker is gone
# wait 20 seconds
d risky2

risky2:0:20

Topic: risky2  Partition: 0  Leader: none  Replicas: 2,3  Isr:   Elr: 2  LastKnownElr: 2

Broker 3 is running, but it has only 10 of the 20 records, so Kafka does not make it leader. The partition is offline. Now force it:

k kafka-leader-election.sh --bootstrap-server $B --election-type unclean --topic risky2 --partition 0
d risky2
k kafka-get-offsets.sh --bootstrap-server $B --topic risky2

Successfully completed leader election (UNCLEAN) for partitions risky2-0
Topic: risky2  Partition: 0  Leader: 3  Replicas: 2,3  Isr: 3
risky2:0:10

The end offset went from 20 back to 10. Write five new records and bring broker 2 back:

seq 21 25 | k kafka-console-producer.sh --bootstrap-server $B --topic risky2
docker start kafka-2
# wait 16 seconds
docker exec kafka-2 /opt/kafka/bin/kafka-dump-log.sh --print-data-log \
  --files /tmp/kraft-combined-logs/risky2-0/00000000000000000000.log | grep -o 'payload: .*'
docker logs kafka-2 2>&1 | grep -i 'risky2-0.*truncat'

values on broker 2:  1 2 3 4 5 6 7 8 9 10 21 22 23 24 25

WARN [UnifiedLog partition=risky2-0] Truncating risky2-0 to offset 10 below high watermark 20

Broker 2 had values 11 to 20. When it returned as a follower, it cut them off to match the new leader. Offsets 10 to 14 now hold the values 21 to 25. A consumer that had committed offset 20 would now be past the end of the log.

One more finding. Setting unclean.leader.election.enable=true on the topic did not elect a leader within 8 seconds in my first attempt. The explicit election command did, at once.

Step 5. Move a replica

k kafka-topics.sh --bootstrap-server $B --create --topic move --replica-assignment 1:2
docker exec kafka-1 sh -c 'echo "{\"version\":1,\"partitions\":[{\"topic\":\"move\",\"partition\":0,\"replicas\":[1,3]}]}" > /tmp/move.json'
k kafka-reassign-partitions.sh --bootstrap-server $B --reassignment-json-file /tmp/move.json --execute
k kafka-reassign-partitions.sh --bootstrap-server $B --reassignment-json-file /tmp/move.json --verify
d move

Successfully started partition reassignment for move-0
Reassignment of partition move-0 is completed.
Topic: move  Partition: 0  Leader: 1  Replicas: 1,3  Isr: 1,3

The replica moved from broker 2 to broker 3. On a real cluster, add --throttle with a byte rate to the execute command.

offset 6

Common mistakes and misconceptions

People think	What is true
Replication factor 3 means no data loss.	Only with `acks=all`, `min.insync.replicas` of 2 or more, and unclean election off. With the default minimum of 1, one broker can hold the only copy of an acknowledged record.
Set `min.insync.replicas` equal to the replication factor for the most safety.	Then one stopped broker blocks all writes, including during a routine restart.
A follower is out of sync when it is many records behind.	The rule is time: it has not caught up for 30 seconds.
Consumers can read a record as soon as the leader has it.	They can read it when it is below the high watermark: when all in-sync replicas have it.
Replication is a backup.	Replication copies every write, including a wrong delete or a bad record. It protects against broker failure, not against mistakes.
Kafka uses majority voting for partition data.	It uses the in-sync set, which can shrink to one broker. Majority voting is used only for cluster metadata (module 05).
Truncation always means data loss.	Truncation above the high watermark removes records that were never committed. Only truncation below it is loss.

offset 7

Staff-level questions

Why is "3 replicas, minimum 2" the standard?

It survives one broker failure with writes still accepted, because two in-sync replicas remain. Every acknowledged record is on at least two brokers, so one permanent loss cannot remove it. And you can restart brokers one at a time without stopping producers.

A minimum of 1 gives up the durability. A minimum of 3 gives up availability during any restart. Two replicas with a minimum of 2 also stops writes on any single failure.

Why does Kafka use an in-sync set and not a majority, as many databases do?

With majority voting, surviving f failures needs 2f + 1 copies. With an in-sync set, f + 1 copies are enough, because any member of the set has every committed record. For large data this saves a great deal of disk.

The cost is that a write waits for the slowest in-sync replica, where a majority system waits only for the faster half. Kafka limits this with the time rule that removes a slow follower. Kafka does use majority voting for its metadata, where the data is small and latency matters more.

You must restart every broker for an upgrade. How do you avoid an outage?

One broker at a time. Before each restart, check that no partition is under-replicated. Stop the broker cleanly so that it hands over leadership first. After it starts, wait until it has rejoined every in-sync set before you touch the next one.

This works only if every topic has at least 3 replicas and a minimum of 2. Find the topics that do not, before the upgrade, with a describe of all topics.

How do you spread a cluster over three zones?

Set broker.rack to the zone. With three replicas, each partition gets one replica in each zone. With a minimum of 2, losing a whole zone leaves two in-sync replicas, and writes continue.

Check the costs. Every write crosses zones twice, which adds latency and network charges. Consumers can read from the replica in their own zone to save the read traffic. The controllers must also be spread over the three zones.

During an incident a partition is offline and the business wants it back now. What do you weigh?

What is lost with an unclean election: every record the chosen replica does not have, and the meaning of every committed consumer offset beyond that point. Downstream systems may already have acted on records that will no longer exist.

What it costs to wait: the time to restart the in-sync broker or recover its disk. If the broker is only stopped, waiting minutes is almost always better. If its disk is destroyed, the data is gone in either case, and electing the best remaining replica is right. The describe output's Elr and LastKnownElr tell you which broker to bring back.

What makes a follower fall out of sync under normal load?

A slow disk on the follower. A saturated network link. Long pauses of the Java runtime. One fetcher thread that copies too many busy partitions. A leader that serves very old data to another reader from disk and evicts recent data from memory.

Look at follower fetch lag and at how often the in-sync set shrinks and expands. Frequent changes with no broker failures point to a broker at its limit.

offset 8

Self-test

1. What is the in-sync set?

The leader plus the followers that have caught up with the leader within replica.lag.time.max.ms.

2. What is the high watermark?

The lowest log end offset among the in-sync replicas. Records below it are committed and visible to consumers.

3. Three replicas, minimum 2. Two brokers are down. What can clients do?

Consumers can read committed records. Producers with acks=all are refused. Producers with acks=1 are accepted, but their records are not visible until the in-sync set reaches the minimum again.

4. What is the preferred leader of a partition?

The first broker in its replica list.

5. What does the leader epoch count?

Leader changes for one partition. It goes up by one with each new leader.

6. A broker returns and cuts records from its log. When is that harmless?

When the records were above the high watermark: they were never committed.

7. What does an unclean leader election trade?

It makes the partition available again and loses the records that the new leader did not have.

8. What does Elr in the describe output list?

Eligible leader replicas: brokers outside the in-sync set that are known to hold every committed record and can be elected safely.

9. Which command lists partitions with fewer in-sync replicas than replicas?

kafka-topics.sh --describe --under-replicated-partitions.

10. Why is replication not a backup?

It copies mistakes as faithfully as good data. A deleted topic is deleted on every replica.