Module 10 · Day 7 · Kafka 4.3

Multi-cluster and design

The last module is about boundaries: copying data between clusters and regions, using Kafka as a work queue with share groups, and knowing when a different tool is the better answer.

Try it: one partition, several workers

Send jobs to a topic with a single partition. Compare how a consumer group and a share group hand them out. The lab ran this with two real workers.

Group type

Workers

offset 0

What it is and why it exists

One cluster in one place is the simple case. Three needs push beyond it.

Surviving the loss of a whole site. Replication inside a cluster (module 04) protects against a broker failure. It does not protect against the loss of the data centre or region that the cluster is in.
Moving data between places. Applications in one region need data produced in another. Analytics needs a copy that heavy queries cannot disturb. A migration needs two clusters in step for a while.
Work queues. A consumer group ties parallel work to the partition count and delivers in order. Some work, such as sending emails or resizing images, wants the opposite: many workers on any record, with each record acknowledged alone.

Kafka's answers are MirrorMaker 2, which copies topics from one cluster to another; stretch clusters, where one cluster spans several sites; and share groups, a second kind of group that gives Kafka queue behaviour.

The module ends with the most senior skill: saying when Kafka is not the right tool.

The one-sentence version. Copy between clusters when sites must fail independently, stretch one cluster when no record may be lost and the sites are close, use share groups when work is independent, and choose something simpler when you do not need a replayable log.

offset 1

Mental model

Two numbers for every disaster plan

Number	Question it answers
Recovery point objective	How much recent data may be lost? Measured in time: zero, seconds, minutes.
Recovery time objective	How long may the service be down? Measured in time: seconds, minutes, hours.

Every design below is a different trade between these two numbers, cost, and complexity.

Try it: three ways to use two or more sites

Design

–data that can be lost

–time to recover

–cost and complexity

A copy is not the original

MirrorMaker 2 is a consumer on one cluster and a producer on the other. So the copy is a new topic with its own offsets. Offset 500 in the source is not offset 500 in the copy. This one fact causes most of the difficulty: a consumer that moves to the other cluster cannot reuse its committed offsets, so they must be translated.

Queue or log?

	Consumer group	Share group
A partition is read by	One member.	Any number of members.
Most useful workers	The partition count.	No such limit.
Order	Kept inside a partition.	Not guaranteed.
Progress is tracked as	One offset for each partition.	A state for each record: available, taken, done.
A record that fails	Blocks the partition until handled.	Is released and tried again, up to a limit, while other records continue.
Use for	Events where order and replay matter.	Independent jobs.

Both kinds of group can read the same topic at the same time. The data is still a log with retention; a share group only changes how it is handed out.

Should this be Kafka at all?

offset 2

How it works internally

MirrorMaker 2

MirrorMaker 2 is a set of Kafka Connect connectors (module 08). The lab's Connect worker listed all three among its plug-ins.

Connector	Job
Source connector	Copies records, and also creates the topics on the target with the same partition count and settings.
Checkpoint connector	Translates consumer group offsets from the source to the target.
Heartbeat connector	Writes a regular marker so that you can check that the link is alive and measure its delay.

You name each cluster with an alias, such as main and dr, and declare a flow: main->dr.
By default the copy of a topic gets the source alias as a prefix: orders becomes main.orders. The prefix shows where data came from and stops a loop when two clusters copy to each other.
A record keeps its key, value, headers, timestamp and partition number. It gets a new offset.
The source connector writes pairs of matching offsets, source and target, to an offset-syncs topic from time to time.
The checkpoint connector reads each group's committed offsets on the source, uses those pairs to compute a safe offset on the target, and writes it to a checkpoints topic. With sync.group.offsets.enabled it also commits it for the group on the target.

Translation is deliberately cautious. In the lab, the group g-new was at offset 1261 on a source partition and was given 1112 on the copy. After a failover it would read 149 records again. It would not skip any. So a consumer that fails over must tolerate repeats.

The copy is asynchronous. A producer gets its answer from the source cluster before the record reaches the target. In the lab a new record reached the second cluster about 2 seconds later. Whatever is in flight when the source is lost, is lost.

Stretch clusters

A stretch cluster is one normal cluster whose brokers and controllers are placed in three sites. With broker.rack set, each partition has a replica in each site. With acks=all and a minimum of 2, a record is in two sites before it is acknowledged. Losing a site loses no committed record, and failover is the ordinary leader election from module 04. Clients do nothing.

The price is in the network. Every write waits for a round trip between sites, and the controllers need low delay to each other. This works between zones of one cloud region, a millisecond or two apart. It does not work between continents.

Share groups

A consumer joins a share group and is assigned partitions. Several members can have the same partition; in the lab both workers showed jobs:0.
When a member fetches, the broker hands it some available records and marks them as acquired by that member, with a time limit: the lock, 30 seconds by default.
The member processes each record and acknowledges it: accept (done), release (give it back for another try), or reject (give up on it).
If the lock runs out with no acknowledgement, the record becomes available again and another member can take it.
Each record has a delivery count. After 5 deliveries without success, the broker stops offering it.
The broker keeps this per-record state in an internal topic, __share_group_state.

Because records are handed out one batch at a time to whoever asks, order is not kept, and there is no per-partition offset for the application to commit. The group's position is a start offset, below which every record is finished.

The lab cluster reports the feature share.version at level 1, so share groups are on. They are new in the 4.x line. Check the release notes for the status in the version you run, and whether your client library supports them. unverified: I did not confirm the release in which they became production-ready

offset 3

Every relevant config

MirrorMaker 2

The values in the "Lab" column are what the lab used and are verified to work. I could not check the defaults against the reference in this session, so that column is from memory. unverified defaults

MirrorMaker 2 config	Default, from memory	Lab	What it does
`clusters`	required	`main, dr`	The aliases of the clusters.
`<alias>.bootstrap.servers`	required	one for each alias	How to reach each cluster.
`<a>-><b>.enabled`	`false`	`main->dr` on	Turns on one direction of copying.
`<a>-><b>.topics`	all topics	`orders\|payments\|events`	Which topics to copy, as a pattern. New matching topics are picked up.
`groups`	all groups	`g-.*`	Which consumer groups get translated offsets.
`replication.factor`	2	1	Copies of the mirrored topics on the target. The lab's second cluster has one broker. Use 3 in production.
`sync.group.offsets.enabled`	`false`	`true`	Commits translated offsets for groups on the target, for groups with no active member there.
`emit.checkpoints.interval.seconds`, `sync.group.offsets.interval.seconds`	60, 60	10, 10	How often offsets are translated and committed.
`refresh.topics.interval.seconds`, `refresh.groups.interval.seconds`	600, 600	10, 10	How often new topics, new partitions and new groups are looked for.
`replication.policy.class`	the default policy, which adds the alias prefix	default	How target topics are named. An identity policy keeps the same names; use it only for one-way copying.
`sync.topic.configs.enabled`	`true`	default	Copies topic settings to the target.
`<alias>.consumer.isolation.level`	`read_uncommitted`	default	The isolation level of the copying consumer. See the failure section: the default copied aborted records in the lab.
`tasks.max`	1	default	Parallel copy tasks.

Share groups

Broker defaults were read from the lab cluster. The consumer default is from the Kafka 4.3 reference.

Config	Where	Default	What it does
`group.share.record.lock.duration.ms`	Broker	30000	How long a member may hold a record before it is offered again. Allowed range 15000 to 60000.
`group.share.delivery.count.limit`	Broker	5	Deliveries of one record before the broker gives up on it. Allowed range 2 to 10.
`group.share.partition.max.record.locks`	Broker	2000	Records of one partition that can be held by members at once.
`group.share.max.size`	Broker	200	Most members in one share group.
`group.share.session.timeout.ms`, `group.share.heartbeat.interval.ms`	Broker	45000, 5000	Liveness of share group members.
`group.share.assignors`	Broker	`simple`	How partitions are given to members.
`share.coordinator.state.topic.replication.factor`, `share.coordinator.state.topic.min.isr`	Broker	the lab sets 3 and 2	Safety of `__share_group_state`.
`share.coordinator.state.topic.num.partitions`	Broker	50	Partitions of that topic.
`share.acknowledgement.mode`	Consumer	`implicit`	`implicit`: records are accepted when the next poll is called. `explicit`: your code acknowledges each record.

offset 4

Failure scenarios

The copy contains aborted transactions

What breaks: Consumers on the second cluster receive records that were rolled back on the first.
Evidence: In the lab, payments had 78 offsets on the source and main.payments had 73 on the copy. The source held 73 data records and 5 transaction markers. Of the 73 data records, 38 belonged to the transaction that was aborted in module 06. They were copied as ordinary records, and the markers were not.
Cause: The copying consumer reads with the default isolation level, read_uncommitted.
Fix: Set the source consumer's isolation level to read_committed in the MirrorMaker 2 configuration for any flow that carries transactional topics. unverified: I did not rerun the lab with this setting

Records are lost in a failover

What breaks: The main site is lost. Records that were acknowledged there but not yet copied do not exist on the standby.
Cause: Copying between clusters is asynchronous.
Fix: Measure the copy delay all the time and agree that this delay is your possible data loss. If no loss is acceptable, you need a stretch cluster, or the producing application must be able to send again from its own source of truth.

Consumers repeat or skip after a failover

What breaks: A consumer starts on the standby at the wrong place.
Cause: Offsets differ between clusters. Translated offsets are cautious, so some records repeat: 149 for one partition in the lab. If offset translation was not set up, the consumer starts from auto.offset.reset, which can skip everything or repeat everything.
Fix: Turn on offset translation for the groups that must fail over. Make consumers safe to repeat. Remember the topic name changes to the prefixed form unless you use the identity policy.

Failback is harder than failover

What breaks: After the main site returns, the standby holds records the main site never saw, and the main site may hold records that were never copied.
Fix: Plan the return path before you need it. Often the simplest safe plan is to treat the standby as the new main, rebuild the old main as its standby from scratch, and switch back in a quiet period. Practise it.

Two active clusters update the same thing

What breaks: The same customer is changed in both regions at once. Each region applies the other's change late. The regions end with different values.
Cause: There is no order between two clusters.
Fix: Give each entity a home region that owns its writes, or design updates that can be merged in any order. This is an application design problem that Kafka cannot solve for you.

A stretch cluster with a slow link

What breaks: Write latency rises for every producer, followers fall out of sync, and controller elections start.
Cause: The delay between sites is too high or varies.
Fix: Use a stretch cluster only across sites with a short, steady delay. Measure the 99th percentile, not the average.

A share group job is done twice, or gives up

What breaks: A job takes longer than the lock. The broker offers it to a second worker while the first is still working. Or a job fails 5 times and is dropped.
Fix: Keep jobs shorter than the lock time, or raise the lock within its allowed range. Make jobs safe to repeat. Decide what happens to a job that reaches the delivery limit: at the least, log it where someone will see it.

Kafka was the wrong choice

You observe: A team spends more time on the cluster than on the product. Topics carry a few records each hour. Applications wait on a reply topic to imitate a function call.
Fix: Use the checklist in this module at design time. For small volumes, a table in the database you already run, or a managed queue, is cheaper to own.

offset 5

Hands-on lab

Every command below was run on 4 October 2026. Part B uses a second cluster of one broker, kafka-dr, which I added to docker-compose.yml. The flows that were copied are the topics from earlier labs.

k()  { docker exec -i kafka-1  /opt/kafka/bin/"$@"; }
dr() { docker exec -i kafka-dr /opt/kafka/bin/"$@"; }
B=kafka-1:19092; D=kafka-dr:19092

Part A, step 1. Two workers share one partition

k kafka-topics.sh --bootstrap-server $B --create --topic jobs --partitions 1 --replication-factor 3

docker exec -d kafka-1 sh -c '/opt/kafka/bin/kafka-console-share-consumer.sh \
  --bootstrap-server kafka-1:19092 --topic jobs --group workers > /tmp/share-a.txt'
docker exec -d kafka-2 sh -c '/opt/kafka/bin/kafka-console-share-consumer.sh \
  --bootstrap-server kafka-1:19092 --topic jobs --group workers > /tmp/share-b.txt'
# wait 12 seconds
for i in $(seq 1 20); do echo "job-$i" | k kafka-console-producer.sh --bootstrap-server $B --topic jobs; done
# wait 6 seconds
docker exec kafka-1 cat /tmp/share-a.txt
docker exec kafka-2 cat /tmp/share-b.txt

worker A:  job-2 job-4 job-6 job-8 job-10 job-12 job-14 job-16 job-18 job-20
worker B:  job-1 job-3 job-5 job-7 job-9 job-11 job-13 job-15 job-17 job-19

Twenty jobs in one partition, ten for each worker. No job went to both.

Part A, step 2. Look at the share group

k kafka-share-groups.sh --bootstrap-server $B --describe --group workers
k kafka-share-groups.sh --bootstrap-server $B --describe --group workers --members

GROUP    TOPIC  PARTITION  START-OFFSET  LAG
workers  jobs   0          19            0

GROUP    CONSUMER-ID             HOST         CLIENT-ID               #PARTITIONS  ASSIGNMENT
workers  KrADfCvxRA2wNNbQ2n91zQ  /172.23.0.3  console-share-consumer  1            jobs:0
workers  wJoM_f5OQyS0TDxCfg_omA  /172.23.0.2  console-share-consumer  1            jobs:0

Both members are assigned jobs:0, which a consumer group never allows. The position is a start offset, not a committed offset.

Part A, step 3. The same with a consumer group

docker exec -d kafka-1 sh -c '/opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server kafka-1:19092 \
  --topic jobs --group classic-workers --from-beginning > /tmp/cg-a.txt'
docker exec -d kafka-2 sh -c '/opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server kafka-1:19092 \
  --topic jobs --group classic-workers --from-beginning > /tmp/cg-b.txt'
# wait 14 seconds
docker exec kafka-1 sh -c 'wc -l < /tmp/cg-a.txt'; docker exec kafka-2 sh -c 'wc -l < /tmp/cg-b.txt'

docker exec kafka-1 sh -c "pkill -f ConsoleShareConsumer; pkill -f ConsoleConsumer"
docker exec kafka-2 sh -c "pkill -f ConsoleShareConsumer; pkill -f ConsoleConsumer"

20
0

One consumer received all 20 jobs and the other received none: one partition, one reader.

Part B, step 1. Start the second cluster and the copy

docker compose up -d kafka-dr

docker exec kafka-1 sh -c 'cat > /tmp/mm2.properties <<EOF
clusters = main, dr
main.bootstrap.servers = kafka-1:19092
dr.bootstrap.servers = kafka-dr:19092
main->dr.enabled = true
main->dr.topics = orders|payments|events
dr->main.enabled = false
groups = g-.*
replication.factor = 1
checkpoints.topic.replication.factor = 1
heartbeats.topic.replication.factor = 1
offset-syncs.topic.replication.factor = 1
offset.storage.replication.factor = 1
status.storage.replication.factor = 1
config.storage.replication.factor = 1
sync.group.offsets.enabled = true
emit.checkpoints.interval.seconds = 10
sync.group.offsets.interval.seconds = 10
refresh.topics.interval.seconds = 10
refresh.groups.interval.seconds = 10
EOF'
docker exec -d kafka-1 sh -c '/opt/kafka/bin/connect-mirror-maker.sh /tmp/mm2.properties > /tmp/mm2.log 2>&1'
# wait about 45 seconds
dr kafka-topics.sh --bootstrap-server $D --list

__consumer_offsets
heartbeats
main.checkpoints.internal
main.events
main.heartbeats
main.orders
main.payments
mm2-configs.main.internal
mm2-offsets.main.internal
mm2-status.main.internal

The three topics exist on the second cluster with the prefix main.. The other topics are MirrorMaker 2's own: translated offsets in main.checkpoints.internal, the link's heartbeat, and its Connect state. The replication factor of 1 is only because this second cluster has one broker.

Part B, step 2. Compare source and copy

k  kafka-get-offsets.sh --bootstrap-server $B --topic orders
dr kafka-get-offsets.sh --bootstrap-server $D --topic main.orders
k  kafka-get-offsets.sh --bootstrap-server $B --topic payments
dr kafka-get-offsets.sh --bootstrap-server $D --topic main.payments

orders:0:12       orders:1:3       orders:2:3       orders:3:0  orders:4:1       orders:5:0
main.orders:0:12  main.orders:1:3  main.orders:2:3  …           main.orders:4:1  …

payments:0:78
main.payments:0:73

The copy has the same six partitions, and each record is in the same partition number. For orders the offsets happen to match, because nothing was ever removed from it. For payments they do not: 78 against 73. See the first failure scenario for what the difference of 5 means, and for what is wrong with the 73.

Part B, step 3. Measure the delay

echo 'user-17:mirrored-2' | k kafka-console-producer.sh --bootstrap-server $B --topic orders \
  --reader-property parse.key=true --reader-property key.separator=:
# repeat until the number changes:
dr kafka-get-offsets.sh --bootstrap-server $D --topic main.orders --partitions 0

main.orders:0:12   →   main.orders:0:13     about 2 seconds later

Part B, step 4. See translated consumer offsets

k  kafka-consumer-groups.sh --bootstrap-server $B --describe --group g-new
dr kafka-consumer-groups.sh --bootstrap-server $D --describe --group g-new

on main:   g-new  events       0  CURRENT-OFFSET 1261  LOG-END-OFFSET 1657  LAG 396
           g-new  events       1  CURRENT-OFFSET  592  LOG-END-OFFSET  928  LAG 336

on dr:     g-new  main.events  0  CURRENT-OFFSET 1112  LOG-END-OFFSET 1657  LAG 545
           g-new  main.events  1  CURRENT-OFFSET  405  LOG-END-OFFSET  928  LAG 523

The group g-new never ran on the second cluster, yet it has committed offsets there, on the prefixed topic. They are behind the source values: 1112 against 1261 for partition 0. If this group failed over now, it would read 149 records of that partition a second time and miss none.

Step 5. Stop the lab processes

docker exec kafka-1 sh -c "pkill -f MirrorMaker; pkill -f ConnectDistributed"
docker compose down        # stop every container, keep nothing running

offset 6

Common mistakes and misconceptions

People think	What is true
Three copies in one cluster is disaster recovery.	It protects against a broker failure. A site failure takes all three unless the cluster spans sites.
A mirrored topic has the same offsets.	It has its own. Consumer positions must be translated.
Copying between clusters loses nothing.	It is asynchronous. Whatever was in flight is lost with the source.
MirrorMaker 2 respects transactions by default.	In the lab it copied aborted records as ordinary records.
Active-active gives the best of everything.	It moves the hard problem, conflicting updates, into your application.
A disaster plan that was never practised is a plan.	It is a hope. Run the failover, and the return, on a schedule.
Share groups replace consumer groups.	They are for independent jobs. Events that need order still need consumer groups.
Every asynchronous problem needs Kafka.	Kafka is a replayable log for many readers at high volume. Without those needs, simpler tools cost less to own.

offset 7

Staff-level questions

The business says "we cannot lose any payment event, even if a region is lost". What do you propose?

First make the requirement precise: zero loss of acknowledged records, and how long may the service be down? Zero loss rules out asynchronous copying alone.

Option one: a stretch cluster over three zones of one region, with a minimum of 2 and acks=all. It survives a zone, not a region. Option two, for a real region loss: keep the source of truth in a database that is itself replicated synchronously, publish through an outbox, and treat Kafka in the second region as rebuildable from that source. Then Kafka's asynchronous copy only decides how fast you recover, not what you lose.

State the cost of each plainly. Synchronous replication across regions makes every payment slower, all year, to protect against a rare event.

Design the failover runbook for an active and standby pair.

Before: copy delay and heartbeat are monitored; offset translation is on for the groups that matter; consumers are safe to repeat; clients can be pointed at the standby by configuration, and know the prefixed topic names or use the identity policy.

During: decide, by a named person and a stated rule, that the main site is lost. Stop the copy so that a half-alive source cannot write. Point producers at the standby. Start consumers there from the translated offsets. Record the time and the last copied offsets for each topic: that is the possible loss.

After: do not switch back in a hurry. Rebuild the old site as the new standby, let it catch up, and return in a planned window. Then review what was lost and what repeated.

When would you choose share groups over a traditional message queue product?

When you already run Kafka and the jobs are produced from data that is in Kafka: one platform, one security model, and the same records can also be read as a log by other applications.

A dedicated queue is still the better fit when you need features that are native there: priorities, a delay for each message, routing by content, a built-in dead-letter queue, or strict order within a group of messages together with many competing workers. Check also the maturity of share groups in your client language.

How do you migrate every application from an old cluster to a new one with no downtime?

Copy all topics from old to new with MirrorMaker 2, using the identity policy so that names stay the same, and with offset translation. Wait until the copy delay is steady and low.

Move consumers first, group by group: stop a group on the old cluster, start it on the new one from its translated offsets. It repeats a few records. Then move producers. For a topic with keys, move all its producers together, or records for one key will be written on both sides out of order. When the old cluster's topics receive nothing and are fully copied, stop the copy and retire the old cluster.

A team wants to use Kafka for request and reply between two services. What do you say?

It works, and it is usually the wrong shape. The caller must create a reply topic or partition, match replies to requests with an id, handle timeouts and late replies, and it pays the latency of two produce and fetch round trips. Failures are harder to see than a failed call.

If the caller needs an answer now, use a direct call. Use Kafka where the sender does not need to wait: publish the fact, and let the other service react. If they want both, do the call directly and publish an event about the result.

What would you put on one page as "how we use Kafka here"?

Topic standards: naming, three copies, a minimum of 2, how partition counts are chosen, who owns each topic. Client standards: acks=all, idempotence, a client id, handling of send errors, consumers that are safe to repeat, explicit auto.offset.reset. Data standards: a schema for every topic with a stated compatibility mode. Failure standards: dead-letter topics with owners, lag alerts in time, a practised recovery plan. And a short list of what Kafka is not to be used for.

These are the decisions from all ten modules. Writing them down once is what stops each team from learning them through an incident.

offset 8

Self-test

1. What do recovery point objective and recovery time objective measure?

How much recent data may be lost, and how long the service may be down.

2. What is MirrorMaker 2 built on?

Kafka Connect: a source connector, a checkpoint connector and a heartbeat connector.

3. The topic orders is copied from a cluster with the alias main. What is it called on the target by default?

main.orders.

4. Why can a consumer not reuse its committed offsets on the other cluster?

The copy is a new topic with its own offsets. They must be translated.

5. Is copying between clusters synchronous?

No. The producer is answered before the record reaches the other cluster.

6. What makes a stretch cluster lose no committed record when a site fails?

It is one cluster with replicas in each site, acks=all and a minimum of 2, so each record is in two sites before it is acknowledged.

7. What can a share group do that a consumer group cannot?

Let several members read the same partition, with each record acknowledged alone.

8. What happens to a share group record whose lock runs out?

It becomes available again and another member can take it, until the delivery limit, 5 by default.

9. What does a share group give up?

Order.

10. Name two signs that a use case does not need Kafka.

Any two of: the caller waits for a reply; low volume with a database already at hand; messages need priorities or delays; nobody will own the cluster.

Multi-cluster and design

Try it: one partition, several workers

What it is and why it exists

Mental model

Two numbers for every disaster plan

Try it: three ways to use two or more sites

A copy is not the original

Queue or log?

Should this be Kafka at all?

Try it: does your use case fit?

How it works internally

MirrorMaker 2

Stretch clusters

Share groups

Every relevant config

MirrorMaker 2

Share groups

Failure scenarios

The copy contains aborted transactions

Records are lost in a failover

Consumers repeat or skip after a failover

Failback is harder than failover

Two active clusters update the same thing

A stretch cluster with a slow link

A share group job is done twice, or gives up

Kafka was the wrong choice

Hands-on lab

Part A, step 1. Two workers share one partition

Part A, step 2. Look at the share group

Part A, step 3. The same with a consumer group

Part B, step 1. Start the second cluster and the copy

Part B, step 2. Compare source and copy

Part B, step 3. Measure the delay

Part B, step 4. See translated consumer offsets

Step 5. Stop the lab processes

Common mistakes and misconceptions

Staff-level questions

Self-test