Failure-drill playbook

Builds on: Module 04.

Goal. See how one stopped follower changes the in-sync set, and measure how long Kafka takes to notice it.

Setup

k kafka-topics.sh --bootstrap-server $B --create --topic ha2 --replica-assignment 3:1:2
seq 1 100 | k kafka-console-producer.sh --bootstrap-server $B --topic ha2
d ha2

Break it

# Part A: a clean stop. Start the stopwatch first.
docker stop kafka-2
d ha2

unverified Not run in the labs: docker kill kafka-2 as the second part (an abrupt stop, with no goodbye to the controller), and the timings for a follower. The module shows a clean stop of a leader, and a clean stop that leaves the in-sync set after 12 seconds. Predict the kill timing from the 9 second and 30 second values in the cheat sheet, then measure it.

Watch

Topic: ha2  Partition: 0  Leader: 3  Replicas: 3,1,2  Isr: 3,1,2  Elr:  LastKnownElr:

Topic: strict  Partition: 0  Leader: 1  Replicas: 3,1,2  Isr: 1,2  Elr: 3  LastKnownElr:

Questions to answer before you repair

1. Which broker leads ha2 now? Did the leader change when you stopped broker 2? Why not?
2. How many seconds passed before Isr lost broker 2 after the clean stop?
3. What did the producer see while broker 2 was down? Try the producer command again.
4. Which two timers decide how fast Kafka notices a dead broker, and which one applies to a clean stop?

Repair

docker start kafka-2
# wait 14 seconds
d ha2

Expect Isr: 3,1,2 again. The module saw a returning broker copy what it missed and rejoin after about 14 seconds.

Write-up prompts

• What was the in-sync set before, during and after?
• How long did detection take for a clean stop, and for a kill if you ran it?
• Which alert would have told you, and how fast?

Open the write-up builder for drill 1

Builds on: Module 02, Module 04.

Goal. Make a topic refuse writes on purpose, then learn what acks=1 does in the same state.

Setup

k kafka-topics.sh --bootstrap-server $B --create --topic strict --partitions 1 \
  --replication-factor 3 --config min.insync.replicas=3

Break it

docker stop kafka-3
# wait 12 seconds
k kafka-topics.sh --bootstrap-server $B --describe --topic strict

echo hello | k kafka-console-producer.sh --bootstrap-server $B --topic strict \
  --command-property delivery.timeout.ms=6000 --command-property request.timeout.ms=3000 \
  --command-property linger.ms=0

Watch

Topic: strict  Partition: 0  Leader: 1  Replicas: 3,1,2  Isr: 1,2  Elr: 3  LastKnownElr:

WARN Got error produce response … on topic-partition strict-0, retrying (2 attempts left). Error: NOT_ENOUGH_REPLICAS
WARN Got error produce response … on topic-partition strict-0, retrying (1 attempts left). Error: NOT_ENOUGH_REPLICAS
WARN Got error produce response … on topic-partition strict-0, retrying (0 attempts left). Error: NOT_ENOUGH_REPLICAS
ERROR Error when sending message to topic strict with key: null, value: 5 bytes with error: …

echo hello | k kafka-console-producer.sh --bootstrap-server $B --topic strict \
  --request-required-acks 1 --command-property enable.idempotence=false
k kafka-get-offsets.sh --bootstrap-server $B --topic strict

strict:0:0      ← while broker 3 is stopped
strict:0:1      ← after broker 3 returns

Questions to answer before you repair

1. Why did the leader refuse the write? State it with the numbers: how many are in sync, and how many does the topic need?
2. The acks=1 producer got no error. Where is the record, and why can no consumer read it yet?
3. Which setting would let this topic survive one broker failure and still accept writes?
4. How long did the producer retry, and which setting set that limit?

Repair

docker start kafka-3
# wait 12 seconds
k kafka-get-offsets.sh --bootstrap-server $B --topic strict

The end offset moves to 1 when the third broker is back in the in-sync set. The permanent fix is a topic setting, not a restart: lower min.insync.replicas to 2 on a 3-copy topic.

Write-up prompts

• What did the application see: errors, silence or both?
• What number made the broker say no?
• What would you change in the topic settings, and what do you give up?

Open the write-up builder for drill 2

Builds on: Module 04.

Goal. Stop a leader, write under the new leader, and read the epoch file that records who led which offsets.

Setup

If you ran drill 1, the topic ha2 already exists. Skip the create line, or use the name ha3 and change it in every command.

k kafka-topics.sh --bootstrap-server $B --create --topic ha2 --replica-assignment 3:1:2
seq 1 100 | k kafka-console-producer.sh --bootstrap-server $B --topic ha2
d ha2
docker exec kafka-1 cat /tmp/kraft-combined-logs/ha2-0/leader-epoch-checkpoint

Break it

docker stop kafka-3
d ha2
seq 101 150 | k kafka-console-producer.sh --bootstrap-server $B --topic ha2
docker exec kafka-1 cat /tmp/kraft-combined-logs/ha2-0/leader-epoch-checkpoint

Watch

Topic: ha2  Partition: 0  Leader: 3  Replicas: 3,1,2  Isr: 3,1,2  Elr:  LastKnownElr:
0
1
0 0                ← epoch 0 started at offset 0

Topic: ha2  Partition: 0  Leader: 1  Replicas: 3,1,2  Isr: 1,2    Elr:  LastKnownElr:

0
2
0 0
1 100              ← epoch 1 started at offset 100

Questions to answer before you repair

1. Which broker leads now, and which epoch is it? How can you tell from the file?
2. Broker 3 is stopped. Which offsets does it have, and which does it lack?
3. When broker 3 returns, how will it use the epoch file to decide where to start copying?
4. Will broker 3 lead again by itself? What does the preferred election do?

Repair

docker start kafka-3
# wait 14 seconds
d ha2
k kafka-leader-election.sh --bootstrap-server $B --election-type preferred --topic ha2 --partition 0
d ha2

Module 04 saw all 150 records present, Isr: 1,2,3, and leader 3 after the preferred election.

Write-up prompts

• What did clients see in the second after the stop?
• What do the two epoch lines say happened?
• What would you add to monitoring so a leader change is visible without a person running a describe?

Open the write-up builder for drill 3

Builds on: Module 04.

Goal. Cause a partition to go offline, force a leader that is missing records, and watch the data disappear.

Setup

This is the one drill where two brokers are down for a while. Keep broker 1 running so the controllers keep a majority.

k kafka-topics.sh --bootstrap-server $B --create --topic risky2 --replica-assignment 2:3 \
  --config min.insync.replicas=1
seq 1 10 | k kafka-console-producer.sh --bootstrap-server $B --topic risky2

Break it

docker stop kafka-3            # broker 3 now misses everything that follows
# wait 8 seconds
seq 11 20 | k kafka-console-producer.sh --bootstrap-server $B --topic risky2
k kafka-get-offsets.sh --bootstrap-server $B --topic risky2

docker stop kafka-2; docker start kafka-3    # the only in-sync broker is gone
# wait 20 seconds
d risky2

Watch

risky2:0:20

Topic: risky2  Partition: 0  Leader: none  Replicas: 2,3  Isr:   Elr: 2  LastKnownElr: 2

k kafka-leader-election.sh --bootstrap-server $B --election-type unclean --topic risky2 --partition 0
d risky2
k kafka-get-offsets.sh --bootstrap-server $B --topic risky2

Successfully completed leader election (UNCLEAN) for partitions risky2-0
Topic: risky2  Partition: 0  Leader: 3  Replicas: 2,3  Isr: 3
risky2:0:10

seq 21 25 | k kafka-console-producer.sh --bootstrap-server $B --topic risky2
docker start kafka-2
# wait 16 seconds
docker exec kafka-2 /opt/kafka/bin/kafka-dump-log.sh --print-data-log \
  --files /tmp/kraft-combined-logs/risky2-0/00000000000000000000.log | grep -o 'payload: .*'
docker logs kafka-2 2>&1 | grep -i 'risky2-0.*truncat'

values on broker 2:  1 2 3 4 5 6 7 8 9 10 21 22 23 24 25

WARN [UnifiedLog partition=risky2-0] Truncating risky2-0 to offset 10 below high watermark 20

Questions to answer before you repair

1. Why did Kafka refuse to make broker 3 the leader on its own?
2. Which records were lost, and who had acknowledged them? Was the producer told?
3. A consumer had committed offset 20 before the election. What does it see now?
4. Which two settings would have prevented the loss, and what would each have cost in availability?

Repair

d risky2
k kafka-get-offsets.sh --bootstrap-server $B --topic risky2

The lost records cannot be recovered from Kafka. Check that both brokers run and the in-sync set has both. The real repair is a decision: replay values 11 to 20 from the source of truth, and raise min.insync.replicas to 2 on a 3-copy topic.

Write-up prompts

• Write the timeline with offsets: what was acknowledged, when, and by whom?
• Who owns the decision to run an unclean election, and what do they need to know first?
• What is the exact data that was lost, in business terms?

Open the write-up builder for drill 4

Builds on: Module 05.

Goal. Stop two of three controllers and see which operations stop, then restore and read the new epoch.

Setup

Read LeaderId first. The module had node 1 as leader. If yours differs, stop that node and run the tools from another one.

k kafka-metadata-quorum.sh --bootstrap-server $B describe --status
k kafka-metadata-quorum.sh --bootstrap-server $B describe --replication

Break it

docker stop kafka-1
docker exec -i kafka-2 /opt/kafka/bin/kafka-metadata-quorum.sh --bootstrap-server kafka-2:19092 describe --status
docker exec -i kafka-2 /opt/kafka/bin/kafka-topics.sh --bootstrap-server kafka-2:19092 \
  --create --topic during-failover --partitions 1 --replication-factor 2

# now lose the majority
docker stop kafka-3
docker exec -i kafka-2 timeout 12 /opt/kafka/bin/kafka-topics.sh --bootstrap-server kafka-2:19092 \
  --describe --topic during-failover
docker exec -i kafka-2 timeout 14 /opt/kafka/bin/kafka-topics.sh --bootstrap-server kafka-2:19092 \
  --create --topic no-quorum --partitions 1 --replication-factor 1
docker logs --since 20s kafka-2 2>&1 | grep -i 'election' | tail -1

Watch

LeaderId: 2   LeaderEpoch: 4            ← about 1 second after the stop

NodeId  LogEndOffset  Lag   Status
2       5875          0     Leader
1       -1            5876  Follower
3       5875          0     Follower

Created topic during-failover.

INFO [RaftManager id=2] Election timed out before receiving sufficient vote responses to become candidate. …

Questions to answer before you repair

1. Why did one failure change nothing for clients, and two failures stopped topic creation?
2. Why did even --describe hang? Which part of the cluster did the tool ask?
3. Would a producer that was already writing to a healthy leader have failed? Check the module's mistakes table before you answer.
4. Why is the epoch higher after the restore than after the first election?

Repair

docker start kafka-1 kafka-3
# wait 20 seconds
k kafka-metadata-quorum.sh --bootstrap-server $B describe --status

The module saw LeaderId: 2, LeaderEpoch: 6 and MaxFollowerLag: 0. The failed election attempts each used up an epoch.

Write-up prompts

• What worked, and what did not, with two of three controllers down?
• How would you have found the cause from the logs alone?
• What would you change about controller placement (machines, racks, zones)?

Open the write-up builder for drill 5

Builds on: Module 03.

Goal. Watch partitions move when a member leaves, and compare a clean exit with a kill.

Setup

k kafka-topics.sh --bootstrap-server $B --create --topic events --partitions 6 --replication-factor 3
k kafka-producer-perf-test.sh --topic events --num-records 6000 --record-size 100 \
  --throughput -1 --command-property bootstrap.servers=$B
docker exec -d kafka-1 /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server $B \
  --topic events --group g-classic --from-beginning --command-property client.id=c1
docker exec -d kafka-2 /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server $B \
  --topic events --group g-classic --from-beginning --command-property client.id=c2
# wait 12 seconds
k kafka-consumer-groups.sh --bootstrap-server $B --describe --group g-classic
k kafka-consumer-groups.sh --bootstrap-server $B --describe --group g-classic --state

Break it

docker exec kafka-2 sh -c "pkill -f 'group g-classic'"
# wait 8 seconds
k kafka-consumer-groups.sh --bootstrap-server $B --describe --group g-classic --members

unverified Not run in the labs: pkill -9 on the consumer (a kill with no goodbye), so the group must wait for the 45 second session timeout. The module shows pkill -9 only for a transaction drill, and shows only the clean stop for a consumer. Restart c2 first, then time how long #MEMBERS takes to drop.

Watch

GROUP      COORDINATOR (ID)    ASSIGNMENT-STRATEGY  STATE   #MEMBERS
g-classic  kafka-1:19092 (1)   range                Stable  2

GROUP      CONSUMER-ID       HOST         CLIENT-ID  #PARTITIONS
g-classic  c1-7b00d4f1-…     /172.23.0.4  c1         6

Questions to answer before you repair

1. Which partitions did the surviving member gain? Did it have to wait for anything?
2. Why did the clean stop not need the 45 second session timeout?
3. What would a client see during the move: duplicates, a pause, or both?
4. The module also ran a group with group.protocol=consumer. What did the strategy name change to, and who now chooses it?

Repair

docker exec kafka-1 sh -c "pkill -f ConsoleConsumer"; docker exec kafka-2 sh -c "pkill -f ConsoleConsumer"
k kafka-consumer-groups.sh --bootstrap-server $B --describe --group g-classic

The group then shows no active members. Its lag is whatever was written since.

Write-up prompts

• How long was the group without a full assignment?
• What did you expect from a kill, and what did you measure?
• Which consumer settings would you tune for a long-running job per record?

Open the write-up builder for drill 6

Builds on: Module 03, Module 09.

Goal. Read lag correctly, find the one partition that does not move, and skip it by moving the group's offset forward.

Setup

unverified Not run in the labs: a consumer that crashes on one bad record. No lab has a consumer with code in it. This drill stands in for the stuck record with lag on a stopped group, and teaches the skip. Drill 9 shows a real poison record in Connect.

# uses topic events and group g-classic from drill 6
docker exec kafka-1 sh -c "pkill -f ConsoleConsumer"; docker exec kafka-2 sh -c "pkill -f ConsoleConsumer"
k kafka-producer-perf-test.sh --topic events --num-records 3000 --record-size 100 \
  --throughput -1 --command-property bootstrap.servers=$B
k kafka-consumer-groups.sh --bootstrap-server $B --describe --group g-classic

Break it

# the group is stopped, so the lag stays. Read it for every group:
k kafka-consumer-groups.sh --bootstrap-server $B --describe --all-groups

Watch

GROUP      TOPIC   PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG
g-classic  events  0          1361            1657            296
g-classic  events  1          592             928             336
g-classic  events  2          934             1230            296
g-classic  events  3          592             1184            592
g-classic  events  4          1480            2072            592
g-classic  events  5          1041            1929            888

k kafka-consumer-groups.sh --bootstrap-server $B --group g-classic --topic events \
  --reset-offsets --to-earliest --dry-run
k kafka-consumer-groups.sh --bootstrap-server $B --group g-classic --topic events:3 \
  --reset-offsets --shift-by 1 --dry-run
k kafka-consumer-groups.sh --bootstrap-server $B --group g-classic --topic events:3 \
  --reset-offsets --shift-by 1 --execute

GROUP      TOPIC   PARTITION  NEW-OFFSET
g-classic  events  3          0
…

Questions to answer before you repair

1. Which partition has the most lag? Is that the same as the partition that is stuck?
2. Why must the group have no active members before a reset?
3. What do you lose by skipping a record, and who must agree to that?
4. Lag in records is not lag in time. What second number would you ask for?

Repair

k kafka-consumer-groups.sh --bootstrap-server $B --describe --group g-classic

Check that CURRENT-OFFSET moved on partition 3 only. Keep the skipped offset in your write-up: someone will ask for that record later. In production the better repair is a dead-letter topic (drill 9).

Write-up prompts

• How did you tell a slow group from a stuck one?
• Which offset did you skip and why was it safe?
• What alert on time since the last commit would have caught this?

Open the write-up builder for drill 7

Builds on: Module 06.

Goal. Kill a transactional producer mid-transaction and see a read_committed consumer stop at the open transaction until the coordinator aborts it.

Setup

k kafka-topics.sh --bootstrap-server $B --create --topic payments --partitions 1 --replication-factor 3
k kafka-producer-perf-test.sh --topic payments --num-records 30 --record-size 20 --throughput 10 \
  --transactional-id pay-1 --transaction-duration-ms 1000 --command-property bootstrap.servers=$B
k kafka-get-offsets.sh --bootstrap-server $B --topic payments

Break it

docker exec -d kafka-2 /opt/kafka/bin/kafka-producer-perf-test.sh --topic payments \
  --num-records 100000 --record-size 20 --throughput 5 --transactional-id pay-2 \
  --transaction-duration-ms 300000 --command-property bootstrap.servers=$B transaction.timeout.ms=40000
# wait 8 seconds
docker exec kafka-2 sh -c "pkill -9 -f 'transactional-id pay-2'"

k kafka-transactions.sh --bootstrap-server $B describe --transactional-id pay-2
for iso in read_uncommitted read_committed; do
  k kafka-console-consumer.sh --bootstrap-server $B --topic payments --from-beginning \
    --isolation-level $iso --timeout-ms 5000 2>&1 | tail -1
done

Watch

CoordinatorId  TransactionalId  ProducerId  ProducerEpoch  TransactionState  TransactionTimeoutMs  …  TopicPartitions
3              pay-2            2000        0              Ongoing           40000                    payments-0

read_uncommitted:  Processed a total of 68 messages
read_committed:    Processed a total of 30 messages

k kafka-transactions.sh --bootstrap-server $B describe --transactional-id pay-2
k kafka-dump-log.sh --print-data-log --files /tmp/kraft-combined-logs/payments-0/00000000000000000000.log | grep endTxnMarker | tail -1
k kafka-get-offsets.sh --bootstrap-server $B --topic payments

… pay-2  2000  …  CompleteAbort

| offset: 71 … endTxnMarker: ABORT coordinatorEpoch: 0
payments:0:72

Questions to answer before you repair

1. Why did the committed reader stop at 30, and not at 68?
2. What would happen to a committed reader of this partition if other producers wrote after the open transaction?
3. What decided the moment of cleanup, and where did you read that number?
4. The 38 aborted records are still in the log. Who can still read them, and for how long?

Repair

k kafka-transactions.sh --bootstrap-server $B list

No manual repair is shown in the labs: you wait for the timeout (here 40 seconds, set by transaction.timeout.ms). The staff-level repair is a shorter timeout and a stable transactional id, so a restarted producer fences the old one.

Write-up prompts

• How long was the committed reader blocked, and by what?
• What is the transaction timeout in your applications, and is it right?
• Which consumers in your system read with the default isolation level?

Open the write-up builder for drill 8

Builds on: Module 08, Module 03.

Goal. Feed a bad record to a sink connector, watch the task fail with no data delivered, then route the bad record to a dead-letter topic.

Setup

docker exec kafka-1 sh -c 'cat > /tmp/connect.properties <<EOF
bootstrap.servers=kafka-1:19092
group.id=connect-lab
key.converter=org.apache.kafka.connect.storage.StringConverter
value.converter=org.apache.kafka.connect.storage.StringConverter
offset.storage.topic=connect-offsets
config.storage.topic=connect-configs
status.storage.topic=connect-status
plugin.path=/opt/kafka/libs/connect-file-4.3.1.jar
listeners=http://0.0.0.0:8083
EOF'
docker exec -d kafka-1 sh -c '/opt/kafka/bin/connect-distributed.sh /tmp/connect.properties > /tmp/connect.log 2>&1'
get()  { docker exec kafka-1 wget -q -O - "http://localhost:8083$1"; }
post() { docker exec kafka-1 wget -q -O - --header 'Content-Type: application/json' --post-data "$2" "http://localhost:8083$1"; }
# wait about 20 seconds
get /

Break it

k kafka-topics.sh --bootstrap-server $B --create --topic json-in --partitions 1 --replication-factor 3
printf '{"id":1}\nthis is not json\n{"id":2}\n' | k kafka-console-producer.sh --bootstrap-server $B --topic json-in

post /connectors '{"name":"json-strict","config":{"connector.class":"org.apache.kafka.connect.file.FileStreamSinkConnector","tasks.max":"1","file":"/tmp/strict.txt","topics":"json-in","value.converter":"org.apache.kafka.connect.json.JsonConverter","value.converter.schemas.enable":"false"}}'
# wait 10 seconds
get /connectors/json-strict/status
docker exec kafka-1 cat /tmp/strict.txt

Watch

{"name":"json-strict","connector":{"state":"RUNNING",…},
 "tasks":[{"id":0,"state":"FAILED",…,"trace":"org.apache.kafka.connect.errors.ConnectException: Tolerance exceeded in error handler …

(strict.txt is empty)

Questions to answer before you repair

1. Which word in the status output tells you the real state: connector or task?
2. The two good records were not delivered. Why not?
3. Which consumer group tracks this sink? Which module 03 command shows its lag?
4. Is a dead-letter topic always right? What does it hide?

Repair

post /connectors '{"name":"json-dlq","config":{"connector.class":"org.apache.kafka.connect.file.FileStreamSinkConnector","tasks.max":"1","file":"/tmp/dlq-ok.txt","topics":"json-in","value.converter":"org.apache.kafka.connect.json.JsonConverter","value.converter.schemas.enable":"false","errors.tolerance":"all","errors.deadletterqueue.topic.name":"json-in-dlq","errors.deadletterqueue.topic.replication.factor":"3","errors.deadletterqueue.context.headers.enable":"true"}}'
# wait 12 seconds
get /connectors/json-dlq/status
docker exec kafka-1 cat /tmp/dlq-ok.txt
k kafka-console-consumer.sh --bootstrap-server $B --topic json-in-dlq --from-beginning \
  --formatter-property print.headers=true --timeout-ms 6000

The module saw {id=1} and {id=2} delivered, and the bad record in the dead-letter topic with headers naming topic json-in, partition 0, offset 1, and the stage VALUE_CONVERTER. When done: docker exec kafka-1 sh -c "pkill -f ConnectDistributed".

Write-up prompts

• What did monitoring show: nothing, or a failed task?
• What do you do with the records in the dead-letter topic, and who owns that?
• Which alert would you add on task state?

Open the write-up builder for drill 9

Builds on: Module 08.

Goal. Try to register a schema that old data cannot satisfy, read the refusal, and test changes without registering them.

Setup

Run this on your own machine, not inside a container: the registry listens on port 8086 and the module uses curl.

docker compose up -d schema-registry
R=http://localhost:8086
H='Content-Type: application/vnd.schemaregistry.v1+json'
curl -s $R/config

V1='{"schema":"{\"type\":\"record\",\"name\":\"Order\",\"fields\":[{\"name\":\"id\",\"type\":\"string\"},{\"name\":\"amount\",\"type\":\"double\"}]}"}'
V2='{"schema":"{\"type\":\"record\",\"name\":\"Order\",\"fields\":[{\"name\":\"id\",\"type\":\"string\"},{\"name\":\"amount\",\"type\":\"double\"},{\"name\":\"currency\",\"type\":\"string\",\"default\":\"CAD\"}]}"}'
V3='{"schema":"{\"type\":\"record\",\"name\":\"Order\",\"fields\":[{\"name\":\"id\",\"type\":\"string\"},{\"name\":\"amount\",\"type\":\"double\"},{\"name\":\"currency\",\"type\":\"string\",\"default\":\"CAD\"},{\"name\":\"customerId\",\"type\":\"string\"}]}"}'
curl -s -X POST -H "$H" -d "$V1" $R/subjects/orders-value/versions
curl -s -X POST -H "$H" -d "$V2" $R/subjects/orders-value/versions

Break it

curl -s -X POST -H "$H" -d "$V3" $R/subjects/orders-value/versions
curl -s $R/subjects/orders-value/versions

Watch

{"error_code":40901,"message":"Schema being registered is incompatible with an earlier schema for subject
 "orders-value", details: [{errorType:'READER_FIELD_MISSING_DEFAULT_VALUE', description:'The field
 'customerId' at path '/fields/3' in the new schema has no default value and is missing in the old schema' …
[1,2]

V4='{"schema":"{\"type\":\"record\",\"name\":\"Order\",\"fields\":[{\"name\":\"id\",\"type\":\"string\"},{\"name\":\"currency\",\"type\":\"string\",\"default\":\"CAD\"}]}"}'
curl -s -X POST -H "$H" -d "$V4" "$R/compatibility/subjects/orders-value/versions/latest?verbose=true"
curl -s -X PUT -H "$H" -d '{"compatibility":"FULL"}' $R/config/orders-value
curl -s -X POST -H "$H" -d "$V4" "$R/compatibility/subjects/orders-value/versions/latest?verbose=true"

{"is_compatible":true,"messages":[]}

{"compatibility":"FULL"}
{"is_compatible":false,"messages":["{errorType:'READER_FIELD_MISSING_DEFAULT_VALUE', description:'The field
 'amount' at path '/fields/1' in the old schema has no default value and is missing in the new schema' …

Questions to answer before you repair

1. Which field did the registry object to, and what does "no default value" mean for an old record?
2. Whose job is it to prevent this: the producer team, the registry, or the build pipeline?
3. Why is removing amount allowed under BACKWARD and refused under FULL?
4. What would have happened with no registry in the way?

Repair

k kafka-topics.sh --bootstrap-server $B --describe --topic _schemas | head -1

Nothing was damaged: the registry refused the change. The repair is to give customerId a default and test it with the compatibility call. Registering that fixed version was not run in the labs. Then set the subject back with the same PUT to the mode you want.

Write-up prompts

• What did the refusal say, in your own words?
• Where in your release process would the test call run?
• Which compatibility mode does each of your subjects use, and why?

Open the write-up builder for drill 10

Builds on: Module 09, Module 02.

Goal. Put a byte-rate quota on one client, compare it with an unlimited one, and learn why a short test shows almost no throttling.

Setup

The topic perf comes from module 02. Skip the create line if it exists.

k kafka-topics.sh --bootstrap-server $B --create --topic perf --partitions 3 --replication-factor 3
k kafka-configs.sh --bootstrap-server $B --alter --add-config 'producer_byte_rate=5242880' \
  --entity-type clients --entity-name slow-client

Break it

k kafka-producer-perf-test.sh --topic perf --num-records 400000 --record-size 200 --throughput -1 \
  --command-property bootstrap.servers=$B client.id=slow-client
k kafka-producer-perf-test.sh --topic perf --num-records 400000 --record-size 200 --throughput -1 \
  --command-property bootstrap.servers=$B client.id=fast-client

Watch

Client        Quota               MB each second   Average latency
slow-client   5 MiB each second   21.64            973 ms
fast-client   none                39.03            422 ms

unverified Not run in the labs: a longer run. Raise --num-records until the run lasts a minute or more, then compare MB each second with the quota. The module only says this is the way to do it.

Questions to answer before you repair

1. Why did the quota not reject anything? What does a quota do instead?
2. Why did a short test mislead? Give the window length and the test length.
3. Which numbers would you collect from a real noisy client before setting its quota?
4. Who should be told when you throttle a client, and how will they find out?

Repair

k kafka-configs.sh --bootstrap-server $B --alter --delete-config producer_byte_rate \
  --entity-type clients --entity-name slow-client

Always remove the test quota. Quotas live in the cluster until someone deletes them.

Write-up prompts

• What did the first short test show, and why was that wrong?
• What quota would you set, and how long would you test it?
• What does the client team see when throttled?

Open the write-up builder for drill 11

Builds on: Module 09, Module 01, Module 02.

Goal. Add partitions to a live topic and see one key land in a different partition, which breaks per-key order.

Setup

These are example records written for this drill. Module 01 builds the same topic; skip the create line if orders exists.

k kafka-topics.sh --bootstrap-server kafka-1:19092 --create --topic orders --partitions 3 --replication-factor 3
printf 'user-42:created\nuser-42:paid\nuser-42:shipped\nuser-17:created\nuser-1:created\n' | \
  k kafka-console-producer.sh --bootstrap-server $B --topic orders \
  --reader-property parse.key=true --reader-property key.separator=:

Break it

k kafka-topics.sh --bootstrap-server $B --alter --topic orders --partitions 6
printf 'user-42:after-change\nuser-17:after-change\nuser-1:after-change\n' | \
  k kafka-console-producer.sh --bootstrap-server $B --topic orders \
  --reader-property parse.key=true --reader-property key.separator=:
k kafka-console-consumer.sh --bootstrap-server $B --topic orders --from-beginning \
  --formatter-property print.key=true --formatter-property print.partition=true \
  --formatter-property print.offset=true --timeout-ms 6000

Watch

Partition:2  Offset:0   user-1   created
Partition:2  Offset:2   user-1   after-change
Partition:0  Offset:0   user-17  created
Partition:0  Offset:11  user-17  after-change
Partition:1  Offset:0   user-42  created
Partition:1  Offset:1   user-42  paid
Partition:1  Offset:2   user-42  shipped
Partition:4  Offset:0   user-42  after-change      ← the same key, a different partition

Questions to answer before you repair

1. Which key moved? Which two partitions now hold its story?
2. What does that do to a consumer that needs all events for one customer in order?
3. Why did the other keys not move?
4. Can you now go back to 3 partitions? Try the alter command and read the error.

Repair

k kafka-topics.sh --bootstrap-server $B --alter --topic orders --partitions 3

Module 09 saw: "The topic orders currently has 6 partition(s); 3 would not be an increase." You cannot undo it.

unverified Not run in the labs: the real repair. The usual plan is a new topic with the final partition count, then a copy and a switch for producers and consumers. The module does not run it.

Write-up prompts

• Which events could have been processed out of order, and what was the business effect?
• What rule would stop a team adding partitions to a keyed topic casually?
• How many partitions will you size for next time?

Open the write-up builder for drill 12

Builds on: Module 09, Module 01.

Goal. Find the topics that use the disk, then free space the supported way by lowering retention and waiting for the check. Do not delete files by hand.

Setup

Module 01 made five 1 MiB segments this way.

k kafka-topics.sh --bootstrap-server kafka-1:19092 --create --topic rolls \
  --partitions 1 --replication-factor 3 --config segment.bytes=1048576
k kafka-producer-perf-test.sh --topic rolls --num-records 20000 --record-size 200 \
  --throughput -1 --command-property bootstrap.servers=kafka-1:19092
docker exec kafka-1 ls /tmp/kraft-combined-logs/rolls-0

Break it

# the filling step itself: not run in the labs
# (fill the disk of one broker; module 09 gives no command for it)

unverified Not run in the labs: filling a broker's disk. Do not attempt it on a laptop shared with other work. The modules support only the steps before and after the fill.

Watch

k kafka-log-dirs.sh --bootstrap-server $B --describe --broker-list 1 --topic-list perf,comp-none,orders,rolls

perf-0        247910144
perf-2        241752474
perf-1        237756878
comp-none-0     1548818
orders-0            713
orders-2            159

k kafka-configs.sh --bootstrap-server kafka-1:19092 --alter \
  --entity-type topics --entity-name rolls --add-config retention.ms=60000
# wait up to 6 minutes
docker exec kafka-1 ls /tmp/kraft-combined-logs/rolls-0
k kafka-get-offsets.sh --bootstrap-server kafka-1:19092 --topic rolls --time earliest

00000000000000000000.log.deleted
00000000000000004992.log.deleted
…
00000000000000019968.log.deleted
rolls:0:20000

Questions to answer before you repair

1. Which topics use the most space on broker 1? What would you ask their owners?
2. Why did the space not come back at once after you lowered retention?
3. Why must you not delete segment files by hand?
4. Which thresholds would you alert on before the disk is full? Check the module for the two values.

Repair

k kafka-configs.sh --bootstrap-server kafka-1:19092 --alter \
  --entity-type topics --entity-name rolls --delete-config retention.ms

Put retention back to its normal value after the emergency. --delete-config returns a setting to its default. Module 09 uses it for a client quota (drill 11).

unverified Not run in the labs: --delete-config retention.ms on a topic. The module uses the flag only for quotas.

Write-up prompts

• What was the first sign, and how late was it?
• How much data did you give up by lowering retention, and who approved it?
• What limit would you set so no single topic can fill a disk?

Open the write-up builder for drill 13

Builds on: Module 10, Module 06, Module 03.

Goal. Start the copy to the second cluster, compare offsets, read translated group offsets, and find that aborted records were copied as normal records.

Setup

The group g-new comes from module 03, step 3. Without it, the last two commands show nothing.

# do drill 8 first, so payments has an aborted transaction
docker compose up -d kafka-dr
dr() { docker exec -i kafka-dr /opt/kafka/bin/"$@"; }
D=kafka-dr:19092

docker exec kafka-1 sh -c 'cat > /tmp/mm2.properties <<EOF
clusters = main, dr
main.bootstrap.servers = kafka-1:19092
dr.bootstrap.servers = kafka-dr:19092
main->dr.enabled = true
main->dr.topics = orders|payments|events
dr->main.enabled = false
groups = g-.*
replication.factor = 1
checkpoints.topic.replication.factor = 1
heartbeats.topic.replication.factor = 1
offset-syncs.topic.replication.factor = 1
offset.storage.replication.factor = 1
status.storage.replication.factor = 1
config.storage.replication.factor = 1
sync.group.offsets.enabled = true
emit.checkpoints.interval.seconds = 10
sync.group.offsets.interval.seconds = 10
refresh.topics.interval.seconds = 10
refresh.groups.interval.seconds = 10
EOF'
docker exec -d kafka-1 sh -c '/opt/kafka/bin/connect-mirror-maker.sh /tmp/mm2.properties > /tmp/mm2.log 2>&1'
# wait about 45 seconds
dr kafka-topics.sh --bootstrap-server $D --list

Break it

k  kafka-get-offsets.sh --bootstrap-server $B --topic payments
dr kafka-get-offsets.sh --bootstrap-server $D --topic main.payments
k  kafka-consumer-groups.sh --bootstrap-server $B --describe --group g-new
dr kafka-consumer-groups.sh --bootstrap-server $D --describe --group g-new

unverified Not run in the labs: the failover itself: stop the copy with docker exec kafka-1 sh -c "pkill -f MirrorMaker; pkill -f ConnectDistributed", write records to orders on the main cluster, and count what is missing on main.orders. The module shows that command only as clean-up.

Watch

payments:0:78
main.payments:0:73

on main:   g-new  events       0  CURRENT-OFFSET 1261  LOG-END-OFFSET 1657  LAG 396
on dr:     g-new  main.events  0  CURRENT-OFFSET 1112  LOG-END-OFFSET 1657  LAG 545

main.orders:0:12   →   main.orders:0:13     about 2 seconds later

Questions to answer before you repair

1. If g-new failed over now, how many records of partition 0 would it read twice, and would it miss any?
2. Why do the offsets differ for payments but match for orders?
3. Why does a reader of main.payments see records that were rolled back on the main cluster?
4. If the main cluster were lost now, which acknowledged records might not exist on the second cluster?

Repair

docker exec kafka-1 sh -c "pkill -f MirrorMaker; pkill -f ConnectDistributed"
docker compose down        # stop every container; the labs keep no data after this

The module names the fix for aborted records as setting the source consumer's isolation level to read_committed in the MirrorMaker 2 configuration for flows that carry transactional topics. The module marks it unverified.

unverified Not run in the labs: that setting. The exact property name is not shown in the module, and the lab was not rerun with it. Look it up before you use it.

Write-up prompts

• What is your recovery point: how many seconds of data could the second cluster lack?
• How would a consumer on the second cluster know where to start?
• What would you do about aborted records in the copy?

Open the write-up builder for drill 14

Builds on: Module 11.

Goal. Meet the three common client failures on a secured cluster: wrong password, missing truststore, and missing permission. Read each one correctly.

Setup

This is the separate cluster in secure/, not the main one. It has its own health check: the list command below, run as admin.

cd ~/Desktop/kafka-knowledge-base/secure
sh make-certs.sh
docker compose up -d
docker logs kafka-sec 2>&1 | grep 'Kafka Server started'

s() { docker exec -i kafka-sec timeout 25 /opt/kafka/bin/"$@"; }
S=kafka-sec:9094
A=/clients/admin.properties
s kafka-topics.sh --bootstrap-server $S --command-config $A --create --topic payments-sec --partitions 1 --replication-factor 1
s kafka-acls.sh --bootstrap-server $S --command-config $A --add --allow-principal User:alice --producer --topic payments-sec
s kafka-acls.sh --bootstrap-server $S --command-config $A --add --allow-principal User:bob --consumer --topic payments-sec --group billing

Break it

s kafka-topics.sh --bootstrap-server $S --command-config /clients/plaintext.properties --list
s kafka-topics.sh --bootstrap-server $S --command-config /clients/no-truststore.properties --list
s kafka-topics.sh --bootstrap-server $S --command-config /clients/wrong-password.properties --list
s kafka-topics.sh --bootstrap-server $S --command-config /clients/alice.properties \
  --create --topic alice-topic --partitions 1 --replication-factor 1
s kafka-console-consumer.sh --bootstrap-server $S --command-config /clients/alice.properties --topic payments-sec --group snoop --from-beginning --timeout-ms 6000

Watch

1. ERROR Uncaught exception in thread 'kafka-admin-client-thread' … java.lang.OutOfMemoryError: Java heap space
2. ERROR … failed authentication due to: SSL handshake failed
   org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
3. ERROR … failed authentication due to: Authentication failed: Invalid username or password
alice create:  Error while executing topic command : Authorization failed.
alice read:    GroupAuthorizationException: Not authorized to access group: snoop

docker exec kafka-sec sh -c 'grep -i denied /opt/kafka/logs/kafka-authorizer.log | tail -3'

INFO Principal = User:alice is Denied operation = DESCRIBE … on resource = Group:LITERAL:snoop for request = FindCoordinator … based on rule DefaultDeny

Questions to answer before you repair

1. Which of the three failures is an authentication problem, and which is authorization? How do the messages differ?
2. The plain client got an out-of-memory error. Why is that message misleading?
3. An empty topic list was returned to alice. Is the list empty, or is alice not allowed to see it?
4. Why is making the application a super user the wrong fix?

Repair

s kafka-acls.sh --bootstrap-server $S --command-config $A --list
s kafka-topics.sh --bootstrap-server $S --command-config $A --list

For the missing permission, add a rule with the same --consumer form as bob's, naming alice and group snoop. For the others, fix the client file, not the broker. When finished: docker compose down in the secure folder.

unverified Not run in the labs: the alice rule for snoop. It uses the same command form as module 11 with another principal and group.

Write-up prompts

• What did each failure look like to an application team, and how would they misread it?
• Which log line did you use to find the cause?
• What would a runbook line say for each of the three?

Open the write-up builder for drill 15