Projects we will work on

Purpose

This is the long project. You build one small order system on the lab cluster: a database, a connector that publishes events, a safe consumer, a Kafka Streams job, monitoring and a weekly failure drill. You keep it running for weeks. It is the only project here that teaches what a system does over time.

The full plan, requirements R1 to R11 and the starter files live on the capstone page. This card does not repeat them. It only tracks the eight weeks.

Spec

• Read capstone.html and capstone/README.md first. Both say the starter files were not run end to end.
• Needs the cdc and monitoring profiles together, as the capstone page says.
• The spec, the acceptance tests and the design choices are on the capstone page. Use them there.

Eight milestones

Acceptance tests

Each week has its own tests on the capstone page. Tick a week here only when its tests pass and you have written down what you saw. The full test list is in capstone/acceptance-tests.md.

What to measure

Each capstone week lists what to measure. Examples from that page: time from a cold start to a ready cluster, delay from database commit to topic, replication slot size while the connector is stopped, and the highest rate with flat lag. Follow that page.

The story it produces

Eight weeks give about eight stories. The capstone page has a table of the five-part shape with an example from week 2. Use the same shape for every week. The template below is a blank copy.

Situation:         I was building ___ . What mattered was ___ .
What I saw:        ___ (output, numbers, symptoms; copy them from the log)
What I decided:    ___ . I rejected ___ because ___ .
What I got wrong:  ___ (the part interviewers trust most)
What I changed:    ___ , and I measured the change as ___ .

Purpose

Module 09 says too many partitions is a failure, but gives no feel for where "too many" starts. In this project you add partitions in steps and watch what each step costs: creation time, memory, restart time, failover time, and the size of the cluster metadata log. You will not find a universal limit. You will learn which number moves first and how to measure it.

Spec

• Main three-broker cluster only. The monitoring profile is optional, and helps for JMX.
• Create topics with replication factor 3 in steps of total partitions: for example 500, 1,000, 2,000 and 5,000. These are targets to attempt.
• After each step: run the three health checks, record every measurement in the table below, then go on.
• Run the restart test and the failover test at the smallest step and again at the largest step you reach.
• Keep the Docker memory stop rule, below.

The loop

The topic command is the one from modules 01 and 03. The loop around it is yours to adapt.

# one step: add 5 topics of 100 partitions each (500 partitions in this step)
for i in $(seq 1 5); do
  k kafka-topics.sh --bootstrap-server $B --create --topic scale-a-$i \
    --partitions 100 --replication-factor 3
done
# next steps: change the topic names (scale-b-..., scale-c-...) and the counts.
# every partition has 3 copies, so 500 partitions mean 1,500 replicas in the cluster.

Eight milestones

Acceptance tests

What to measure

The story it produces

The story is about a limit, not a success. A good one reads: "I expected X to break first. Y moved first. I changed how I measured, and found Z." Put real numbers in it.

Situation:         I wanted to know what many partitions cost on a 3-broker lab.
What I saw:        at ___ partitions: create ___ s, broker memory ___, restart ___ s,
                   failover ___ s, metadata log ___ . (copy from the table)
What I decided:    I stopped at ___ because ___ . I did not claim a real-world limit because ___ .
What I got wrong:  I expected ___ to move first, but ___ did. Or: I measured too soon after the step.
What I changed:    I waited ___ before measuring / changed the step size to ___ .

Purpose

Staff-level interviews ask how you handle a bad day. In this project you pick three incidents from the table, break the lab on purpose, observe, repair, and then write a post-incident report for each. The report is the point. Writing a timeline from your own log shows you what you missed.

Spec

• Pick three incidents from the table. Pick at least one that touches replication or the controller, and one from the client side.
• Use the matching drill on drills.html for the commands. Do not skip its questions.
• For each incident: reproduce, observe, repair, then write the report. Use the template below.
• As a reading step, open case-studies.html and compare each incident with a published case.
• Run the three health checks before and after each incident.

Choose from these lab-supported incidents

Post-incident report template

POST-INCIDENT REPORT: <title>
Date and who ran it:
Timeline     (clock times from your log; first sign, detection, action, recovery)
Impact       (what a user or a downstream system would have seen; numbers)
Cause        (the one mechanism, in plain words; point at the module section)
Fix          (the exact commands you ran, and the output that proved it worked)
Prevention   (a setting, an alert or a habit; name who would own it)
What the monitoring showed  (which metric or alert moved, how fast, or "nothing")
What I would still not know in production

Eight milestones

Acceptance tests

What to measure

• Time from the break to the first sign you noticed, and to the first alert if one fired.
• Time from the repair command to clean health checks.
• The number the drill asks for: epoch, message count, lag, or throughput.
• How many of your predictions were wrong. This is the most useful number.

The story it produces

You will have three stories. Choose the one where your prediction was most wrong. Speak it aloud in under two minutes.

Situation:         I replayed ___ (incident) on a 3-broker lab.
What I saw:        timeline ___ ; first sign at ___ ; numbers ___ .
What I decided:    I repaired by ___ and rejected ___ because ___ .
What I got wrong:  I predicted ___ . The lab showed ___ .
What I changed:    I would add ___ (alert, setting or runbook line) and check it by ___ .

Purpose

Module 12 found that the lab exporter could not read consumer groups from Kafka 4.3, so the lab has no lag metric and no lag alert. Real teams meet this: a tool stops working after an upgrade. You will build a small replacement in a language you know. It reads lag from the command-line tool, turns it into metrics, and raises one useful alert.

This card gives the spec and the tests only. You write the tool.

Spec

• Run kafka-consumer-groups.sh --describe --all-groups (module 03) every 30 seconds. Run it through docker exec on a broker, as the modules do.
• Parse the columns GROUP, TOPIC, PARTITION, CURRENT-OFFSET, LOG-END-OFFSET and LAG for every group.
• Compute lag in records for each partition. Check it against the LAG column.
• Estimate lag in time: lag in records divided by the end-offset rate (records per second, measured over a window you choose). Decide what to publish when the rate is zero.
• Publish the numbers where Prometheus can read them: a text file or a small HTTP endpoint. Name the metrics yourself and label them with group, topic and partition.
• Write one alert rule: a partition's committed offset has not moved for a time you choose, while its end offset has moved.
• Survive broker restarts. If one broker is down, try another, as module 05 does with kafka-2.
• Say what happens when a partition has no committed offset yet. unverified not run in the labs: the output for a group with no commit on a partition. Run it and record what you see.

Eight milestones

Acceptance tests

What to measure

• Seconds your tool takes to run one poll, and whether it finishes inside the 30-second gap.
• Time from the stop of a consumer to the alert firing. Include the wait time you chose.
• False alerts in a 24-hour run with a healthy group. Write down the count, even if it is zero.
• Lag in time estimate against the real measure in test D6.
• Time for the tool to recover after the cluster restart in D5.

The story it produces

The strong story is about the gap between what a metric says and what is true: lag in records looked small while the group was stuck, or the estimate was wrong at a zero rate. Tell what you saw in the numbers.

Situation:         The lab exporter had no lag metric on Kafka 4.3, so ___ .
What I saw:        my first version showed ___ ; the stuck group showed ___ ; idle showed ___ .
What I decided:    I chose a wait time of ___ s because ___ ; I rejected ___ .
What I got wrong:  ___ (a false alert, a stale number, a parse error)
What I changed:    ___ , and the false alerts went from ___ to ___ .

Purpose

Module 10 says a copy is not the original. Offsets differ, transactions are not preserved the way you hope, and failback is harder than failover. In this project you rehearse a real failover with a runbook and a stopwatch, then count what it cost. You also reproduce one problem module 10 reports and test its proposed fix.

Spec

• Copy topics from main to kafka-dr with MirrorMaker 2, using the module 10 part B steps. Start the second cluster with docker compose up -d kafka-dr.
• Translate group offsets (module 10 shows this with sync.group.offsets.enabled).
• Stop the main cluster. Start consumers on the standby from the translated offsets. Switch producers to the standby.
• Measure copy delay and repeated records.
• Reproduce the aborted-records problem from module 10, then test a read_committed copying consumer.
• Module 10 marks this fix unverified unverified: it did not rerun the lab. Your job is to find the right property name, set it, and record what you find. Do not trust a name from memory. Do not trust the module. Write down where you found it and what happened.
• Write the runbook before the rehearsal, and time the rehearsal against a target of 15 minutes. The target is a number to attempt. No module measured a failover time.

Runbook template

FAILOVER RUNBOOK: main -> kafka-dr
Decision rule   (what exact condition makes us fail over? e.g. main unreachable for ___ minutes)
Who decides     (a named role and a named backup; who tells the others)
Stopwatch       start ___ : ___ ; target 15 minutes total; stop ___ : ___
1. Stop the copy        (command; confirm the copy process has ended)
2. Record last offsets  (source end offsets if readable; copy end offsets; translated group offsets)
3. Switch producers     (new address; which topic name on the standby; who changes the settings)
4. Start consumers      (group name; the offsets they start from; how you check them)
5. Verify               (health checks on the standby; sample records; count of repeats)
6. Rebuild the old site (what to start; what is stale; how to resync; who confirms)
Known losses    (records not copied; repeated records; aborted records visible?)

Failback (return to the main cluster) is harder than failover. Module 10 explains why. Write down what you would do, and mark each step you did not run. unverified not run in the labs: any failback.

Eight milestones

Acceptance tests

What to measure

• Copy delay in seconds, at least five samples, and the range.
• Failover time on the stopwatch, step by step. Which step took longest?
• Repeated records after the switch, and how you counted them.
• Records not copied at the moment you stopped main.
• The count of aborted records visible on the copy before the fix, and after it.
• Time to rebuild the old site, and what you still could not tell.

The story it produces

The best story here is the gap between your runbook and your rehearsal: the step that took longer than planned, the number of repeats you did not predict, or the fix that behaved differently from the module. State what you found, not what you expected.

Situation:         The standby cluster had to take over, and the target was 15 minutes.
What I saw:        copy delay ___ s; failover took ___ min ___ s; ___ repeated records; ___ uncopied.
What I decided:    I moved consumers to translated offsets and accepted ___ repeats because ___ .
What I got wrong:  ___ (a runbook step, a topic name, an offset I trusted)
What I changed:    the runbook now says ___ ; the read_committed setting I found was ___ and it ___ .