Capstone: order-to-settlement
Build one small real system on the lab cluster and keep it running. You will break it on purpose, fix it, and collect stories you can tell in an interview.
Status of the starter files. The folder capstone/ holds a setup script, a database schema, a connector config and the acceptance tests. None of it has been run end to end. The script passed a syntax check only. The tables orders and outbox and the connector settings are as run in the cdc lab. Fix what you find and write it down: that is part of the exercise.
Why this project
Reading explains how Kafka works. It does not give you the memory of a system that went wrong. A staff-level interview asks "tell me about a time". You need your own answers.
- One system forces every module to meet: producers, consumers, transactions, Kafka Streams, the ecosystem, operations, monitoring.
- A system that runs for weeks fails in ways a one-hour lab does not: a slot that grows, an alert that cries wolf, a restart at a bad moment.
- Each week ends with a written note. By week 8 you hold about eight stories in a fixed shape.
Architecture
The order service never talks to Kafka. It writes to PostgreSQL. A connector reads the database log and publishes. Two readers consume the topic.
pgTables orders, outbox, processed_events.cdc-connect, port 8087. Reads the outbox rows from the database log.order-eventsKey is the order id. 3 copies, minimum 2.processed_events. Bad records go to order-events-dlq and raise an alert.Schema registry (port 8086) holds the contract for the event. With the connector settings from the lab, the registry does not check records on the wire. See week 5.
Requirements and acceptance tests
Each requirement has a test. The full commands and what a pass looks like are in capstone/acceptance-tests.md. A requirement is done only when its test passes and you wrote down what you saw.
| # | Requirement | Acceptance test |
|---|---|---|
| R1 | Topic order-events: key is the order id, 3 copies, min.insync.replicas 2. | Describe the topic and read the settings. Stop two brokers and send with acks=all: the send fails. |
| R2 | The service saves an order and its event in one database transaction. | Force an error before commit: neither row exists. |
| R3 | Events reach Kafka only through the outbox and the connector. | Search the service code for a Kafka client: none. Stop the connector, insert three orders, restart: all three arrive. |
| R4 | A registered schema with a compatibility mode. | Ask the registry about a change that removes a required field: it answers not compatible. |
| R5 | The consumer is safe to repeat. | Reset the group to the start and replay: row counts in processed_events and your totals do not change. |
| R6 | Bad records go to order-events-dlq and raise an alert. | Send an unparseable record: it appears on the dead-letter topic, the consumer keeps going, the alert fires. |
| R7 | Daily total for each merchant, correct after a crash. | Kill the job mid-stream and restart: totals equal your hand-computed sums. |
| R8 | Every stateful operator and repartition step has a name. State is on a durable disk. exactly_once_v2 is on. | List the internal topics: all names are yours. Delete the state directory: totals come back from the changelog. |
| R9 | Monitoring from the monitoring profile and a lag check from the command line. | The rules show on port 9095. kafka-consumer-groups.sh --describe shows lag that rises when you stop the consumer. |
| R10 | A written throughput limit. | One page that names the highest rate with flat lag and the part that limited it. |
| R11 | One failure drill every week of week 8, and ideally every week from week 3. | A story for each drill. |
Design choices you must make
- Merchant in the payload. The
orderstable from the lab has no merchant column. Put the merchant in the outbox payload, or add a column and say so in your notes. - Event id. The outbox
iddefaults to a generated value. Insert it yourself and copy it into the payload aseventId. The consumer then does not depend on any header the connector may add. - Topic name from the connector. The connector decides which topic it writes to. unverified No module shows the default name or how to change it. Watch which topic appears in week 2, then make it match
order-events. - Language. Use any language you know that has a Kafka client. The plan does not depend on it.
The 8-week plan
Four weeks is the minimum: do weeks 1 to 4 and 8. Eight weeks is better. Tick items as you finish them. Your ticks are saved in this browser.
Environment and topics
Deliverables
Measure. Time for the cluster to be ready from a cold start. How many seconds a broker restart takes.
Write down. A one-page diagram in your own words. Why 3 copies and a minimum of 2 (module 04).
Outbox and capture
Deliverables
Measure. Delay between the database commit and the record on the topic. Size of the replication slot while the connector is stopped.
Write down. The real topic name and key the connector produced. Compare them with what you wanted, and write down the fix.
Consumer safe to repeat
Deliverables
Measure. Events per second the consumer handles. How many repeats the table skipped after a restart.
Write down. Why commit order matters: effect and event id first, offset after (module 03).
Streams job
Deliverables
Measure. Time to restart with state kept, and with state deleted. Size of the changelog topic.
Write down. What happens to a late order that belongs to yesterday. Decide, then write the rule.
Schemas and a breaking-change rehearsal
Deliverables
Measure. Which changes the registry accepted and refused under each mode.
Write down. Who breaks first when a field changes: the consumer, the Streams job or the connector? Write the order you saw.
Monitoring and alerts
Deliverables
Measure. How long each alert takes to fire. How many false alerts you get in one week.
Write down. For each alert: who acts, what they do first, and how they know it is fixed.
Load test and the written limit
Deliverables
Measure. Records per second and lag at each step. Which part limited you: producer, broker disk, consumer or database.
Write down. A one-page limit statement: "This system holds X orders per second because Y." Say how you would raise it.
Failure-drill week and the stories
Deliverables
Measure. Time to detect and time to recover for each drill.
Write down. Finish the stories table below. Speak each one aloud in under two minutes.
Use the commands in the modules for the work. Setup, k and B are on the cheat sheet.
Stories to collect
Every story has five parts. Fill one row for each week, and for each drill. The fourth part, what you got wrong, is the one interviewers trust most.
| Part | Question to answer | Example from week 2 |
|---|---|---|
| 1 Situation | What were you building or running, and what mattered? | Events must never be lost when the service crashes. |
| 2 What I saw | The facts: output, numbers, symptoms. | The record landed on a topic with a different name than I expected. |
| 3 What I decided | The choice and the reason, with the options you rejected. | I changed the connector routing instead of the consumer. |
| 4 What I got wrong | A wrong guess, a missed check, a mistake. | I assumed the key was the order id and did not read it back. |
| 5 What I changed | The fix, and the test or alert that now guards it. | Added acceptance test B2 that reads the key. |
Seed ideas for stories
- A consumer that processed an event twice after a restart.
- The slot that grew while the connector was stopped.
- A Streams restart that was slow because the state directory was gone.
- A schema change that one reader could not handle.
- An alert that fired at the wrong time, or not at all.
- The load test: what actually limited you.
What this will not teach you
| Gap | Why it matters | What to do about it |
|---|---|---|
| Real traffic | Your load is invented. Real data is uneven: hot keys, bursts, odd sizes. | Make the load test skewed: send most orders for one merchant and see one partition lag. Say plainly in an interview that the traffic was synthetic. |
| Many teams | You own every part. In a company, a schema change needs agreement across teams. | Write a one-page change process for the event schema: who reviews, how long notice is, how to retire a field. Ask a friend to play the consumer team and object. |
| Cloud scale | Three small containers on one machine show no network limits, no disk limits and no cost trade-offs. | Read the tiered storage and multi-cluster modules and the numbers in them. Estimate, on paper, the disk and network for 100 times your load. |
| On-call pressure | You fail on purpose, calmly, with the runbook open. A night page is different. | Ask someone to start a drill without telling you which one. Set a timer. Practise saying your first three checks aloud. |
Related pages. Failure drills, Cheat sheet, Glossary. Starter files: capstone/README.md, setup.sh, schema.sql, connector-outbox.json, acceptance-tests.md.