Kafka, from the log up
Ten modules that go from "what is a topic" to the design decisions a staff engineer makes. Each one has the theory, the configs, the failures, a lab that was run on a real three-broker cluster, and 175 activities in all: animations to watch, incidents to handle, settings to tune, numbers to work out and designs to defend.
This is the whole idea: a list that only grows at the end. Records written: 0
Seven-day reading order
Plan 6 to 8 hours each day. Read a section, then do its activities before you read on. Do the lab on the real cluster, and finish with the boss round. Your progress is kept in this browser.
01 The log
Topics, partitions, offsets, segments, indexes, retention, compaction.
02 Producers
Partitioner, batching, compression, acknowledgements, idempotence, retries, ordering.
03 Consumers
Groups, both rebalance protocols, offset commits, lag, delivery guarantees.
04 Replication
Leaders and followers, in-sync replicas, high watermark, leader epoch, unclean election.
05 KRaft and the controller
The metadata log, the controller quorum, the life of a broker.
06 Transactions and exactly-once
The transactional producer, read-process-write, and where the guarantee stops.
07 Kafka Streams
Streams and tables, state stores, windows, joins, changelog topics.
08 Ecosystem
Schema Registry, Kafka Connect, change data capture, the outbox pattern, dead-letter topics.
09 Operations
Sizing, partition count, key metrics, tiered storage, quotas, security, upgrades.
10 Multi-cluster and design
MirrorMaker 2, multi-region, share groups, and when not to use Kafka.
11 Security lab
A locked cluster: TLS, user names and passwords, and permissions for each user, with every rejection tested.
12 Monitoring lab
Broker metrics by their real names, Prometheus with alert rules, and a fault to make the alerts fire.
13 Change data capture and outbox
Debezium and PostgreSQL: snapshot, insert, update, delete, an outbox with a rolled-back transaction, and a stopped connector.
14 Tiered storage
A broker with a tiered storage plug-in: segments move to remote storage, old data is read back, then expires.
The lab cluster
Every lab uses the same three-broker cluster, defined in docker-compose.yml in this folder. It also starts a web console at http://localhost:8085 and, for module 08, a schema registry at http://localhost:8086.
cd ~/Desktop/kafka-knowledge-base
docker compose up -dThat starts six containers: three brokers, the web console, the schema registry, and a one-broker second cluster for module 10. To start only the three brokers, run docker compose up -d kafka-1 kafka-2 kafka-3. Do the labs in module order: later labs use topics that earlier labs create. To start again from nothing, run docker compose down and then up again.
Practice and projects
Reading is not enough for a staff interview. These four pages turn the modules into practice.
Failure drills
15 scripted incidents to run on the lab cluster, with a stopwatch and a write-up builder for your own stories.
Mock interview
75 timed questions to answer aloud, with model answers, self-scoring and a 45-minute mock loop.
Capstone project
Build one small system on the lab cluster, keep it running, and collect real stories from it.
Case studies
Nine public Kafka cases, and seven laptop recipes for the experience and scale you do not have yet. No invented stories.
Projects board
Five projects with milestones and acceptance tests: the capstone, a scale lab, incident replay, a lag monitor and a failover rehearsal.
Reading guide
The design proposals behind the main features, checked against the Apache pages, with what to take from each.
Reference pages
Glossary
77 terms, each with the module that explains it. You can filter by typing.
Cheat sheet
The commands from every lab, the settings to start from, and the default numbers worth remembering.
Unverified claims
The 87 statements I could not confirm against the Kafka 4.3 documentation or a lab cluster.