The order builds ideas in sequence: the log, then producers, replication, consumers, the controller, and last the newest features. The times are my rough guess for reading the Motivation and Proposed Changes sections with notes. I did not measure them. Expect about 25 to 30 hours in all. Long proposals such as KIP-848 can take longer.
Your read marks are saved in this browser only. The filter does not hide the five non-KIP sources.
- Status (wiki page)
- Adopted
- First Kafka version
- 0.11.0.0 (released 2017-06-28). Source: the Jira ticket KAFKA-4815, titled "Idempotent/transactional Producer (KIP-98)".
The problem it solved
Kafka gave "at least once" delivery, so a retry could write the same record twice. Stream processing also needed to write to several partitions as one atomic unit.
Read these parts first
Motivation, Proposed Changes, Rejected Alternatives. The page also has Public Interfaces, Message Format and RPC Protocol Summary. Most useful: Proposed Changes, then Message Format.
What to take from it
- The producer API gains five methods: initTransactions, beginTransaction, sendOffsets, commitTransaction and abortTransaction.
- The message format moves to version 2. It adds a producer ID (64 bits), an epoch (16 bits) and a sequence number (32 bits) once per batch, not once per record.
- A new consumer setting, isolation.level, has two values: read_uncommitted (the default) and read_committed.
Question to ask yourself
A producer times out and retries. How does the broker know the retry is a duplicate, and what must stay the same for that to work?
Where this knowledge base covers it
Module 02 Producers, Module 06 Transactions
Sources used: KIP page · Jira KAFKA-4815
- Status (wiki page)
- Approved
- First Kafka version
- Not fully confirmed unverified. The Jira sub-task KAFKA-15449 ("Verify transactional offset commits (KIP-890 part 1)") is fixed in 3.7.0. The Kafka 4.0 announcement says the second phase is complete. The umbrella ticket KAFKA-14402 lists 4.1.0. I did not find one page that states the first release.
The problem it solved
A delayed message can arrive after its transaction has aborted. That leaves a hanging transaction: the last stable offset (the point up to which read_committed consumers may read) stops moving, and the log cannot be cleaned.
Read these parts first
Motivation, Proposed Changes, Rejected Alternatives. The page has no delivery plan, so use the release notes to see what shipped when. Most useful: Motivation, because it lists the exact failure cases.
What to take from it
- The producer epoch goes up after every commit or abort marker. The pair (producer id, epoch) then names one transaction.
- New clients add a partition to the transaction on the first produce request. They no longer need a separate AddPartitionsToTxn call.
- For older clients, the broker checks with a verify-only AddPartitionsToTxn request and returns INVALID_TXN_STATE if no transaction is open.
Question to ask yourself
Why was the transaction coordinator alone not enough to stop a delayed write from joining the next transaction?
Where this knowledge base covers it
Module 06 Transactions
Sources used: KIP page · Jira KAFKA-14402 · Jira KAFKA-15449 · Kafka 4.0 announcement
- Status (wiki page)
- Accepted
- First Kafka version
- 4.0.0, generally available. Sources: the Kafka 4.0 upgrade guide and Jira KAFKA-14048 (fix version 4.0.0, released 2025-03-18).
The problem it solved
The old rebalance protocol put the assignment logic in the clients. A fix needed a client release, problems were hard to debug without client logs, and every rebalance made the whole group stop and wait.
Read these parts first
Motivation first. Then the sections on the heartbeat call and the assignors. I did not record the exact section names for this page unverified. Most useful: Motivation.
What to take from it
- A new call, ConsumerGroupHeartbeat, carries membership, state and partition assignment. The group coordinator drives it.
- Brokers run the assignors on the server. Range and uniform ship by default. Kafka Streams keeps client-side assignment.
- Offset commit and fetch use topic IDs, so a deleted and re-created topic is not mistaken for the old one.
Question to ask yourself
In the old protocol, why did one slow member delay the whole group? What removes that dependency in the new one?
Where this knowledge base covers it
Module 03 Consumers
Sources used: KIP page · Jira KAFKA-14048 · Kafka 4.0 upgrade guide
- Status (wiki page)
- Accepted (2.4.0), as written on the page
- First Kafka version
- 2.4.0. Sources: the status line on the KIP page and Jira KAFKA-8179 (fix version 2.4.0).
The problem it solved
In a rebalance, every member gave up all its partitions, even those it would keep. Stateful applications such as Kafka Streams paid a high price for that pause.
Read these parts first
Motivation first, then the sections on the changed protocol and the upgrade plan. I did not record the exact section names for this page unverified. Most useful: The upgrade plan, because it shows why two rolling restarts are needed.
What to take from it
- Members report the partitions they own in their subscription and assignment messages, so only moved partitions are revoked.
- Upgrading takes two rolling restarts. First add CooperativeStickyAssignor beside the old assignor. Then remove the old one.
- A new callback, onPartitionsLost(), tells an application it was removed from the group, as opposed to a normal rebalance.
Question to ask yourself
Why does a rebalance in two rounds stop fewer partitions than one rebalance that revokes everything?
Where this knowledge base covers it
Module 03 Consumers
Sources used: KIP page · Jira KAFKA-8179
- Status (wiki page)
- Accepted
- First Kafka version
- 2.4.0 unverified. The wiki page gives no version. Jira KAFKA-7018 ("persist memberId for consumer restart") is fixed in 2.4.0, but its title does not name the KIP.
The problem it solved
A restarting consumer got a new identity, so each restart caused a rebalance. A rolling restart of N consumers caused about 2N rebalances.
Read these parts first
Motivation, Background of Consumer Rebalance, Proposed Changes, Rejected Alternatives. The page also has a Non Goal and a Future Works section. Most useful: Background of Consumer Rebalance.
What to take from it
- A consumer setting, group.instance.id, names each client. A non-null value turns on static membership.
- The broker keeps an in-memory map from group.instance.id to member.id. A returning member gets its old assignment without a rebalance.
- The page raises the maximum session timeout to 30 minutes, so a restart can finish before the broker removes the member.
Question to ask yourself
You turn on static membership and set a long session timeout. What do you gain, and what do you risk when a member really dies?
Where this knowledge base covers it
Module 03 Consumers
Sources used: KIP page · Jira KAFKA-7018
- Status (wiki page)
- Accepted
- First Kafka version
- 0.11.0.0. Source: Jira KAFKA-1211, whose title names KIP-101 (fix version 0.11.0.0).
The problem it solved
A follower that restarted cut its log back to its high watermark (the last offset known to be safe). It could then become leader and lose records that were in fact committed. Replica logs could also end up different.
Read these parts first
Definition of Terms, Motivation, Proposed Changes, Rejected Alternatives. The page also has Additional Concerns and an Appendix. Most useful: Motivation, because it walks through the failure step by step.
What to take from it
- A leader epoch is a 32-bit number that rises with each period of leadership for one partition. The leader stamps it on the messages it accepts.
- The core failure: a follower truncates to its high watermark, becomes leader, and permanently loses data before the old leader passes on the commit.
- Each replica keeps a leader epoch sequence file, which maps epochs to offsets and is cached in memory. Truncation uses it in place of the high watermark.
Question to ask yourself
A follower restarts and its high watermark is behind the leader. Walk through what it truncates under the old rule and under the epoch rule.
Where this knowledge base covers it
Module 04 Replication
Sources used: KIP page · Jira KAFKA-1211
- Status (wiki page)
- Accepted
- First Kafka version
- 2.0.0. Source: Jira KAFKA-6361, "Fast leader fail over can lead to log divergence between leader and follower" (fix version 2.0.0).
The problem it solved
KIP-101 left some cases where two replicas keep different logs after a quick leader change. This KIP closes those cases for both clean and unclean elections.
Read these parts first
Motivation, Proposed Changes, Rejected Alternatives. The page also covers the effect on compacted topics. Most useful: Proposed Changes. Read KIP-101 first.
What to take from it
- The OffsetForLeaderEpoch response gains a leader epoch field. Brokers use it to find the longest common log prefix where offsets and epochs both match.
- In rare unclean elections a follower may need several request and response rounds to converge. The common case needs only a few.
- If the leader has no epoch information, it answers UNKNOWN_OFFSET_FOR_LEADER_EPOCH. The follower then falls back to truncating at the high watermark.
Question to ask yourself
Why is an offset alone not enough to say two logs match? What does the epoch add?
Where this knowledge base covers it
Module 04 Replication
Sources used: KIP page · Jira KAFKA-6361
- Status (wiki page)
- Accepted
- First Kafka version
- 2.4.0. Source: Jira KAFKA-8443 (fix version 2.4.0, released 2019-12-16).
The problem it solved
Consumers could read only from the partition leader. When a cluster spans data centres, that sends read traffic across them and costs money.
Read these parts first
Motivation, Proposed Changes, Rejected Alternatives. The page also has Protocol Changes. Most useful: Proposed Changes.
What to take from it
- Consumers may fetch from any replica. The high watermark still limits what they can see, so replication guarantees hold.
- The broker chooses the replica through a plug-in set by replica.selector.class, using client location such as the rack ID.
- A new error code, OFFSET_NOT_AVAILABLE, marks an offset that exists but is not yet available on that replica. The consumer retries.
Question to ask yourself
A consumer reads from a follower. Could it see a record that is later lost? Why or why not?
Where this knowledge base covers it
Module 04 Replication
Sources used: KIP page · Jira KAFKA-8443
- Status (wiki page)
- Accepted
- First Kafka version
- 2.8.0, from the fix version of the Jira ticket KAFKA-9119. How that release labelled the feature (for example early access): not confirmed unverified.
The problem it solved
Kafka needed a separate ZooKeeper cluster to keep its metadata. That limited the number of partitions, added a second system to run, and made the cluster harder to set up.
Read these parts first
Motivation, Architecture, Compatibility Deprecation and Migration Plan, Rejected Alternatives, Follow-on Work. Most useful: Architecture.
What to take from it
- Three controller nodes form a Raft quorum that manages the metadata log. They replace three ZooKeeper nodes.
- Brokers fetch metadata updates from the active controller through a new MetadataFetch API. The controller does not push them.
- The plan needs a bridge release, where ZooKeeper stays but sits only behind the controller, before it is removed.
Question to ask yourself
What failure does the single metadata log remove, compared with controllers that push state changes to brokers?
Where this knowledge base covers it
Module 05 KRaft
The KIP is the vision. KIP-595 holds the protocol and KIP-833 the timeline.
Sources used: KIP page · Jira KAFKA-9119
- Status (wiki page)
- Accepted
- First Kafka version
- Not confirmed unverified. The Jira ticket KAFKA-9876 is still open and has no fix version.
The problem it solved
KIP-500 needed a way for the controllers to agree on one metadata log. This KIP adapts Raft, a consensus protocol, to Kafka while keeping its pull-based replication and log structure.
Read these parts first
Motivation, Key Concepts, Leader Election and Data Replication, Log Compaction and Snapshots, Rejected Alternatives. Most useful: Key Concepts, then Leader Election and Data Replication. Read the Raft paper first.
What to take from it
- Replication is pull-based. Voters and observers send fetch requests to the leader, which simplifies bootstrap and observers.
- A record is identified by its offset and the epoch of the leader that appended it. That lets logs reconcile after a leader change.
- The page requires an fsync (a forced write to disk) on every append to the local log of the metadata topic.
Question to ask yourself
Raft pushes entries from the leader. Kafka pulls. What does that change for a new or lagging controller?
Where this knowledge base covers it
Module 05 KRaft
Sources used: KIP page · Jira KAFKA-9876
- Status (wiki page)
- Accepted
- First Kafka version
- Removal of ZooKeeper mode: 4.0.0. Source: the Kafka 4.0 upgrade guide ("ZooKeeper mode has been removed"). I found no separate proposal that is titled as the removal itself unverified. KIP-833 is the plan that names Kafka 4.0.
The problem it solved
Kafka had to maintain two controller code paths, one for ZooKeeper and one for KRaft. This KIP set the plan to mark KRaft ready, deprecate ZooKeeper mode and then remove it.
Read these parts first
The plan and timeline sections. I did not record the section names for this page unverified. Most useful: The timeline. It shows how a plan changes after a proposal is accepted.
What to take from it
- The page says the plan is to remove ZooKeeper mode entirely in Kafka 4.0. Reason given: avoid maintaining parallel code for important infrastructure such as the controller.
- The page lists 3.7 as the "Final release with ZK mode". Do not rely on that. The 4.0 upgrade guide is the source of truth for what shipped.
- The 4.0 guide says only KRaft mode is supported. This is the clearest example on this page of a proposal that the release notes overrule.
Question to ask yourself
You must upgrade a ZooKeeper-based cluster to 4.x. What is the path, and which release notes would you check first?
Where this knowledge base covers it
Module 05 KRaft, Module 09 Operations
Sources used: KIP page · Kafka 4.0 upgrade guide
- Status (wiki page)
- Accepted
- First Kafka version
- 3.9.0. Source: Jira KAFKA-14094 (fix version 3.9.0, released 2024-11-07).
The problem it solved
The set of controllers was fixed in a configuration file. An operator could not add a controller or replace a failed one without a manual, disruptive change.
Read these parts first
Motivation, Key terms, Proposed changes, User explanation, Reference explanation. The page also lists the new RPCs and the Admin client changes. Most useful: User explanation, then Reference explanation.
What to take from it
- An AddVoter RPC adds a controller to the voter set. The cluster does not need to stop all controllers.
- Two new control records, KRaftVersionRecord and VotersRecord, store the voter set in the log and in snapshots. This replaces reliance on controller.quorum.voters.
- With controller.quorum.auto.join.enable=true, a controller can join as an observer and then become a voter without a command-line step.
Question to ask yourself
Why is changing the voter set of a consensus group harder than changing any other piece of state?
Where this knowledge base covers it
Module 05 KRaft
Module 05 covers the quorum. It does not cover dynamic membership in depth.
Sources used: KIP page · Jira KAFKA-14094
- Status (wiki page)
- Accepted
- First Kafka version
- 4.0, as a preview. Source: the Kafka 4.0 announcement ("introduces Eligible Leader Replicas (ELR) in preview in 4.0"). The 4.0 upgrade guide says KIP-966 Part 1 is in 4.0. The Jira ticket KAFKA-15332 is open. The release that made it the default: not confirmed unverified.
The problem it solved
In a "last replica standing" case, an unclean shutdown can turn the data loss on one broker into loss for the whole partition. The KIP aims to tolerate such an event when min.insync.replicas is the limit.
Read these parts first
Motivation, Proposed Changes, Delivery Plan, Rejected Alternatives. Most useful: Proposed Changes. The Delivery Plan shows the parts that ship in stages.
What to take from it
- The high watermark advances only if the in-sync replica set (ISR) is at least min.insync.replicas in size.
- ELR are replicas outside the ISR that are eligible to lead. They are guaranteed to hold the data at least up to the high watermark.
- Three unclean recovery modes replace random unclean election: Aggressive, Balanced and None. None needs a manual step.
Question to ask yourself
What is the difference between being in the ISR and being an eligible leader replica? Why keep both?
Where this knowledge base covers it
Module 04 Replication, Module 05 KRaft
Sources used: KIP page · Jira KAFKA-15332 · Kafka 4.0 announcement
- Status (wiki page)
- Accepted
- First Kafka version
- 3.6.0 (Jira KAFKA-7739, fix version 3.6.0, released 2023-10-10). The KIP page says it is production-ready since 3.9. How 3.6.0 labelled it: not confirmed unverified.
The problem it solved
Large clusters hold all retained data on broker disks. That makes recovery and rebalancing slow and forces you to add brokers just to keep more history.
Read these parts first
Motivation, Goals and Non-Goals, High-level Design, RemoteLogManager, Follower Replication, Consumer Fetch Requests. The page has no Rejected Alternatives heading in my notes unverified. Most useful: High-level Design.
What to take from it
- There are two tiers. The local tier uses broker disks. The remote tier, such as HDFS or S3, stores completed log segments.
- Segment metadata goes in an internal, non-compacted topic named __remote_log_metadata, with 50 partitions by default.
- Followers copy only from the leader local storage. They build helper state from remote segments and may see OFFSET_MOVED_TO_TIERED_STORAGE.
Question to ask yourself
A consumer reads data that now lives only in remote storage. Which component serves it, and what does the broker disk still hold?
Where this knowledge base covers it
Module 09 Operations, Module 14 Tiered storage lab
Sources used: KIP page · Jira KAFKA-7739
- Status (wiki page)
- Accepted
- First Kafka version
- Early access in 4.0, preview in 4.1, production-ready in 4.2. Sources: the 4.0.0 announcement, the wiki preview notes (4.1) and the 4.2.0 announcement. The Jira ticket KAFKA-16092 shows no fix version.
The problem it solved
In a consumer group, the number of useful consumers is capped by the number of partitions, and a consumer owns a partition. Queue-style work needs many workers and per-record acknowledgement.
Read these parts first
Motivation, Concepts, Record delivery and acknowledgement, Managing durable share-partition state. The page has no Rejected Alternatives heading in my notes unverified. Most useful: Concepts, then Record delivery and acknowledgement.
What to take from it
- Share groups use only server-side assignors, such as SimpleAssignor. Consumers do not own partitions exclusively.
- Each acquired record has a lock that lasts 30 seconds by default. A delivery count caps redelivery at 5 by default.
- State is kept in a new internal topic, __share_group_state, with 50 partitions by default. It is separate from __consumer_offsets.
Question to ask yourself
What do you lose by moving from a consumer group to a share group? Think about ordering.
Where this knowledge base covers it
Module 03 Consumers
Early-access and preview builds were not compatible with each other. Read the 4.1 and 4.2 notes before you rely on old advice.
Sources used: KIP page · Preview release notes (4.1) · Kafka 4.2.0 announcement · Jira KAFKA-16092
- Status (wiki page)
- Accepted
- First Kafka version
- 2.4.0. Source: Jira KAFKA-7500 (fix version 2.4.0, released 2019-12-16).
The problem it solved
The first MirrorMaker could not sync configuration between clusters and had no support for active/active use. MirrorMaker 2 is built on Kafka Connect and fixes both.
Read these parts first
Motivation, Proposed Changes, Remote Topics Partitions, Internal Topics, Rejected Alternatives. Most useful: Remote Topics, Partitions.
What to take from it
- Remote topics carry the source cluster alias as a prefix, for example us-west.topic1. That allows active/active replication and keeps partitioning and order.
- Three internal topics hold heartbeats, checkpoints and offset syncs. All three are log-compacted.
- Four ways to run it: a dedicated MirrorMaker cluster, a connector on a Connect cluster, a standalone worker, and a legacy mode.
Question to ask yourself
Why does MirrorMaker 2 prefix topic names? What goes wrong in a two-way setup if it does not?
Where this knowledge base covers it
Module 10 Multi-cluster
Sources used: KIP page · Jira KAFKA-7500