Kafka 4.3 · public cases, studied from the outside
Case studies for engineers without production experience
Nine published cases, each tied to the module that explains the mechanism. Then seven recipes to earn a little real experience on a laptop.
What this is. A set of public write-ups from other teams. All sources were fetched on 5 October 2026. This page reports what each source says and nothing more. You read them from the outside.
What this is not. It is not your experience. Never tell an interviewer that you lived one of these. Each case has a spoken version in the third person and a line that shows how to say it honestly.
What I dropped. Three items were not used because they could not be confirmed. A “Kafka 4.0 on Kubernetes” postmortem on dev.to: the page returned 404. A “$2.3M fintech” story: it is illustrative and has no source. Uber’s multi-region disaster recovery blog: the page returned 404.
9 of 9 cases shown. 0 of 9 studied.
No case matches these filters.
Part A. Nine published cases
1. Apache Kafka supports 200,000 partitions per cluster
The Kafka project recommended up to 4,000 partitions on each broker and up to 200,000 in a whole cluster. It then measured how two slow operations improved between Kafka 1.0.0 and 1.1.0.
Root cause, as stated
The post says the main reason for the cluster-wide limit is "to accommodate for the rare event of a hard failure of the controller". The controller is the part of the cluster that tracks partition leaders.
What changed
Kafka 1.1.0 made its calls to ZooKeeper asynchronous and batched the messages that tell brokers about new leaders. ZooKeeper is the older system that held cluster metadata.
The number that mattered
Controlled shutdown of a broker with 50,000 partitions across 5 brokers took 6.5 minutes on 1.0.0 and 3 seconds on 1.1.0. Reloading controller state for 100,000 partitions took 28 seconds on 1.0.0 and 14 seconds on 1.1.0.
This is the ZooKeeper era. Kafka 4 uses KRaft instead. Module 05 measured a controller failover of about 1 second on 3 nodes, which is a small cluster. This page makes no claim for large clusters.
Questions to ask yourself
Why does a limit for the whole cluster exist when each broker already has its own limit?
What work must a new controller do before it can serve requests, and how does the partition count change it?
Which of these two measured numbers would you report first to a manager, and why?
How to say this in an interview
Spoken version, in third person: A published Apache Kafka blog post describes a cluster that was allowed to grow to many partitions. It reports that the team recommended up to 4,000 partitions on each broker and 200,000 in a cluster. The team gave a cluster-wide limit mainly because a hard controller failure is rare but costly. In their test, shutting down a broker with 50,000 partitions across 5 brokers took 6.5 minutes on Kafka 1.0.0. Version 1.1.0 used asynchronous ZooKeeper calls and batched leader messages, and the same test took 3 seconds.
How to say it honestly: Start with "In a published case I studied, ..." and name the source. Do not say "we" or "I saw". You read this case. You did not run it.
2. LinkedIn: improving Kafka at a trillion messages a day
LinkedIn described a Kafka deployment that grew 1,200 times in four years, from 1 billion messages a day in July 2011. It listed the problems that growth caused and the tools the team built.
Root cause, as stated
Several causes, all stated in the post. One application could hurt others on a shared cluster. In its words, "if one application starts abusing Kafka, it can have an adverse impact on the performance and SLAs for other applications that share the cluster". A service level agreement (SLA) is a promise about performance. Partitions were spread unevenly, which made hotspot brokers that risked disk and network. Topic deletion caused instability. Mirror Maker, the tool that copies data between clusters, could lose messages. Replica lag calculations failed during traffic spikes.
What changed
They built quotas that throttle producers and consumers over a bytes-per-second threshold. They built a new consumer that does not use ZooKeeper, fixes to make Mirror Maker lossless, time-based replica lag, and safe topic deletion. They also built Burrow for consumer lag, a monitoring framework, and Simoorg for failure testing.
The number that mattered
1 trillion published messages a day, peaks of 4.5 million messages per second, about 1.34 PB a week, and each message consumed by about four applications.
Not confirmed in the source: the post as summarised here gives no incident dates, no outage durations, and no figures for the effect of the fixes.
Questions to ask yourself
Which of the listed problems comes from sharing one cluster, and which comes from the size of the cluster?
Why would a lag measured in messages fail during a traffic spike, and why would a lag measured in time hold up better?
Which of the tools they built would you build first, and what would you watch to know it worked?
How to say this in an interview
Spoken version, in third person: A 2015 LinkedIn engineering post describes a Kafka deployment that carried 1 trillion messages a day. It says traffic grew 1,200 times in four years, and that one abusive application could hurt every other application on a shared cluster. The team added quotas that throttle producers and consumers above a bytes-per-second threshold. They also listed problems that quotas did not solve, such as hotspot brokers, unsafe topic deletion and Mirror Maker losing messages. Their fixes included time-based replica lag, safe topic deletion, lossless Mirror Maker changes, and tools named Burrow and Simoorg.
How to say it honestly: Start with "In a published case I studied, ..." and name the source. Do not say "we" or "I saw". You read this case. You did not run it.
3. Honeycomb: brokers throttled by the cloud network
Delays appeared in a processing service and persisted even after the team scaled it. The team traced the problem to the Kafka brokers.
Root cause, as stated
As stated: the brokers were "being throttled over network allowances by AWS". AWS is the cloud provider.
What changed
The post says "We needed to scale our cluster vertically to get more network capacity". Vertical scaling means bigger machines rather than more of them. The team moved high-impact partitions to larger instances first and then moved the rest gradually. They added "50% extra capacity", but delays recurred.
The number that mattered
50% extra capacity was added, and the delays still came back.
Not confirmed in the source: the exact instance types and the duration of the incident.
Questions to ask yourself
Why does adding more consumers not help when the brokers are the limit?
Which broker metrics would show a network limit before users see delay?
Why move the high-impact partitions first rather than all partitions at once?
How to say this in an interview
Spoken version, in third person: A Honeycomb engineering post describes a month of incidents that included delays in a processing service. The team saw that those delays persisted even when they scaled the service itself. They decided the Kafka brokers were being throttled over network allowances by AWS, so they moved high-impact partitions to larger instances and then the rest. The problem was not over: after adding 50% extra capacity, the delays recurred. The change they described was vertical scaling for more network capacity, and the post goes on to find a second cause.
How to say it honestly: Start with "In a published case I studied, ..." and name the source. Do not say "we" or "I saw". You read this case. You did not run it.
4. Honeycomb: one connection per broker in the client library
After the brokers were scaled, delays continued. The team looked at how their consumers talked to the brokers. The post also mentions a Kafka auto-balancer, a tool that moves partitions between brokers, that repeatedly got stuck and needed manual restarts.
Root cause, as stated
As stated: with the Sarama client library, "Kafka consumer groups open only a single connection to any leader, regardless of how many partitions are going to be fed from it". The cause was contention for the connection buffer when several partitions from one broker were assigned to a single consumer. The root cause of the auto-balancer getting stuck is not stated.
What changed
The team tried increasing buffer settings and throughput over several days. The post gives no numbers for this.
The number that mattered
One connection to each leader broker, no matter how many partitions it feeds. That fact is the number that mattered.
Not confirmed in the source: the cause of the auto-balancer getting stuck, and any numbers for the buffer changes.
Questions to ask yourself
What happens to fetch throughput when many partitions share one connection and one buffer?
Which consumer setting or assignment choice could spread partitions from one broker across consumers?
How would you tell connection contention apart from a slow consumer, using only metrics?
How to say this in an interview
Spoken version, in third person: The same Honeycomb post describes a second problem after the brokers were scaled. It says delays continued, and the team looked at the Sarama client library. They found that a consumer group opens one connection to each leader broker, so several partitions from one broker compete for one connection buffer. The fix was not quick: they tried larger buffer settings and more throughput over several days. The post also says a Kafka auto-balancer kept getting stuck and needed manual restarts, and it does not state why.
How to say it honestly: Start with "In a published case I studied, ..." and name the source. Do not say "we" or "I saw". You read this case. You did not run it.
Uber built a proxy that sits between Kafka and consumer services and pushes messages to them. Over 1,000 consumer services are onboarded.
Root cause, as stated
Three problems, as stated. First, head-of-line blocking: one message that cannot be processed blocks the messages behind it. Messages fail because of size limits such as the 4 MB gRPC default, invalid consumer instances or request filtering. They become poison-pill messages, which are messages that keep failing and block the consumer. Second, partition scalability and rebalancing do not fit queue-like use. Third, hardware efficiency: the proxy fleet grew from "two digits of servers" to "hundreds, eventually thousands".
What changed
Context-aware routing through message headers. Active head-of-line blocking resolution: when tracker utilisation exceeds 90% and uncommitted messages are below 2%, it cancels requests and routes the messages to dead-letter queues. A consumer auto-rebalancer that scales up fast, scales down slowly and keeps placement sticky. Delay processing that pauses only specific partitions.
The number that mattered
Thresholds of 90% tracker utilisation and 2% uncommitted messages trigger the blocking resolution. Over 1,000 consumer services are onboarded.
Not confirmed in the source: any throughput, latency or cost numbers. The post gives none.
Questions to ask yourself
Why is a poison-pill message worse in a partitioned log than in a classic queue?
What does a dead-letter queue give up, and what does it protect?
Why scale down slowly but up fast?
How to say this in an interview
Spoken version, in third person: A February 2026 Uber engineering post describes a consumer proxy called uForwarder. It says messages that fail on limits such as the 4 MB gRPC default can block every message behind them. Uber decided to detect that state actively: when tracker utilisation exceeds 90% and uncommitted messages are below 2%, the proxy cancels requests and routes messages to dead-letter queues. The post also lists a fleet that grew from two digits of servers to hundreds and eventually thousands, which drove the work on efficiency and auto-rebalancing. It reports that over 1,000 consumer services are onboarded.
How to say it honestly: Start with "In a published case I studied, ..." and name the source. Do not say "we" or "I saw". You read this case. You did not run it.
Uber moved to tiered storage, where recent data stays on the broker and older data moves to cheaper remote storage.
Root cause, as stated
Storage and compute were tightly coupled. The post says "Kafka cluster storage is typically scaled by adding more broker nodes to the cluster. But this also adds needless memory and CPUs". Longer retention needed more nodes and separate pipelines to offload old data.
What changed
Tiered storage (KIP-405, the Kafka improvement proposal for it). A local tier holds recent data, for hours. A remote tier on S3, GCS or Azure holds history, for days to months. It has run in production for "~1-2 years in different workloads".
The number that mattered
None. The post gives no measured numbers: no retention, no broker count, no recovery time and no cost. Say that plainly.
Not confirmed in the source: retention, broker count, recovery time and cost. The post gives no measured numbers.
Questions to ask yourself
Which resource does a broker need for storage, and which does it waste when you add nodes only for disk?
What gets slower when a consumer reads old data from the remote tier?
What would you measure before and after a move like this to prove it helped?
How to say this in an interview
Spoken version, in third person: A July 2024 Uber engineering post describes why storage and compute were tightly coupled in their Kafka clusters. It says that adding broker nodes for storage also adds needless memory and processors, and longer retention needed separate offload pipelines. Uber chose tiered storage, with recent data on a local tier and history on remote object storage. The post does not report measured results, so any claim about savings would go beyond it. It says the feature has run in production for about one to two years in different workloads.
How to say it honestly: Start with "In a published case I studied, ..." and name the source. Do not say "we" or "I saw". You read this case. You did not run it.
Cloudflare engineers described lessons from running Kafka for many services. The cluster handled "1 trillion messages".
Root cause, as stated
Two problems. Services "still ended up being tightly coupled because of an unstructured approach to schema management". A schema is the agreed shape of a message. Separately, during growth in load in the pandemic, a critical consumer "had begun to breach its Service Level Agreements", and the cause was hard to find at first.
What changed
They moved from JSON to Protobuf and built a client-side validation library. They added OpenTelemetry tooling, a standard for traces and metrics, to their SDKs. Their four lessons: balance configurability with standardisation, use strict 1:1 contract interfaces, invest in metrics, and write clear adoption documentation.
Not confirmed in the source: cluster and node counts.
Questions to ask yourself
How can services be tightly coupled even though they only talk through Kafka?
Why validate messages in the client library and not only on the broker?
What metrics would have helped find a consumer that was breaching its agreement?
How to say this in an interview
Spoken version, in third person: A 2023 QCon London talk, reported by InfoQ, describes Cloudflare running Kafka for a very large number of messages. The speakers said services were still tightly coupled because schema management was unstructured. The team decided to move from JSON to Protobuf and to build a client-side validation library. During pandemic load growth, a critical consumer began to breach its service level agreements, and the cause was hard to find at first. The team added OpenTelemetry tooling to their SDKs and listed four lessons, including investing in metrics.
How to say it honestly: Start with "In a published case I studied, ..." and name the source. Do not say "we" or "I saw". You read this case. You did not run it.
8. KAFKA-3861: a shrunk in-sync set before a leader crash
design flawaffected version 0.10.0.0; status as fetched 5 October 2026
A leader shrinks the in-sync set, the group of replicas that are up to date, down to itself. Then it crashes.
Root cause, as stated
As described in the ticket: the controller cannot elect a new leader from followers that are outside the in-sync set. With unclean election disabled, the partition can be unavailable. Restarting the crashed broker may even trigger cascading failures.
What changed
Nothing yet, as far as the ticket shows. The fix version is none and the status was open when fetched on 5 October 2026.
The number that mattered
Affected version 0.10.0.0. Fix version: none. Status: open.
Module 04 explains eligible leader replicas, which keep recently fenced replicas as election candidates. This page does not claim that this ticket was closed by them. The ticket was still open when fetched.
Not confirmed in the source: whether any later change closed this ticket. The page only reports that its status was open.
Questions to ask yourself
Why does Kafka refuse to elect a replica that is outside the in-sync set?
What is the trade between availability and losing data in this scenario?
What would you check to know whether a partition is in this state right now?
How to say this in an interview
Spoken version, in third person: A Jira ticket, KAFKA-3861, reports a design flaw seen on Kafka 0.10.0.0. It describes a leader that shrinks the in-sync set to only itself and then crashes. The controller then cannot choose a new leader from followers outside the in-sync set, so with unclean election disabled the partition can be unavailable. The ticket also says that restarting the crashed broker may trigger cascading failures. It showed no fix version and an open status on 5 October 2026.
How to say it honestly: Start with "In a published case I studied, ..." and name the source. Do not say "we" or "I saw". You read this case. You did not run it.
A walk-through of one failure with replication factor 3 and minimum in-sync replicas 2. The leader is R1 and the in-sync set is [R1, R2, R3]. R3 is fenced, which removes it from the set: [R1, R2]. R1 acknowledges records that only R2 has. Then R2 is fenced and the set is [R1], the "last replica standing". An operating-system-level failure kills R1, and its page cache and unflushed data are lost.
Root cause, as stated
When R1 restarts as leader, R2 and R3 truncate their logs to match. In the post's words: "it forces its followers to truncate their logs to match its own - this turns a local data loss issue into a global one."
What changed
KIP-966 introduces the Eligible Leader Replicas (ELR) set, so that recently fenced replicas stay as election candidates.
The number that mattered
Replication factor 3, minimum in-sync replicas 2, and an in-sync set that shrinks to one replica.
The lab in module 04 shows the Elr column in the describe output.
Questions to ask yourself
Why can a record be acknowledged and still be lost in this sequence?
Why does the truncation on R2 and R3 make the loss global?
What does keeping R2 and R3 as candidates change about who may be elected?
How to say this in an interview
Spoken version, in third person: A 2023 blog post by Jack Vanlightly explains a data-loss path called the last replica standing. With replication factor 3 and minimum in-sync replicas 2, two followers are fenced one after the other, and the leader acknowledges records held only by itself and one follower, then ends up alone. An operating-system-level failure kills that leader and loses its unflushed data. When it restarts as leader, the other replicas truncate their logs to match, so a local loss becomes a global one. KIP-966 changed this by adding the Eligible Leader Replicas set, which keeps recently fenced replicas as election candidates.
How to say it honestly: Start with "In a published case I studied, ..." and name the source. Do not say "we" or "I saw". You read this case. You did not run it.
Part B. Earning the experience on a laptop
Each recipe covers one gap. Commands come from the modules unless the page says unverified or “not run in the labs”. The helper and address used in every command:
These recipes use the 3-broker lab cluster on one laptop. Results describe that laptop only.
Recipe 1. Partition count at scale
The gap
You have never seen a broker carry thousands of partitions. Case 1 shows why that matters.
What to run
Create a topic with many partitions, using the create command from module 09 and the 3-broker lab cluster. Raise the count in steps and keep the replication factor at 3.
The step sizes (100 and then more), timing the commands with time, and reading memory with docker stats --no-stream are not run in the labs. unverified Whether a laptop can hold large counts is also unverified.
What to measure
The create time for each step. Broker memory from docker stats at each step. The time from docker start until the under-replicated check is empty.
What you may honestly claim
You may claim the three numbers for your own steps, on your own laptop. You may not claim a safe partition count for any real cluster.
Template
I measured topic creation time, broker memory and restart time at [N] partitions on a laptop with [CPU, memory, Docker setting]; I do not know how it behaves on real hardware; to check I would repeat it on a staging cluster with the same hardware as production.
Recipe 2. Cross-zone latency
The gap
All lab brokers share one machine, so there is no network distance between them.
What to run
You cannot measure cross-zone latency truthfully on one laptop. The brokers talk over a local virtual network that adds almost no delay.
You can set the rack labels that module 04 describes (broker.rack on brokers and client.rack on consumers, module 03) and check where replicas land. That tests placement only, not delay.
Setting rack labels in the lab compose file is not run in the labs. unverified This page names no tool for adding artificial delay, because none was run. Any such tool would be unverified.
What to measure
Replica placement from the describe output. Nothing about latency.
What you may honestly claim
You may claim that you understand rack placement and that every write crosses zones (module 04). You may not give a latency number.
Template
I checked replica placement with rack labels on a laptop with [setup]; I do not know how latency behaves on real hardware; to check I would measure produce latency with acks=all on a cluster whose brokers sit in three real zones.
Recipe 3. Controller load
The gap
The controller tracks every partition. You want to see how its work grows.
What to run
Use the quorum and metadata-log commands from module 05 before and after creating many topics (recipe 1).
All of these commands come from module 05. Comparing the numbers before and after many topics is not run in the labs. unverified
What to measure
HighWatermark and LogEndOffset from the quorum status. The count of each record type in the metadata log. The size of the metadata log directory.
What you may honestly claim
You may claim how the metadata log grew with your partition count, and that module 05 measured a failover of about 1 second on 3 nodes. You may not claim anything about a controller at 200,000 partitions.
Template
I measured metadata log record counts and the quorum high watermark before and after creating [N] topics on a laptop with [setup]; I do not know how it behaves on real hardware; to check I would watch controller failover time on a large staging cluster.
Recipe 4. Broker failover time
The gap
You know failover happens. You have not timed it, or compared a clean stop with a crash.
What to run
Use the failover steps from module 04 on a topic with 3 replicas. Stop a broker with docker stop and watch the describe output change.
d() { k kafka-topics.sh --bootstrap-server $B --describe --topic $1 | tail -1; }
docker stop kafka-3
d ha2
To see a crash instead of a clean stop, repeat with docker kill and poll the describe output once a second with timestamps.
docker kill, the once-a-second loop and the timestamps are not run in the labs. unverified Module 04 only uses docker stop.
What to measure
Seconds from the stop or kill until the describe line shows a new leader and a smaller in-sync set. Compare stop with kill.
What you may honestly claim
You may claim the two timings on your laptop for one topic. You may not claim failover time for a loaded production cluster.
Template
I measured leader change time after docker stop and docker kill on a laptop with [setup]; I do not know how it behaves on real hardware; to check I would repeat it under production-like load.
Recipe 5. Disk and page-cache pressure
The gap
Module 09 says a cluster is fast while consumers read recent data from the operating system cache, and slow when reads go to disk. You have not seen that.
What to run
Advice only. Not run. Fill a topic with the performance tool from module 09, then read the old data from the start while producing new data.
Not run. Running the cold read at the same time as the producer is not in the labs. unverified Docker may run inside a virtual machine whose cache differs from the host. unverified
What to measure
Produce latency from the performance tool output with and without the cold read. Disk use from kafka-log-dirs.sh (module 09).
What you may honestly claim
If you run it, you may claim the change in produce latency on your laptop. Until then, claim nothing beyond what module 09 explains.
Template
I [would measure] produce latency during a cold read of an old topic on a laptop with [setup]; I do not know how it behaves on real hardware; to check I would repeat it on brokers with the production disk and memory.
Recipe 6. Noisy neighbours
The gap
Case 2 shows one application hurting others. You want to see a quota work.
A short test shows no throttle. Module 09 reports that its first two attempts, with 6 MB and 20 MB of data, showed no slowing. A quota limits a steady rate, not a burst. Run for a minute or more.
Module 09 ran the commands above. A longer run of one minute or more is not run in the labs. unverified
What to measure
MB each second and average latency for the limited and the unlimited client.
What you may honestly claim
You may claim what module 09 measured: the limited client averaged 21.64 MB each second against 39.03 for the unlimited one, in a short run that was mostly burst. You may not claim a steady-rate result until you run the longer test.
Template
I measured throughput for a quota-limited and an unlimited client on a laptop with [setup]; I do not know how it behaves on real hardware; to check I would run it for several minutes alongside real consumer load.
Recipe 7. Upgrades
The gap
Staff engineers plan rolling changes. You have not done one.
What to run
Run a rolling restart as module 09 describes: three health checks first, then one broker at a time.
k kafka-topics.sh --bootstrap-server $B --describe --unavailable-partitions
k kafka-topics.sh --bootstrap-server $B --describe --under-min-isr-partitions
k kafka-topics.sh --bootstrap-server $B --describe --under-replicated-partitions
docker stop kafka-1
docker start kafka-1
# wait, then run the three checks again before touching kafka-2
Read the cluster versions before and after with k kafka-features.sh --bootstrap-server $B describe (module 05). Plan to restart the active controller last (module 09).
A version upgrade itself is not run. Only the restart is. The wait time between restarts and the order of the three brokers are not run in the labs. unverified
What to measure
Whether all three checks print nothing after each restart, and how long that takes.
What you may honestly claim
You may claim you practised the restart procedure and the health checks on a laptop. You may not claim you have upgraded a Kafka version.
Template
I measured the time for the health checks to clear after each broker restart on a laptop with [setup]; I do not know how it behaves on real hardware; to check I would run the same procedure on a staging cluster, including a real version change.