Operations
Running Kafka well is mostly arithmetic and discipline: size the cluster from real numbers, watch a short list of signals, and change one thing at a time. This module gives you the numbers, the signals and the procedures.
Try it: size a cluster
Enter the workload. The result is a first estimate to test, not a final answer.
What it is and why it exists
The earlier modules explained how Kafka works. Operations is the work of keeping it working: enough capacity, safe changes, quick detection of problems, and protection from clients that misbehave.
Kafka is forgiving in one way and strict in another. It keeps serving through a broker failure, a slow disk or a restart, which hides problems until two happen together. And several decisions cannot be undone: a partition count cannot go down, and deleted data does not come back. So operations is about seeing a problem while the cluster still has spare protection, and about thinking before irreversible changes.
This module covers seven areas: sizing, partition counts, the signals to watch, tiered storage, quotas, security, and upgrades.
The one-sentence version. Size from write rate, retention and copies; keep spare capacity for one broker failure; alert on lost redundancy before lost availability; and roll every change through the cluster one broker at a time.
Mental model
A broker has four resources. Each has a typical way to run out.
| Resource | Used for | How it runs out |
|---|---|---|
| Disk space | Write rate × retention × copies. | Traffic grows, or retention is raised, and nobody recalculates. |
| Disk and network throughput | Client writes, follower copies, consumer reads. | A consumer reads old data from disk, or a broker failure moves load to the others. |
| Memory | Mostly the operating system's page cache, which serves recent data. | Readers fall behind what the cache holds, and reads go to disk. |
| Processor | Request handling, encryption, compression work. | Many small requests, encryption without hardware support, or the broker re-compressing batches. |
The page cache explains a common surprise. A cluster is fast while every consumer reads recent data, because that data is still in memory. One consumer that starts from the beginning of a large topic forces reads from disk, pushes recent data out of memory, and slows everyone.
Size for the day a broker fails
When one broker of N stops, its leaderships move to the others. Each of them carries N ÷ (N − 1) of its normal load. With three brokers that is 50% more. A cluster that runs at 70% on a normal day has no room for this. So the target on a normal day is well below that.
How many partitions
Try it: choose a partition count
Change one broker at a time
Almost every change, a setting, a software version, a hardware repair, is applied as a rolling restart. Step through the procedure. The check before each restart is the part people skip, and it is the part that prevents outages.
Step through it: a rolling restart of three brokers
How it works internally
Sizing, with the arithmetic shown
- Stored data = write rate × retention time × copies ÷ compression ratio. Use the compressed rate if producers compress.
- Add the overshoot. Retention deletes whole segments (module 01), so add one segment for each partition, and a few percent for index files.
- Add free space. Plan for disks at most about 60% full. The rest absorbs growth, a reassignment, and the extra a broker receives when another one fails.
- Network. Each megabyte written by a client is also sent to each follower. Inbound traffic to the cluster's brokers is write rate × copies. Outbound is the follower traffic plus every consumer group's reads.
- Memory. Give the broker's Java process a modest fixed amount and leave the rest of the machine's memory to the page cache. A useful target is enough cache to hold the last 30 seconds to few minutes of writes. unverified rule of thumb
- Test. Run the performance tools from module 02 with your record sizes on the real hardware. Published numbers from other people's hardware are a starting point only.
The signals to watch
Brokers publish hundreds of metrics. A short list catches most problems. Module 12 reads each of these from a live broker by its exact name.
| Signal | Healthy | What a bad value means |
|---|---|---|
| Offline partitions | 0 | Some data cannot be read or written now. Page someone. |
| Active controller count, summed over controllers | exactly 1 | 0: no metadata changes possible. More than 1 for long: a serious fault. |
| Partitions under the in-sync minimum | 0 | Writes with acks=all are refused for them. |
| Under-replicated partitions | 0 | Redundancy is reduced. Nothing is broken yet. The best early warning. |
| In-sync set shrink and expand rate | near 0 | A broker is at its limit or the network is unstable. |
| Request handler idle share | well above 30% | The broker's request threads are saturated. |
| Produce and fetch request time, 99th percentile | stable | Rising time without rising load points to a slow disk or follower. |
| Disk use for each data directory | below about 60% | Act before 80%. A full disk stops the broker. |
| Consumer lag, in time | within each application's limit | A consumer is too slow or stuck (module 03). |
| Quorum leader epoch | constant | Controller elections are happening (module 05). |
Without a metrics system, three command-line checks cover the top of the list. The lab runs them.
Tiered storage
Keeping months of data on broker disks is costly, and a broker with a lot of local data takes long to replace. With tiered storage, a broker copies closed segments to cheaper remote storage, such as an object store, and deletes its local copy after a short local retention. Consumers that read old offsets are served from the remote copy by the broker; clients do not change.
- The broker switch is
remote.log.storage.system.enable, off by default. A storage plug-in for your object store must be installed. - A topic opts in with
remote.storage.enable=trueand setslocal.retention.msfor the local part.retention.msstays the total. - Only closed segments are copied. The active segment is always local.
- Reads of old data are slower, and have a cost in requests to the object store.
The lab cluster has no remote storage plug-in, so I did not run it in this module. Module 14 runs it with a plug-in and shows segments moving to remote storage and being read back.
Quotas
A quota limits what one client can take, so that one application cannot starve the others. You can limit bytes produced each second, bytes fetched each second, and the share of broker request time. A quota is attached to a user, to a client id, or set as a default.
The broker does not reject requests over quota. It measures the rate over a window, 11 samples of 1 second by default, and when a client is over, it delays its responses and tells the client to wait. So a short burst passes, and a steady overload is slowed to the limit. The lab shows the burst effect.
Security in three layers
| Layer | Question | Mechanism |
|---|---|---|
| Encryption | Can someone on the network read the traffic? | TLS on the listener. |
| Authentication | Who is this client? | TLS client certificates, or SASL with a password, a token, or Kerberos. |
| Authorization | What may this client do? | Access control lists, checked by an authorizer on the broker. |
Each listener is a port with its own security settings. A common layout has one listener for clients with TLS and authentication, one for traffic between brokers, and one for controllers on a private network.
The lab cluster has none of this. Its listeners are PLAINTEXT and its authorizer.class.name is empty, so anyone who can reach a port can read, write and delete everything. That is acceptable only on your own machine. Module 11 builds and tests a secured cluster.
Changing settings
Topic settings and many broker settings can be changed while the cluster runs, with kafka-configs.sh. The change is stored in the metadata log and applies to every broker without a restart. Other broker settings are read only at start and need a rolling restart. The tool's output shows where each value comes from: a default, the broker's file, or a dynamic change.
Every relevant config
All defaults below were read from the lab cluster with kafka-configs.sh --describe --all.
| Broker config | Default | What it does | When to change it |
|---|---|---|---|
num.network.threads | 3 | Threads that read requests from and write answers to the network. | Raise when the network threads' idle share is low, often with TLS. |
num.io.threads | 8 | Threads that process requests, including disk work. | Raise on machines with many cores and disks when the request handler idle share is low. |
background.threads | 10 | Threads for background work such as deleting files. | Rarely. |
num.recovery.threads.per.data.dir | 2 | Threads that check logs at start and flush them at stop. | Raise to shorten restarts on brokers with many partitions. |
queued.max.requests | 500 | Requests that may wait for a processing thread before the network threads stop reading. | Rarely. |
socket.request.max.bytes | 104857600 (100 MiB) | Largest request the broker accepts. | Rarely. |
message.max.bytes | 1048588 | Largest record batch, as a broker default for topics. | Module 01. |
max.connections.per.ip | 2147483647 | Connections allowed from one address. | Lower it to protect against a client that opens connections in a loop. |
connections.max.idle.ms | 600000 (10 minutes) | Idle connections are closed after this time. | Rarely. |
auto.create.topics.enable | true | Creates a topic on first use. | Set false in production. The lab created a topic by accident this way in module 04. |
delete.topic.enable | true | Allows topic deletion. | Protect deletion with access control, not with this switch. |
num.partitions, default.replication.factor | 1, 1 | Defaults for topics created without explicit values. | Set the second to 3. |
compression.type | producer | Keep batches as the producer compressed them. | Leave it. |
quota.window.num, quota.window.size.seconds | 11, 1 | The window over which client rates are measured for quotas. | Rarely. |
producer_byte_rate, consumer_byte_rate, request_percentage | no limit | Quotas, set with kafka-configs.sh for users or client ids. | On every shared cluster. |
remote.log.storage.system.enable | false | Turns on tiered storage on the broker. | With a storage plug-in installed. |
log.local.retention.ms | −2 (same as total retention) | Broker default for how long data stays on local disk with tiered storage. | Hours to a few days, when tiered storage is on. |
listener.security.protocol.map | the lab maps every listener to PLAINTEXT | The security protocol of each listener. | Use SSL or SASL_SSL for anything outside your own machine. |
inter.broker.listener.name | the lab sets PLAINTEXT | The listener that brokers use among themselves. | A dedicated, secured listener. |
authorizer.class.name | empty: no access control | The class that checks access control lists. | Set the standard authorizer in production. |
ssl.client.auth | none | Whether clients must present a certificate. | required for certificate-based authentication. |
controlled.shutdown.enable | true | Hand over leaderships before stopping. | Leave it on. |
log.cleaner.threads | 1 | Compaction threads. It can be changed while running; the lab sets 2. | Raise when compaction falls behind. |
Failure scenarios
A disk fills
- What breaks
- The broker marks the data directory as failed and stops serving its partitions, or stops completely.
- Cause
- Growth without a recalculation; retention raised on a large topic; a compacted topic whose cleaner stopped; a reassignment that copied more than expected.
- Fix
- In an emergency, lower
retention.mson the largest topics and wait for the retention check, up to 5 minutes. Do not delete files by hand. Afterwards, alert at 60% and 75%, and setretention.bytesas a limit on topics that can grow without bound.
A second broker fails during a restart
- What breaks
- With three copies and a minimum of 2, one broker down is fine. Two down stops writes for partitions on both.
- Cause
- The operator restarted the next broker before the first had rejoined every in-sync set, or a broker failed by itself during the maintenance.
- Fix
- Check for zero under-replicated partitions before each restart. Automate the check so that it cannot be skipped.
One reader of old data slows the cluster
- You observe
- Produce latency rises on a broker with no change in write rate. Disk reads jump.
- Cause
- A consumer reads from far back. Its data is not in the page cache, so the broker reads from disk and recent data is pushed out of memory.
- Fix
- Put a fetch quota on bulk readers. Run large replays outside peak hours. With tiered storage, old reads go to remote storage and do not disturb the local disk.
Adding partitions breaks ordering
- What breaks
- Events for one key are processed out of order after a partition increase.
- Evidence
- In the lab, the topic
orderswent from 3 to 6 partitions. The keyuser-42had been in partition 1 and its next record went to partition 4. The keysuser-17anduser-1stayed where they were. - Fix
- Choose the count with growth in mind at creation. For a keyed topic that must grow, create a new topic and migrate. The count can never go down: the tool answers "3 would not be an increase".
Too many partitions
- What breaks
- Broker restarts and failovers take long. The controller is busy. Producers make many tiny batches.
- Cause
- Hundreds of topics created with a large default count "to be safe".
- Fix
- Size each topic. Keep the total for each broker inside what you have tested. Small topics are fine with 1 to 3 partitions.
A topic that nobody protected
- What breaks
- A topic with one copy loses its data when its broker's disk fails.
- Evidence
- The lab's check for partitions under the in-sync minimum found a topic named
no-quorumwith one replica, left over from module 05, on a cluster whose minimum is 2. Nothing can be written to it withacks=all. - Fix
- Review all topics for replication factor and minimum regularly. Turn off automatic topic creation and create topics through a reviewed process.
An open cluster
- What breaks
- Any process on the network can delete topics or read every record.
- Cause
- Plain-text listeners and no authorizer, as in the lab.
- Fix
- TLS, authentication and access control lists before any real data arrives. Adding security later needs a careful migration with two listeners.
Hands-on lab
Every command below was run on 4 October 2026 against the lab cluster, after the labs of modules 01 to 08. Tiered storage and security are not in this lab, for the reasons given above.
k() { docker exec -i kafka-1 /opt/kafka/bin/"$@"; }
B=kafka-1:19092Step 1. Three health checks
Each prints the partitions in that state. No output means healthy.
k kafka-topics.sh --bootstrap-server $B --describe --unavailable-partitions
k kafka-topics.sh --bootstrap-server $B --describe --under-min-isr-partitions
k kafka-topics.sh --bootstrap-server $B --describe --under-replicated-partitions(no unavailable partitions) Topic: no-quorum Partition: 0 Leader: 2 Replicas: 2 Isr: 2 Elr: LastKnownElr: (no under-replicated partitions)
The second check found something real. The topic no-quorum has one replica. Its in-sync set, one broker, is below the cluster minimum of 2. It is not "under-replicated", because it has all the replicas it was given: one. This shows why you run all three checks.
Step 2. Where the disk went
k kafka-log-dirs.sh --bootstrap-server $B --describe --broker-list 1 --topic-list perf,comp-none,ordersperf-0 247910144 perf-2 241752474 perf-1 237756878 comp-none-0 1548818 orders-0 713 orders-2 159
The tool prints one line of JSON; each partition has a size and a lag behind its leader. The three perf partitions hold about 730 MB from the throughput tests. On a real cluster, sort this output to find the topics to look at when a disk fills.
Step 3. Lag for every group
k kafka-consumer-groups.sh --bootstrap-server $B --describe --all-groupsg-classic events 0 … lag 1657 g-classic events 1 … lag 928 g-classic events 2 … lag 1230 …
The group reset in module 03 is still at the start of the topic with nobody running.
Step 4. Add partitions and watch a key move
k kafka-topics.sh --bootstrap-server $B --alter --topic orders --partitions 6
printf 'user-42:after-change\nuser-17:after-change\nuser-1:after-change\n' | \
k kafka-console-producer.sh --bootstrap-server $B --topic orders \
--reader-property parse.key=true --reader-property key.separator=:
k kafka-console-consumer.sh --bootstrap-server $B --topic orders --from-beginning \
--formatter-property print.key=true --formatter-property print.partition=true \
--formatter-property print.offset=true --timeout-ms 6000
k kafka-topics.sh --bootstrap-server $B --alter --topic orders --partitions 3Partition:2 Offset:0 user-1 created Partition:2 Offset:2 user-1 after-change Partition:0 Offset:0 user-17 created Partition:0 Offset:11 user-17 after-change Partition:1 Offset:0 user-42 created Partition:1 Offset:1 user-42 paid Partition:1 Offset:2 user-42 shipped Partition:4 Offset:0 user-42 after-change ← the same key, a different partition Error while executing topic command : The topic orders currently has 6 partition(s); 3 would not be an increase.
The history of user-42 is in partition 1 and its new record is in partition 4. Two consumers now hold parts of one customer's story, and nothing orders them against each other.
Step 5. A quota, and why a short test misleads
k kafka-configs.sh --bootstrap-server $B --alter --add-config 'producer_byte_rate=5242880' \
--entity-type clients --entity-name slow-client
k kafka-producer-perf-test.sh --topic perf --num-records 400000 --record-size 200 --throughput -1 \
--command-property bootstrap.servers=$B client.id=slow-client
k kafka-producer-perf-test.sh --topic perf --num-records 400000 --record-size 200 --throughput -1 \
--command-property bootstrap.servers=$B client.id=fast-client
k kafka-configs.sh --bootstrap-server $B --alter --delete-config producer_byte_rate \
--entity-type clients --entity-name slow-client| Client | Quota | MB each second | Average latency |
|---|---|---|---|
slow-client | 5 MiB each second | 21.64 | 973 ms |
fast-client | none | 39.03 | 422 ms |
The limited client was slowed, but it still averaged four times its quota. The test sent 80 MB in under 4 seconds, and the broker measures over an 11-second window, so most of the test was an allowed burst. My first two attempts, with 6 MB and 20 MB of data, showed no slowing at all. A quota limits a steady rate, not a burst. To see the steady limit, run a test that lasts a minute or more.
Step 6. Change a broker setting without a restart
k kafka-configs.sh --bootstrap-server $B --alter --entity-type brokers --entity-default \
--add-config log.cleaner.threads=2
k kafka-configs.sh --bootstrap-server $B --describe --entity-type brokers --entity-defaultCompleted updating default config for brokers in the cluster.
Default configs for brokers in the cluster are:
log.cleaner.threads=2 synonyms={DYNAMIC_DEFAULT_BROKER_CONFIG:log.cleaner.threads=2}
min.insync.replicas=2 synonyms={DYNAMIC_DEFAULT_BROKER_CONFIG:min.insync.replicas=2}
DYNAMIC_DEFAULT_BROKER_CONFIG means the value was set while running and applies to all brokers.
Step 7. Delete records before an offset
Sometimes data must go before retention removes it, for example after a bad batch. This moves a partition's start offset forward.
docker exec kafka-1 sh -c 'echo "{\"partitions\":[{\"topic\":\"ha2\",\"partition\":0,\"offset\":100}],\"version\":1}" > /tmp/del.json'
k kafka-delete-records.sh --bootstrap-server $B --offset-json-file /tmp/del.json
k kafka-get-offsets.sh --bootstrap-server $B --topic ha2 --time earliestpartition: ha2-0 low_watermark: 100 ha2:0:100
Offsets 0 to 99 are no longer readable. This cannot be undone.
Common mistakes and misconceptions
| People think | What is true |
|---|---|
| Give the broker's Java process most of the memory. | Kafka relies on the operating system's page cache. A large Java heap takes memory away from it and makes pauses longer. |
| Disk at 85% is fine while it works. | One broker failure or one reassignment can fill it. Keep about 40% free. |
| More partitions is the safe default. | Each has a cost, and the count cannot be reduced. |
| Under-replicated partitions will fix themselves. | Often, after a restart. A count above zero that stays is a broker in trouble. |
| A quota rejects requests over the limit. | It delays responses, measured over a window. Bursts pass. |
| We will add security later. | Later means a migration with every client. Start with it. |
| Tiered storage is a backup. | It is the same data in a cheaper place. A deleted topic is deleted there too. |
| A change that worked on one broker is safe to apply to all at once. | Roll it, one broker at a time, with a health check between. |
Staff-level questions
Size a cluster for 100 MB each second, 7 days of retention, 3 copies.
Storage: 100 MB × 604,800 seconds × 3 = about 181 TB without compression. With a 4-times ratio, about 45 TB. With 40% free, provide about 76 TB. With 4 TB for each broker that is 19 brokers by disk alone; with 8 TB, 10 brokers.
Network: clients send 100 MB each second and followers receive 200 more, so about 300 MB each second inbound across the cluster, plus consumer reads outbound. Check each broker's share against its network link, after one broker has failed.
Then say what you would do next: confirm the compression ratio on real data, test one broker's limits on the real hardware, and consider tiered storage, which could cut local disk to a day or two.
One large cluster or several smaller ones?
One cluster is cheaper to run, makes data easy to share, and uses hardware well. But every tenant shares its failures, its upgrades and its limits, and one bad client or topic can hurt all.
Several clusters isolate failure and let teams upgrade at their own pace, at the cost of more operations work and of copying data between clusters. A common result: a few clusters split by criticality or by data sensitivity, not one for each team. Quotas and access control are what make sharing a cluster workable.
When is tiered storage worth it?
When retention is long compared with how far back most consumers read. Then most local disk holds data that is rarely touched. Moving it out cuts disk cost and makes brokers faster to replace and to rebalance, because there is less local data to copy.
It is not worth it for short retention, or when consumers often read old data and need the speed of local disk. Check also how it combines with compacted topics in your version before you plan on it.
How do you add security to a running open cluster with no outage?
Add a second, secured listener beside the open one, with a rolling restart. Move clients to it application by application. Turn on the authorizer in a mode that still allows clients without rules, add access rules for each application, and watch the authorizer's log for requests that would be denied. When nothing would be denied, switch to deny by default. Finally remove the open listener with another rolling restart.
The slow part is finding every client. Use broker request logs and connection metrics by listener.
Which alerts page a person at night, and which wait for the morning?
Night: offline partitions; no active controller; partitions under the in-sync minimum on critical topics; a disk about to fill within hours; consumer lag beyond a business limit on a critical flow.
Morning: under-replicated partitions with one broker down; a rising trend in disk use; frequent in-sync set changes; one consumer group slowly falling behind. These mean reduced protection, not lost service. Paging for them trains people to ignore pages.
A team asks for 500 partitions for a new topic. What do you ask?
Their peak rate and how much one consumer instance processes, which gives the real need. Whether records have keys, since that decides if the count can ever change. How many consumer instances they will actually run. And how many other topics of this size the cluster already has.
Most requests like this come from a guess. A measured answer is often 12 to 48, with room to grow.
Self-test
1. Give the basic formula for stored data.
Write rate × retention time × copies, divided by the compression ratio.
2. Why keep about 40% of the disk free?
For growth, for the retention overshoot, for reassignments, and for the extra load when another broker fails.
3. What memory serves recent data to consumers?
The operating system's page cache, not the broker's Java heap.
4. Which three command-line checks show partition health?
kafka-topics.sh --describe with --unavailable-partitions, --under-min-isr-partitions and --under-replicated-partitions.
5. What must be true before you restart the next broker in a rolling restart?
No partition is under-replicated: the last broker has rejoined every in-sync set.
6. What happens to a key's partition when you add partitions?
It can change, because the partition is the key's hash modulo the count. In the lab user-42 moved from 1 to 4.
7. How does a broker enforce a quota?
It measures the client's rate over a window and delays responses when the client is over the limit.
8. What does tiered storage move, and what stays local?
Closed segments are copied to remote storage and removed locally after the local retention. The active segment stays local.
9. Name the three layers of security.
Encryption, authentication and authorization.
10. How do you change a topic setting on a running cluster?
With kafka-configs.sh --alter. No restart is needed.