Producers
A producer does much more than send a message. It groups records into batches, chooses when to send, waits for a chosen level of confirmation, and tries again after a failure without creating duplicates.
Steps 1 to 4 run in your thread and return at once. Steps 5 and 6 happen later, in the producer's own sender thread. This is why send() returns before the record is safe.
What it is and why it exists
A producer is the client library that writes records to Kafka. It runs inside your application. It finds which broker leads each partition, sends records there, and handles failures.
The producer exists as a smart client because of speed and safety. If each record made its own network request, Kafka would be slow. So the producer collects records into batches. And networks fail, so the producer retries. A retry can write the same record twice, so the producer and the broker count records to detect a repeat. This is idempotence: doing the same thing twice has the same result as doing it once.
Three decisions matter most. How long to wait for a batch to fill. How many brokers must have the record before the producer treats it as written. And what to do when a write fails.
The one-sentence version. The producer trades a few milliseconds of delay for large batches, waits for the level of confirmation you choose, and retries safely because every batch carries a sequence number.
Mental model
Think of a post office counter. You hand over a letter and leave at once. Behind the counter is one bag for each destination. A van leaves when a bag is full, or when the bag has waited long enough. The driver brings back a receipt, and only then do you know the letter arrived.
- The bag is a batch: one for each partition.
- Full is
batch.size, measured in bytes. - Waited long enough is
linger.ms. - The receipt is the acknowledgement, set by
acks.
Use the demo to feel the trade. A slow stream of records with no waiting gives one tiny batch for each record. A little waiting gives large batches and far fewer requests.
Try it: when does a batch leave?
Records are 200 bytes each and all go to one partition. The bar is one batch filling. The animation is slowed down or speeded up so that you can see it.
With linger.ms=0 the real producer still forms batches when it is busy, because records pile up while the sender thread is working. The demo shows the idle case.
What "written" means: acks
Each partition has one leader broker and some follower brokers that copy it. The acks setting says how many of them must have the record before the leader answers.
| acks | The leader answers when | Risk |
|---|---|---|
0 | It does not answer. The producer does not wait. | Records are lost without any error. |
1 | The leader has written the record to its own log. | If the leader fails before a follower copies the record, it is lost. The producer thinks it succeeded. |
all (default) | Every in-sync follower has copied the record. | None, when at least min.insync.replicas brokers are in sync. Module 04 explains "in sync". |
Try it: the leader fails just after a write
How it works internally
The path of one record
- Metadata. On the first send to a topic, the producer asks any broker which broker leads each partition. It keeps the answer and refreshes it every 5 minutes (
metadata.max.age.ms) or after an error. - Serialize. The key serializer and value serializer turn your objects into bytes.
- Partition. With a key: murmur2 hash of the key bytes, remainder after division by the partition count. Without a key: the producer picks one partition and stays on it until about
batch.sizebytes have gone there, then picks another. It prefers brokers that answer quickly. - Accumulate. The record joins the open batch for its partition, in a memory pool of
buffer.memorybytes. If the pool is full,send()waits up tomax.block.msand then throws an error. - Send. The sender thread takes every batch that is full or older than
linger.ms, groups them by broker, and sends one request to each broker. Up to 5 requests can be waiting for an answer on one connection. - Answer. On success, the producer completes the record's future and calls your callback with the partition and offset. On a failure that can be retried, the batch goes back in the queue.
Compression
The producer compresses a whole batch, not single records. Larger batches compress better. The broker stores the batch as it arrives, and the consumer decompresses it. So the processor cost is paid by the clients, and the saving applies to the network, the disk and the copies on other brokers.
Retries and the delivery timeout
The producer retries almost without limit: retries is 2147483647. The real limit is time. delivery.timeout.ms, 2 minutes by default, counts from the moment send() returns. It covers the wait in the batch, every attempt, and every pause between attempts. When it ends, the producer gives up and reports an error for the record.
Some errors are worth a retry: the leader moved, a request timed out, too few brokers are in sync. Others are not, and fail at once: the record is too large, serialization failed, the client is not allowed to write.
Idempotence: how a retry avoids duplicates
A retry is dangerous. Suppose the broker wrote the batch but its answer was lost. The producer sends the batch again, and the log now holds it twice. A second danger is order: batch 1 fails, batch 2 succeeds, then the retry of batch 1 lands after batch 2.
Idempotence, on by default, fixes both:
- At start, the broker gives the producer a producer id.
- The producer numbers the records for each partition: sequence 0, 1, 2, and so on. Each batch carries its first sequence number.
- The broker remembers the last five batches from each producer id for each partition. unverified
- If a batch arrives with a sequence number the broker has already written, the broker answers "success" and writes nothing.
- If a batch arrives with a sequence number that leaves a gap, the broker rejects it. The producer then sends the missing batch first.
Try it: a retry with and without idempotence
The limit of idempotence. It protects against duplicates from the producer's own retries, inside one producer session. It does not help if your application calls send() twice for the same event, or if the application restarts and sends again: a new session gets a new producer id. For those cases you need transactions (module 06) or a consumer that can detect repeats.
Idempotence needs acks=all, at most 5 requests in flight, and retries above 0. If you set a value that conflicts with these and also set enable.idempotence=true, the producer refuses to start.
Every relevant config
All defaults below come from the Kafka 4.3 producer configuration reference.
| Producer config | Default | What it does | When to change it |
|---|---|---|---|
bootstrap.servers | none (required) | A few broker addresses used to find the rest of the cluster. | List two or three brokers so that one failed broker does not block start-up. |
key.serializer, value.serializer | none (required) | Classes that turn keys and values into bytes. | Always set. Module 08 covers schema-based serializers. |
acks | all | How many brokers must have the record before the leader answers. | Use 1 or 0 only for data you can lose, such as metrics. |
enable.idempotence | true | Sequence numbers that stop duplicates and reordering on retry. | Leave it on. |
retries | 2147483647 | Maximum number of attempts. | Do not lower it. Control retries with delivery.timeout.ms. |
delivery.timeout.ms | 120000 (2 minutes) | Total time allowed for one record, from send() to a final answer. | Lower it when a late record has no value. Raise it to ride through a longer broker failure. |
request.timeout.ms | 30000 (30 seconds) | Time to wait for the answer to one request before a retry. | Rarely. |
retry.backoff.ms, retry.backoff.max.ms | 100, 1000 | Pause before a retry. It grows from the first value up to the second. | Rarely. |
max.in.flight.requests.per.connection | 5 | Requests sent to one broker that have no answer yet. | Keep at 5 or lower. Above 5, idempotence cannot be used. |
batch.size | 16384 | Upper size of one batch in bytes. A full batch is sent at once. | Raise to 65536 or 131072 for high throughput. |
linger.ms | 5 | How long a batch waits for more records. It was 0 before Kafka 4.0. unverified | Raise to 20 to 100 for throughput. Set 0 only when each millisecond counts. |
compression.type | none | Compression for each batch: gzip, snappy, lz4 or zstd. | Turn on lz4 or zstd for text or JSON. The lab shows a 10-times saving. |
compression.gzip.level, compression.lz4.level, compression.zstd.level | −1, 9, 3 | Effort level for each algorithm. | Raise for a better ratio at more processor cost. |
buffer.memory | 33554432 (32 MiB) | Memory for records that wait to be sent. | Raise when you have many partitions or large batches. |
max.block.ms | 60000 (1 minute) | How long send() can wait for memory or for metadata. | Lower it when your thread must not be held, for example in a request handler. |
max.request.size | 1048576 | Largest request, and so the largest record, the producer will send. | Raise together with the topic's max.message.bytes. |
partitioner.class | null | Your own partitioner. Null means the built-in behaviour. | Only when the key hash is not enough, for example to isolate one large customer. |
partitioner.ignore.keys | false | When true, keys are not used to pick the partition. | When you need keys for compaction but want an even spread. |
partitioner.adaptive.partitioning.enable | true | Records without a key go more often to brokers that answer quickly. | Rarely. |
partitioner.availability.timeout.ms | 0 | When above 0, a broker that does not accept requests for this long gets no records without keys. | Set it when one slow broker hurts everyone. |
transactional.id, transaction.timeout.ms | null, 60000 | Turn on transactions. | Module 06. |
client.id | empty | A name that appears in broker logs, metrics and quotas. | Always set it. It makes problems much easier to trace. |
metadata.max.age.ms | 300000 (5 minutes) | How often the producer refreshes its view of the cluster. | Lower it if you add partitions and need producers to see them sooner. |
Two tested starting points
| Goal | Settings |
|---|---|
| Safety first (payments, orders) | Defaults: acks=all, idempotence on. On the topic: 3 copies and min.insync.replicas=2. |
| Throughput (logs, clickstream) | batch.size=131072, linger.ms=20, compression.type=lz4, and keep acks=all. |
Failure scenarios
The result of send() is ignored
- What breaks
- Records are lost and nobody knows. This is the most common producer bug.
- Cause
send()returns a future. An error arrives later, in the future or the callback. Code that reads neither never sees it.- Fix
- Always pass a callback that logs and counts errors, or wait for the future.
Too few brokers are in sync
- What breaks
- Every write with
acks=allfails. - You observe
NOT_ENOUGH_REPLICASin the producer log, repeated until the delivery timeout. The lab shows it.- Cause
- Fewer brokers are in sync than
min.insync.replicas. The broker refuses the write because it cannot keep the promise. - Fix
- Bring the broker back. Do not lower the minimum to 1 as a quick fix: that removes the protection you set it for.
The delivery timeout expires
- You observe
- An error like "Expiring 12 record(s) for orders-3: 120000 ms has passed since batch creation".
- Cause
- The leader was unreachable or too slow for 2 minutes. Or the producer creates records faster than the broker accepts them.
- Fix
- Find why the broker is slow before you raise the timeout. Decide what your application does with a record that failed: retry later, write to a fallback store, or stop.
The producer's memory is full
- What breaks
send()stops returning at once. Aftermax.block.msit throws a timeout error.- Cause
- Your code produces faster than the sender thread can send, so
buffer.memoryfills. - Fix
- Treat it as back-pressure: slow the source. Larger batches and compression raise the send rate. More memory only delays the problem.
acks=1 loses a confirmed record
- What breaks
- The producer received success, but the record is not in the topic.
- Cause
- The leader answered and then failed before any follower copied the record. The new leader never had it.
- Fix
- Use
acks=allfor data you cannot lose.
One partition is much busier than the others
- You observe
- One partition grows fast and its consumer falls behind, while others are idle.
- Cause
- Key skew. One key, or a null-like value such as "unknown", carries a large share of the records.
- Fix
- Choose a key with more distinct values, or add a second part to the key. For one very large customer, use a custom partitioner that spreads that customer over several partitions, if you can accept less ordering for it.
Duplicates after an application restart
- What breaks
- The same event is in the topic twice, with idempotence on.
- Cause
- The application crashed after
send()but before it recorded that the event was sent. After the restart it sends again with a new producer id. - Fix
- Use the outbox pattern (module 08) or transactions (module 06), or make the consumer ignore repeats by an event id.
Hands-on lab
Every command below was run on 4 October 2026 against the lab cluster. The numbers are from one laptop where all three brokers share one disk. Your numbers will differ; the ratios are what matter.
k() { docker exec -i kafka-1 /opt/kafka/bin/"$@"; }
B=kafka-1:19092
k kafka-topics.sh --bootstrap-server $B --create --topic perf --partitions 3 --replication-factor 3Step 1. Measure what batching does
The performance tool sends 300,000 records of 200 bytes. Give it producer settings after --command-property, separated by spaces.
# one record in each batch
k kafka-producer-perf-test.sh --topic perf --num-records 300000 --record-size 200 \
--throughput -1 --command-property bootstrap.servers=$B batch.size=1 linger.ms=0
# defaults
k kafka-producer-perf-test.sh --topic perf --num-records 300000 --record-size 200 \
--throughput -1 --command-property bootstrap.servers=$B
# larger batches
k kafka-producer-perf-test.sh --topic perf --num-records 300000 --record-size 200 \
--throughput -1 --command-property bootstrap.servers=$B batch.size=131072 linger.ms=20| Settings | Records each second | MB each second | Average latency |
|---|---|---|---|
batch.size=1, linger.ms=0 | 13,244 | 2.53 | 7,046 ms |
| Defaults (16384, 5) | 400,000 | 76.29 | 70 ms |
batch.size=131072, linger.ms=20 | 710,900 | 135.59 | 2.3 ms |
The same, with compression.type=lz4 | 545,455 | 104.04 | 2.1 ms |
The same, with acks=1 | 656,455 | 125.21 | 1.5 ms |
Batching is the large effect: 54 times more throughput from the first row to the third. Note also that latency went down as batches grew, because the tool sends as fast as it can and small batches make a long queue.
Do not read much into the last two rows. On one laptop the network is not a limit, so compression only adds processor work, and the test data is random bytes that do not compress. acks=1 was not faster than acks=all here. On a real network the picture changes.
Step 2. See compression on disk
Send the same 20,000 JSON lines to two topics, one without compression and one with zstd. The console producer has its own flag for compression.
gen() { awk 'BEGIN{for(i=0;i<20000;i++) printf "{\"orderId\":%d,\"status\":\"created\",\"currency\":\"CAD\",\"country\":\"CA\"}\n", i}'; }
for c in none zstd; do
k kafka-topics.sh --bootstrap-server $B --create --topic comp-$c --partitions 1 --replication-factor 3
gen | k kafka-console-producer.sh --bootstrap-server $B --topic comp-$c --compression-codec $c \
--command-property linger.ms=50 --command-property batch.size=65536
done
docker exec kafka-1 sh -c 'ls -l /tmp/kraft-combined-logs/comp-*-0/*.log'1548818 comp-none-0/00000000000000000000.log 157335 comp-zstd-0/00000000000000000000.log
The compressed log is about one tenth of the size. The dump tool shows the codec on each batch:
baseOffset: 0 lastOffset: 821 count: 822 … size: 3416 magic: 2 compresscodec: zstd …
822 records in 3,416 bytes: about 4 bytes for each record.
Step 3. See idempotence in the log
k kafka-topics.sh --bootstrap-server $B --create --topic idem --partitions 1 --replication-factor 3
echo a | k kafka-console-producer.sh --bootstrap-server $B --topic idem --command-property enable.idempotence=false
echo b | k kafka-console-producer.sh --bootstrap-server $B --topic idem
k kafka-dump-log.sh --files /tmp/kraft-combined-logs/idem-0/00000000000000000000.log | grep baseOffsetbaseOffset: 0 lastOffset: 0 count: 1 baseSequence: -1 lastSequence: -1 producerId: -1 producerEpoch: -1 baseOffset: 1 lastOffset: 1 count: 1 baseSequence: 0 lastSequence: 0 producerId: 16 producerEpoch: 0
The first batch has no producer id and no sequence number: the broker cannot detect a repeat of it. The second has producer id 16 and sequence 0. Run the second command again and you get a new producer id, which is the limit described above.
Step 4. Records without a key stay together
k kafka-topics.sh --bootstrap-server $B --create --topic nokey --partitions 3 --replication-factor 3
seq 1 9 | k kafka-console-producer.sh --bootstrap-server $B --topic nokey
k kafka-get-offsets.sh --bootstrap-server $B --topic nokeynokey:0:0 nokey:1:9 nokey:2:0
All nine records went to partition 1. They were not spread one by one. The producer stays on one partition until about batch.size bytes have gone there.
Step 5. Break the in-sync minimum
Create a topic that needs all three brokers in sync, then stop one broker.
k kafka-topics.sh --bootstrap-server $B --create --topic strict --partitions 1 \
--replication-factor 3 --config min.insync.replicas=3
docker stop kafka-3
# wait 12 seconds
k kafka-topics.sh --bootstrap-server $B --describe --topic strict
echo hello | k kafka-console-producer.sh --bootstrap-server $B --topic strict \
--command-property delivery.timeout.ms=6000 --command-property request.timeout.ms=3000 \
--command-property linger.ms=0Topic: strict Partition: 0 Leader: 1 Replicas: 3,1,2 Isr: 1,2 Elr: 3 LastKnownElr: WARN Got error produce response … on topic-partition strict-0, retrying (2 attempts left). Error: NOT_ENOUGH_REPLICAS WARN Got error produce response … on topic-partition strict-0, retrying (1 attempts left). Error: NOT_ENOUGH_REPLICAS WARN Got error produce response … on topic-partition strict-0, retrying (0 attempts left). Error: NOT_ENOUGH_REPLICAS ERROR Error when sending message to topic strict with key: null, value: 5 bytes with error: …
Only brokers 1 and 2 are in sync, the topic needs 3, so the leader refuses the write. Now send with acks=1:
echo hello | k kafka-console-producer.sh --bootstrap-server $B --topic strict \
--request-required-acks 1 --command-property enable.idempotence=false
k kafka-get-offsets.sh --bootstrap-server $B --topic strict
docker start kafka-3
# wait 12 seconds
k kafka-get-offsets.sh --bootstrap-server $B --topic strictstrict:0:0 ← while broker 3 is stopped strict:0:1 ← after broker 3 returns
This result surprised me, so read it with care. The producer got no error: the leader accepted the write and put it in its log. But the end offset that clients see did not move while fewer brokers were in sync than the minimum. Consumers could not read the record until broker 3 came back. So acks=1 below the minimum is "accepted but not visible". Module 04 explains the mechanism, the high watermark.
Common mistakes and misconceptions
| People think | What is true |
|---|---|
send() returned, so the record is written. | It is only in the producer's memory. It is written when the future completes without an error. |
| Idempotence gives exactly-once delivery. | It removes duplicates from the producer's own retries in one session. Exactly-once from end to end needs transactions and care at the consumer. |
linger.ms adds that delay to every record. | It is the longest wait. A full batch leaves at once, and under load batches fill before the time ends. |
batch.size is a number of records. | It is a number of bytes. |
acks=all means all brokers. | It means all brokers that are in sync at that moment. That can be one broker, unless min.insync.replicas says otherwise. |
| Records without a key are spread one by one over partitions. | They go in groups: the producer stays on a partition for about one batch. |
Lower retries to avoid duplicates. | Idempotence removes the duplicates. Fewer retries only lose more records. |
| One producer for each thread is safer. | The producer is safe to share between threads, and one shared producer makes better batches. |
Staff-level questions
Product wants "no lost orders and no duplicate orders". What do you build?
For loss: acks=all, three copies, min.insync.replicas=2, and code that handles the send error. For duplicates from retries: idempotence, which is on by default.
Then name the gap. The order is first saved in a database and then sent to Kafka. A crash between the two steps loses or repeats the event. Close the gap with the outbox pattern: write the event to a table in the same database transaction, and let a separate process copy the table to Kafka. Consumers still get the event at least once, so give each event an id and make consumers ignore repeats.
How do you tune a producer for latency below 10 milliseconds?
Set linger.ms=0 and keep batches small enough that they do not wait. Keep acks=all if the data matters, and make replication fast instead: brokers close to each other, fast disks. Turn off compression or use lz4. Measure the 99th percentile, not the average, and watch for pauses in the client's runtime.
Then ask if 10 milliseconds is really needed from end to end, because the consumer's fetch settings add delay too.
A broker is slow but not dead. What happens to producers?
Batches for the partitions it leads wait longer. They fill buffer.memory, and then send() blocks for every partition, also the healthy ones. One slow broker can stall the whole producer.
Defences: for records without keys, the adaptive partitioner sends less to the slow broker, and partitioner.availability.timeout.ms can exclude it. For records with keys there is no escape, so keep max.block.ms low enough that your application fails fast, and alert on the producer's buffer and request latency metrics.
Why can max.in.flight be 5 and order still be safe?
The broker keeps the sequence numbers of the last five batches for each producer and partition. If batch 1 fails while batches 2 to 5 are in flight, the broker rejects 2 to 5 because they leave a gap, and the producer resends all of them in order. More than five could not be checked, which is why the limit is 5.
What is a good key?
The thing whose events must stay in order: an order id, an account id, a device id. It should have many distinct values, with no single value that dominates. It should be stable: the same entity always gives the same key.
Check the two failure cases. A key with few values, such as a country, gives hot partitions. A random key gives a perfect spread and no useful order.
Where should compression happen?
In the producer. It then saves network, broker disk, replication traffic and consumer network at once. The topic setting compression.type=producer, the default, tells the broker to keep the batch as it arrived.
If the topic names a different codec, the broker must decompress and compress again, which costs broker processor time. Avoid that.
Self-test
1. When is a batch sent?
When it reaches batch.size bytes, or when it has waited linger.ms, whichever comes first.
2. What is the default for linger.ms in Kafka 4?
5 milliseconds. Before 4.0 it was 0.
3. What does acks=all wait for?
All brokers that are in sync for that partition at that moment have copied the record.
4. What happens to a write with acks=all when fewer brokers are in sync than min.insync.replicas?
The leader refuses it with NOT_ENOUGH_REPLICAS. The producer retries until the delivery timeout.
5. Which setting really limits retries?
delivery.timeout.ms, 2 minutes by default.
6. Which three values identify a batch for duplicate detection?
The producer id, the producer epoch, and the sequence number for that partition.
7. Name one duplicate that idempotence does not prevent.
The application sends the same event again after a restart, or calls send() twice. A new session has a new producer id.
8. Where do records without a key go?
To one partition for about one batch, then to another. The producer prefers brokers that answer quickly.
9. send() blocks and then throws a timeout. Name two causes.
The memory pool buffer.memory is full, or the producer cannot get metadata for the topic. Both wait up to max.block.ms.
10. Why do larger batches compress better?
Compression works on the whole batch, and more records give more repeated content to remove.