Module 01 · Day 1 · Kafka 4.3

The log

Everything in Kafka is built on one data structure: a file that only grows at the end. Learn this module well and the other nine become much easier.

A partition. Each new record goes to the end and gets the next number, its offset. Nothing is edited in place. Records written so far: 0

offset 0

What it is and why it exists

Kafka stores records. A record is one message: a key, a value, a timestamp, and optional headers. Records with the same purpose go into a topic, which is a named stream such as orders.

A topic is split into partitions. Each partition is a log: an ordered list of records that only grows at the end. Kafka gives every record in a partition a number, the offset. The first record is offset 0, the next is offset 1, and so on. An offset never changes and is never reused.

This design exists because it is simple and fast. Writing to the end of a file is the cheapest thing a disk can do. Reading is also cheap: a reader only needs to remember one number, the next offset it wants. Many readers can read the same data at different speeds, and a reader can go back and read old data again. The data stays until a retention rule removes it. Reading a record does not delete it.

The one-sentence version. A topic is a set of append-only files; producers add to the end, consumers remember how far they have read, and the broker deletes old data by a time or size rule.

offset 1

Mental model

Think of a topic as several notebooks, one for each partition. You can only write on the next empty line. Each line has a line number. Readers keep a bookmark. When a notebook is old enough, Kafka tears out whole pages from the front.

Three facts follow from this picture.

Order exists only inside one partition. There is no order between two partitions.
The key decides the partition. Records with the same key go to the same partition, so they stay in order.
Offsets belong to a partition. Offset 5 in partition 0 and offset 5 in partition 2 are different records.

Try it: the key picks the partition

Send records with different keys. The same key always lands in the same lane. This demo uses murmur2, the hash function of the Java client. I checked it against three values from Kafka's own tests.

Key

Press "Send record".

A record with no key has no hash. The producer then fills one batch for one partition and moves to another partition for the next batch. Module 02 covers this "sticky" behaviour.

What a partition looks like on disk

A broker is one Kafka server. On a broker, each partition is a directory. The directory name is the topic name, a dash, and the partition number. Inside, the log is cut into files called segments. Only the newest segment, the active segment, receives writes.

/tmp/kraft-combined-logs/                ← log.dirs: where this broker keeps data
└── orders-0/                            ← topic "orders", partition 0
    ├── 00000000000000000000.log         ← segment 1: records with offsets 0 … 999
    ├── 00000000000000000000.index       ← offset → byte position in the .log file
    ├── 00000000000000000000.timeindex   ← timestamp → offset
    ├── 00000000000000001000.log         ← segment 2: offsets 1000 … 1999
    ├── 00000000000000001000.index
    ├── 00000000000000001000.timeindex
    ├── 00000000000000002000.log         ← active segment: the only file being written
    ├── 00000000000000002000.index
    ├── 00000000000000002000.timeindex
    ├── 00000000000000002000.snapshot    ← producer state saved at the roll (module 02)
    ├── leader-epoch-checkpoint          ← used by replication (module 04)
    └── partition.metadata               ← the topic's unique id

The file name is the base offset: the offset of the first record in that segment, written with 20 digits. The offsets 1000 and 2000 are examples; real segments roll by size or age, not by record count.

offset 2

How it works internally

A write

The producer groups records into a record batch and sends the batch to the broker that leads the partition.
The broker checks the batch (size, checksum) and gives it the next offsets.
The broker appends the batch to the end of the active segment. The bytes go to the operating system's memory first (the page cache). The operating system writes them to disk later.
Each time about 4 KiB of data has been added (index.interval.bytes), the broker adds one entry to the .index file and one to the .timeindex file.
When the active segment is full or old enough, the broker closes it and starts a new one. This is a roll.

Kafka does not force each write to disk by default. The defaults for flush.messages and flush.ms are the largest possible number, which means "let the operating system decide". Kafka gets its safety from copies on other brokers, which is module 04.

Why it is fast

Sequential writes. The disk head, or the solid-state drive controller, never jumps around.
Page cache. Recent data is served from memory without Kafka managing its own cache.
Batches. One network request and one disk write carry many records. A batch is stored exactly as the producer sent it, including its compression.
Zero-copy. For connections without encryption, the operating system sends file bytes straight to the network socket. The bytes do not pass through the Kafka process.

A read: how the broker finds an offset

A consumer asks for "partition 0, from offset 1740". The broker must find the byte position of that record without reading the whole log. It does this in three steps. Run the demo to see each one.

Try it: find a record by offset

Offset to find (0 to 2599)

Pick the segment. Choose the segment with the largest base offset that is not greater than the target.
Search the index. The .index file is sparse: it has an entry for only some offsets. Binary search finds the nearest entry at or before the target.
Scan the log file. Start at that byte position and read forward until the target offset.

segments

.index entries

Press "Find".

The index entries in this demo are 125 offsets apart to keep the picture readable. In a real segment the gap depends on record size, because the broker adds an entry about every 4 KiB of data.

The .timeindex file works the same way for time. It answers "what is the first offset at or after 14:00 yesterday?", which is how a consumer rewinds to a point in time.

The offsets you must know

Name	Meaning
Log start offset	The oldest offset that still exists. It moves forward when retention deletes a segment.
Log end offset	The offset the next record will get. It is one more than the newest record.
High watermark	The newest offset that is safely copied to enough brokers. Consumers can read only up to here. Module 04 explains it.
Committed offset	A consumer group's saved bookmark: the next offset it will read after a restart. Module 03 explains it.
Lag	Log end offset minus committed offset. It is the number of records the consumer has not processed yet.

Retention: deleting old data

With cleanup.policy=delete, the default, a background task checks each partition every 5 minutes (log.retention.check.interval.ms). It removes segments that break a rule:

Time: the newest record in the segment is older than retention.ms (default 7 days).
Size: the partition is larger than retention.bytes (default −1, which means no limit). The oldest segments go first.

The important detail: Kafka deletes whole segments, never part of one. A record stays until every record in its segment has expired. So data usually lives longer than retention.ms. Move the slider to see how much longer.

Try it: retention works on whole segments

Retention is 3 days. Records arrive at a steady rate. Change how often segments roll and watch the age of the oldest record that is still on disk.

A segment rolls every

Now: day 12

segments

active segment closed, still on disk deleted

When a segment is deleted, the broker first renames its files with a .deleted ending and removes them one minute later (file.delete.delay.ms). The log start offset moves forward at once.

The active segment is not exempt. If every record in it has expired, the broker starts a new segment and deletes the old one. Step 7 of the lab shows this on the real cluster. An active segment survives only because new records keep arriving in it.

Compaction: keeping the latest value for each key

Some topics hold state, not events. An example is "the current address of each user". For these, age is the wrong rule. You want to keep the newest record for each key for ever and drop the older ones. This is cleanup.policy=compact.

A background thread, the log cleaner, does the work:

It looks only at closed segments. It never touches the active segment.
It reads the part not yet cleaned and builds a map: key → newest offset.
It copies the segments into new files and leaves out every record that has a newer record with the same key.
It swaps the new files in. The offsets of the remaining records do not change, so the log now has gaps.

A record with a key and a null value is a tombstone. It means "delete this key". The cleaner keeps a tombstone for delete.retention.ms (default 1 day) so that consumers have time to see it, and then removes it too.

Try it: run the log cleaner

Each cell is offset key=value. The value ∅ is a tombstone.

closed segments

active segment

Six records in closed segments, two in the active segment.

A topic can use both rules: cleanup.policy=compact,delete. Kafka then keeps the latest value for each key, and also drops anything older than the retention time.

offset 3

Every relevant config

These are topic settings. Each has a broker setting that supplies the default for topics that do not set their own. I checked every default below against the Kafka 4.3 documentation, except where marked.

Topic config	Default	What it does	When to change it
`cleanup.policy`	`delete`	Chooses the rule for old data: `delete`, `compact`, or both.	Use `compact` for topics that hold the latest state for each key.
`retention.ms`	604800000 (7 days)	A closed segment can be deleted when its newest record is older than this. −1 means keep for ever.	Lower it to save disk. Raise it when consumers must be able to replay further back.
`retention.bytes`	−1 (no limit)	Maximum size of one partition before the oldest segments are deleted.	Set it when disk space, not time, is the hard limit. Multiply by the partition count to get the topic total.
`segment.bytes`	1073741824 (1 GiB)	The active segment rolls when it reaches this size.	Lower it for small topics so that retention and compaction can act sooner.
`segment.ms`	604800000 (7 days)	The active segment rolls when it is this old, even if it is not full.	Lower it on low-volume topics. This is the usual fix when data outlives retention or compaction never runs.
`segment.jitter.ms`	0	Adds a random delay to time-based rolls.	Raise it when many partitions would roll at the same moment and cause a burst of disk work.
`segment.index.bytes`	10485760 (10 MiB)	Maximum size of the offset index file. A full index also forces a roll.	Rarely.
`index.interval.bytes`	4096 (4 KiB)	How much data is written between two index entries.	Rarely. A smaller value gives a larger index and a shorter scan.
`min.cleanable.dirty.ratio`	0.5	Compaction starts when at least this share of the log is not yet cleaned.	Lower it to compact sooner, at the cost of more disk work.
`delete.retention.ms`	86400000 (1 day)	How long a tombstone stays after cleaning.	Raise it if a consumer can be offline for more than a day and must still see deletes.
`min.compaction.lag.ms`	0	A record cannot be compacted until it is at least this old.	Raise it when consumers need to see every change for some time.
`max.compaction.lag.ms`	9223372036854775807	A record becomes eligible for compaction after this time, whatever the dirty ratio.	Set it when a rule says old values must be removed within a fixed time.
`file.delete.delay.ms`	60000 (1 minute)	Wait between marking a file deleted and removing it.	Rarely.
`max.message.bytes`	1048588	Largest record batch the topic accepts, measured after compression.	Raise it for large records, and raise the matching producer and consumer limits too.
`message.timestamp.type`	`CreateTime`	Whether the record time is set by the producer (`CreateTime`) or by the broker (`LogAppendTime`).	Use `LogAppendTime` when you cannot trust producer clocks.
`flush.messages`, `flush.ms`	9223372036854775807	Force a write to disk after this many records or this much time.	Almost never. Use replication for safety.
`remote.storage.enable`, `local.retention.ms`	`false`, −2	Tiered storage: move old segments to cheaper remote storage.	Module 09.

Broker settings for this module

Broker config	Default	What it does
`log.dirs`	none; falls back to `log.dir`	Directories where the broker stores partitions. The lab cluster sets `/tmp/kraft-combined-logs`.
`log.retention.hours`	168	Broker default for retention, in hours. `log.retention.ms` overrides it when set.
`log.retention.check.interval.ms`	300000 (5 minutes)	How often the broker checks for segments to delete.
`log.roll.hours`	168	Broker default for the time-based roll.
`log.cleaner.threads`	1	Number of compaction threads.
`auto.create.topics.enable`	`true`	Creates a topic when a client first uses an unknown name. Turn it off in production.
`num.partitions`	1	Partition count for topics created automatically.

offset 4

Failure scenarios

Data stays long after the retention time

What breaks: A topic that gets a few records each hour has retention.ms of 1 day, but still holds records that are a week old.
You observe: One large segment file. The log start offset does not move.
Cause: The segment rolls only every 7 days or 1 GiB, and retention acts on whole closed segments.
Fix: Set segment.ms to a value near the retention time, for example 1 hour to 1 day.

The broker disk fills

What breaks: The broker stops when a data directory has no free space.
You observe: Disk use grows steadily; errors about failed log directories in the broker log.
Cause: Retention is by time only, and traffic grew. retention.bytes is unlimited by default.
Fix: Set retention.bytes as a safety limit and alert on disk use. Size the disk for peak traffic multiplied by retention time and by the number of copies.

A consumer falls behind retention

What breaks: A consumer was offline for longer than the retention time. Its saved offset now points to deleted data.
You observe: An "offset out of range" error. The consumer then jumps according to auto.offset.reset: to the newest data (records are skipped silently) or to the oldest (many records are processed again).
Fix: Alert when lag, measured in time, comes near the retention time. Choose auto.offset.reset on purpose for each consumer.

A compacted topic never shrinks

What breaks: Old values for a key stay for ever; a new consumer reads millions of outdated records.
You observe: The topic grows. A read from the start shows many records for one key.
Cause: The data sits in the active segment, which the cleaner skips. Or the dirty ratio is not reached. Or the cleaner thread stopped after an error.
Fix: Lower segment.ms. Set max.compaction.lag.ms. Look in the broker's log-cleaner.log file for errors.

A record is too large

What breaks: The producer gets a "record too large" error and the record is not written.
Cause: The batch is larger than max.message.bytes, about 1 MiB.
Fix: Prefer to store the large object elsewhere and send a reference. If you must raise the limit, raise it on the topic, the producer, and the consumers together.

Order is lost after adding partitions

What breaks: Events for one customer arrive out of order.
Cause: The partition is the hash of the key divided by the partition count, taking the remainder. A new count sends the same key to a different partition. Old records stay where they were.
Fix: Choose the partition count with room to grow. If you must change it, create a new topic and migrate. You cannot reduce the partition count at all.

A wrong clock blocks retention

What breaks: A producer with a clock set in the future writes a record. Its segment does not look old until that future time has passed, so retention keeps it.
You observe: One old segment stays while newer segments are deleted.
Limit: Kafka 4.3 rejects a record whose time is more than 1 hour ahead of the broker (message.timestamp.after.max.ms, default 3600000). The problem returns if someone raises that limit.
Fix: Keep the default limit, or use LogAppendTime so that the broker sets the time.

offset 5

Hands-on lab

Every command below was run on 4 October 2026 against this folder's cluster (image apache/kafka:4.3.1). The outputs are the real ones, shortened where marked. Broker numbers, ids and timestamps will differ on your run.

Step 1. Start the cluster

Run this in the knowledge base folder, where docker-compose.yml is.

docker compose up -d
docker compose ps

You should see kafka-1, kafka-2, kafka-3 and kafka-ui running. The web console is at http://localhost:8085. To type less, define a short name for running a Kafka tool inside the first broker:

k() { docker exec -it kafka-1 /opt/kafka/bin/"$@"; }

Step 2. Create a topic and look at it

k kafka-topics.sh --bootstrap-server kafka-1:19092 \
  --create --topic orders --partitions 3 --replication-factor 3

k kafka-topics.sh --bootstrap-server kafka-1:19092 --describe --topic orders

Expected output

Topic: orders   TopicId: …   PartitionCount: 3   ReplicationFactor: 3   Configs: min.insync.replicas=2
    Topic: orders   Partition: 0   Leader: 2   Replicas: 2,3,1   Isr: 2,3,1   Elr:   LastKnownElr:
    Topic: orders   Partition: 1   Leader: 3   Replicas: 3,1,2   Isr: 3,1,2   Elr:   LastKnownElr:
    Topic: orders   Partition: 2   Leader: 1   Replicas: 1,2,3   Isr: 1,2,3   Elr:   LastKnownElr:

Leader, Replicas, Isr and Elr describe the copies of each partition. Module 04 explains them.

Step 3. Write records with keys

Type each line and press Enter. The text before the colon is the key. Press Ctrl+D to finish.

k kafka-console-producer.sh --bootstrap-server kafka-1:19092 --topic orders \
  --reader-property parse.key=true --reader-property key.separator=:

user-17:created
user-42:created
user-1:created
user-42:paid
user-42:shipped

Step 4. Read them back with partition and offset

k kafka-console-consumer.sh --bootstrap-server kafka-1:19092 --topic orders \
  --from-beginning --formatter-property print.key=true \
  --formatter-property print.partition=true --formatter-property print.offset=true

Output (the order between partitions can differ)

Partition:1   Offset:0   user-42   created
Partition:1   Offset:1   user-42   paid
Partition:1   Offset:2   user-42   shipped
Partition:0   Offset:0   user-17   created
Partition:2   Offset:0   user-1    created

Check two things. All user-42 records are in one partition and in the order you typed them. Each partition starts its own offsets at 0. The partitions match the demo above: I computed them with the same hash. Press Ctrl+C to stop the consumer.

Many older guides use --property for both tools. Kafka 4.3 still accepts it but prints a warning: the producer now wants --reader-property and the consumer wants --formatter-property. The consumer also prints one line about the newer rebalance protocol, which module 03 covers.

Step 5. Look at the files

docker exec kafka-1 ls -l /tmp/kraft-combined-logs/orders-1

k kafka-dump-log.sh --print-data-log \
  --files /tmp/kraft-combined-logs/orders-1/00000000000000000000.log

Output (shortened)

-rw-r--r-- 1 appuser appuser 10485760 … 00000000000000000000.index
-rw-r--r-- 1 appuser appuser      121 … 00000000000000000000.log
-rw-r--r-- 1 appuser appuser 10485756 … 00000000000000000000.timeindex
-rw-r--r-- 1 appuser appuser        8 … leader-epoch-checkpoint
-rw-r--r-- 1 appuser appuser       43 … partition.metadata

baseOffset: 0 lastOffset: 2 count: 3 baseSequence: 0 lastSequence: 2 producerId: 0 … position: 0 CreateTime: 1791154194544 size: 121 magic: 2 compresscodec: none crc: 3354041928 isvalid: true
| offset: 0 CreateTime: 1791154194543 keySize: 7 valueSize: 7 sequence: 0 headerKeys: [] key: user-42 payload: created
| offset: 1 CreateTime: 1791154194543 keySize: 7 valueSize: 4 sequence: 1 headerKeys: [] key: user-42 payload: paid
| offset: 2 CreateTime: 1791154194544 keySize: 7 valueSize: 7 sequence: 2 headerKeys: [] key: user-42 payload: shipped

Each baseOffset line is one record batch. position is the byte position in the file, the value the index points to. Here the three user-42 records arrived together, so they are one batch of 121 bytes. If you type slowly, you get three batches.

The index files look like 10 MiB each. The broker reserves the full segment.index.bytes for the active segment and cuts the file to its real size when the segment closes.

Step 6. Make segments roll

Create a topic with the smallest segment size that Kafka 4 allows, 1 MiB, and write about 4 MiB into it.

k kafka-topics.sh --bootstrap-server kafka-1:19092 --create --topic rolls \
  --partitions 1 --replication-factor 3 --config segment.bytes=1048576

k kafka-producer-perf-test.sh --topic rolls --num-records 20000 --record-size 200 \
  --throughput -1 --command-property bootstrap.servers=kafka-1:19092

docker exec kafka-1 ls /tmp/kraft-combined-logs/rolls-0

Output: five segments, named by base offset (index files left out)

20000 records sent, 31897.9 records/sec (6.08 MB/sec), 250.74 ms avg latency, 406.00 ms max latency …

00000000000000000000.log
00000000000000004992.log      00000000000000004992.snapshot
00000000000000009984.log      00000000000000009984.snapshot
00000000000000014976.log      00000000000000014976.snapshot
00000000000000019968.log      00000000000000019968.snapshot

Each segment holds 4,992 records, about 1 MiB. The .snapshot files save producer state at each roll; module 02 explains why. A smaller segment is not possible: segment.bytes=1000 fails with "Value must be at least 1048576".

Step 7. Watch retention delete whole segments

k kafka-configs.sh --bootstrap-server kafka-1:19092 --alter \
  --entity-type topics --entity-name rolls --add-config retention.ms=60000

# wait up to 6 minutes: 1 minute of retention plus the 5-minute check
docker exec kafka-1 ls /tmp/kraft-combined-logs/rolls-0
k kafka-get-offsets.sh --bootstrap-server kafka-1:19092 --topic rolls --time earliest

Output, about 4 minutes after the config change

00000000000000000000.log.deleted
00000000000000004992.log.deleted
…
00000000000000019968.log.deleted
00000000000000020000.log
rolls:0:20000

Whole files went, not single records. The earliest offset moved from 0 to 20000. The files carry the .deleted ending for one minute before they disappear.

Look at the last segment. The active segment, base offset 19968, had also expired. The broker started a new empty segment at offset 20000 and deleted the old one. So an active segment is protected only while it still receives records that are newer than the retention time.

Step 8. Watch compaction

k kafka-topics.sh --bootstrap-server kafka-1:19092 --create --topic addresses \
  --partitions 1 --replication-factor 3 \
  --config cleanup.policy=compact --config segment.ms=10000 \
  --config min.cleanable.dirty.ratio=0.01 --config delete.retention.ms=1000

k kafka-console-producer.sh --bootstrap-server kafka-1:19092 --topic addresses \
  --reader-property parse.key=true --reader-property key.separator=:

user-1:Paris
user-2:Rome
user-1:Berlin
user-3:Oslo
user-3:Lima

Wait 30 seconds. Then send one more record, user-4:Cairo, so that the first segment closes. Wait about one more minute and read from the start:

k kafka-console-consumer.sh --bootstrap-server kafka-1:19092 --topic addresses \
  --from-beginning --formatter-property print.key=true --formatter-property print.offset=true

Output: the offsets have gaps

Offset:1   user-2   Rome
Offset:2   user-1   Berlin
Offset:4   user-3   Lima
Offset:5   user-4   Cairo

If you still see all six records, wait longer. The cleaner checks for work every 15 seconds (log.cleaner.backoff.ms) and gives no exact time.

Step 9. Stop the cluster

docker compose down      # keep the data
docker compose down -v   # delete the data too

offset 6

Common mistakes and misconceptions

People think	What is true
A record is removed when a consumer reads it.	Reading changes nothing in the log. Only retention or compaction removes data.
A topic keeps records in order.	Only one partition does. A topic with several partitions has no total order.
Retention of 7 days means data is deleted at 7 days.	It is deleted some time after 7 days, when its whole segment has expired and the next check runs.
A compacted topic holds one record for each key.	It holds at least the latest record for each key. The active segment and the part not yet cleaned still hold older ones.
Offsets in a partition have no gaps.	Compaction leaves gaps, and so do transactions (module 06). Never compute a record count from two offsets and trust it exactly.
More partitions are always better.	Each partition costs files, memory, and recovery time. And you cannot reduce the count later.
A record without a key is fine on a compacted topic.	Compaction needs a key. The broker rejects records without one on a compacted topic.
Kafka writes every record to disk before it answers.	By default it writes to memory and lets the operating system write to disk. Copies on other brokers give the safety.

offset 7

Staff-level questions

Try to answer each one aloud before you open it.

How much disk does a topic need?

Start with: write rate × retention time × number of copies. For 5 MB each second, 7 days, and 3 copies: 5 MB × 604,800 seconds × 3 ≈ 9 TB.

Then add what the simple formula misses. Add one segment for each partition, because retention overshoots by up to a segment. Add index files, a small percentage. Add free space, often 30 to 40 percent, so that a broker can take over partitions from a failed one. Use the compressed size if producers compress.

How do you choose the partition count?

Find the number you need for throughput: target rate divided by what one partition handles on the slowest side, which is usually the consumer. Then find the number you need for parallel consumers, because one partition is read by at most one consumer in a group. Take the larger number and add room for growth, because changing the count later breaks key ordering.

Then check the cost: more files, more memory for indexes, slower recovery after a broker failure, and more small batches at the producer. A count in the tens is normal. A count in the thousands for one topic needs a reason.

A team wants "keep data for ever". What do you tell them?

Ask what they will do with it. If they need the current state for each key, use compaction: size is bounded by the number of keys. If they need the full history, use retention.ms=-1 with tiered storage so that old segments move to cheap storage (module 09).

Point out the costs: a new replica must copy everything, and a consumer that reads from the start takes a long time. Often a database or a data lake is the right home for history, with Kafka as the transport.

A privacy rule says a user's data must be gone in 30 days. Can a Kafka topic meet it?

On a delete topic, set retention plus one segment period below 30 days, for example retention.ms of 25 days and segment.ms of 1 day.

On a compacted topic, write a tombstone for the user's key and set max.compaction.lag.ms so that the cleaner must act in time. The tombstone itself goes after delete.retention.ms. Remember the copies outside the topic: consumer databases, backups, and remote storage.

Another design avoids the problem: encrypt each user's data with a key for that user, and delete the key.

Why does Kafka trust the page cache, and when is that a risk?

Forcing each write to disk would remove most of the speed. Kafka accepts that one broker can lose its newest data in a power failure, and protects the data by copying it to other brokers before it confirms the write.

The risk is a failure that hits all copies at once, such as one power source for all brokers. The answer is to spread brokers across racks or zones, not to force writes to disk.

What do small segments cost?

Each segment is three open files and two memory-mapped indexes. Ten thousand partitions with one-hour segments and seven days of data is more than five million files. The broker can reach the operating system's limit for open files, and start-up and recovery become slow.

So small segments are right for a few low-volume or compacted topics, and wrong as a default for the whole cluster.

When is a log the wrong tool?

When you need to find a record by something other than its offset, a database is better. When each message must be handled once by any free worker, with separate retries for each message, a queue fits better; Kafka's share groups (module 10) cover part of this. When messages are very large, store the object elsewhere. When the volume is small and you already have a database, a table may be enough.

offset 8

Self-test

Ten questions. Answer first, then open.

1. What does an offset identify?

The position of one record in one partition. It is not unique across partitions or topics.

2. Two records have the same key. What does Kafka guarantee?

They go to the same partition, so consumers see them in the order they were written. This holds while the partition count stays the same.

3. What does the name 00000000000000001000.log tell you?

The first record in this segment has offset 1000.

4. Name three things that make the active segment roll.

It reaches segment.bytes. It reaches the age segment.ms. Its index file is full.

5. How does the broker find offset 1740 quickly?

It picks the segment by base offset, binary-searches the sparse .index for the nearest earlier entry, and scans forward in the .log file from that byte position.

6. retention.ms is 1 day and segment.ms is 7 days on a quiet topic. About how old can a record get?

About 8 days. The segment closes at 7 days. Its newest record must then be 1 day old before the segment can go.

7. What is a tombstone and how long does it live?

A record with a key and a null value. It tells compaction to delete the key. It stays for delete.retention.ms after cleaning, 1 day by default.

8. Why can a compacted topic still hold two values for one key?

The cleaner skips the active segment and runs only when enough of the log is not yet cleaned. Until it runs, older values remain.

9. What is lag?

The log end offset minus the consumer group's committed offset: the records not yet processed.

10. You raise a topic from 3 to 6 partitions. What breaks?

Keys map to different partitions from now on. New records for a key can land in a different partition from its old records, so order for that key is no longer guaranteed across the change.