Module 14 · Depth lab · Kafka 4.3

Tiered storage lab

Module 09 described tiered storage but could not run it, because Apache Kafka does not ship the storage plug-in. This lab installs an open-source plug-in, copies real segments to a folder, and shows local data disappear while every record stays readable.

Try it: where does each segment live?

A model, not a measurement. One segment closes every hour. Move the three sliders and read each segment's state. The letter in each cell repeats its colour.

Now: hour 30

Local retention (hours) 4

Total retention (hours) 24

A: active, local only B: local and remote R: remote only X: deleted

Cell numbers count segments from the oldest. The model assumes a segment is copied the moment it closes and ignores check intervals. A real broker does neither.

offset 0

What it is and why it exists

A Kafka partition keeps its data on the disks of the brokers that hold it. Long retention therefore means large disks, and a broker with a lot of data takes long to replace. Many consumers, though, read only recent data. Old data sits on expensive disk and is rarely touched.

Tiered storage splits retention in two. A broker copies each closed segment to a second place, called remote storage. After a short local retention the broker deletes its own copy. The data stays readable from the remote copy until the total retention ends. Clients do not change.

Module 09 described this from general knowledge and said so. This module runs it. One fact surprised me and is worth knowing before you plan anything: the feature is half in Apache Kafka and half not. Kafka contains the machinery that decides what to copy and when. The code that talks to the remote store is a plug-in, and Apache Kafka does not include one.

The one-sentence version. With a plug-in installed, a topic keeps closed segments on local disk for the local retention, keeps them in remote storage for the total retention, and reads from either place without the client knowing.

What this lab did not test. It used one broker and a folder, not an object store. It measured no read speed, no outage, no cost, no compacted topic and no replication above one copy. Those parts below carry an unverified tag.

offset 1

Mental model

Think of two clocks on every closed segment. Both start when the segment closes. The short clock, local.retention.ms, decides when the local copy goes. The long clock, retention.ms, decides when the data is gone for good.

State	Where the bytes are	When
Active, local only	Local disk only.	The newest segment, still receiving writes. The lab showed it is never copied.
Local and remote	Both.	Closed, copied, and younger than the local retention.
Remote only	Remote storage only.	Older than the local retention, younger than the total retention.
Deleted	Nowhere.	Older than the total retention.

Use the demo above. Set local retention to 4 and total retention to 24. Drag "Now" forward and watch each segment move through the four states from the right to the left.

Two start offsets

A partition now has two different "oldest" offsets. The earliest offset is the oldest record you can still read, from any tier. The earliest-local offset is the oldest record still on local disk. In the lab they were 0 and 30000. Everything between them was served from remote storage.

offset 2

How it works internally

Three parts

The remote log manager runs inside each broker, in Apache Kafka. It finds closed segments, asks the plug-in to copy them, and later deletes expired ones. The broker log names it: [RemoteLogManager=1 partition=...].
The remote storage manager is the plug-in. It writes segments to, and reads them from, the remote store. Its class name is a broker setting. Apache Kafka does not provide one. This lab used the open-source project tiered-storage-for-apache-kafka from Aiven, release v1.1.1.
The metadata topic, __remote_log_metadata, is an internal topic. The lab broker created it on its own. It records which segments exist remotely. unverified: I did not read its contents, so what it stores is from the topic name and module 09's description

The life of one segment, as observed

Producers write. The active segment fills up to segment.bytes and closes. A new active segment starts.
The remote log manager copies the closed segment. In the lab, each segment became three files in the remote folder: the records (.log), .indexes and .rsm-manifest. The names start with the segment's base offset, then a unique id.
The broker logged "Copied 00000000000000019968.log to remote storage" and the same for 24960 and 29952.
After local.retention.ms, the normal retention task removed the local copy. The files first got the ending .deleted, as module 01 describes.
A consumer reading from offset 0 got all 30000 records, although no local file held them.
When the total retention retention.ms passed, the remote files were deleted too.

What the remote folder looked like

/tmp/remote/history-<topicid>/0/
  <baseoffset>-<uuid>.log
  <baseoffset>-<uuid>.indexes
  <baseoffset>-<uuid>.rsm-manifest
  ... 7 segments x 3 files = 21 files, 6.1 MB

The plug-in stores one segment as one data file plus its indexes and a manifest. The topic id in the path is the topic's unique id, not its name.

Where the plug-in lives

The plug-in is a set of Java library files, called jars. The lab folder tiered/plugins/core held 124 jars including core-1.1.1.jar. The folder tiered/plugins/filesystem held 3 files: storage-core-1.1.1.jar, storage-filesystem-1.1.1.jar and commons-io-2.19.0.jar. The compose file mounts both read-only at /tiered, and a broker setting gives the class path. In production you would swap the filesystem backend for one that writes to S3, Google Cloud Storage or Azure. unverified: I did not test any object-store backend, and I did not check which backends this release includes

offset 3

Every relevant config

The lab sets broker settings as environment variables. The Kafka image turns KAFKA_X_Y into x.y, so KAFKA_REMOTE_LOG_STORAGE_SYSTEM_ENABLE becomes remote.log.storage.system.enable. The first table lists what ran. A "what it does" cell with a tag means I took it from the setting's name, not from a test.

Setting	Lab value	Default seen	What it does
`remote.log.storage.system.enable` (broker)	true	false	Broker switch for tiered storage. Without it, a topic cannot use it.
`remote.log.storage.manager.class.name` (broker)	`io.aiven.kafka.tieredstorage.RemoteStorageManager`	not recorded	The plug-in class the broker loads.
`remote.log.storage.manager.class.path` (broker)	`/tiered/core/:/tiered/filesystem/`	not recorded	Where the broker finds the plug-in jars.
`remote.log.metadata.manager.listener.name` (broker)	`PLAINTEXT`	not recorded	The listener the metadata manager connects to. unverified: meaning taken from the name
`rlmm.config.remote.log.metadata.topic.replication.factor` (broker)	1	not recorded	Copies of the internal metadata topic. The lab has one broker, so it needs 1. unverified: reason inferred
`remote.log.manager.task.interval.ms` (broker)	5000	not recorded	How often the remote log manager runs its tasks. unverified: meaning taken from the name The lab lowered it to see results fast.
`log.retention.check.interval.ms` (broker)	5000	300000 (module 01)	How often the broker looks for segments to delete. The lab lowered it so deletions show within seconds.
`rsm.config.chunk.size` (plug-in)	4194304	not recorded	A plug-in setting. unverified: I do not know exactly what the plug-in does with it
`rsm.config.storage.backend.class` (plug-in)	`io.aiven.kafka.tieredstorage.storage.filesystem.FileSystemStorage`	not recorded	Which backend writes the remote files. This one writes to a folder.
`rsm.config.storage.root` (plug-in)	`/tmp/remote`	not recorded	The folder used as "remote". It is on a memory-backed file system in the container, so it vanishes with the container.
`rsm.config.storage.overwrite.enabled` (plug-in)	true	not recorded	A plug-in setting. unverified: meaning taken from the name
`remote.storage.enable` (topic)	true	false	The topic opts in to tiering.
`local.retention.ms` (topic)	10000	−2, meaning the same as total retention	How long a closed segment stays on local disk.
`retention.ms` (topic)	3600000, later 15000	604800000 (module 01)	Total retention. Changing it also deleted the remote data.
`segment.bytes` (topic)	1048576	1073741824 (module 01)	Segment size. 1 MiB is the smallest value Kafka accepts (module 01).
`log.local.retention.ms` (broker)	not set	−2	The broker-wide default for `local.retention.ms`.

Exist, not tested

The topic-creation tool lists these as valid topic settings. I did not use them, so I state no behaviour.

local.retention.bytes
remote.log.copy.disable
remote.log.delete.on.disable

The topic from the lab

--partitions 1 --replication-factor 1
--config remote.storage.enable=true
--config segment.bytes=1048576
--config local.retention.ms=10000
--config retention.ms=3600000

These numbers are for a quick demonstration. Ten seconds of local retention would be absurd in production. unverified: I have no tested guidance for production values

offset 4

Failure scenarios

The feature needs a plug-in that Apache Kafka does not include

You observe: In module 09 the stock lab cluster had the settings but no plug-in, so tiering could not run. Here it ran only after I downloaded a separate project and mounted it into the container.
Cause: The plug-in is outside Apache Kafka. It has its own releases, its own owner and its own bugs.
Fix: Pin the plug-in version. Test it against each Kafka upgrade, as module 12 says for monitoring tools. unverified: I tested one plug-in release with one Kafka release. I did not test any other pair

Shortening total retention deletes the remote data too

You observe: I set retention.ms=15000 with kafka-configs.sh --alter. After 40 seconds the remote folder held 0 files and the earliest offset was 30000.
Cause: Total retention governs both tiers. The remote copy is not a separate archive.
Fix: Treat retention.ms as the setting that destroys data, as before. Change it with the same care as always.

A topic with big segments and little traffic never tiers

What breaks: Local disk does not shrink for a quiet topic, although tiering is on.
Reason: Two facts combine. Only closed segments are copied, shown in the lab. A segment closes only at segment.bytes or segment.ms (module 01, 1 GiB and 7 days by default). A quiet topic stays in one active segment for days.
Fix: Lower segment.ms, as module 01 advises for retention. unverified: I did not run a quiet topic

The remote store is down

Not tested: I did not stop or break the remote folder.
What I expect: Copies stop, so local disk grows, and reads of old data fail while recent data still works. unverified: expectation from the design, not an observation
What to do: Test it before you depend on it: make the store unreachable, then watch disk, copy errors in the broker log, and consumer errors.

Reading cold data is slower than reading local data

Not measured: The read of all 30000 records worked. I took no timing, and the lab "remote" is a local folder, so a timing would mislead anyway.
What to expect: A network call to an object store is slower than a local disk, and many stores charge for requests. unverified: general knowledge, no number from this lab

offset 5

Hands-on lab

This lab was run on 5 October 2026 with apache/kafka:4.3.1 and Aiven's tiered-storage plug-in v1.1.1. It uses its own single node, kafka-tier, from the folder tiered/. The compose file and the plug-in folders are already there. The plug-in is not part of Apache Kafka. Remote storage is a folder, not an object store.

If the plug-in folders are missing, these are the download commands I used. They fetch from the project's GitHub releases page.

cd ~/Desktop/kafka-knowledge-base/tiered
mkdir -p plugins/core plugins/filesystem
curl -sL -o core.tgz https://github.com/Aiven-Open/tiered-storage-for-apache-kafka/releases/download/v1.1.1/core-1.1.1.tgz
curl -sL -o filesystem.tgz https://github.com/Aiven-Open/tiered-storage-for-apache-kafka/releases/download/v1.1.1/filesystem-1.1.1.tgz
tar -xzf core.tgz -C plugins/core --strip-components=1
tar -xzf filesystem.tgz -C plugins/filesystem --strip-components=1

Step 1. Start the node

cd ~/Desktop/kafka-knowledge-base/tiered
docker compose up -d

The broker started with no errors and logged "Kafka Server started". The compose file carries the settings from the config table. Use the bootstrap address from that file in the commands below; I write it as <address>.

Step 2. Create a topic that tiers

kafka-topics.sh --bootstrap-server <address> --create --topic history \
  --partitions 1 --replication-factor 1 \
  --config remote.storage.enable=true \
  --config segment.bytes=1048576 \
  --config local.retention.ms=10000 \
  --config retention.ms=3600000

The internal topic __remote_log_metadata appeared.

Step 3. Write 30000 records

kafka-producer-perf-test.sh --topic history --num-records 30000 \
  --record-size 200 --throughput -1 ...

30000 records, 11.61 MB/s

The "..." stands for the connection options the tool needs. Right after the write, the partition had 7 .log files on local disk.

Step 4. Wait 35 seconds and look at both sides

Local partition directory, the .log files

00000000000000000000.log.deleted
00000000000000004992.log.deleted
00000000000000009984.log.deleted
00000000000000014976.log.deleted
00000000000000019968.log.deleted
00000000000000024960.log.deleted
00000000000000029952.log.deleted
00000000000000030000.log

Remote folder /tmp/remote/history-<topicid>/0/

21 files = 7 segments x 3 files each, 6.1 MB total

Broker log, three of the copy lines

[RemoteLogManager=1 partition=<topicid>:history-0] Copied 00000000000000019968.log to remote storage with segment-id: ...
(the same line for 24960 and 29952)

Read this carefully. All seven closed segments were copied and then deleted locally. The only live local file is the active segment, 00000000000000030000.log. Nothing was copied from it.

Step 5. Compare the offsets

kafka-get-offsets.sh --bootstrap-server <address> --topic history --time earliest
kafka-get-offsets.sh --bootstrap-server <address> --topic history --time earliest-local
kafka-get-offsets.sh --bootstrap-server <address> --topic history --time latest

earliest:        0   (history:0:0)
earliest-local:  30000
latest:          30000

The oldest readable record is 0. The oldest local record is 30000. The data between is remote only.

Step 6. Read everything from the beginning

A console consumer reading history from the beginning returned all 30000 messages. No timing was taken. It did not fail, so reads of the remote range work in this setup.

Step 7. Shorten total retention and watch the remote folder

kafka-configs.sh --bootstrap-server <address> --alter --entity-type topics \
  --entity-name history --add-config retention.ms=15000
# wait 40 seconds

remote folder: 0 files
earliest offset: 30000

The remote data was gone and the earliest offset moved up to 30000. Total retention applied to the remote tier.

Step 8. Clean up

docker compose down

The remote folder is memory-backed, so it disappears with the container.

offset 6

Common mistakes and misconceptions

People think	What is true
Tiered storage ships with Kafka.	Kafka has the machinery. The plug-in that writes to remote storage is a separate project.
Setting the topic option is enough.	The broker switch and the plug-in settings must be in place first. The topic option alone does nothing useful.
Tiered storage is a backup.	Total retention deleted the remote data in the lab. A deleted topic is deleted there too (module 09).
The active segment is copied too.	Only closed segments were copied. The active file stayed local.
Local retention applies before the copy.	The lab copied first. Local files were deleted after the local retention.
There is one start offset.	There are two: earliest and earliest-local. They were 0 and 30000.
`retention.ms` now means local time.	It stays the total. `local.retention.ms` is the local part.
Tiering removes the need for replication.	Recent data is only on broker disks, so it still needs copies on several brokers. unverified: the lab used one copy and tested no broker loss
A folder test proves object-store behaviour.	It proves the Kafka side only. Latency, errors and cost need a real store.

offset 7

Staff-level questions

When does tiered storage pay off, and what does reading cold data cost?

It pays off when retention is long compared with how far back most consumers read. Then most local disk holds data nobody touches. Moving it out lets you keep fewer or smaller disks. It does not pay off when retention is short, or when consumers often replay old data.

The cost is on cold reads. I measured no timing and no price. Expect a slower read than local disk, and in a real object store, a charge per request. unverified: expectation, not measured Ask for the share of reads that go beyond the local window. If it is large, tiering moves cost rather than removing it.

What happens if the remote store is down?

I did not test it, and I will not pretend to. The questions to answer by experiment are: does the broker keep accepting writes, does local disk fill because copies stop, do reads of recent data still work, and how long until the broker recovers after the store returns?

The design suggests recent data keeps working while old reads fail, and local disk grows until the store returns. unverified: reasoning, not a test The staff answer is to size local disk for a remote outage of a stated length and to alert on copy failures.

How does tiering change replica rebuild and rebalancing?

A new or replaced replica must catch up. With tiering, the data older than the local window is already remote, so a replica may need to copy only the local part. That makes rebuilds and rebalances shorter and lighter on the network. unverified: I ran one broker, so no replica was ever rebuilt

The saving depends on how much of the data is local. With a local retention near the total, there is no saving.

What do retention.ms and local.retention.ms mean now?

retention.ms is the total lifetime of data across both tiers. local.retention.ms is how long a closed segment stays on local disk after it closes. The default of −2 means "same as total", so local data is never cut short until you set it.

The lab confirmed both. Local files went away after the local retention, with the data still readable. Shortening retention.ms emptied the remote folder and moved the earliest offset to 30000. The segment rule from module 01 still applies: whole segments only.

What is the risk of the plug-in dependency?

The plug-in is not part of Apache Kafka. Its owner, release schedule and supported Kafka versions are separate from yours. A Kafka upgrade can break it, and a plug-in bug sits in your data path. If the project stops, you must adopt another plug-in or move the data off it.

Reduce the risk by pinning versions, testing every Kafka upgrade with the plug-in, and keeping a tested exit plan. unverified: I checked one plug-in release only. I did not review its compatibility policy or support terms

If old data is remote, do I still need replication, and is this a backup?

Replication still matters. Recent data lives only on broker disks until its segment closes and is copied, and the active segment is never copied. A broker loss needs copies on other brokers for that data. unverified: the lab used one copy, so I did not see replicas and tiering together

It is not a backup. The remote copy follows the same retention and the same topic lifetime. The lab showed that shortening retention deleted it. A backup must survive a wrong setting, a deleted topic and a bad deploy.

offset 8

Self-test

1. Which part of tiered storage is not in Apache Kafka?

The plug-in that writes to and reads from remote storage.

2. Which segments does the broker copy?

Closed segments only. The active segment stayed local in the lab.

3. What does local.retention.ms control?

How long a closed segment stays on local disk. The default −2 means the same as total retention.

4. What does retention.ms control when tiering is on?

The total lifetime across both tiers. In the lab, shortening it emptied the remote folder.

5. What were earliest and earliest-local in the lab?

0 and 30000. Offsets 0 to 29999 were remote only.

6. What is the smallest segment.bytes you can set?

1048576, which is 1 MiB (module 01).

7. Name two things this lab did not test.

Any of: a real object store, read speed, a remote outage, compacted topics, more than one copy, cost.

8. Why is tiered storage not a backup?

The remote data follows the same retention and topic lifetime. Shortening retention deleted it.