77 terms · each with the module that explains it
Glossary
Every term the ten modules define, in one place. The number on the right is the module that explains the term in full.
| Term | Meaning | Module |
|---|---|---|
| acks | The producer setting that says how many brokers must have a record before the leader answers: 0, 1 or all. | 02 |
| Active controller | The one controller that is currently the leader of the quorum and makes all metadata decisions. | 05 |
| Active segment | The newest segment of a partition, and the only one being written. | 01 |
| At-least-once | A delivery guarantee: no record is lost, but a record can be processed more than once. Process first, then commit. | 03 |
| At-most-once | A delivery guarantee: no record is processed twice, but a record can be lost. Commit first, then process. | 03 |
| Batch | A group of records for one partition that the producer sends, and the broker stores, as one unit. | 02 |
| Broker | One Kafka server that stores partitions and serves clients. | 01 |
| Change data capture | Reading a database's own log of changes and publishing each change as a record. | 08 |
| Changelog topic | A compacted internal topic that holds every change to one Kafka Streams state store. | 07 |
| Co-partitioning | Two topics with the same partition count and the same partitioning by key, as a join needs. | 07 |
| Combined mode | One process that is both broker and controller. For development only. | 05 |
| Committed offset | A consumer group's saved position for a partition: the next offset it will read. | 03 |
| Committed record | A record that every in-sync replica has. Only committed records are shown to consumers. | 04 |
| Compaction | A cleanup policy that keeps at least the newest record for each key and removes older ones. | 01 |
| Connector | A ready-made component for Kafka Connect that copies data from a system into Kafka (source) or out of it (sink). | 08 |
| Consumer | The client library that reads records. | 03 |
| Consumer group | A set of consumers with the same group id that share the partitions of a topic, one owner for each partition. | 03 |
| Controller | A server that votes in the metadata quorum and stores the metadata log. | 05 |
| Converter | The part of Kafka Connect that turns records into bytes and back. | 08 |
| Coordinator | A broker that manages one consumer group, one transactional id, or one share group. | 03 |
| Dead-letter topic | A topic that receives records that could not be processed, with the reason. | 08 |
| Eligible leader replicas | Replicas outside the in-sync set that are known to hold every committed record and can be elected safely. Shown as Elr. | 04 |
| Epoch | A counter that goes up at each change of leader or producer session, so that messages from an old one can be refused. | 04 |
| Exactly-once | For a read-process-write application in Kafka: each input affects the outputs and the committed offset exactly one time. | 06 |
| Fenced | Refused by the cluster. A fenced broker leads nothing; a fenced producer cannot write. | 05 |
| Follower | A replica that copies the leader. | 04 |
| High watermark | The lowest log end offset among the in-sync replicas. Records below it are committed. | 04 |
| Idempotence | Producer and broker count records with sequence numbers so that a retry cannot create a duplicate. | 02 |
| In-sync replicas | The leader plus the followers that have caught up with it within the allowed time. Written ISR. | 04 |
| Isolation level | A consumer setting: read_uncommitted shows all records; read_committed hides records of open and aborted transactions. | 06 |
| KRaft | Kafka's own way to store cluster metadata, with a quorum of controllers and the Raft method. It replaced ZooKeeper in Kafka 4.0. | 05 |
| Lag | The log end offset minus a group's committed offset: the records not yet processed. | 03 |
| Last stable offset | The offset of the first record of the oldest open transaction. Committed readers read only below it. | 06 |
| Leader | The replica of a partition that accepts writes and normally serves reads. | 04 |
| Leader epoch | The number of the current leadership of one partition. It rises by one at each leader change. | 04 |
| Listener | A network port of a broker with its own security settings. | 09 |
| Log | An ordered list of records that only grows at the end. Each partition is one log. | 01 |
| Log end offset | The offset that the next record of a partition will get. | 01 |
| Log start offset | The oldest offset of a partition that still exists. | 01 |
| Marker | A control record that says a transaction was committed or aborted. It uses one offset. | 06 |
| Metadata log | The single-partition log, __cluster_metadata, that holds every change to the cluster's metadata. | 05 |
| min.insync.replicas | The smallest in-sync set for which the leader accepts writes with acks=all. | 04 |
| MirrorMaker 2 | Kafka Connect connectors that copy topics, and translate consumer offsets, from one cluster to another. | 10 |
| Offset | The position of a record in its partition. It never changes and is never reused. | 01 |
| Outbox pattern | Writing an event to a table in the same database transaction as the business change, and publishing it from there. | 08 |
| Page cache | The operating system's memory for file data. Kafka relies on it to serve recent records. | 09 |
| Partition | One of the logs that a topic is split into. | 01 |
| Preferred leader | The first broker in a partition's replica list. | 04 |
| Producer | The client library that writes records. | 02 |
| Quorum | The voting group of controllers. A majority must run for metadata to change. | 05 |
| Quota | A limit on what one client or user may send, fetch or use of broker time. | 09 |
| Rebalance | The moment partitions change owner inside a consumer group. | 03 |
| Record | One message: a key, a value, a timestamp and optional headers. | 01 |
| Recovery point objective | How much recent data may be lost in a disaster, as a time. | 10 |
| Recovery time objective | How long the service may be down in a disaster. | 10 |
| Repartition topic | An internal Kafka Streams topic that regroups records by a new key. | 07 |
| Replica | One copy of a partition on one broker. | 04 |
| Replication factor | The number of copies of each partition. | 04 |
| Retention | The rule that removes old data, by time or by size, in whole segments. | 01 |
| Schema registry | A separate service that stores the agreed shape of each topic's records and checks changes for compatibility. | 08 |
| Segment | One of the files that a partition's log is cut into. | 01 |
| Share group | A group in which several members can read the same partition and acknowledge each record alone. Queue behaviour. | 10 |
| Snapshot | The full metadata state at one offset, so that older metadata records can be deleted. | 05 |
| Standby replica | A warm copy of a Kafka Streams state store on another instance. | 07 |
| State store | Local storage, by default RocksDB, that a Kafka Streams step uses to remember things. | 07 |
| Static membership | A consumer with a fixed group.instance.id, so that a quick restart causes no rebalance. | 03 |
| Stretch cluster | One cluster whose brokers and controllers are placed in several nearby sites. | 10 |
| Task | In Kafka Streams: processing for one sub-topology and one partition. In Kafka Connect: one unit of a connector's copying work. | 07 |
| Tiered storage | Copying closed segments to cheaper remote storage and keeping only recent data on broker disks. | 09 |
| Tombstone | A record with a key and a null value. In a compacted topic it means "delete this key". | 01 |
| Topic | A named stream of records, split into partitions. | 01 |
| Transaction | Writes to several partitions, and offset commits, that take effect together or not at all. | 06 |
| Transactional id | A stable name for a producer that lets Kafka recognise it after a restart and fence the old instance. | 06 |
| Truncation | A replica cutting records from the end of its log to match the leader. | 04 |
| Unclean leader election | Electing a replica that was not in sync. It restores availability and loses records. | 04 |
| Window | A piece of a stream cut by time: tumbling, hopping or session. | 07 |
| Zero-copy | Sending file bytes straight to the network without passing them through the Kafka process. | 01 |