Monitoring lab

What it is and why it exists

Kafka hides trouble well. A cluster with one broker down still serves every request. A follower that is falling behind causes no error. A disk that fills in six hours looks fine now. Without measurements you learn about each of these when the second fault arrives and something stops.

Monitoring means three things: metrics that the brokers publish, a system that collects them over time, and alerts that tell a person when a number means trouble. The skill is not in collecting everything. It is in choosing a few signals, knowing what each one means, and deciding which ones are worth waking someone for.

Mental model

Sort every signal into one of three levels. The level decides the response.

Use the tiles at the top of the page. In each situation, find the first tile that turned red and decide its level.

Two sources of numbers

• Inside the broker. Each broker publishes hundreds of metrics through the Java management interface, JMX. These describe what that broker sees and does: its request times, its thread use, the partitions it leads.
• From outside. An exporter connects like a client and asks the cluster about topics, partitions and offsets. It sees the cluster as applications see it.

The two can give different numbers for what sounds like the same thing. In the lab, with one broker stopped, the exporter counted 57 under-replicated partitions for the cluster, and broker 1's own metric said 31. Both were right: a broker counts only the partitions it leads.

How it works internally

Metric names

A broker metric has a domain, a type and a name, written like kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions. Some have more labels, such as the request type. Each has one or more attributes: a simple Value, or for rates and timings a Count, rates over 1, 5 and 15 minutes, and percentiles.

The signals, by their real names

Every name in this table was read from a lab broker with kafka-jmx.sh. The value column shows what a healthy lab cluster returned.

From metrics to alerts

1. A collector, here Prometheus, reads the numbers every few seconds and stores them with a timestamp.
2. A rule is an expression over those numbers, for example "the sum of under-replicated partitions is above 0".
3. A rule has a for time. The condition must hold that long before the alert fires. This filters out a restart that lasts ten seconds.
4. A firing alert carries a severity label, which decides if it pages a person or opens a ticket.
5. When the condition stops, the alert clears by itself.

Lag is measured elsewhere

Consumer lag is not a broker metric. It is the difference between a partition's end offset and a group's committed offset, so something must read both. The command-line tool does this (module 03). An exporter can do it too; the one in this lab could not read groups from Kafka 4.3, so the lab has no lag metric.

Failure scenarios

Hands-on lab

Step 1. Start the cluster with monitoring

cd ~/Desktop/kafka-knowledge-base
docker compose --profile monitoring up -d
k() { docker exec -i kafka-1 /opt/kafka/bin/"$@"; }
B=kafka-1:19092

Prometheus is at http://localhost:9095 and Grafana at http://localhost:3005.

Step 2. Read broker metrics by name

J() { docker exec -i kafka-1 timeout 20 /opt/kafka/bin/kafka-jmx.sh \
  --jmx-url service:jmx:rmi:///jndi/rmi://localhost:9101/jmxrmi \
  --one-time true --report-format properties --object-name "$1"; }

J 'kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions'
J 'kafka.controller:type=KafkaController,name=ActiveControllerCount'
J 'kafka.controller:type=KafkaController,name=OfflinePartitionsCount'
J 'kafka.server:type=KafkaRequestHandlerPool,name=RequestHandlerAvgIdlePercent'
J 'kafka.server:type=raft-metrics'

kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions:Value=0
kafka.controller:type=KafkaController,name=ActiveControllerCount:Value=0
kafka.controller:type=KafkaController,name=OfflinePartitionsCount:Value=0
kafka.server:type=KafkaRequestHandlerPool,name=RequestHandlerAvgIdlePercent:FifteenMinuteRate=0.9778922787520811
kafka.server:type=raft-metrics:current-leader=2
kafka.server:type=raft-metrics:current-epoch=1
kafka.server:type=raft-metrics:current-state=follower

Look at the second line. The active controller count on node 1 is 0, and that is healthy: the quorum's leader is node 2, as the last lines say. Run the same command in kafka-2 and you get 1.

Step 3. Check the collector

curl -s http://localhost:9095/api/v1/targets | python3 -c "import sys,json;[print(t['labels']['job'],t['health']) for t in json.load(sys.stdin)['data']['activeTargets']]"
curl -s http://localhost:9095/api/v1/rules   | python3 -c "import sys,json;[print(r['name'],r['state']) for g in json.load(sys.stdin)['data']['groups'] for r in g['rules']]"

kafka up

KafkaBrokerDown inactive
KafkaUnderReplicatedPartitions inactive
KafkaPartitionBelowMinimum inactive

The exporter publishes these metric names: kafka_brokers, kafka_topic_partitions, kafka_topic_partition_current_offset, kafka_topic_partition_oldest_offset, kafka_topic_partition_leader, kafka_topic_partition_leader_is_preferred, kafka_topic_partition_replicas, kafka_topic_partition_in_sync_replica and kafka_topic_partition_under_replicated_partition.

Step 4. Cause a fault and watch the alerts

k kafka-topics.sh --bootstrap-server $B --create --topic watched --partitions 6 --replication-factor 3
q() { curl -s --data-urlencode "query=$1" http://localhost:9095/api/v1/query; echo; }

q 'kafka_brokers'
docker stop kafka-3
# wait 50 seconds
q 'kafka_brokers'
q 'sum(kafka_topic_partition_under_replicated_partition)'
q 'ALERTS{alertstate="firing"}'
J 'kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions'

before:   kafka_brokers = 3

after:    kafka_brokers = 2
          under-replicated partitions, cluster total = 57
          firing: KafkaBrokerDown, KafkaUnderReplicatedPartitions
          broker 1's own UnderReplicatedPartitions = 31

Two alerts fired within 50 seconds. KafkaPartitionBelowMinimum did not, and that is correct: with one broker down, every partition still has two in-sync replicas. The cluster total, 57, counts internal topics too. Broker 1 reports only the 31 partitions it leads.

Step 5. Repair and watch them clear

docker start kafka-3
# wait 40 seconds
q 'kafka_brokers'
q 'sum(kafka_topic_partition_under_replicated_partition)'
q 'ALERTS{alertstate="firing"}'

kafka_brokers = 3
under-replicated partitions = 0
firing: (nothing)

Step 6. Look in the browser, then stop

Open http://localhost:9095/alerts to see the three rules and their state. In Grafana, the Prometheus data source is already connected: make a panel with the query kafka_brokers and repeat step 4 to watch the line drop.

docker compose --profile monitoring down

Common mistakes and misconceptions

Staff-level questions

Self-test