Module 11 · Depth lab · Kafka 4.3

Security lab

Module 09 described security and ran none of it. This lab builds a locked cluster and tests every lock: encryption, user names and passwords, and permissions for each user. Every result on this page comes from a real run.

Try it: how far does each client get?

A request must pass three gates. Pick a client and see where it stops. The messages are the real ones from the lab.

Client

1 EncryptionTLS: is the channel private, and is this really the broker?

2 AuthenticationSASL: who is this client?

3 AuthorizationAccess control: may this user do this?

DoneThe request is served.

offset 0

What it is and why it exists

A cluster with default settings accepts anyone who can reach a port. That person can read every record, write to every topic, and delete every topic. The main lab cluster in this knowledge base is like that, on purpose, so that the first ten modules stay simple.

Securing a cluster means adding three things, each of which answers a different question.

Encryption with TLS. Nobody on the network can read or change the traffic, and the client can check that it is talking to the real broker.
Authentication with SASL or client certificates. The broker knows which user each connection belongs to. A user is called a principal, written like User:alice.
Authorization with access control lists. For each principal, rules say which operations are allowed on which topics, groups and other resources.

They depend on each other. Permissions mean nothing if anyone can claim to be alice. A password means little if it crosses the network in clear text.

The one-sentence version. TLS makes the channel private and proves the broker, SASL proves the client, and access control lists decide what that client may do, with everything denied unless a rule allows it.

offset 1

Mental model

Think of an office building. The corridor to the door is private, and the sign on the door is checked: that is TLS. At the door you show a badge: that is authentication. Inside, your badge opens some rooms and not others: that is authorization. A list of who entered which room is the audit log.

Listeners

A broker can listen on several ports, and each one, a listener, has its own security protocol. This is how a cluster offers a secured port to applications while brokers and controllers talk on their own ports.

Protocol	Encrypted	Client proves who it is
`PLAINTEXT`	No	No. Every client is `User:ANONYMOUS`.
`SSL`	Yes	Only if client certificates are required.
`SASL_PLAINTEXT`	No	Yes, but the exchange is not protected.
`SASL_SSL`	Yes	Yes. The usual choice for clients.

Ways to authenticate

Mechanism	How it works	Use it when
SASL PLAIN	A user name and password, sent inside TLS.	Labs, or with a password store behind it. This lab uses it.
SASL SCRAM	A password that is never sent; both sides prove they know it.	Passwords managed in Kafka itself.
SASL OAUTHBEARER	A signed token from an identity service.	Companies with a central identity provider.
Client certificates	The client presents its own certificate in TLS.	Service-to-service identity with an existing certificate system.

This lab ran SASL PLAIN only. The other three rows are described from general knowledge. unverified

What a permission looks like

One rule says: this principal is allowed (or denied) this operation on this resource. A real rule from the lab:

(principal=User:alice, host=*, operation=WRITE, permissionType=ALLOW)
on ResourcePattern(resourceType=TOPIC, name=payments-sec, patternType=LITERAL)

With allow.everyone.if.no.acl.found=false, anything without a matching allow rule is denied. A super user skips all rules.

offset 2

How it works internally

The TLS handshake

The broker has a private key and a certificate that names it, here CN=kafka-sec with the host names kafka-sec and localhost.
The certificate is signed by a certificate authority, here a lab authority named "Kafka Lab CA".
The client has a truststore: the list of authorities it believes. In the lab it holds exactly one entry, the lab authority.
On connection, the broker sends its certificate. The client checks the signature against its truststore and checks that the host name it dialled is in the certificate.
Both sides agree on keys, and everything after that is encrypted.

The client never needs the broker's private key, and it does not need the broker's certificate in advance: only the authority's certificate. That is why one truststore file works for every broker signed by the same authority.

Authentication, then the principal

Inside the encrypted channel, the client sends its user name and password. The broker checks them against the users it was configured with. From then on, every request on that connection carries the principal User:alice. The client does not send its name with each request and cannot change it.

Authorization on every request

For each request, the broker asks the authorizer: may this principal perform this operation on this resource? The standard authorizer stores the rules in the cluster metadata log (module 05), so every broker has the same rules. Each denial is written to a log file. A real line from the lab:

Principal = User:alice is Denied operation = CREATE from host = 172.23.0.2
on resource = Topic:LITERAL:alice-topic for request = CreateTopics … based on rule DefaultDeny

What a simple action really needs

One client action needs several permissions. The tool's shortcuts add them together:

Shortcut	Rules it created in the lab
`--producer --topic T`	`WRITE`, `DESCRIBE` and `CREATE` on topic T
`--consumer --topic T --group G`	`READ` and `DESCRIBE` on topic T, and `READ` on group G

A consumer needs permission on its group as well as on the topic. In the lab, bob could read the topic with the group billing and was refused with any other group name.

Brokers and controllers are clients too

A broker connects to other brokers and to the controllers. Those connections must also authenticate. In the lab, the broker uses the user admin for both, and the controller listener requires a password as well. A common mistake is to secure the client port and leave the controller port open.

offset 3

Every relevant config

The "Lab" column is what secure/docker-compose.yml sets. The cluster started and passed every test with exactly these values.

Broker config	Lab	What it does
`listeners`	`SECURE://:9094, CONTROLLER://:9093`	The ports and their names.
`listener.security.protocol.map`	`SECURE:SASL_SSL, CONTROLLER:SASL_PLAINTEXT`	The protocol of each listener. In production, make the controller listener `SASL_SSL` too.
`inter.broker.listener.name`	`SECURE`	Brokers talk to each other on the secured listener.
`ssl.keystore.location`, `ssl.keystore.password`, `ssl.keystore.type`	`broker.p12`, PKCS12	The broker's private key and signed certificate.
`ssl.truststore.location`, `ssl.truststore.password`	`truststore.p12`	The authorities the broker believes, for its own outgoing connections.
`sasl.enabled.mechanisms`	`PLAIN`	Which login mechanisms the broker accepts. The default is `GSSAPI`, which is Kerberos.
`sasl.mechanism.inter.broker.protocol`, `sasl.mechanism.controller.protocol`	`PLAIN`	How the broker logs in to other brokers and to controllers.
`listener.name.secure.plain.sasl.jaas.config`	users admin, alice, bob	For PLAIN: the list of users and passwords, and the broker's own login. In production, passwords do not belong in a configuration file.
`authorizer.class.name`	`org.apache.kafka.metadata.authorizer.StandardAuthorizer`	Turns on access control. The default is empty: no checks.
`super.users`	`User:admin`	Principals that skip all rules. Keep the list very short.
`allow.everyone.if.no.acl.found`	`false`	Deny when no rule matches. `true` is useful only while migrating an open cluster.
`ssl.client.auth`	default `none`	`required` makes clients present a certificate. Not used in this lab.
`auto.create.topics.enable`	`false`	Topics are created only by someone with permission.

Client settings

Client config	Lab value	What it does
`security.protocol`	`SASL_SSL`	Must match the listener. The default is `PLAINTEXT`.
`sasl.mechanism`	`PLAIN`	The login mechanism.
`sasl.jaas.config`	the user name and password	The client's credentials.
`ssl.truststore.location`, `ssl.truststore.password`, `ssl.truststore.type`	`truststore.p12`, PKCS12	The authority that signed the broker's certificate.
`ssl.endpoint.identification.algorithm`	default `https`	Checks the host name against the certificate. Never set it to empty in production. unverified default

offset 4

Failure scenarios

A client without TLS connects to a TLS port

You observe: The client dies with java.lang.OutOfMemoryError: Java heap space. Nothing mentions security.
Cause: The client expects a plain Kafka answer that starts with a length. It receives TLS bytes, reads them as a huge length, and tries to allocate that much memory.
Fix: Set security.protocol on the client. When a client runs out of memory at connection time, check the protocol before the heap size.

The client does not trust the broker

You observe: "failed authentication due to: SSL handshake failed", SslAuthenticationException.
Cause: No truststore, the wrong authority in it, an expired certificate, or a host name that is not in the certificate.
Fix: Give the client the right authority certificate. Check the certificate's dates and names with keytool -list -v.

Wrong user name or password

You observe: "Authentication failed: Invalid username or password".
Note: The broker does not say which of the two was wrong. That is deliberate.

Authenticated but not allowed

You observe: For a topic: "Authorization failed". For a consumer group: GroupAuthorizationException: Not authorized to access group: snoop. Listing topics does not fail: it returns only what the user may see, in the lab an empty list.
Fix: Read the authorizer log to see exactly which operation on which resource was denied, and add that rule. Do not make the user a super user.

A read-only user gets a "cluster" error when it tries to write

You observe: ClusterAuthorizationException: Cluster authorization failed, not a topic error.
Cause: The producer has idempotence on by default. Its first step is to ask for a producer id, and the broker allows that only for a user who may write to at least one topic. Bob may write to none. unverified: the error is from the lab; this explanation is from memory
Fix: The denial is correct. Know the message, so that you do not go looking for a cluster-level problem.

Certificates expire

What breaks: On one day, every client fails the handshake at the same moment. The lab certificates last 365 days.
Fix: Alert weeks before expiry. Practise rotation: brokers can load a new keystore without a restart through a dynamic setting. unverified: rotation was not run in the lab

The controller port was forgotten

What breaks: Clients are locked down, but anyone who reaches the controller listener can act with full power.
Fix: Secure every listener, and put the controller listener on a network that only brokers and controllers can reach.

offset 5

Hands-on lab

Every command below was run on 5 October 2026. The secured cluster is separate from the main lab cluster, in the folder secure/. Its passwords are examples for a lab; they are written in the files and are not secrets.

Step 1. Create certificates and start

cd ~/Desktop/kafka-knowledge-base/secure
sh make-certs.sh          # a lab certificate authority, a broker certificate, a truststore
docker compose up -d
docker logs kafka-sec 2>&1 | grep 'Kafka Server started'

s() { docker exec -i kafka-sec timeout 25 /opt/kafka/bin/"$@"; }
S=kafka-sec:9094

Read make-certs.sh: it is three short steps with keytool. The folder clients/ holds one properties file for each kind of client used below.

Step 2. Look at the certificate

docker exec kafka-sec keytool -list -keystore /etc/kafka/secrets/truststore.p12 -storepass labpass
docker exec kafka-sec keytool -list -v -keystore /etc/kafka/secrets/broker.p12 -storepass labpass -alias broker | grep -E 'Owner|Issuer|DNSName'

ca, Oct 5, 2026, trustedCertEntry

Owner: CN=kafka-sec
Issuer: CN=Kafka Lab CA
  DNSName: kafka-sec
  DNSName: localhost

Step 3. Three clients that are turned away at the door

s kafka-topics.sh --bootstrap-server $S --command-config /clients/plaintext.properties --list
s kafka-topics.sh --bootstrap-server $S --command-config /clients/no-truststore.properties --list
s kafka-topics.sh --bootstrap-server $S --command-config /clients/wrong-password.properties --list

1. ERROR Uncaught exception in thread 'kafka-admin-client-thread' … java.lang.OutOfMemoryError: Java heap space

2. ERROR … failed authentication due to: SSL handshake failed
   org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed

3. ERROR … failed authentication due to: Authentication failed: Invalid username or password

The first is the one to remember: a plain client on a TLS port fails with an out-of-memory error. The timeout in the helper stops these commands when they would retry for ever.

Step 4. alice gets in, and may do nothing

s kafka-topics.sh --bootstrap-server $S --command-config /clients/alice.properties --list
s kafka-topics.sh --bootstrap-server $S --command-config /clients/alice.properties \
  --create --topic alice-topic --partitions 1 --replication-factor 1

(an empty list)
Error while executing topic command : Authorization failed.

Step 5. admin creates a topic and grants permissions

A=/clients/admin.properties
s kafka-topics.sh --bootstrap-server $S --command-config $A --create --topic payments-sec --partitions 1 --replication-factor 1
s kafka-acls.sh --bootstrap-server $S --command-config $A --add --allow-principal User:alice --producer --topic payments-sec
s kafka-acls.sh --bootstrap-server $S --command-config $A --add --allow-principal User:bob --consumer --topic payments-sec --group billing

Created topic payments-sec.
Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=payments-sec, patternType=LITERAL)`:
    (principal=User:alice, host=*, operation=DESCRIBE, permissionType=ALLOW)
    (principal=User:alice, host=*, operation=WRITE, permissionType=ALLOW)
    (principal=User:alice, host=*, operation=CREATE, permissionType=ALLOW)
Adding ACLs for resource `ResourcePattern(resourceType=GROUP, name=billing, patternType=LITERAL)`:
    (principal=User:bob, host=*, operation=READ, permissionType=ALLOW)
Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=payments-sec, patternType=LITERAL)`:
    (principal=User:bob, host=*, operation=DESCRIBE, permissionType=ALLOW)
    (principal=User:bob, host=*, operation=READ, permissionType=ALLOW)

Step 6. Each user can do exactly its job

printf 'p1\np2\np3\n' | s kafka-console-producer.sh --bootstrap-server $S --command-config /clients/alice.properties --topic payments-sec
s kafka-console-consumer.sh --bootstrap-server $S --command-config /clients/bob.properties --topic payments-sec --group billing --from-beginning --timeout-ms 8000

echo x | s kafka-console-producer.sh --bootstrap-server $S --command-config /clients/bob.properties --topic payments-sec
s kafka-console-consumer.sh --bootstrap-server $S --command-config /clients/alice.properties --topic payments-sec --group snoop --from-beginning --timeout-ms 6000
s kafka-console-consumer.sh --bootstrap-server $S --command-config /clients/bob.properties --topic payments-sec --group other --from-beginning --timeout-ms 6000

alice writes:               (no output: success)
bob reads with "billing":   p1  p2  p3   Processed a total of 3 messages

bob tries to write:         ClusterAuthorizationException: Cluster authorization failed.
alice tries to read:        GroupAuthorizationException: Not authorized to access group: snoop
bob with group "other":     GroupAuthorizationException: Not authorized to access group: other

Step 7. List the rules and read the denials

s kafka-acls.sh --bootstrap-server $S --command-config $A --list
docker exec kafka-sec sh -c 'grep -i denied /opt/kafka/logs/kafka-authorizer.log | tail -3'

Current ACLs for resource `ResourcePattern(resourceType=GROUP, name=billing, patternType=LITERAL)`:
    (principal=User:bob, host=*, operation=READ, permissionType=ALLOW)
Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=payments-sec, patternType=LITERAL)`:
    (principal=User:bob, … operation=READ …)  (principal=User:alice, … operation=WRITE …)  …

INFO Principal = User:alice is Denied operation = CREATE … on resource = Topic:LITERAL:alice-topic for request = CreateTopics … based on rule DefaultDeny
INFO Principal = User:alice is Denied operation = DESCRIBE … on resource = Group:LITERAL:snoop for request = FindCoordinator … based on rule DefaultDeny
INFO Principal = User:bob is Denied operation = DESCRIBE … on resource = Group:LITERAL:other for request = FindCoordinator … based on rule DefaultDeny

The authorizer log is your audit trail and your debugging tool. Each line names the user, the operation, the resource, and the rule that decided.

Step 8. Stop

docker compose down

offset 6

Common mistakes and misconceptions

People think	What is true
TLS means the cluster is secure.	TLS hides the traffic. Without authentication and rules, everyone who connects can still do everything.
A user who can read a topic can consume it.	It also needs permission on its consumer group.
An out-of-memory error at start is a memory problem.	On a TLS port it usually means the client is not using TLS.
Making the application a super user is a quick fix.	It removes every protection for that application, and those "quick fixes" are never undone.
Listing topics shows everything on the cluster.	It shows what the user may describe. An empty list can mean "no permission".
Securing the client port is enough.	Broker-to-broker and controller listeners need the same care.
Passwords in the broker's configuration are fine.	They are fine in this lab. In production, use SCRAM, tokens or certificates, and a secrets store.

offset 7

Staff-level questions

How do you add security to a running open cluster with no outage?

Add a secured listener beside the open one, with a rolling restart. Move applications to it one by one. Turn the authorizer on with allow.everyone.if.no.acl.found=true, so that nothing is denied yet, and add rules for each application while you watch the authorizer log for requests that have no rule. When every application has its rules, switch that setting to false. Last, remove the open listener.

The slow part is finding every client. Connection counts for each listener tell you who is still on the old port.

Which authentication mechanism do you choose?

Use what the company already runs for identity. With a central identity provider: tokens, so that users are created and removed in one place. With an internal certificate system for services: client certificates. With neither: SCRAM, where Kafka holds salted passwords and no password crosses the network.

Avoid PLAIN with passwords in configuration files outside a lab. Whatever you choose, decide how a credential is rotated before you issue the first one.

How do you organise permissions for thirty teams?

By naming. Each team owns a prefix, and one prefixed rule gives the team's principal rights on every topic and group under it. Sharing between teams is an explicit rule that the owning team approves. Rules live in version control and are applied by automation, so that the cluster's rules always match a reviewed file.

Keep super users to the platform team's automation. Review the list of rules on a schedule; unused rules are how access spreads.

What does encryption cost, and where?

Processor time on brokers and clients for every byte. And the loss of zero-copy: an encrypted response must pass through the broker process, where a plain one is sent from the file straight to the network (module 01). Size brokers with TLS turned on, and raise the network thread count if their idle share drops.

It is still the right default. Unencrypted traffic inside a "trusted" network is the usual way data leaks.

A team says "we are denied, please just open it up". How do you respond?

Ask for the exact error and read the authorizer log line: principal, operation, resource. Almost always one rule is missing, often the consumer group, or the idempotent producer's needs. Add that one rule. Then improve the self-service path so that the next team does not need to ask.

What else does a regulated company need beyond these three layers?

Encryption of data on disk, at the volume or by the application for sensitive fields. An audit trail kept outside the brokers. Quotas, so that a valid user cannot overload the cluster. Network limits on who can reach each listener. And a process: who approves access, how it is removed when someone leaves, how keys are rotated.

offset 8

Self-test

1. Which question does each of the three layers answer?

Encryption: can anyone else read this? Authentication: who is this client? Authorization: may this user do this?

2. What does a client need in its truststore?

The certificate of the authority that signed the broker's certificate.

3. What is a principal?

The identity the broker attaches to a connection after authentication, such as User:alice.

4. Which two resources does a consumer need rules for?

The topic and its consumer group.

5. What happens to a request with no matching rule in this lab?

It is denied, because allow.everyone.if.no.acl.found is false.

6. Where do you find out why a request was denied?

In the broker's kafka-authorizer.log.

7. What does a plain client see when it connects to a TLS port?

In the lab, an out-of-memory error.

8. What is a super user?

A principal that skips all access rules.