
Why KRaft Mode Matters
Kafka Raft Metadata (KRaft) is a new mode for Apache Kafka that eliminates the need for Zookeeper. It was introduced in KIP-500 to remove the kafka dependency on Zookeeper. Apache Kafka traditionally required Zookeeper for cluster coordination - until KRaft (Kafka Raft Metadata) mode arrived. This update eliminates Zookeeper dependencies while improving:

Image from Kafka Raft Metadata (KRaft) mode
- Simplified architecture: Fewer moving parts
- Faster recovery: Metadata changes happen in milliseconds
- Easier scaling: More predictable cluster behavior
I recently wanted to setup Kafka for my local development, and after quite a few tries, ngl, I was finally able to do it. Letโs show you how it went.
Prerequisites
- Docker (Latest version?)
- 4GB+ free memory (Kafka containers can be hungry)
Complete Docker Compose Setup
services:
kafka:
image: apache/kafka:latest
container_name: broker
environment:
KAFKA_NODE_ID: 1
KAFKA_PROCESS_ROLES: broker,controller
KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093,EXTERNAL://:9094
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:9092,EXTERNAL://localhost:9094
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,EXTERNAL:PLAINTEXT
KAFKA_CONTROLLER_QUORUM_VOTERS: 1@broker:9093
KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
ports:
- "9094:9094"
healthcheck:
test:
[
"CMD-SHELL",
"./opt/kafka/bin/kafka-broker-api-versions.sh --bootstrap-server localhost:9092 > /dev/null 2>&1",
]
interval: 10s
timeout: 10s
retries: 5
kafka-ui:
image: provectuslabs/kafka-ui:latest
container_name: kafka-ui
ports:
- "8088:8080"
environment:
KAFKA_CLUSTERS_0_NAME: local
KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: broker:9092
KAFKA_CLUSTERS_0_READONLY: "false"
depends_on:
kafka:
condition: service_healthy
Kafka Configuration:
KAFKA_NODE_ID
: Unique identifier for the brokerKAFKA_PROCESS_ROLES
: Broker and controller rolesKAFKA_LISTENERS
: Listeners for different protocolsKAFKA_ADVERTISED_LISTENERS
: Addresses advertised to clientsKAFKA_LISTENER_SECURITY_PROTOCOL_MAP
: Security protocol mapKAFKA_INTER_BROKER_LISTENER_NAME
: Inter-broker listener nameKAFKA_CONTROLLER_QUORUM_VOTERS
: Controller quorum votersKAFKA_CONTROLLER_LISTENER_NAMES
: Controller listener namesKAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR
: Offsets topic replication factorKAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR
: Transaction state log replication factorKAFKA_TRANSACTION_STATE_LOG_MIN_ISR
: Transaction state log minimum ISR
Cluster Management Commands
Start all services:
docker-compose up -d
Verify running containers:
docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"
Expected output:
NAMES STATUS PORTS
broker Up 4 minutes (healthy) 9092/tcp, 0.0.0.0:9094->9094/tcp
kafka-ui Up 4 minutes 0.0.0.0:8088->8080/tcp
Monitoring with Kafka UI
After starting containers, access http://localhost:8088 to:
- View real-time broker status
- Inspect topic configurations
- Monitor consumer lag
- Explore message payloads
- Manage ACLs (access control lists)
Connection URLs to use in your applications
You will need to connect to the broker using the following URL:
- If running outside of docker:
localhost:9094
- If running inside of docker:
broker:9092
Topic Management: CLI vs UI
Command-line creation:
docker exec broker /opt/kafka/bin/kafka-topics.sh \
--topic orders \
--create \
--bootstrap-server localhost:9092 \
--partitions 3 \
--replication-factor 1
UI alternative:
- Navigate to Topics โ Create Topic
- Set partitions/replication
- Click โSubmitโ

Message Production Patterns
Basic text messages:
docker exec -it broker /opt/kafka/bin/kafka-console-producer.sh \
--topic orders \
--bootstrap-server localhost:9092

Structured JSON:
# Using jq for JSON formatting
echo '{"id":1001, "amount":49.99}' | jq -c | docker exec -i broker /opt/kafka/bin/kafka-console-producer.sh \
--topic orders \
--bootstrap-server localhost:9092

You can also see the messages on the UI:

Troubleshooting Checklist
Brokers not connecting?
- Check port conflicts with
netstat -tuln | grep 9092
UI not showing topics?
- Ensure bootstrap servers match in UI config
- Check container logs:
docker logs kafka-ui
When to Choose This Setup
Good for:
- Local development
- CI/CD pipelines
- Prototyping new features
Not for:
- Production deployments
- High-availability requirements
- Sensitive data handling
Learn More
Senior Backend Engineer available for Kafka and distributed systems projects. Contact: [email protected]