Table of Contents
Laptop showing Kafka and Docker logos with KRaft mode enabled

Why KRaft Mode Matters

Kafka Raft Metadata (KRaft) is a new mode for Apache Kafka that eliminates the need for Zookeeper. It was introduced in KIP-500 to remove the kafka dependency on Zookeeper. Apache Kafka traditionally required Zookeeper for cluster coordination - until KRaft (Kafka Raft Metadata) mode arrived. This update eliminates Zookeeper dependencies while improving:

Kafka Raft Metadata (KRaft) mode

Image from Kafka Raft Metadata (KRaft) mode

  • Simplified architecture: Fewer moving parts
  • Faster recovery: Metadata changes happen in milliseconds
  • Easier scaling: More predictable cluster behavior

I recently wanted to setup Kafka for my local development, and after quite a few tries, ngl, I was finally able to do it. Letโ€™s show you how it went.

Prerequisites

  • Docker (Latest version?)
  • 4GB+ free memory (Kafka containers can be hungry)

Complete Docker Compose Setup

services:
  kafka:
    image: apache/kafka:latest
    container_name: broker
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093,EXTERNAL://:9094
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:9092,EXTERNAL://localhost:9094
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,EXTERNAL:PLAINTEXT
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@broker:9093
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
    ports:
      - "9094:9094"
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "./opt/kafka/bin/kafka-broker-api-versions.sh --bootstrap-server localhost:9092 > /dev/null 2>&1",
        ]
      interval: 10s
      timeout: 10s
      retries: 5

  kafka-ui:
    image: provectuslabs/kafka-ui:latest
    container_name: kafka-ui
    ports:
      - "8088:8080"
    environment:
      KAFKA_CLUSTERS_0_NAME: local
      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: broker:9092
      KAFKA_CLUSTERS_0_READONLY: "false"
    depends_on:
      kafka:
        condition: service_healthy

Kafka Configuration:

  • KAFKA_NODE_ID: Unique identifier for the broker
  • KAFKA_PROCESS_ROLES: Broker and controller roles
  • KAFKA_LISTENERS: Listeners for different protocols
  • KAFKA_ADVERTISED_LISTENERS: Addresses advertised to clients
  • KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: Security protocol map
  • KAFKA_INTER_BROKER_LISTENER_NAME: Inter-broker listener name
  • KAFKA_CONTROLLER_QUORUM_VOTERS: Controller quorum voters
  • KAFKA_CONTROLLER_LISTENER_NAMES: Controller listener names
  • KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: Offsets topic replication factor
  • KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: Transaction state log replication factor
  • KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: Transaction state log minimum ISR

Cluster Management Commands

Start all services:

docker-compose up -d

Verify running containers:

docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"

Expected output:

NAMES     STATUS              PORTS
broker            Up 4 minutes (healthy)    9092/tcp, 0.0.0.0:9094->9094/tcp
kafka-ui          Up 4 minutes              0.0.0.0:8088->8080/tcp

Monitoring with Kafka UI

After starting containers, access http://localhost:8088 to:

  • View real-time broker status
  • Inspect topic configurations
  • Monitor consumer lag
  • Explore message payloads
  • Manage ACLs (access control lists)

Connection URLs to use in your applications

You will need to connect to the broker using the following URL:

  • If running outside of docker:
localhost:9094
  • If running inside of docker:
broker:9092

Topic Management: CLI vs UI

Command-line creation:

docker exec broker /opt/kafka/bin/kafka-topics.sh \
  --topic orders \
  --create \
  --bootstrap-server localhost:9092 \
  --partitions 3 \
  --replication-factor 1

UI alternative:

  1. Navigate to Topics โ†’ Create Topic
  2. Set partitions/replication
  3. Click โ€œSubmitโ€
Kafka UI topic creation

Message Production Patterns

Basic text messages:

docker exec -it broker /opt/kafka/bin/kafka-console-producer.sh \
  --topic orders \
  --bootstrap-server localhost:9092
Kafka console producer

Structured JSON:

# Using jq for JSON formatting
echo '{"id":1001, "amount":49.99}' | jq -c | docker exec -i broker /opt/kafka/bin/kafka-console-producer.sh \
  --topic orders \
  --bootstrap-server localhost:9092
Kafka console producer

You can also see the messages on the UI:

Kafka messages on UI

Troubleshooting Checklist

Brokers not connecting?

  • Check port conflicts with netstat -tuln | grep 9092

UI not showing topics?

  • Ensure bootstrap servers match in UI config
  • Check container logs: docker logs kafka-ui

When to Choose This Setup

Good for:

  • Local development
  • CI/CD pipelines
  • Prototyping new features

Not for:

  • Production deployments
  • High-availability requirements
  • Sensitive data handling

Learn More

โ„น๏ธ

Senior Backend Engineer available for Kafka and distributed systems projects. Contact: [email protected]