Kafka Essential Commands

Kafka is a distributed system consisting of servers and clients that communicate via a high-performance TCP network protocol. This cheatsheet provides essential commands for managing and interacting with Apache Kafka topics.

List Kafka Topics

View all available topics in your Kafka cluster.

kafka-topics --list --bootstrap-server kafka-broker:9092

Create a New Kafka Topic

Create a new topic with specified configurations.

kafka-topics --create --topic test-topic --bootstrap-server kafka-broker:9092 --partitions 3 --replication-factor 1

Describe Kafka Topic Details

Get detailed information about a specific Kafka topic, including partitions, replicas, and leader information.

kafka-topics --describe --topic test-topic --bootstrap-server kafka-broker:9092

Produce Messages to Kafka Topic

Send messages to a Kafka topic using the console producer.

echo "hello" | kafka-console-producer --bootstrap-server kafka-broker:9092 --topic test-topic

Produce JSON Messages to a Kafka topic:

cat > file.json << EOF
{ "id": 1, "first_name": "John"}
{ "id": 2, "first_name": "Peter"}
{ "id": 3, "first_name": "Nate"}
{ "id": 4, "first_name": "Frank"}
EOF

kafka-console-producer --bootstrap-server kafka-broker:9092 --topic test-topic < file.json

Consume Messages from Kafka Topic

Read messages as they arrive in a Kafka topic.

kafka-console-consumer --bootstrap-server kafka-broker:9092 --topic test-topic

Read all messages from the beginning of a Kafka topic:

kafka-console-consumer --bootstrap-server kafka-ops:9092 --topic test-topic --from-beginning

Count Messages in a Kafka Topic

Calculate the total number of messages in a Kafka topic.

kafka-run-class kafka.tools.GetOffsetShell --bootstrap-server kafka-broker:9092 --topic test-topic | awk -F  ":" '{sum += $3} END {print "Result: "sum}'

External Resources

• Apache Kafka Documentation
• MDN JSON Guide
• Stack Overflow Kafka Questions