Kafka Cheatsheet
Kafka Essential Commands
Kafka is a distributed system consisting of servers and clients that communicate via a high-performance TCP network protocol. This cheatsheet provides essential commands for managing and interacting with Apache Kafka topics.
List Kafka Topics
View all available topics in your Kafka cluster.
kafka-topics --list --bootstrap-server kafka-broker:9092
Create a New Kafka Topic
Create a new topic with specified configurations.
kafka-topics --create --topic test-topic --bootstrap-server kafka-broker:9092 --partitions 3 --replication-factor 1
Describe Kafka Topic Details
Get detailed information about a specific Kafka topic, including partitions, replicas, and leader information.
kafka-topics --describe --topic test-topic --bootstrap-server kafka-broker:9092
Produce Messages to Kafka Topic
Send messages to a Kafka topic using the console producer.
echo "hello" | kafka-console-producer --bootstrap-server kafka-broker:9092 --topic test-topic
Produce JSON Messages to a Kafka topic:
cat > file.json << EOF
{ "id": 1, "first_name": "John"}
{ "id": 2, "first_name": "Peter"}
{ "id": 3, "first_name": "Nate"}
{ "id": 4, "first_name": "Frank"}
EOF
kafka-console-producer --bootstrap-server kafka-broker:9092 --topic test-topic < file.json
Consume Messages from Kafka Topic
Read messages as they arrive in a Kafka topic.
kafka-console-consumer --bootstrap-server kafka-broker:9092 --topic test-topic
Read all messages from the beginning of a Kafka topic:
kafka-console-consumer --bootstrap-server kafka-ops:9092 --topic test-topic --from-beginning
Count Messages in a Kafka Topic
Calculate the total number of messages in a Kafka topic.
kafka-run-class kafka.tools.GetOffsetShell --bootstrap-server kafka-broker:9092 --topic test-topic | awk -F ":" '{sum += $3} END {print "Result: "sum}'