Creating Kafka Topics and Testing with Command-Line Tools

Apache Kafka is one of the most popular distributed event-streaming platforms used for building scalable and real-time data pipelines. While Kafka can be integrated with Java, Python, or other programming languages, it’s always a good idea to first get comfortable with Kafka command-line tools. These tools allow you to create topics, send messages, and consume data directly from your terminal or command prompt without writing any code.

In this blog post, you will learn step by step how to:

• Create a Kafka topic from the command line
• List, describe, and delete Kafka topics using the kafka-topics command
• Send messages to a Kafka topic using the console producer
• Consume messages from a topic using the Kafka console consumer
• Understand common pitfalls and troubleshooting tips

By the end, you’ll have a solid foundation in working with the Kafka CLI Tool and be ready to move toward application-level development.

Prerequisites

• Kafka is installed and running locally (if not installed, see the Step-by-step Kafka installation Guide).
• A running Kafka broker and Zookeeper (if you’re using the traditional setup).
• Access to either a Unix-like terminal (Linux/Mac) or Windows Command Prompt/PowerShell.

Introduction to Kafka Command Line Tools

Kafka comes with a rich set of CLI tools that make it easy to manage your Kafka environment. These tools are found in the bin directory of your Kafka installation. If you are on:

• Unix/Linux/Mac, you’ll use scripts ending with .sh (for example, kafka-topics.sh).
• Windows, you’ll use .bat scripts (for example, kafka-topics.bat).

The Kafka command line tools include:

• kafka-topics: To create, list, describe, and delete topics
• kafka-console-producer: To send messages to a topic
• kafka-console-consumer: To read messages from a topic

These tools are extremely useful for developers who want to test Kafka locally before moving to Java APIs.

How to Create Kafka Topics from the Command Line?

A Kafka topic is like a logical channel or feed where messages are published and consumed. Topics are the backbone of Kafka, and understanding how to create them is the first step.

Command to create a topic:

bin/kafka-topics.sh --create --topic first_topic --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1

bin\windows\kafka-topics.bat --create --topic first_topic --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1

Parameters explanation

• --topic first_topic: The name of the Kafka topic.
• --bootstrap-server localhost:9092 : Connects to the Kafka broker.
• --partitions 1: Defines the number of partitions for scalability.
• --replication-factor 1: Number of replicas for fault tolerance.

If the topic is created successfully, you’ll see the following output on the console:

Created topic first_topic.

List Topics in Kafka Command Windows

Once you’ve created a topic, you might want to list all available topics in Kafka. This is where the --list option of the kafka-topics command comes in handy.

bin/kafka-topics.sh --list --bootstrap-server localhost:9092

bin\windows\kafka-topics.bat --list --bootstrap-server localhost:9092

Expected Output:

first_topic

This confirms that your topic exists. If you create multiple topics, you’ll see all of them listed here. This step is often searched as “List topics in Kafka command windows”, making it a very common troubleshooting and learning step.

Describe a Kafka Topic

You can get detailed information about a topic using the --describe command.

bin/kafka-topics.sh --describe --topic first_topic --bootstrap-server localhost:9092

bin\windows\kafka-topics.bat --describe --topic first_topic --bootstrap-server localhost:9092

Expected Output:

Topic: first_topic PartitionCount:1 ReplicationFactor:1 Configs:
Topic: first_topic Partition: 0 Leader: 0 Replicas: 0 Isr: 0

This command shows the number of partitions, replicas, and leader brokers for the topic. It’s an essential part of managing Kafka.

Delete a Kafka Topic

Sometimes, you’ll want to delete a topic (for example, during testing). Use the --delete command:

bin/kafka-topics.sh --delete --topic first_topic --bootstrap-server localhost:9092

bin\windows\kafka-topics.bat --delete --topic first_topic --bootstrap-server localhost:9092

How to Send a Message to a Kafka Topic in the Command Line?

Once your topic is ready, you can send messages using the Kafka console producer.

bin/kafka-console-producer.sh --topic first_topic --bootstrap-server localhost:9092

bin\windows\kafka-console-producer.bat --topic first_topic --bootstrap-server localhost:9092

Once the producer starts, type your message:

Hello Kafka!
This is my first Kafka message.

Press Enter after each message. Each line is sent as a new record into the Kafka topic.

Consuming Messages Using Kafka Console Consumer

Now that messages are in your topic, let’s read them using the Kafka console consumer.

bin/kafka-console-consumer.sh --topic first_topic --from-beginning --bootstrap-server localhost:9092

bin\windows\kafka-console-consumer.bat --topic first_topic --from-beginning --bootstrap-server localhost:9092

Expected Output:

Hello Kafka!
This is my first Kafka message.

The --from-beginning flag tells Kafka to read messages from the very start of the topic. Without it, the consumer will only read new messages.

This is one of the most important commands for beginners working with the Kafka CLI Tool.

Common Pitfalls and Troubleshooting

Working with Kafka for the first time can be tricky. Here are some common issues:

1. Zookeeper vs. Bootstrap Server Confusion
   • Older Kafka versions required --zookeeper localhost:2181
   • Newer versions use --bootstrap-server localhost:9092
   • Always check which version of Kafka you’re using.
2. Topic Deletion Not Working
   • Add delete.topic.enable=true in server.properties.
3. Broker Not Running
   • Make sure Kafka is started (bin/kafka-server-start.sh config/server.properties).
4. Producer/Consumer Not Receiving Messages
   • Check if your topic name is correct (case-sensitive).

Conclusion

In this post, we explored how to use Kafka command line tools for managing topics and sending/receiving data. You learned:

• How to create a Kafka topic from the command line
• How to list topics in Kafka command windows
• How to use the kafka-topics command to describe and delete topics
• How to send messages to a topic with the Kafka console producer
• How to read messages with the Kafka console consumer

By practicing these steps, you’ve learned the basics of Kafka CLI Tool operations. This flow is essential for testing and debugging before moving to Java or other client APIs.

Next, you can try writing Java-based Kafka producers and consumers to integrate Kafka into your applications.