Spring Boot with Apache Kafka on Windows.
Apache Kafka® is used for building real-time data pipelines and streaming apps. It is horizontally scalable, fault-tolerant, wicked fast, and runs in production in thousands of companies. (https://kafka.apache.org/)
ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. All of these kinds of services are used in some form or another by distributed applications. Each time they are implemented there is a lot of work that goes into fixing the bugs and race conditions that are inevitable. Because of the difficulty of implementing these kinds of services, applications initially usually skimp on them, which make them brittle in the presence of change and difficult to manage. Even when done correctly, different implementations of these services lead to management complexity when the applications are deployed. (https://zookeeper.apache.org/)
Spring Boot is an open-source Java-based framework used to create a micro Service. It is developed by Pivotal Team and is used to build stand-alone and production-ready spring applications. (https://spring.io/projects/spring-boot)
1. Apache Kafka and Zookeeper Setup
a. Download Zookeeper and Kafka from below links -
Kafka — https://www.apache.org/dyn/closer.cgi?path=/kafka/2.5.0/kafka_2.12-2.5.0.tgz
Zookeeper — https://downloads.apache.org/zookeeper/zookeeper-3.6.0/apache-zookeeper-3.6.0-bin.tar.gz
b. Setup Zookeeper -
After download, please extract the zookeeper binary and follow the below steps:
- Copy and Rename “zoo_sample.cfg” to “zoo.cfg” in your_path\zookeeper\conf
- Create a data directory in zookeeper folder.
- Find & edit dataDir=/tmp/zookeeper to /your-path/zookeeper/data using any text editor like notepad.
dataDir=/your-path/zookeeper/data - Add entries in System Environment Variables.
Add in System Variables ZOOKEEPER_HOME = your-path\zookeeper
Edit System Variable named “Path” and append this in the last ;%ZOOKEEPER_HOME%\bin; - Open command prompt and type zkserver.
- zkserver will start the zookeeper on the default Ip and port which is 0.0.0.0:2181, you can change the default port in zoo.cfg file
c. Setup Apache Kafka
After download, please extract the “Kafka binary” and follow the steps below :
- Go to config folder in “Apache Kafka” and edit “server.properties” using any text editor.
- Find log.dirs and replace after “=/tmp/kafka-logs” to “= your-path\kafka\kafka-logs”.
- Change the “zookeeper.connect” property to zookeeper.connect=0.0.0.0:2181
- By default “Apache Kafka” will run on port 9092 and “Apache Zookeeper” will run on port 2181.
- Open the command prompt and go to “Apache Kafka directory” and run the following command.
.\bin\windows\kafka-server-start.bat .\config\server.properties
- Create a “Kafka Topic” by execution below command in command prompt:
kafka-topics.bat — create — zookeeper localhost:2181 — replication-factor 1 — partitions 1 — topic KafkaTest
- To start a “Producer”, please execute the command below:
.\bin\windows\kafka-console-producer.bat — broker-list 0.0.0.0:9092 — topic KafkaTest
- To start a “Consumer”, please execute the command below:
kafka-console-consumer.bat -bootstrap-server 0.0.0.0:9092 -topic KafkaTest
2. Spring Boot Application set up to integrate with Apache Kafka
a. Create a Spring Boot Application using https://start.spring.io/ and add Spring-Kafka & Web dependencies.
b. Create a Rest Controller class with API to publish the message to KafkaProducer.
c. Create a Kafka Producer class using KafkaTemplate.
d. Create a Kafka Consumer class using @KafkaListener.
e. Create an application.yaml with Kafka Producer and Consumer details.
f. Now, run the Spring Boot application and hit the Rest API to publish the message to Kafka Producer.
localhost:8080/bootkafka/kafkapublish?message=Hello from Author
Thank you for reading this article.
