# streaming-wikipedia-with-kafka

**Repository Path**: mirrors_codecentric/streaming-wikipedia-with-kafka

## Basic Information

- **Project Name**: streaming-wikipedia-with-kafka
- **Description**: Repository for sample Kotlin implementations for Apache Kafka components
- **Primary Language**: Unknown
- **License**: Apache-2.0
- **Default Branch**: main
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2022-08-17
- **Last Updated**: 2025-10-18

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# Streaming Wikipedia with Apache Kafka
This repository contains a few samples how Kafka can be used. All samples are implemented using Kotlin and Gradle. 

- kafka-playground-producer: Receives *server sent events* from Wikimedia via a Webflux client and sends the events to a Kafka topic.
- kafka-playground-streams: Uses the Kafka Stream API to make aggregations on received events. Sends the aggregations to other topics.
- kafka-playground-consumer: Consumes events from Kafka topics. Consuming means in this particular case: logging.
- kafka-playground-connect: Implements a Kafka Connector to replace the producer.

## Installation and Setup
Prerequisites: you need docker and java on your machine. 

The following steps are necessary to get the apps up and running:
1. Build the connector. Therefore execute `cd kafka-playground-connect && ./gradlew clean shadowJar && cd ../`
2. Start the Kafka environment. Therefore execute `docker-compose -f docker/docker-compose.yaml up -d zookeeper kafka kafka-ui connect`
3. Wait until all 4 containers are running (check by using `docker ps -a`)
4. When opening the browser at `localhost:9080` you should see a web UI
5. To activate our custom connector you have to execute `curl -X POST -H "Content-Type:application/json" -d @./kafka-playground-connect/http/sse-source-connect.json http://localhost:8083/connectors`. In this JSON file our custom configuration parameters are set.
6. In the web UI you should now see a topic *wikipedia-via-connect* with an increasing number of messages in it. We are now successfully receiving events from wikimedia. Yay!
7. You can now start the *streams* and *consumer* applications by running `docker-compose -f docker/docker-compose.yaml up -d consumer streams-consumer`. Both apps have a multistage docker build, so you don't have to build the applications on your own machine first.
8. After both containers have started you can check the logs by running `docker logs consumer` or `docker logs streams-consumer`. You should find information about the aggregation made in the streams application in both log files (every 30 seconds).
9. Furthermore, you can see new topics in the web UI. The UI is awful at printing numbers, so you can't really see the aggregated results there.
10. Stopping the application is possible via `docker-compose -f docker/docker-compose.yaml down`.