# kafka-log-study
**Repository Path**: linestyle007/kafka-log-study
## Basic Information
- **Project Name**: kafka-log-study
- **Description**: No description available
- **Primary Language**: Unknown
- **License**: Not specified
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No
## Statistics
- **Stars**: 0
- **Forks**: 0
- **Created**: 2021-09-13
- **Last Updated**: 2021-11-02
## Categories & Tags
**Categories**: Uncategorized
**Tags**: None
## README
# kafka-log-study
kafka日志收集
## 准备工作
### 配置zookeeper
```scale
# 1.安装zookeeper
$ wget https://archive.apache.org/dist/zookeeper/zookeeper-3.4.14/zookeeper-3.4.14.tar.gz
$ tar -zxvf zookeeper-3.4.14.tar.gz
$ cd zookeeper-3.4.14
$ mkdir data
# 2.配置zoo.cfg
$ cd conf
$ mv zoo_sample.cfg zoo.cfg
$ vim zoo.cfg
dataDir=/root/zookeeper-3.4.14/data
# 3.启动zookeeper
./zkServer.sh start
./zkServer.sh stop
./zkServer.sh status
```
### 配置nginx
```scala
# 1. 安装git
$ yum install -y git
# 2. 安装相关依赖
$ yum install -y gcc gcc-c++ zlib zlib-devel openssl openssl-devel pcre pcre-devel
# 3. kafka的客户端源码
$ cd /root/software
//$ git clone https://github.com/edenhill/librdkafka
$ git clone https://gitee.com/mirrors/librdkafka.git
# 4. 编译
$ cd /root/software/librdkafka
$ ./configure
$ make && make install
# 5. 安装
$ cd /root/software
$ wget http://nginx.org/download/nginx-1.18.0.tar.gz
# 6. 解压
$ tar -zxf nginx-1.18.0.tar.gz
# 7. 下载模块源码
$ cd /root/software
$ git clone https://github.com/brg-liuwei/ngx_kafka_module
# 8. 编译
$ cd /root/software/nginx-1.18.0
$ ./configure --add-module=/root/software/ngx_kafka_module/
$ make && make install
# 9.加载
$ echo "/usr/local/lib" >> /etc/ld.so.conf
$ ldconfig
```
### 修改nginx.conf
```scala
# 1. 修改 nginx.conf 配置
$ vi /usr/local/nginx/conf/nginx.conf
#pid logs/nginx.pid;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
#log_format main '$remote_addr - $remote_user [$time_local] "$request" '
# '$status $body_bytes_sent "$http_referer" '
# '"$http_user_agent" "$http_x_forwarded_for"';
#access_log logs/access.log main;
sendfile on;
#tcp_nopush on;
#keepalive_timeout 0;
keepalive_timeout 65;
#gzip on;
kafka;
kafka_broker_list 192.168.3.11:9092;
server {
#charset koi8-r;
#access_log logs/host.access.log main;
location = /kafka/log {
add_header 'Access-Control-Allow-Origin' $http_origin;
add_header 'Access-Control-Allow-Credentials' 'true';
add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
kafka_topic tp_individual;
}
#error_page 404 /404.html;
}
}
```
### 创建kafka topic
```scala
# 启动 kafka 和 zk 容器
$ zkServer.sh start
$ ./bin/kafka-server-start.sh -daemon /root/kafka_2.11-1.0.2/config/server.properties
# 列出所有主题
$ ./bin/kafka-topics.sh --list --zookeeper localhost:2181/myKafka
# 创建 `topic` ： tp_individual
$ ./bin/kafka-topics.sh --zookeeper localhost:2181/myKafka --create --topic tp_individual --partitions 1 --replication-factor 1
# 启动消费者
$ ./bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic tp_individual --from-beginning
```
### 测试
```scala
# 1.客户端
$ curl 192.168.3.11:80/kafka/log -d "test message" -v
# 2.服务端：在kafka消费端，可以监听到消息
[root@localhost ~]# $ ./bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic tp_individual --from-beginning
{"user_id":"u_donald","act_time":"2021-2-2 21:44:41","action":"click","job_code":"test"}
{"user_id":"u_donald","act_time":"2021-2-2 21:44:42","action":"job_collect","job_code":"test"}
{"user_id":"u_donald","act_time":"2021-2-2 21:44:42","action":"cv_send","job_code":"test"}
{"user_id":"u_donald","act_time":"2021-2-2 21:44:43","action":"cv_upload","job_code":"test"}
```
## 实现步骤
### 创建html
```html
index