# kafkaWork

**Repository Path**: zhayan/kafka-work

## Basic Information

- **Project Name**: kafkaWork
- **Description**: kafka学习作业
- **Primary Language**: Unknown
- **License**: Not specified
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2022-03-16
- **Last Updated**: 2022-03-18

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# 环境搭建

## 运行效果

[运行模拟视频](doc/kafka日志收集作业.rar)



![](doc/img/page.png)





![](doc/img/idea.png)



![](doc/img/opt.png)

## profile

```
# java_env
export JAVA_HOME=/usr/java/jdk1.8.0_311
export CLASSPATH=.:${JAVA_HOME}/jre/lib/rt.jar:${JAVA_HOME}/lib/dt.jar:${JAVA_HOME}/lib/tools.jar
export PATH=$PATH:${JAVA_HOME}/bin

# maven 
MAVEN_HOME=/usr/maven/apache-maven-3.8.4
export PATH=${MAVEN_HOME}/bin:${PATH}

export KAFKA=/opt/kafka_2.12-1.0.2
export PATH=$PATH:$KAFKA/bin

export ZOOKEEPER=/opt/apache-zookeeper-3.6.0
export PATH=$PATH:$ZOOKEEPER/bin
export ZOO_LOG_DIR=/opt/log/zookeeper
```



```
# 修改vim
vim /etc/profile

# 修改后生效
source /etc/profile
```



## git

```
yum install -y git

# 更新git
sudo yum update nss curl
```



## zookeeper

```
# 下载zk
wget https://archive.apache.org/dist/zookeeper/zookeeper-3.6.0/apache-zookeeper-3.6.0.tar.gz
tar -zxvf zookeeper-3.6.0.tar.gz
cd /opt/apache-zookeeper-3.6.0/bin
./zkServer.sh start


# zk配置，没有就使用zoo_sample.cfg复制一个zoo.cfg
cd /opt/apache-zookeeper-3.6.0/conf
vim zoo.cfg
# 调整
dataDir=/opt/data/zookeeper
dataLogDir=/opt/log/zookeeper

# 配置/etc/profile
vim /etc/profile
export ZOOKEEPER=/opt/apache-zookeeper-3.6.0
export PATH=$PATH:$ZOOKEEPER/bin
export ZOO_LOG_DIR=/opt/log/zookeeper

```



```
# 启动zk
cd /opt/apache-zookeeper-3.6.0/bin
zkServer.sh start
 
# 查看状态
zkServer.sh status
```



![](doc/img/zk.png)

## kafka

```
#下载kafka并解压
http://archive.apache.org/dist/kafka/

# 修改配置
vim /opt/kafka_2.12-1.0.2/config/server.properties

zookeeper.connect=localhost:2181/myKafka
advertised.listeners=PLAINTEXT://192.168.174.138:9092
log.dirs=/opt/kafkaLogs

# 修改环境变量/etc/profile
vim /etc/profile
export KAFKA=/opt/kafka_2.12-1.0.2
export PATH=$PATH:$KAFKA/bin

# 生效配置
source /etc/profile
```



```
# 后台启动
cd /opt/kafka_2.12-1.0.2/bin
kafka-server-start.sh -daemon /opt/config/server.properties
```



```
# 部分操作
cd /opt/kafka_2.12-1.0.2/bin

# 查看现有主题
kafka-topics.sh --list --zookeeper localhost:2181/myKafka

#新增主题
kafka-topics.sh --zookeeper localhost:2181/myKakfa --create --topic topic_1 --partitions 1 --replication-factor 1

# 删除主题
kafka-topics.sh --zookeeper localhost:2181/myKafka --delete --topic topic_1

# 查看主题
kafka-topics.sh --zookeeper localhost:2181/myKafka --describe --topic topic_1
```



```
# 生产者
kafka-console-producer.sh --topic topic_1 --broker-list localhost:9092

# 消费者
kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic topic_1 [--from-beginning]
```



## nginx_kafka_module

安装好该组件后，才能在nginx中使用**kafka标识**

```
# /opt目录下载
cd /opt
git clone https://github.com/edenhill/librdkafka

cd librdkafka
./configue  # 可能需要其他依赖[yum install -y gcc gcc-c++ pcre-devel zlib-devel]
make 
make install
```



```
# 下载ngx_kafka_module
git clone https://github.com/brg-liuwei/ngx_kafka_module  

# 安装
cd /opt/nginx-1.18.0 ./configure --add-module=/opt/ngx_kafka_module 
make
make install
```



## nginx



```
# 依赖包
yum -y install make zlib zlib-devel gcc-c++ libtool  openssl openssl-devel

# nginx下载
https://nginx.org/en/download.html
wget http://nginx.org/download/nginx-1.8.0.tar.gz
tar -zcvf nginx-1.8.0.tar.gz

# nginx编译
cd /opt/nginx-1.8.0
./configure 
make && make install

# 编译后的默认目录
/usr/local/nginx/
```



```

#user  nobody;
worker_processes  1;

#error_log  logs/error.log;
#error_log  logs/error.log  notice;
#error_log  logs/error.log  info;

#pid        logs/nginx.pid;

events {
    worker_connections  1024;
}


http {
        kafka;
        kafka_broker_list 192.168.174.138:9092;
    server {
        listen       80;
        server_name  localhost;

		# 跨域设置
    	add_header Access-Control-Allow-Origin *;
    	add_header Access-Control-Allow-Methods 'GET, POST, OPTIONS';
    	add_header Access-Control-Allow-Headers 'DNT,X-Mx-ReqToken,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Authorization';

        location / {
            root   html;
            index  index.html index.htm;
        }
        
        # kafka采集日志api
        location /log {
            # kafka主题
			kafka_topic topic_1;
        }

        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }
    }
}

```



```
# 重启
nginx -s reload

# 启动
cd /usr/local/nginx/sbin
nginx

# 查看日志相关:如跨域、404等
cd /usr/local/nginx/logs
tail -f access.log
tail -f error.log
```



## nginx跨域



```
# nginx.conf

# 跨域设置
    	add_header Access-Control-Allow-Origin *;
    	add_header Access-Control-Allow-Methods 'GET, POST, OPTIONS';
    	add_header Access-Control-Allow-Headers 'DNT,X-Mx-ReqToken,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Authorization';

```



```
# ajax.post调整
    function doClick(type) {
        $.ajax({
            url: 'http://192.168.174.138:80/log',
            type: 'POST',
            // contentType防止跨域
            contentType: "application/x-www-form-urlencoded",
            dataType: "json",
            data: JSON.stringify({
                userId: 'wsz',
                actTime: new Date(),
                action: type,
                jobCode: 'wszJobCode'
            }),
            success: function (res) {
                alert(res)
            },
            error: function (res) {
                alert(res)
            }
        });
    }
```



## 其它问题

1. 最好直接在linux中下载相关依赖，librdkafka，nginx_kafka_module，防止./configure时失败
2. 跨域问题：多尝试