# ELK

**Repository Path**: iuhao/ElasticSearch-Logstash-Kibana

## Basic Information

- **Project Name**: ELK
- **Description**: Kafka收集到的日志存放于ES（ElasticSearch，一款基于Apache Lucene的开源分布式搜索引擎）中便于查找和分析，ELK（ElasticSearch, Logstash, Kibana）平台实现日志收集、日志搜索和日志分析的功能,
- **Primary Language**: Unknown
- **License**: Not specified
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2018-01-14
- **Last Updated**: 2020-12-19

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# ELK(ElasticSearch, Logstash, Kibana)搭建实时日志分析平台
- 尝试了一周的Log4j+Kafka 日志分析,最终听朋友介绍外加公司内部大数据组所用的ES反映LK（ElasticSearch, Logstash, Kibana）平台恰好可以同时实现日志收集、日志搜索和日志分析的功能,所以打算搭建一个属于物联网卡管路业务以及数据平台的实时日志分析平台
![输入图片说明](https://gitee.com/uploads/images/2018/0110/105147_ab2c7de2_1468963.png "123049_meyt_1434710.png")
### 系统环境 ###
ElasticSearch: 2.1.2

Logstash: 2.1.2

Kibana: 4.3.0

为了方便统一整理上传到CSDN资源库：
[下载](http://download.csdn.net/download/li1669852599/10197925)
![输入图片说明](https://gitee.com/uploads/images/2018/0110/105109_8b397bdd_1468963.png "QQ截图20180110104806.png")

### ElasticSearch [端口：9200] ###
    tar -zxvf elasticsearch-2.1.2.tar.gz
    cd elasticsearch-2.1.2

- 安装Head插件（Optional）：
- 
    ./bin/plugin install mobz/elasticsearch-head

- 安装kopf插件
-
    ./bin/plugin install lmenezes/elasticsearch-kopf
      
- 启动
-
    ./bin/elasticsearch
![输入图片说明](https://gitee.com/uploads/images/2018/0110/105248_0999c281_1468963.png "QQ截图20180110105058.png")
 _注：这里有一个分片空置属于缓存问题_ 
![输入图片说明](https://gitee.com/uploads/images/2018/0110/105507_19274c1e_1468963.png "QQ截图20180110105211.png")

### Logstash ###

    tar -xvf logstash-2.3.4.tar.gz
    cd logstash-2.3.4

- 配置
- 
将配置文件放置在config文件夹下面

	input {
	  log4j {
	    mode => "server"
	    host => "172.16.0.112"
	    port => 4567
	  }
	}
	
	filter {
	}
	
	output {
	  elasticsearch {
	    action => "index"          #The operation on ES
	    hosts  => "172.16.0.112:9200"   #ElasticSearch host, can be array.
	    index  => "applog"         #The index to write data to.
	  }
	}

- 启动
- 

    ./bin/logstash agent -f config/log4j_to_es.conf
    Settings: Default filter workers: 12
    log4j:WARN No appenders could be found for logger (org.apache.http.client.protocol.RequestAuthCache).
    log4j:WARN Please initialize the log4j system properly.
    log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
    Logstash startup completed


### kibana###

    tar -zxvf kibana-4.3.0-linux-x86.tar.gz
    cd kibana-4.3.0-linux-x86
    vi config/kibana.yml

修改以下几项（由于是单机版的，因此host的值也可以使用localhost来代替，这里仅仅作为演示）：

    server.port: 5601
    server.host: “172.16.0.112”
    elasticsearch.url: http://172.16.0.112:9200
    kibana.index: “.kibana”

启动kibana
    ./bin/kibana

![输入图片说明](https://gitee.com/uploads/images/2018/0110/105534_430c47ee_1468963.png "QQ截图20180110105356.png")

# 启动脚本 #

- logstash 

```
#!/bin/bash

#   code by Li.Shangzhi
#   email lishangzhi@jimilab.net
#   samples:./bin/logstash agent -f config/log4j_to_es.conf

export logstash_conf="config/log4j_to_es.conf"
export logstash_bin="./bin/logstash"

echo 'Start logstash...'
nohup ${logstash_bin} agent --verbose -f ${logstash_conf} > logstash.log 2>&1 &
echo 'Start logstash!'

rm -rf logstash.log
```

- kibana
 
```
#!/bin/bash

#   code by Li.Shangzhi
#   email lishangzhi@jimilab.net
#   samples:./bin/kibana 

export logstash_conf="config/log4j_to_es.conf"
export logstash_bin="./bin/kibana "

echo 'Start kibana...'
nohup ${logstash_bin} > kibana.log 2>&1 &
echo 'Start kibana!'

rm -rf kibana.log
```


 **案例 ** 
### gitee.samples 节点

log4j.properties

```
log4j.rootLogger=INFO,console

# for package com.demo.elk, log would be sent to socket appender.

log4j.logger.com.ibyte.logstash=DEBUG, socket

# appender socket

log4j.appender.socket=org.apache.log4j.net.SocketAppender
log4j.appender.socket.Port=4567
log4j.appender.socket.RemoteHost=172.16.0.112
log4j.appender.socket.layout=org.apache.log4j.PatternLayout
log4j.appender.socket.layout.ConversionPattern=%d [%-5p] [%l] %m%n
log4j.appender.socket.ReconnectionDelay=10000

# appender console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.out
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d [%-5p] [%l] %m%n
```
logstash 日志采集写入ES 
```
package com.ibyte.logstash;
import org.apache.log4j.Logger;
/**
 * @FileName LogstashApplication.java
 * @Description: 
 *
 * @Date 2018年1月10日 上午10:03:02
 * @author Li.Shangzhi
 * @version 1.0
 */
public class LogstashApplication {

	private static final Logger LOGGER = Logger.getLogger(LogstashApplication.class);

	public static void main(String[] args) {
		// SpringApplication.run(SpringbootLogstashApplication.class, args);
		// TODO Auto-generated method stub
		for (int i = 0; i < 10; i++) {
			LOGGER.info("info log 客户端A测试信息[" + i + "].iByte");
			try {
				Thread.sleep(500);
			} catch (InterruptedException e) {
				e.printStackTrace();
			}
		}
	}
}
```
![输入图片说明](https://gitee.com/uploads/images/2018/0110/111228_efb02851_1468963.png "QQ截图20180110110058.png")
![输入图片说明](https://gitee.com/uploads/images/2018/0110/110427_5ea0ef31_1468963.png "QQ截图20180110110242.png")
![输入图片说明](https://gitee.com/uploads/images/2018/0110/110531_da58e46e_1468963.png "QQ截图20180110110352.png")