# elasticSearch-springboot-demo

**Repository Path**: zippobai/elasticSearch-springboot-demo

## Basic Information

- **Project Name**: elasticSearch-springboot-demo
- **Description**: Elasticsearch+springboot2 demo,  ELK（elasticsearch+logstash+kibana） 搭建教程
- **Primary Language**: Unknown
- **License**: Not specified
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 3
- **Created**: 2021-11-20
- **Last Updated**: 2021-11-20

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

### elasticSearch-springboot-demo，实现常规的elk模式[ELK = Elasticsearch, Logstash, Kibana 是一套实时数据收集，存储，索引，检索，统计分析及可视化的解决方案]
- elasticsearch 6.5.3【检索，存储工具】
- jdk8+
- elasticSearch-analysis-ik 6.5.3 【中文分词】
- kibana-6.5.3-windows-x86_64 【可视化操作】
- logstash-6.5.3 【日志处理】
- logstash-input-jdbc 【日志处理的插件】 `安装`：`logstash-plugin.bat install logstash-input-jdbc`

> 有机会想实现一下新浪目前的技术架构

![want_to_commplet_技术架构](images/want_to_commplet_tech.png)

#### 特性
- 加入一次性mock100万条数据的接口，模拟生成数据还挺快的
- 优化单元测试的顺序

#### 准备工作
- 下载elasticSearch[中文网的下载地址](https://elasticsearch.cn/download/)，选择版本`elasticsearch-6.5.3`(因为我用的是jdk8)
- 下载elasticSearch-analysis-ik中文分词[下载地址](https://github.com/medcl/elasticsearch-analysis-ik/releases/tag/v6.5.3)，选择版本`elasticsearch-analysis-ik-6.5.3.zip`

#### 启动elasticsearch
- 修改解压后的文件`D:\elasticsearch\elasticsearch-6.5.3\config\elasticsearch.yml`

```
cluster.name: my-application 
network.host: 0.0.0.0
http.port: 9200
```

- 解压elasticSearch-analysis-ik到`D:\elasticsearch\elasticsearch-6.5.3\plugins\ik`

- window直接运行`D:\elasticsearch\elasticsearch-6.5.3\bin\elasticsearch.bat`

#### 启动springboot
- 运行单元测试`com.hcj.essearch.EssearchApplicationTests`


#### 启动kibana
- 解压文件
- 修改配置文件
```jshelllanguage
# kibana.yml
server.port: 5601
server.host: 0.0.0.0
server.name: "your-hostname-Kibana"
elasticsearch.url: "http://localhost:9200"
```
- 运行
```jshelllanguage
D:\elasticsearch\kibana-6.5.3-windows-x86_64\bin\kibana.bat

# 访问
http://localhost:5601/
```

#### 启动logstash
- 解压文件
- 修改配置文件
```jshelllanguage
# 修改文件logstash.yml
node.name: test
http.host: "127.0.0.1"
http.port: 9600
# 开启配置文件自动加载
config.reload.automatic: true
```

```jshelllanguage
# logstash-sample.conf的配置
input {
  beats {
    port => 5044
  }
  jdbc {
        jdbc_connection_string => "jdbc:mysql://172.16.0.191:3306/lbz-office?useUnicode=true&useJDBCCompliantTimezoneShift=true&useLegacyDatetimeCode=false&serverTimezone=UTC"
        jdbc_user => "root"
        jdbc_password => "*****"
        jdbc_driver_library => "D:\elasticsearch\logstash-6.5.3\mysql-connection\mysql-connector-java-8.0.17.jar"
        jdbc_driver_class => "com.mysql.jdbc.Driver"
        jdbc_paging_enabled => "true"
        jdbc_page_size => "50000"
        statement_filepath => "D:\elasticsearch\logstash-6.5.3\mysql-connection\jdbc.sql"
        schedule => "* * * * *"
        type => "jdbc"
    }
}

filter {
    json {
        source => "message"
        remove_field => ["message"]
    }
}

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    #index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    index => "sys_log_db"
    document_type => "sys_log"
    document_id => "%{id}"
    #user => "elastic"
    #password => "changeme"
  }
}
```


```
# D:\elasticsearch\logstash-6.5.3\mysql-connection\jdbc.sql文件内容为
select * from sys_log
```
- 安装插件[https://github.com/logstash-plugins](https://github.com/logstash-plugins)
```jshelllanguage
# 安装 logstash-input-jdbc
logstash-plugin.bat install logstash-input-jdbc
```
- 运行
```jshelllanguage
cd D:\elasticsearch\logstash-6.5.3\bin
 .\logstash -f ..\config\logstash-sample.conf
```


#### 相关截图
![elasticSearch.png](images/elasticSearch.png)
***
![elasticSearch_1.png](images/elasticSearch_1.png)
***
- kibana的api调用，可参考[https://blog.csdn.net/qq_38225558/article/details/85863157](https://blog.csdn.net/qq_38225558/article/details/85863157)
- kibana的api调用，[官方文档](https://www.elastic.co/guide/en/elasticsearch/reference/6.5/search-request-scroll.html)

```jshelllanguage
GET /orders/product/_search
{
  "query": {
    "match_phrase_prefix": {
      "productName": "aa"
    }

  },
  "sort": [
    {
      "createTime": {
        "order": "desc"
      }
    }
  ],
  "from": 0,
  "size": 20,
  "_source": ["productName","productDesc"]
  
}
```
![elasticSearch_kibana1.png](images/elasticSearch_kibana1.png)

****
- `接入logstash+jdbc插件后, 实现数据同步`

```jshelllanguage
http://localhost:8088/query_hit_page?pageNo=1&pageSize=10&keyword=POST&indexName=sys_log_db&fields=operation,exception_msg
```

![elasticSearch_logstash_1.png](images/elasticSearch_logstash_1.png)
![elasticSearch_logstash_2.png](images/elasticSearch_logstash_2.png)