4 Star 13 Fork 5

kevwan / go-stash

Create your Gitee Account
Explore and code with more than 5 million developers,Free private repositories !:)
Sign up
Clone or download
Cancel
Notice: Creating folder will generate an empty file .keep, because not support in Git
Loading...
readme-en.md

English | 简体中文

go-stash

go-stash is a high performance, free and open source server-side data processing pipeline that ingests data from Kafka, processes it, and then sends it to ElasticSearch.

go-stash is about 4x throughput more than logstash, and easy to deploy, only one executable file.

go-stash

Quick Start

gostash -f etc/config.yaml

config.yaml example as below:

Clusters:
- Input:
    Kafka:
      Name: gostash
      Brokers:
        - "172.16.186.16:19092"
        - "172.16.186.17:19092"
      Topics:
        - k8slog
      Group: pro
      Consumers: 16
  Filters:
    - Action: drop
      Conditions:
        - Key: k8s_container_name
          Value: "-rpc"
          Type: contains
        - Key: level
          Value: info
          Type: match
          Op: and
    - Action: remove_field
      Fields:
        - message
        - _source
        - _type
        - _score
        - _id
        - "@version"
        - topic
        - index
        - beat
        - docker_container
    - Action: transfer
      Field: message
      Target: data
  Output:
    ElasticSearch:
      Hosts:
        - "172.16.141.4:9200"
        - "172.16.141.5:9200"
      # {.event} is the value of the json attribute from input
      # {{yyyy-MM-dd}} means date, like 2020-09-09
      Index: {.event}-{{yyyy-MM-dd}}

Comments ( 0 )

Sign in for post a comment

About

go-stash是一个高效的从Kafka获取,根据配置的规则进行处理,然后发送到ElasticSearch集群的工具 spread retract
Go
MIT
Cancel

Releases

No release

Gitee Metrics

Contributors

All

Activities

load more
can not load any more
Go
1
https://gitee.com/kevwan/go-stash.git
git@gitee.com:kevwan/go-stash.git
kevwan
go-stash
go-stash
master

Search