# flume-dev

**Repository Path**: alexancy/flume-dev

## Basic Information

- **Project Name**: flume-dev
- **Description**: 定制化开发flume的相关组件
- **Primary Language**: Java
- **License**: Not specified
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 1
- **Created**: 2018-06-07
- **Last Updated**: 2020-12-19

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# flume-dev

#### 项目介绍
定制化开发flume的相关组件


### 说明
实现准实时的检测的文件 并且可以实现的断点续传
如果使用的flume 1.7.0 可以使用的 [Taildir Source](http://flume.apache.org/FlumeUserGuide.html#taildir-source)

## conf 配置文件
``` 
## 实现flume 断点续传的source的测试配置文件
a1.sources = r1
a1.sinks = k1
a1.channels = c1

# Describe/configure the source
a1.sources.r1.type =org.yonggan.flume.TailFileSource
a1.sources.r1.filePath =/usr/local/nginx/logs/access.log
a1.sources.r1.offsetFile =/cloud/flume/offset
a1.sources.r1.interval =1000
a1.sources.r1.charset =utf-8

# Describe the sink
a1.sinks.k1.type = logger

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
```


flume 1.7.0 Taildir Source 配置文件conf

```
a1.sources = r1
a1.channels = c1
a1.sources.r1.type = TAILDIR
a1.sources.r1.channels = c1
a1.sources.r1.positionFile = /var/log/flume/taildir_position.json
a1.sources.r1.filegroups = f1 f2
a1.sources.r1.filegroups.f1 = /var/log/test1/example.log
a1.sources.r1.headers.f1.headerKey1 = value1
a1.sources.r1.filegroups.f2 = /var/log/test2/.*log.*
a1.sources.r1.headers.f2.headerKey1 = value2
a1.sources.r1.headers.f2.headerKey2 = value2-2
a1.sources.r1.fileHeader = true

```
Taildir Source 存在的问题

1.  不支持window 原因就是： flume 使用json文件记录偏移量 使用的linux 的inode来唯一标识文件
2.  不支持多级文件监控
需要优化Source 源