# hive-hook-plugin-cdh6.3.2

**Repository Path**: ch3un9/hive-hook-plugin-cdh6.3.2

## Basic Information

- **Project Name**: hive-hook-plugin-cdh6.3.2
- **Description**: CDH6.3.2上hive血缘关系
- **Primary Language**: Java
- **License**: Not specified
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 1
- **Created**: 2023-06-10
- **Last Updated**: 2023-06-10

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

## hive-hook-plugin-cdh6.3.2

#### 介绍
CDH6.3.2上hive血缘关系

#### 1、项目编译后，将生成的jar包复制到hive的lib目录下，应该只要放在HS2对应的节点上
hive-hook-plugin-1.0.jar

#### 2、修改hive-site.xml，添加相应配置，在CM上进行修改，修改Hive和HiveServer2 高级配置代码段
```
<property>
  <name>hive.metastore.event.listeners</name>
  <value>org.data.meta.hive.listener.MetaStoreListener</value>
</property>
<property>
  <name>hive.metastore.pre.event.listeners</name>
  <value>org.data.meta.hive.listener.MetaStorePreAuditListener</value>
</property>
<property>
  <name>hive.exec.post.hooks</name>
  <value>org.apache.hadoop.hive.ql.hooks.LineageLogger,org.data.meta.hive.hook.LineageLoggerHook</value>
</property>
```
#### 3、重启Hive服务
#### 4、生成的log会在HS2节点的/data1/log/hive/lineage目录下，每台HS2都会有对应的log
#### 5、生成的血缘json格式如下
```
--测试SQL
INSERT into test.test06 
SELECT a.col as id,
       b.col2 as name
from ods.test0108 a
LEFT JOIN test.test04 b ON a.col = b.col1
where b.col1 = 1
;
```
```
EVENT##U-bdf53c3c-04f3-40af-9a7a-a2177c2922cc##1664349820789##HIVE##LineageHookInfo##{"database":"test","duration":17737,"engine":"mr","hash":"96e2411124dc3373b560b382185afac5","jobIds":["job_1660100089000_0034"],"queryText":"INSERT into test.test06 \r\nSELECT a.col as id,\r\n       b.col2 as name\r\nfrom ods.test0108 a\r\nLEFT JOIN test.test04 b ON a.col \u003d b.col1\r\nwhere b.col1 \u003d 1","timestamp":1664349803,"user":"hive","userGroupNames":["hive"],"version":"1.0","columnLineages":[{"srcDatabase":"ods","destDatabase":"test","edgeType":"PROJECTION","sources":[{"table":"test0108","column":"col"}],"targets":[{"table":"test06","column":"id"}]},{"srcDatabase":"test","destDatabase":"test","edgeType":"PROJECTION","sources":[{"table":"test04","column":"col2"}],"targets":[{"table":"test06","column":"name"}]},{"srcDatabase":"test","destDatabase":"test","expression":"(a.col \u003d b.col1)","edgeType":"PREDICATE","sources":[{"table":"test0108","column":"col"},{"table":"test04","column":"col1"}],"targets":[{"table":"test06","column":"id"},{"table":"test06","column":"name"}]},{"srcDatabase":"test","destDatabase":"test","expression":"(b.col1 \u003d 1)","edgeType":"PREDICATE","sources":[{"table":"test04","column":"col1"}],"targets":[{"table":"test06","column":"id"},{"table":"test06","column":"name"}]}],"tableLineages":[{"srcDatabase":"test","destDatabase":"test","srcTable":"test04","destTable":"test06"},{"srcDatabase":"ods","destDatabase":"test","srcTable":"test0108","destTable":"test06"}]}
```
目前血缘只能统计hive组件，无法统计impala组件运行SQL产生的血缘关系