# flinkStudy

**Repository Path**: febbird/flink-study

## Basic Information

- **Project Name**: flinkStudy
- **Description**: No description available
- **Primary Language**: Java
- **License**: Not specified
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 3
- **Created**: 2023-11-03
- **Last Updated**: 2023-11-03

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# flinkStudy

#### 介绍
学习Flink的一点点积累，主要是看b站上尚硅谷视频+Flink官方文档+Flink中文邮件列表学习

需求是用纯SQL的方式从kafka获取json格式的数据，通过flink处理后实时输出到hive中去，并且需要提交到远程Flink去执行，可以手动在项目中取消job任务等

（hadoop和hive都需要经过kerberos认证）

#### 软件架构
java:1.8
flink-1.13.6-bin-scala_2.11
hive:1.1.1
hadoop:2.6.0-cdh5.16.2
kafka:2.7.0

#### 使用说明

flink-sql-connector-hive-1.2.2_2.11-1.13.6.jar包没办法通过pom依赖引入，需要到https://nightlies.apache.org/flink/flink-docs-release-1.13/zh/docs/connectors/table/hive/overview/页面去下载，然后添加到运行代码项目的lib中去


flink集群的lib包中需要自己添加所需的jar包，可以参照下面的

flink-connector-jdbc_2.11-1.13.6.jar            hadoop-annotations-2.6.0-cdh5.16.2.jar                 hive-common-1.1.1.jar
flink-connector-kafka_2.11-1.13.6.jar           hadoop-auth-2.6.0-cdh5.16.2.jar                        hive-exec-1.1.1.jar
flink-csv-1.13.6.jar                            hadoop-aws-2.6.0-cdh5.16.2.jar                         hive-jdbc-1.1.1.jar
flink-dist_2.11-1.13.6.jar                      hadoop-client-2.6.0-cdh5.16.2.jar                      hive-metastore-1.1.1.jar
flink-json-1.13.6.jar                           hadoop-common-2.6.0-cdh5.16.2.jar                      kafka-clients-2.7.0.jar
flink-shaded-hadoop-2-uber-2.8.3-10.0.jar       hadoop-hdfs-2.6.0-cdh5.16.2.jar                        libfb303-0.9.3.jar
flink-shaded-zookeeper-3.4.14.jar               hadoop-mapreduce-client-app-2.6.0-cdh5.16.2.jar        log4j-1.2-api-2.17.1.jar
flink-sql-connector-hive-1.2.2_2.11-1.13.6.jar  hadoop-mapreduce-client-common-2.6.0-cdh5.16.2.jar     log4j-api-2.17.1.jar
flink-sql-connector-kafka_2.11-1.13.6.jar       hadoop-mapreduce-client-core-2.6.0-cdh5.16.2.jar       log4j-core-2.17.1.jar
flink-table_2.11-1.13.6.jar                     hadoop-mapreduce-client-jobclient-2.6.0-cdh5.16.2.jar  log4j-slf4j-impl-2.17.1.jar
flink-table-blink_2.11-1.13.6.jar               hadoop-mapreduce-client-shuffle-2.6.0-cdh5.16.2.jar    mysql-connector-java-5.1.48.jar