1 Star 0 Fork 340

黑鸭子 / PiFlow

加入 Gitee
与超过 1200万 开发者一起发现、参与优秀开源项目,私有仓库也完全免费 :)
免费加入
克隆/下载
PiFlow_V0.7_Deployment_Instructions.md 5.48 KB
一键复制 编辑 原始数据 按行查看 历史

1.Piflow Server

1.1 环境要求

1.已部署Spark2,Hadoop,Yarn,Hive(可选)

2.JDK1.8

3.scala 2.11.8

1.2 Release说明

https://github.com/cas-bigdatalab/piflow/releases/tag/v0.7

1.3安装文件

1.3.1 下载安装文件:

https://github.com/cas-bigdatalab/piflow/releases/download/v0.7/piflow-server-v0.7.tar.gz

将piflow-server-v0.7.tar.gz解压,如下图所示:

  1. bin为PiFlow命令行工具;

  2. classpath为用户自定开发组件Stop放置路径;

  3. config.properties为配置文件;

  4. lib为piflowServer所需jar包;piflow-server-0.9.jar为piflowServer本身jar

  5. logs为PiFlow日志目录

  6. start.sh、restart.sh、stop.sh、status.sh为piflowServer启动停止脚本。

1.3.2 Config.properties配置文件如下;

#Spark master and deploy mode
spark.master=yarn
spark.deploy.mode=cluster

#hdfs default file system
fs.defaultFS=hdfs://10.0.88.23:9000

#yarn resourcemanager.hostname
yarn.resourcemanager.hostname=10.0.88.23

#if you want to use hive, set hive metastore uris
hive.metastore.uris=thrift://10.0.88.23:9083

#show data in log, set 0 if you do not want to show data in logs
data.show=10

#server port
server.port=8002

#h2db port
h2.port=50002

1.3 环境配置

配置集群的环境变量(自定义按需配置)

export JAVA_HOME=/opt/java
export JRE_HOME=/opt/java/jre
export CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib:$CLASSPATH
export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATH

export HADOOP_HOME=/opt/hadoop-2.6.0
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

export HIVE_HOME=/opt/apache-hive-2.3.6-bin
export PATH=$PATH:$HIVE_HOME/bin

export SPARK_HOME=/opt/spark-2.1.0-bin-hadoop2.6
export PATH=$PATH:$SPARK_HOME/bin

export SCALA_HOME=/opt/scala-2.11.8
export PATH=$PATH:$SCALA_HOME/bin

1.4 运行

  ./start.sh   

2.Piflow Web

1.1 环境要求

  1. MYSQL5.7

  2. JDK1.8

1.2 项目部署

下载安装文件:

https://github.com/cas-bigdatalab/piflow-web/releases/download/v0.7/piflow-web-v0.7.tar.gz

将piflow-web-v0.7.tar.gz解压,如下图所示:

解压后内容说明:

(1)、piflow-web.jar 为 piflow-web的启动jar包。

(2)、config.properties为配置文件。

(3)、srart.sh 为启动脚本

(4)、stop.sh 为停止脚本

(5)、status.sh 为查看状态脚本

(6)、restart.sh 为重启脚本

(7)、piflowWeb.log 为日志文件

(8)、temp_v0.7.sh 为平滑升级的补丁脚本(不需要执行)

(9)、error.log 为平滑升级的补丁脚本的日志文件

config.properties配置文件如下;

如果想使用MySQL数据库则使用这份配置文件

server.port=6001
server.servlet.session.timeout=3600

syspara.interfaceUrlHead=http://10.0.86.191:8002
syspara.isIframe=true

# data source
sysParam.datasource.type=mysql
# MySQL Configuration
#Configure the connection address of MySQL
spring.datasource.url = jdbc:mysql://10.0.88.109:3306/piflow_web?useUnicode=true&characterEncoding=UTF-8&useSSL=false&allowMultiQueries=true&autoReconnect=true&failOverReadOnly=false
#Configure database user name
spring.datasource.username=root
#Configuration database password
spring.datasource.password=123456
#Configure JDBC Driver
# Can not be configured, according to the URL automatic identification, recommended configuration
spring.datasource.driver-class-name=com.mysql.jdbc.Driver

spring.flyway.locations=classpath:db/flyway-mysql/

# Log Coordination Standard
logging.level.org.flywaydb=debug
logging.level.com.nature.mapper=debug
logging.level.root=info
logging.level.org.springframework.security=info
logging.level.org.hibernate.SQL=DEBUG

如果想使用H2DB数据库则使用这份配置文件


server.port=6001
server.servlet.session.timeout=3600

syspara.interfaceUrlHead=http://10.0.86.191:8002
syspara.isIframe=true

# data source
sysParam.datasource.type=h2
# h2 Configuration
#Configure the connection address of H2DB
spring.datasource.url=jdbc:h2:file:/media/nature/linux_disk_0/PiFlow_DB/piflow_web
#Configure database user name
spring.datasource.username=Admin
#Configuration database password
spring.datasource.password=Admin
#Configure JDBC Driver
# Can not be configured, according to the URL automatic identification, recommended configuration
spring.datasource.driver-class-name=org.h2.Driver
#H2DB web console settings
spring.datasource.platform=h2
#After this configuration, h2 web consloe can be accessed remotely. Otherwise it can only be accessed locally.
spring.h2.console.settings.web-allow-others=true
#With this configuration, you can access h2 web consloe through YOUR_URL / h2. YOUR_URL is the access URL of your program.
spring.h2.console.path=/h2
#With this configuration, h2 web consloe will start when the program starts. Of course this is the default. If you #don't want to start h2 web consloe when you start the program, then set it to false.
spring.h2.console.enabled=true

spring.flyway.locations=classpath:db/flyway-h2db/

# Log Coordination Standard
logging.level.org.flywaydb=debug
logging.level.com.nature.mapper=debug
logging.level.root=info
logging.level.org.springframework.security=info
logging.level.org.hibernate.SQL=DEBUG

运行

  cd piflow-web 
  ./start.sh  

访问进行登陆注册:http://serverIp:serverPort/piflow-web/

Scala
1
https://gitee.com/Lhx11187/piflow.git
git@gitee.com:Lhx11187/piflow.git
Lhx11187
piflow
PiFlow
master

搜索帮助