2 Unstar Star 1 Fork 2

opensourceway / om-bigdata

Create your Gitee Account
Explore and code with more than 5 million developers,Free private repositories !:)
Sign up
This repository doesn't specify license. Without author's permission, this code is only for learning and cannot be used for other purposes.
New framework of om collector spread retract

Clone or download
Cancel
Notice: Creating folder will generate an empty file .keep, because not support in Git
Loading...
README.md

om-bigData

介绍

以下是om-bigData介绍

om-bigData是针对om-collect工程收集的数据进行批量处理或实时处理的大数据处理框架。

软件架构

  • spark

  • kafka

  • storm

  • elasticsearch

安装教程

  1. 克隆工程

    git clone https://gitee.com/opensourceway/om-bigdata.git

  2. 修改打包文件assembly.xml

    • 如果是spark程序,则将pom.xml标签中内容修改为com.huawei.opensourway.AnalysisForBatch
    • 如果是storm程序,则修改成com.huawei.opensourway.AnalysisForStreaming
  3. 使用mvn命令编译

    mvn package (编译.java)

    mvn assembly:single (执行之后会在target目录下生成对应的jar包,如om-bigdata-1.0-SNAPSHOT.jar)

使用说明

  1. spark程序,上传jar包到集群master节点/root目录,执行 which spark-submit,找到脚本路径,并切换到对应路径下执行如下命令:

    ./spark-submit --class com.huawei.om.AnalysisForBatch --num-executors 1 --executor-cores 4 --master yarn-cluster /root/om-analysis-1.0-SNAPSHOT.jar ak sk 4 inputfilepath vhost indexname (如果配置项如果不存在,则使用conf.properties中对应属性)

  2. storm程序

    将打包文件上传到storm所在节点/root目录,执行 which storm 并切换到对应目录,执行如下命令:

    ./storm jar /root/stormkafkagitee-1.0-SNAPSHOT.jar com.huawei.opensourway.AnalysisForStreaming

Comments ( 0 )

Sign in for post a comment

1
https://gitee.com/opensourceway/om-bigdata.git
git@gitee.com:opensourceway/om-bigdata.git
opensourceway
om-bigdata
om-bigdata
master

Search

132457 8cb2edc1 1899542 131848 70c8d3a4 1899542