在eclipse下开发spark程序环境配置

https://blog.csdn.net/lihaitao000/article/details/51461277

soft 软件支持

https://pan.baidu.com/s/1HCmtxxblEOzTBY8WKzdPDg

apark centos集群安装 https://blog.csdn.net/weixin_36394852/article/details/76030317

1. Spark下载 进入到了/usr/local文件夹 wget https://d3kbcqa49mib13.cloudfront.net/spark-2.2.0-bin-hadoop2.7.tgz

Spark安装之前的准备 tar -zxvf spark-2.2.0-bin-hadoop2.7.tgz rm -rf spark-2.2.0-bin-hadoop2.7.tgz mv spark-2.2.0-bin-hadoop2.7 spark-2.2.0

配置环境变量 vi /etc/profile

在最尾巴加入 export SPARK_HOME=/usr/local/spark-2.2.0 export PATH=$PATH:$SPARK_HOME/bin

配置Spark环境 cd spark-2.2.0 spark-env.sh和slaves

首先我们把缓存的文件spark-env.sh.template改为spark识别的文件spark-env.sh cp conf/spark-env.sh.template conf/spark-env.sh

vi conf/spark-env.sh export JAVA_HOME=/usr/java/jdk1.8.0_141 export SCALA_HOME=/usr/scala-2.11.7 export HADOOP_HOME=/usr/local/hadoop-2.7.2 export HADOOP_CONF_DIR=/usr/local/hadoop-2.7.2/etc/hadoop export SPARK_MASTER_IP=SparkMaster export SPARK_WORKER_MEMORY=4g export SPARK_WORKER_CORES=2 export SPARK_WORKER_INSTANCES=1

java 安装 yum -y install java-1.8.0-openjdk* java_home 查找 whereis javac ll /usr/bin/javac ll /etc/alternatives/javac ... /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-3.b10.el6_9.x86_64

scala wget http://www.scala-lang.org/files/archive/scala-2.11.7.tgz tar xvf scala-2.11.7.tgz # vi /etc/profile

hadoop http://www.apache.org/dyn/closer.cgi/hadoop/common 参考 https://blog.csdn.net/cafebar123/article/details/73500014 wget http://www-eu.apache.org/dist/hadoop/common/hadoop-2.7.2/hadoop-2.7.2.tar.gz tar zxvf hadoop-2.7.2.tar.gz 配置hadoop 修改/usr/hadoop/hadoop-2.7.3/etc/hadoop/hadoop-env.sh 文件的java环境，将java安装路径加进去： export JAVA_HOME=/alidata/server/java-1.7.0
配置hadoop环境变量 vi /etc/profile export HADOOP_HOME=/usr/local/hadoop-2.7.2 export PATH=$PATH:$HADOOP_HOME/bin

添加如下参数： export SCALA_HOME=/usr/local/scala-2.11.7 export PATH=$PATH:$SCALA_HOME/bin

export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk.x86_64

 export HADOOP_HOME=/usr/local/hadoop-2.7.2
 export PATH=$PATH:$HADOOP_HOME/bin  

source /etc/profile
scala -version