Watch 1 Star 0 Fork 0

youlixishi / hadoop-demoJava

Create your Gitee Account
Explore and code with more than 5 million developers,Free private repositories !:)
Sign up
This repository doesn't specify license. Without author's permission, this code is only for learning and cannot be used for other purposes.
hadoop相关demo spread retract

Clone or download
install-hadoop.txt 6.98 KB
Copy Edit Web IDE Raw Blame History
youlixishi authored 2017-12-04 00:45 . Upload install-hadoop.txt
http://blog.csdn.net/xu470438000/article/details/50512442
6.
docker run -it -d --name hadoop-master --hostname hadoop-master -P -p 50070:50070 -p 8088:8088 -v /sofware_shared/:/sofware_shared/ --privileged=true francis/hadoop:0.0.1
docker run -it -d --name hadoop1 --hostname hadoop1 -P -v /sofware_shared/:/sofware_shared/ --privileged=true francis/hadoop:0.0.1
docker run -it -d --name hadoop2 --hostname hadoop2 -P -v /sofware_shared/:/sofware_shared/ --privileged=true francis/hadoop:0.0.1
docker run -it -d --name hadoop3 --hostname hadoop3 -P -v /sofware_shared/:/sofware_shared/ --privileged=true francis/hadoop:0.0.1
1. 修改静态的hostname 永久的
vi /etc/hostname
2. 修改动态的hostname, 暂时的, 关机则失效
hostname xxxx
3. 添加java 环境变量
export JAVA_HOME=/usr/local/jdk1.8.0_144
export PATH=$JAVA_HOME/bin:$PATH
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
4. 添加docker 主机
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a8e03a018d9b registry.cn-hangzhou.aliyuncs.com/acc/base-centos7 "/bin/bash" 14 minutes ago Up 14 minutes 0.0.0.0:10003->10000/tcp hadoop3
954158d5ec4e registry.cn-hangzhou.aliyuncs.com/acc/base-centos7 "/bin/bash" 14 minutes ago Up 14 minutes 0.0.0.0:10002->10000/tcp hadoop2
ba18a2489b14 registry.cn-hangzhou.aliyuncs.com/acc/base-centos7 "/bin/bash" 15 minutes ago Up 15 minutes 0.0.0.0:10001->10000/tcp hadoop1
cbdc0180799a registry.cn-hangzhou.aliyuncs.com/acc/base-centos7 "/bin/bash" 15 minutes ago Up 15 minutes 0.0.0.0:10000->10000/tcp hadoop-master
5. 分别修改hostname为docker 名字
7.
vi /etc/hosts
172.17.0.3 hadoop1
172.17.0.4 hadoop2
172.17.0.5 hadoop3
172.17.0.2 hadoop-master
8.
yum install openssh*
/etc/init.d/sshd start
9.
cd ~
mkdir .ssh
cd .ssh
ssh-keygen -t rsa
master:
ssh-copy-id -i localhost
ssh-copy-id -i hadoop-master
ssh-copy-id -i hadoop1
ssh-copy-id -i hadoop2
ssh-copy-id -i hadoop3
slave1
cd ~
cd .ssh
ssh-copy-id -i localhost
ssh-copy-id -i hadoop1
slave2
cd ~
mkdir .ssh
cd .ssh
ssh-keygen -t rsa
ssh-copy-id -i localhost
ssh-copy-id -i hadoop2
slave3
cd ~
mkdir .ssh
cd .ssh
ssh-keygen -t rsa
ssh-copy-id -i localhost
ssh-copy-id -i hadoop3
10.
mkdir -p /home/hadoop/tmp
mkdir -p /home/hadoop/hdfs/data
mkdir -p /home/hadoop/hdfs/name
11.
core-site.xml
<property>
<name>fs.defaultFS</name>
<value>hdfs://hadoop-master:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>file:/home/hadoop/tmp</value>
</property>
<property>
<name>io.file.buffer.size</name>
<value>131702</value>
</property>
hdfs-site.xml
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/home/hadoop/dfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/home/hadoop/dfs/data</value>
</property>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>hadoop-master:9001</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
mapred-site.xml
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>hadoop-master:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>hadoop-master:19888</value>
</property>
yarn-site.xml
<property>
<name>yarn.resourcemanager.hostname</name>
<value>heres04</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.auxservices.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>hadoop-master:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>hadoop-master:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>hadoop-master:8031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>hadoop-master:8033</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>hadoop-master:8088</value>
</property>
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>768</value>
</property>
<property>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>2048</value>
</property>
<property>
<name>yarn.nodemanager.vmem-pmem-ratio</name>
<value>2.1</value>
</property>
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<property>
<name>yarn.log-aggregation.retain-seconds</name>
<value>604800</value>
</property>
12. vi hadoop-env.sh
export JAVA_HOME=/usr/local/jdk1.8.0_144/
vi yarn-env.sh
export JAVA_HOME=/usr/local/jdk1.8.0_144/
13. slaves
删除默认的localhost,增加新的从节点
14. 将配置好的hadoop 目录传送到 从节点
scp -r /usr/local/hadoop-2.9.0 hadoop1:/usr/local/
scp -r /usr/local/hadoop-2.9.0 hadoop2:/usr/local/
scp -r /usr/local/hadoop-2.9.0 hadoop3:/usr/local/
15.
yum install -y which
16. master
(1)初始化,输入命令,bin/hdfs namenode -format
(2)全部启动sbin/start-all.sh,也可以分开sbin/start-dfs.sh、sbin/start-yarn.sh
(3)停止的话,输入命令,sbin/stop-all.sh
(4)输入命令,jps,可以看到相关信息
17.
cd ~/.ssh/
ssh-copy-id -i localhost
ssh-copy-id -i hadoop-master
18.
创建 demo
vi a.txt
hello you
hello me
放入hdfs
/usr/local/hadoop-2.9.0/bin/hdfs dfs -put /tmp/a.txt /
/usr/local/hadoop-2.9.0/bin/hadoop jar /usr/local/hadoop-2.9.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.9.0.jar wordcount /a.txt /out
/usr/local/hadoop-2.9.0/bin/hadoop jar hadoop-mapreduce-examples-2.9.0.jar wordcount /a.txt /out
/usr/local/hadoop-2.9.0/bin/hadoop jar hadoop-mapreduce-examples-2.9.0.jar wordcount / /out
19. 查看结果
/usr/local/hadoop-2.9.0/bin/hdfs -text /out/part-r-0000
vi /usr/local/hadoop-2.9.0/etc/hadoop/hadoop-env.sh
# export HADOOP_OPTS="-Djava.net.preferIPv4Stack=true -Dsun.security.krb5.debug=true -Dsun.security.spnego.debug"
-Xmx2000m
20.
http://hadoop-master:8088/cluster/nodes
http://hadoop-master:50070/explorer.html#/
21.
查看文件
/usr/local/hadoop-2.9.0/bin/hadoop fs -text /out/part-r-00000
上传文件
/usr/local/hadoop-2.9.0/bin/hdfs dfs -put /tmp/a.txt /

Comment ( 0 )

Sign in for post a comment

Java
1
https://gitee.com/youlixishi/hadoop-demo.git
git@gitee.com:youlixishi/hadoop-demo.git
youlixishi
hadoop-demo
hadoop-demo
master

Search