hadoop分布式部署

机器列表：
10.200.25.154   主
10.200.4.212        从
10.200.27.126        从
1.建立信用机制,ssh无需密码登录
ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
ssh localhost
------------
2安装好是jdk
------------
3.hadoop配置：
cd /home/zjy/hadoop/etc/hadoop
3.1:core-site.xml:
<property>
          <name>fs.default.name</name>
          <value>hdfs://10.200.25.154:9000</value>
        </property>
        <property>
           <name>hadoop.tmp.dir</name>
           <value>/home/zjy/hadoop/tmp</value>
        </property>

3.2: hdfs-site.xml:
 <property>
         <name>dfs.replication</name>
         <value>1</value>
  </property>
  <property>
   <name>dfs.permissions</name>
   <value>false</value>
  </property>

3.3 mapred-site.xml :
<configuration>  
    <property>  
        <name>mapreduce.jobtracker.address</name>  
        <value>10.200.25.154:9001</value>  
        <description>jobtracker's address</description>  
    </property>  
</configuration>
----------------
清单配置成下面这种形式：
etc/hadoop/mapred-site.xml
<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
</configuration>

 清单 4.Yarn-site.xml 配置：   

 <!-- Site specific YARN configuration properties --> 
  <property> 
  <name>Yarn.nodemanager.aux-services</name> 
  <value>mapreduce.shuffle</value> 
  </property> 
  <property> 
  <description>The address of the applications manager interface in the RM.</description> 
  <name>Yarn.resourcemanager.address</name> 
  <value>10.200.25.154:18040</value> 
  </property> 

  <property> 
  <description>The address of the scheduler interface.</description> 
  <name>Yarn.resourcemanager.scheduler.address</name> 
  <value>10.200.25.154:18030</value> 
  </property> 

  <property> 
  <description>The address of the RM web application.</description> 
  <name>Yarn.resourcemanager.webapp.address</name> 
  <value>10.200.25.154:18088</value> 
  </property> 

  <property> 
  <description>The address of the resource tracker interface.</description> 
  <name>Yarn.resourcemanager.resource-tracker.address</name> 
  <value>10.200.25.154:9025</value> 
  </property> 

------
masters 文件配置
10.200.25.154
slaves 文件配置：
10.200.4.212
10.200.27.126 

-------

每台机器的/etc/hosts添加设置：
###hadoop host list#
10.200.25.154 YFCS-S6-APP
10.200.4.212    BOSS20CS-DB
10.200.27.126   NGCRM-S2-APP

-----------------
4.环境变更配置：vi ~/.profile            vi ~/.bashsrc

export JAVA_HOME=/opt/jdk1.8.0_25
export JRE_HOME=/opt/jdk1.8.0_25/jre
export HADOOP_HOME=/hadoop/hadoop
export HADOOP_HOME_WARN_SUPPRESS=1
export CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib:$HADOOP_HOME/lib:$CLASSPATH
export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$HADOOP_HOME/bin:$PATH
export HADOOP_DEV_HOME=/hadoop/hadoop
export HADOOP_COMMON_HOME=/hadoop/hadoop
export HADOOP_HDFS_HOME=/hadoop/hadoop
export HADOOP_CONF_DIR=/hadoop/hadoop/etc/hadoop
export HADOOP_PREFIX=/home/zjy/hadoop
export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_PREFIX}/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_PREFIX/lib"
-----------------------
/*find ./ -depth -name "*"|xargs chown zjy 
find ./ -depth -name "*"|xargs chgrp hadoop 
*/
------------
5。执行格式化：
hadoop namenode -format
注意：core-site.xml、hdfs-site.xml、mapred-site.xml等配置文件变化后都需要执行格式化
------------
6。启动：
cd $HADOOP_HOME/sbin
./start-all.sh
---------------
7,检查：
查看hadoop启动的进程
[root@centoshadoop1~]#jps
查看hadoop集群状态
[root@centoshadoop1~]#hadoop dfsadmin  -report
查看hadoop中的目录情况
[root@centoshadoop1~]#hadoop fs –ls /

现在就可以在浏览器中输入地址：
http://10.200.25.154:50030
http://10.200.25.154:50070
完成。
查看yarn是否启动成功
http://10.200.25.154:8088/cluster

查看nodemanager 是否启动成功
http://10.200.25.154:8042/

//设置与修改日志级别

命令方式：bin/hadoop daemonlog -setlevel ${nodemanager-host}:8042 org.apache.hadoop.yarn.server.nodemanager.NodeManager DEBUG
--网页形式：http://10.200.25.154:8042/logLevel

Log: “org.apache.hadoop.yarn.server.nodemanager.NodeManager"  Level:  "DEBUG" 点击"Set Log Level"
--修改配置文件：/home/zjy/hadoop/etc/hadoop/log4j.properties 
添加 log4j.logger.org.apache.hadoop.yarn.server.nodemanager.NodeManager=DEBUG
----------------------------------------------------------------------------------------------------

hadoop dfsadmin -safemode leave        离开安全模式。

10.200.27.126 root/126!scsb
10.200.4.212  root/212w#bsc

10.200.25.154 YFCS-S6-APP
10.200.4.212        BOSS20CS-DB
10.200.27.126        NGCRM-S2-APP
172.20.23.101 ycs
172.20.23.130 zjy        

10.200.4.212上操作：指令：
vi /etc/hosts
172.20.23.130   master
10.200.25.154   YFCS-S6-APP
---
useradd  -c "zjy-hadoop-test" zjy
passwd zjy

usermod -G mysql,oinstall,sshd,ftp,www,tester,dialout,video zjy
mkdir /home/zjy
chown zjy /home/zjy
chgrp tester zjy

usermod -G oinstall,sshd,ftp,www,dialout,video zjy

10.200.27.126:
------------------
export PATH=:$PATH:$WAS_HOME/bin:$JAVA_HOME/bin:$ANT_HOME/bin:$FINDBUGS_HOME/bin:/bin:/usr/bin:.
export LANG="zh_CN.UTF-8"
export LANGUAGE="zh_CN:zh_GB:zh"
export LC_CTYPE="zh_CN.GBK"
export LD_LIBRARY_PATH=/opt/gnome/lib64:/opt/gnome/lib/

###hadoop########

export JAVA_HOME=/opt/jdk1.7.0_45
export JRE_HOME=/opt/jdk1.7.0_45/jre
export HADOOP_HOME=/home/zjy/hadoopp
export HADOOP_HOME_WARN_SUPPRESS=1
export CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib:$HADOOP_HOME/lib:/usr/java/jdk1.7.0_25/lib/tools.jar:/usr/java/jdk1.7.0_25/lib/dt.jar:/usr/java/jdk1.7.0_25/jre/lib/rt.jar:$CLASSPATH
export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$HADOOP_HOME/bin:$PATH
export HADOOP_DEV_HOME=/home/zjy/hadoop
export HADOOP_COMMON_HOME=/home/zjy/hadoop
export HADOOP_HDFS_HOME=/home/zjy/hadoop
export HADOOP_CONF_DIR=/home/zjy/hadoop/etc/hadoop
alias p='ps -ef |grep -v grep |grep '
export PS1='$LOGNAME@`hostname`:`pwd`$'
hostname=master
alias l='ls -lrt'
alias L='ls -lrt'
set -o vi

172.20.23.101 ysc:
-------------------------------
export JAVA_HOME=/usr/local/jdk1.6.0_38
export JRE_HOME=$JAVA_HOME/jre
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib
export HADOOP_PREFIX=/usr/local/hadoop
export PATH=.:$PATH
export HADOOP_HOME=/usr/local/hadoop
export HADOOP_HOME_WARN_SUPPRESS=1
export CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib:$HADOOP_HOME/lib:$CLASSPATH
export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$HADOOP_HOME/bin:$PATH
export HADOOP_DEV_HOME=/usr/local/hadoop
export HADOOP_COMMON_HOME=/usr/local/hadoop
export HADOOP_HDFS_HOME=/usr/local/hadoop
export HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop

alias l='ls -lrt'
alias p='ps -ef|grep -v grep |grep '

scp 10.200.25.154:/home/zjy/.ssh/id_dsa.pub ./id_dsa.pub_154
cat id_dsa.pub_154 >>.ssh/authorized_keys

scp mapred-site.xml 10.200.4.212:/home/zjy/hadoop/etc/hadoop/mapred-site.xml
scp mapred-site.xml 10.200.27.126:/home/zjy/hadoop/etc/hadoop/mapred-site.xml
scp mapred-site.xml 172.20.23.101:/home/zjy/hadoop/etc/hadoop/mapred-site.xml

scp yarn-site.xml 10.200.4.212:/home/zjy/hadoop/etc/hadoop/yarn-site.xml
scp yarn-site.xml 10.200.27.126:/home/zjy/hadoop/etc/hadoop/yarn-site.xml
scp yarn-site.xml 172.20.23.101:/home/zjy/hadoop/etc/hadoop/yarn-site.xml

scp hdfs-site.xml 10.200.4.212:/home/zjy/hadoop/etc/hadoop/hdfs-site.xml
scp hdfs-site.xml 10.200.27.126:/home/zjy/hadoop/etc/hadoop/hdfs-site.xml

yarn-site.xml
---------------------

启停：

# MR(Resource Manager, Node Manager & Job History Server).部分：
$ sbin/yarn-daemon.sh start resourcemanager       //启动resourcenamager
$ sbin/yarn-daemon.sh start nodemanager                                                //启动nodemanager
$ sbin/mr-jobhistory-daemon.sh start historyserver 

Hadoop Startup
To start a Hadoop cluster you will need to start both the HDFS and YARN cluster.

Format a new distributed filesystem:

$ $HADOOP_PREFIX/bin/hdfs namenode -format <cluster_name>Start     //格式化文件系统
the HDFS with the following command, run on the designated NameNode:

$ $HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs start namenode  启动namenode
Run a script to start DataNodes on all slaves:

$ $HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs start datanode  启动dataname
Start the YARN with the following command, run on the designated ResourceManager:

$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR start resourcemanager                         启动 resourcemanager 

----        在slaves上执行的命令：
Run a script to start NodeManagers on all slaves:

$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR start nodemanager  //启动所有 nodemanager
Start a standalone WebAppProxy server. If multiple servers are used with load balancing it should be run on each of them:

$ $HADOOP_YARN_HOME/bin/yarn start proxyserver --config $HADOOP_CONF_DIRS                                                //启动proxyserver
tart the MapReduce JobHistory Server with the following command, run on the designated server:

$ $HADOOP_PREFIX/sbin/mr-jobhistory-daemon.sh start historyserver --config $HADOOP_CONF_DIR

---------------------
Hadoop Shutdown
Stop the NameNode with the following command, run on the designated NameNode:

$ $HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs stop 
namenodeRun a script to stop DataNodes on all slaves:

$ $HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs stop datanode
Stop the ResourceManager with the following command, run on the designated ResourceManager:

$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR stop resourcemanager
Run a script to stop NodeManagers on all slaves:

$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR stop nodemanager
Stop the WebAppProxy server. If multiple servers are used with load balancing it should be run on each of them:

$ $HADOOP_YARN_HOME/bin/yarn stop proxyserver --config $HADOOP_CONF_DIR
Stop the MapReduce JobHistory Server with the following command, run on the designated server:

$ $HADOOP_PREFIX/sbin/mr-jobhistory-daemon.sh stop historyserver --config $HADOOP_CONF_DIR

-----------
运行自带的例子：

hdfs dfs -mkdir /input        // 创建文件目录
hdfs dfs -mkdir /output                                //

hadoop fs -ls /                        //查看文件
hadoop dfs -put /home/zjy/hadoop/tmp/file1 /input   //上传文件
hadoop dfs -get /input/file1 ./                                                                                                                //下载文件

zjy@YFCS-S6-APP:/home/zjy/hadoop/tmp/input$more file1
hello hadoop test start
--
hadoop dfs -ls /input/file1
-rw-r--r--   3 zjy supergroup         24 2014-11-07 18:25 /input/file1

hadoop jar /home/zjy/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.4.1.jar wordcount /input /output
hadoop jar /home/zjy/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.4.1.jar wordcount /input /output
hadoop jar /home/zjy/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.4.1-tests.jar wordcount /input /output

hadoop jar /home/zjy/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.4.1.jar wordcount /input /output

hadoop fs -cat hdfs://127.0.0.1:9000/output

hadoop fs -rmdir hdfs://127.0.0.1:9000/output

hadoop fs -ls hdfs://127.0.0.1:9000/output

ps -ef |grep -v grep |grep hadoop|awk '{print $2}'|xargs kill -9