Hadoop常见重要命令行操作及命令作用

Hadoop常见重要命令行操作及命令作用

关于Hadoop

[root@master ~]# hadoop --help

Usage: hadoop [--config confdir] COMMAND

       where COMMAND is one of:

  fs                   run a generic filesystem user client

  version              print the version

  jar <jar>            run a jar file

  checknative [-a|-h]  check native hadoop and compression libraries availability

  distcp <srcurl> <desturl> copy file or directories recursively

  archive -archiveName NAME -p <parent path> <src>* <dest> create a hadoop archive

  classpath            prints the class path needed to get the

                       Hadoop jar and the required libraries

  daemonlog            get/set the log level for each daemon

or

  CLASSNAME            run the class named CLASSNAME

Most commands print help when invoked w/o parameters.查看版本

[root@master ~]# hadoop version

Hadoop 2.2.0.2.0.6.0-101

Subversion git@github.com:hortonworks/hadoop.git -r b07b2906c36defd389c8b5bd22bebc1bead8115b

Compiled by jenkins on 2014-01-09T05:18Z

Compiled with protoc 2.5.0

From source with checksum 704f1e463ebc4fb89353011407e965

This command was run using /usr/lib/hadoop/hadoop-common-2.2.0.2.0.6.0-101.jar运行jar文件

[root@master liguodong]# hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.2.0.2.0.6.0-101.jar pi 10 100 Number of Maps = 10 Samples per Map = 100 Wrote input for Map #0 Wrote input for Map #1 Wrote input for Map #2 ... Job Finished in 19.715 seconds Estimated value of Pi is 3.14800000000000000000检查Hadoop本地库和压缩库的可用性

[root@master liguodong]# hadoop checknative -a

15/06/03 10:28:07 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native

15/06/03 10:28:07 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library

Native library checking:

hadoop: true /usr/lib/hadoop/lib/native/libhadoop.so.1.0.0

zlib:   true /lib64/libz.so.1

snappy: true /usr/lib64/libsnappy.so.1

lz4:    true revision:43

bzip2:  true /lib64/libbz2.so.1文件归档 Archive

hadoop不适合小文件的存储，小文件本身就占用了很多