hadoop 3.1.2 源码编译


编译最好是在 mac 或者 linux 环境下进行,windows 环境编译如下


* Windows System
* JDK 1.8
* Maven 3.0 or later
* ProtocolBuffer 2.5.0
* CMake 3.1 or newer
* Visual Studio 2010 Professional or Higher
* Windows SDK 8.1 (if building CPU rate control for the container executor)
* zlib headers (if building native code bindings for zlib)
* Internet connection for first build (to fetch all Maven and Hadoop dependencies)
* Unix command-line tools from GnuWin32: sh, mkdir, rm, cp, tar, gzip. These
  tools must be present on your PATH.
* Python ( for generation of docs using 'mvn site')

从这个网站下载 windows 下支持执行 git,sh 之类命令的 git-scm http://git-scm.com/downloads

如果用 Visual Studio, 必须是专业版或者更高版本.
不要用 Visual Studio Express. 该版本不支持 64-bit, 编译源码是 64 位的

The Windows SDK 8.1 is available to download at:


不需要 Cygwin.


因为 windows 的最大路径长度的限制,保持源码在一个较短的没有中文字符的路径中,如: C:\hdc

命令文件放置在 dev-support 目录下的 win-paths-eg.cmd.

win-paths-eg.cmd 是为了设置可用环境. 你需要修改这个文件. 它会将所有需要的内容添加到命令行环境中, 按需修改.

Several tests require that the user must have the Create Symbolic Links

All Maven goals are the same as described above with the exception that
native code is built by enabling the ‘native-win’ Maven profile. -Pnative-win
is enabled by default when building on Windows since the native components
are required (not optional) on Windows.

设置环境变量 ZLIB_HOME
set ZLIB_HOME=C:\zlib-1.2.7

zlib1.dll 必须在 PATH 中. Hadoop has been tested
with zlib 1.2.7, built using Visual Studio 2010 out of contrib\vstudio\vc10 in
the zlib 1.2.7 source tree.


Building distributions:

  • Build distribution with native code : mvn package [-Pdist][-Pdocs][-Psrc][-Dtar][-Dmaven.javadoc.skip=true]

Running compatibility checks with checkcompatibility.py

Invoke ./dev-support/bin/checkcompatibility.py to run Java API Compliance Checker
to compare the public Java APIs of two git objects. This can be used by release
managers to compare the compatibility of a previous and current release.

As an example, this invocation will check the compatibility of interfaces annotated as Public or LimitedPrivate:

./dev-support/bin/checkcompatibility.py –annotation org.apache.hadoop.classification.InterfaceAudience.Public –annotation org.apache.hadoop.classification.InterfaceAudience.LimitedPrivate –include “hadoop.*” branch-2.7.2 trunk

Changing the Hadoop version declared returned by VersionInfo

If for compatibility reasons the version of Hadoop has to be declared as a 2.x release in the information returned by
org.apache.hadoop.util.VersionInfo, set the property declared.hadoop.version to the desired version.
For example: mvn package -Pdist -Ddeclared.hadoop.version=2.11

If unset, the project version declared in the POM file is used.

ubuntu 条件下的编译方法


  $ mvn package -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true


  $ mvn package -Pdist,native,docs -DskipTests -Dtar


  $ mvn package -Psrc -DskipTests


  $ mvn package -Pdist,native,docs,src -DskipTests -Dtar

在 /tmp/hadoop-site 目录中创建网页内容

  $ mvn clean site -Preleasedocs; mvn site:stage -DstagingDirectory=/tmp/hadoop-site