Praveen's blog

Maven repository refers to ~/.m2/repository, whereas Maven repository manager refers to an artifact repository manager like Apache Archiva or Artifactory.

Installing an artifact is installing it in the Maven repository, whereas deploying an artifact means publishing the artifact in a Maven repository manager. For more information, please refer to Maven reference.

Installing artifacts refers to installing them in Maven repository, whereas installing binaries refers to installing the entire binary distribution on the cluster.

Checkout Hadoop common from 0.20-append branch.

$ svn co http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-append/ hadoop-common-0.20-append

Hadoop uses Apache Ant as a build tool. In order to build hadoop-common, you need to create a hadoop-common-0.20-append/build.properties file that looks something like this.

resolvers=internal
version=0.20-append-r1057313-yourversion
project.version=${version}
hadoop.version=${version}
hadoop-core.version=${version}
hadoop-hdfs.version=${version}
hadoop-mapred.version=${version}

Note that at the time of creation of this article, the latest revision that was available in 0.20-append branch was r1057313.

Also, try to assign a meaningful suffix in place of yourversion so that you can distinguish between the official artifacts and the artifacts that are deployed by you.

Please follow this step only if you are running a Maven repository manager for team collaboration and you want to deploy the Hadoop common artifacts to that repository manager.

Edit hadoop-common-0.20.append/build.xml and add two new targets.

<target name="mvn-deploy-internal" depends="mvn-taskdef, bin-package, set-version, simpledeploy-internal"
   description="To deploy hadoop core and test jar's to apache maven repository"/>

<target name="simpledeploy-internal" unless="staging">
   <artifact:pom file="${hadoop-core.pom}" id="hadoop.core"/>
   <artifact:pom file="${hadoop-test.pom}" id="hadoop.test"/>
   <artifact:pom file="${hadoop-examples.pom}" id="hadoop.examples"/>
   <artifact:pom file="${hadoop-tools.pom}" id="hadoop.tools"/>
   <artifact:pom file="${hadoop-streaming.pom}" id="hadoop.streaming"/>

   <artifact:install-provider artifactId="wagon-http" version="${wagon-http.version}"/>
   <artifact:deploy file="${hadoop-core.jar}">
       <remoteRepository id="yourrepo.internal" url="http://yourreposerver.com:port/path"/>
       <pom refid="hadoop.core"/>
   </artifact:deploy>
   <artifact:deploy file="${hadoop-test.jar}">
       <remoteRepository id="yourrepo.internal" url="http://yourreposerver.com:port/path"/>
       <pom refid="hadoop.test"/>
   </artifact:deploy> 
   <artifact:deploy file="${hadoop-examples.jar}">
       <remoteRepository id="yourrepo.internal" url="http://yourreposerver.com:port/path"/>
       <pom refid="hadoop.examples"/>
   </artifact:deploy>
   <artifact:deploy file="${hadoop-tools.jar}">
       <remoteRepository id="yourrepo.internal" url="http://yourreposerver.com:port/path"/>
       <pom refid="hadoop.tools"/>
   </artifact:deploy>
   <artifact:deploy file="${hadoop-streaming.jar}">
       <remoteRepository id="yourrepo.internal" url="http://yourreposerver.com:port/path"/>
       <pom refid="hadoop.streaming"/>
   </artifact:deploy>
</target>

Note that yourrepo.internal is the same ID that you have configured authentication for in the ~m2/settings.xml file earlier.

Now, build and install/deploy Hadoop common artifacts using the Maven ant tasks.

You need to generate a binary tarball to install on the cluster. This is achieved by running tar target.

$ ant tar -Djava5.home=<Java 5 SE Home> -Dforrest.home=<Forrest 0.8 Home>

Please note that you need Java SE/EE 5 and Apache Forrest 0.8 for this step. Substituting Java SE/EE 5 or Apache Forrest 0.9 will result in a build failure.

This will generate the hadoop-common-0.20-append/build/hadoop-common-0.20-append-r1057313-yourversion.tar.gz tarball.

Copy the tarball that was generated in the previous step to your cluster, and unpack them in desired location.

This ensures that you have a consistent Hadoop installation because you are not mixing and matching artifacts from a Hadoop common official release and artifacts that you built.

Checkout HBase from 0.90.3 tag.

$ svn co http://svn.apache.org/repos/asf/hbase/tags/0.90.3 hbase-0.90.3

Now, edit hbase-0.90.3/pom.xml and modify HBase and Hadoop versions.

...
<groupId>org.apache.hbase</groupId>
<artifactId>hbase</artifactId>
<packaging>jar</packaging>
<version>0.90.3-yourversion</version>
...
 <hadoop.version>0.20-append-r1057313-yourversion</hadoop.version>
...

Note that you should be using the same revision number for Hadoop that you have assigned while building Hadoop.

Also, try to assign a meaningful suffix in place of yourversion so that you can distinguish between the official artifacts and the artifacts that are deployed by you.

If you are running an internal repository manager for team collaboration, it is the time to specify in the hbase-0.90.3/pom.xml. Add the following section to it.

<project>
  ...
  <distributionManagement>
    <repository>
      <id>yourrepo.internal</id>
      <name>Your internal repository</name>
      <url>http://yourreposerver.com:port/path</url>
    </repository>
  </distributionManagement>
  ...
</project>

Note that yourrepo.internal is the same ID that you have configured authentication for in the ~m2/settings.xml file earlier.

Now, build and install/deploy HBase artifacts using the Maven goals.

You need to generate a binary tarball to install on the cluster. This is achieved by invoking assembly:single goal.

$ mvn assembly:single

This will generate the hbase-0.90.3/target/hbase-0.90.3-yourversion.tar.gz tarball.

Copy the tarball that was generated in the last step to your cluster and unpack them in the desired location.

This ensures that you have a consistent HBase installation with the right version of Hadoop artifacts that you have built. There is no need of replacing any artifact by hand because, Maven automatically pulled the right version of the artifact that you have built.