Java – Hadoop : Code shipped from master to slave

Hadoop : Code shipped from master to slave… here is a solution to the problem.

Hadoop : Code shipped from master to slave

I started a Hadoop cluster and submitted a job to the master. The jar file is only contained in the master. Does Hadoop ship the JAR to all slaves at the start of the job? Is it possible for a slave to run with a previous version of code released during the last run?

Thanks
Bala

Solution

From the MapReduce tutorial:

The framework will copy the necessary
files to the slave node
before any
tasks for the job are executed on that
node. Its efficiency stems from the
fact that the files are only copied
once per job
and the ability to cache
archives which are un-archived on the
slaves.

More information here:

http://hadoop.apache.org/common/docs/current/mapred_tutorial.html

Related Problems and Solutions