Hadoop : Code shipped from master to slave… here is a solution to the problem.
Hadoop : Code shipped from master to slave
I started a Hadoop cluster and submitted a job to the master. The jar file is only contained in the master. Does Hadoop ship the JAR to all slaves at the start of the job? Is it possible for a slave to run with a previous version of code released during the last run?
Thanks
Bala
Solution
From the MapReduce tutorial:
The framework will copy the necessary
files to the slave node before any
tasks for the job are executed on that
node. Its efficiency stems from the
fact that the files are only copied
once per job and the ability to cache
archives which are un-archived on the
slaves.
More information here:
http://hadoop.apache.org/common/docs/current/mapred_tutorial.html