Hadoop stream: PYTHONPATH not working when mapper runs
- I
- SET PYTHONPATH IN IT AND IT WORKS FINE TOO, UNLESS I RUN THE MAP-REDUCE JOB
It doesn’t say
Retrospective (last of the most recent call):
The file “/work/app/hadoop/tmp/mapred/local/taskTracker/hduser/jobcache/job_201203091218_0006/attempt_201203091218_0006_m_000020_0/work/./mapper.py”, line 57, in
Import utilities from src.utilities
Import error: There is no module named src.utilities
java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): The child process failed with code 1
In org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:311)
At org.apache.hadoop.streaming.PipeMapRed.mapRedFinished (PipeMapRed.java:545)
In org.apache.hadoop.streaming.PipeMapper.map(PipeMapper.java:121)
In org.apache.hadoop.mapred.MapRunner.run (MapRunner.java:50)
In org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:36)
In org.apache.hadoop.mapred.MapTask.runOldMapper (MapTask.java:436)
In org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
in org.apache.hadoop.mapred.Child$4.run(Child.java:261)
In java.security.AccessController.doPrivileged( native method)
In javax.security.auth.Subject.doAs (Subject.java:396)
In org.apache.hadoop.security.UserGroupInformation.doAs (UserGroupInformation.java:1059)
In org.apache.hadoop.mapred.Child.main(Child.java:255)
java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): The child process failed with code 1
In org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:311)
At org.apache.hadoop.streaming.PipeMapRed.mapRedFinished (PipeMapRed.java:545)
In org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:132)
In org.apache.hadoop.mapred.MapRunner.run (MapRunner.java:57)
In org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:36)
In org.apache.hadoop.mapred.MapTask.runOldMapper (MapTask.java:436)
In org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
in org.apache.hadoop.mapred.Child$4.run(Child.java:261)
In java.security.AccessController.doPrivileged( native method)
In javax.security.auth.Subject.doAs (Subject.java:396)
In org.apache.hadoop.security.UserGroupInformation.doAs (UserGroupInformation.java:1059)
At org.apache.hadoop.mapred.Child.main (Child.java:255).
Question:
– Do we have to set the Python path specifically during hadoop-streaming? Where is it?
Solution
I’m having the same issue, I think the problem is that the Hadoop virtual environment doesn’t recognize your system’s pythonpath. If you install packages to /Library/Python/2.7/site-packages, Hadoop will pick them up and it will work.