Python – Hadoop stream: PYTHONPATH not working when mapper runs

Hadoop stream: PYTHONPATH not working when mapper runs… here is a solution to the problem.

Hadoop stream: PYTHONPATH not working when mapper runs

    I

  • SET PYTHONPATH IN IT AND IT WORKS FINE TOO, UNLESS I RUN THE MAP-REDUCE JOB
  • It doesn’t say

    Retrospective (last of the most recent call):
    The file “/work/app/hadoop/tmp/mapred/local/taskTracker/hduser/jobcache/job_201203091218_0006/attempt_201203091218_0006_m_000020_0/work/./mapper.py”, line 57, in
    Import utilities from src.utilities
    Import error: There is no module named src.utilities
    java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): The child process failed with code 1
    In org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:311)
    At org.apache.hadoop.streaming.PipeMapRed.mapRedFinished (PipeMapRed.java:545)
    In org.apache.hadoop.streaming.PipeMapper.map(PipeMapper.java:121)
    In org.apache.hadoop.mapred.MapRunner.run (MapRunner.java:50)
    In org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:36)
    In org.apache.hadoop.mapred.MapTask.runOldMapper (MapTask.java:436)
    In org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
    in org.apache.hadoop.mapred.Child$4.run(Child.java:261)
    In java.security.AccessController.doPrivileged( native method)
    In javax.security.auth.Subject.doAs (Subject.java:396)
    In org.apache.hadoop.security.UserGroupInformation.doAs (UserGroupInformation.java:1059)
    In org.apache.hadoop.mapred.Child.main(Child.java:255)
    java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): The child process failed with code 1
    In org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:311)
    At org.apache.hadoop.streaming.PipeMapRed.mapRedFinished (PipeMapRed.java:545)
    In org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:132)
    In org.apache.hadoop.mapred.MapRunner.run (MapRunner.java:57)
    In org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:36)
    In org.apache.hadoop.mapred.MapTask.runOldMapper (MapTask.java:436)
    In org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
    in org.apache.hadoop.mapred.Child$4.run(Child.java:261)
    In java.security.AccessController.doPrivileged( native method)
    In javax.security.auth.Subject.doAs (Subject.java:396)
    In org.apache.hadoop.security.UserGroupInformation.doAs (UserGroupInformation.java:1059)
    At org.apache.hadoop.mapred.Child.main (Child.java:255).

Question:
– Do we have to set the Python path specifically during hadoop-streaming? Where is it?

Solution

I’m having the same issue, I think the problem is that the Hadoop virtual environment doesn’t recognize your system’s pythonpath. If you install packages to /Library/Python/2.7/site-packages, Hadoop will pick them up and it will work.

Related Problems and Solutions