Python – Exit Code : 127 with fabric2 while executing a distant ssh command

I came across the strange behavior of fabric2 command module. These commands are like a charm :

connect = Connection(host=h, user=u ,connect_kwargs={"password":p})
connect.run('mkdir temp_streaming')
connect.put(m, "temp_streaming/mapper.py")
connect.put(r, "temp_streaming/reducer.py")

But speaking of this:

input_path = input("Please choose the input of your program (on your HDFS File System) ")
output_path = input("Please choose a name for your output folder ")

main_command = str('yarn jar '+ jar_path+' -files mapper.py,reducer.py -mapper temp_streaming/mapper.py -reducer temp_streaming/reducer.py -input '+ input_path + ' -output '+ output_ path)

connect.run(main_command)

I’m getting this error I can’t figure out :

bash: yarn: command not found
Traceback (most recent call last):
  File "__main.py__", line 77, in <module>
    main()
  File "__main.py__", line 65, in main
   RunMapReduce(mapper, reducer, jar_path)
  File "__main.py__", line 46, in RunMapReduce
   connect.run(main_command)
 File "<decorator-gen-3>", line 2, in run
 File "/usr/local/lib/python3.5/dist-packages/fabric2/connection.py", line 30, in opens
return method(self, *args, **kwargs)
 File "/usr/local/lib/python3.5/dist-packages/fabric2/connection.py", line 586, in run
return self._run(runner, command, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/invoke/context.py", line 100, in _run
return runner.run(command, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/invoke/runners.py", line 268, in run
return self._run_body(command, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/invoke/runners.py", line 401, in _run_body
raise UnexpectedExit(result)
invoke.exceptions.UnexpectedExit: Encountered a bad command exit code!

Command: 'yarn jar /home/hadoop/hadoop/share/hadoop/tools/lib/hadoop-streaming-2.8.4.jar -files mapper.py,reducer.py -mapper temp_streaming/mapper.py -reducer temp_streaming/ reducer.py -input /books/kafka_metamorphosis.txt -output /test'

Exit code: 127

Stdout: already printed

Stderr: already printed

When I log in to the server directly with ssh and run it, even though this bash command works, it looks like the remote server doesn’t know about the yarn command. It seems to me that it is related to this library, but I can’t solve my problem with documentation.

I’ve also had this problem: it turns out that Fabric runs SSH as a non-interactive shell. $PATH variable contains the path to your yarn bin, set in /etc/profile or ~/.bash_profile or ~/.bashrc, neither of which has a source.

Here is the link, and some solutions.

Personally, I don’t like playing with these files => What I just did (before stumbling upon this link) was that I used the full path to the binary: /usr/bin/yarn [the rest of your command].