hadoop streams: how to see application logs?… here is a solution to the problem.
hadoop streams: how to see application logs?
I can see all the hadoop logs on my /usr/local/hadoop/logs
path
But where can I see application-level logs? For example:
Mapper .py
import logging
def main():
logging.info("starting map task now")
-- do some task -- //
print statement
reducer .py
import logging
def main():
for line in sys.stdin:
logging.info("received input to reducer - " + line)
-- do some task -- //
print statement
Where can I see the relevant log statements for logging.info
or my application?
I’m using Python
and using hadoop-streaming
Thanks
Solution
Hadoop collects stderr and can be viewed on the Hadoop map/reduce status site. So you can just write to stderr.