Python – hadoop streams: how to see application logs?

hadoop streams: how to see application logs?… here is a solution to the problem.

hadoop streams: how to see application logs?

I can see all the hadoop logs on my /usr/local/hadoop/logs path

But where can I see application-level logs? For example:

Mapper .py

import logging

def main():
    logging.info("starting map task now")
     -- do some task -- //
    print statement  

reducer .py

import logging
def main():
    for line in sys.stdin:
        logging.info("received input to reducer - " + line)  
         -- do some task -- //
        print statement

Where can I see the relevant log statements for logging.info or my application?
I’m using Python and using hadoop-streaming

Thanks

Solution

Hadoop collects stderr and can be viewed on the Hadoop map/reduce status site. So you can just write to stderr.

Related Problems and Solutions