Python – Airflow: How to force the failure of the bash operator

Airflow: How to force the failure of the bash operator… here is a solution to the problem.

Airflow: How to force the failure of the bash operator

I’m running a series of python scripts (ex: script1.py, script2.py) in a script (e.g., do_stuff.sh) that I’m running with Airflow BashOperator. I wonder if there is a way to fail BashOperator in a python script if certain conditions are not met? I don’t need the script itself to fail because of BashOperator, so I can trigger clean_up task.

Script 1.py:

def main(x)
    if x == 0:
        raise ValueError('BashOperator FAILS')
    else:
        print x
if __name__ == '__main__':
    import plac
    plac.call(main)

Sorry, I’m still new to Airflow/scripting if my question is basic.

Thanks for your help!

Solution

You should be able to do this in your bash command:

exit 123

In this case, you will exit with error code 123, but you can use any error code you want.

Edit:

In python, throwing an exception the way you say it also fails the task:

raise ValueError('This will exit bash with an error.')

Related Problems and Solutions