Airflow: How to force the failure of the bash operator… here is a solution to the problem.
Airflow: How to force the failure of the bash operator
I’m running a series of python scripts (ex: script1.py, script2.py) in a script (e.g., do_stuff.sh) that I’m running with Airflow BashOperator. I wonder if there is a way to fail BashOperator in a python script if certain conditions are not met? I don’t need the script itself to fail because of BashOperator, so I can trigger clean_up task.
Script 1.py:
def main(x)
if x == 0:
raise ValueError('BashOperator FAILS')
else:
print x
if __name__ == '__main__':
import plac
plac.call(main)
Sorry, I’m still new to Airflow/scripting if my question is basic.
Thanks for your help!
Solution
You should be able to do this in your bash command:
exit 123
In this case, you will exit with error code 123, but you can use any error code you want.
Edit:
In python, throwing an exception the way you say it also fails the task:
raise ValueError('This will exit bash with an error.')