airflow
How to retrieve default args in python callable
I need to be able to access default_args defined as part of DAG definition in a Python Operator, python_callable. Maybe it's my unfamiliartiy with python or airflow in general, but could someone guide on how to achieve this. Following is a code sample of what am trying to achieve default_args = { 'owner': 'airflow', 'depends_on_past': False, 'email': 'xyz#xyz.com', 'email_on_failure': 'xyz#xyz.com', 'email_on_retry': False, 'retries': 1, 'retry_delay': timedelta(minutes=5), 'start_date': datetime(2017, 5, 15, 23, 20), 'end_date': datetime(2017, 5, 16, 23, 45), 'touchfile_path': '/user/myname/touchfiles/', } dag = DAG( 'test', default_args=default_args, template_searchpath=['/Users/myname/Desktop/utils/airflow/resources'], user_defined_macros=dict(SCHEMA_NAME='abc'), #schedule_interval='*/2 * * * * ') schedule_interval='#once') def webhdfs_touchfile_create(ds, *args, **kwargs): web_hdfs_hook = WebHDFSHook('webhdfs_default') client = web_hdfs_hook.get_conn() client.write("/user/myname/airflow_hdfs","stringToWrite") pp.pprint(kwargs) task1 = PythonOperator( task_id='task1', provide_context=True, #enabling this would allow to pass arguments automatically to your callable function python_callable=webhdfs_touchfile_create, templates_dict={'attr1': {{ default_args['touchfile_path'] }}}, dag=dag) Since the template_dict for PythonOperator is the only attribute which jinja templating works, how can i retrieve the 'touchfile_path' paramter in there?
Related Links
Example of using a parameter in Airflow?
Raw SQL and file transformation in airflow
Assigning tasks to specific machines with airflow
Removing Airflow task logs
Accessing the 'ds' variable in airflow
[airflow]hive_hook: can not connect to hive metastore
Airflow: New DAG is not found by webserver
Airflow scheduler marked deleted DAG as active
celery redis:airflow celery with redis broker :Error connecting to redis Int or String expected
Chaining together 2 Airflow macros
Run code before all Airflow DAGs
Refreshing dags without web server restart apache airflow
Airflow BashOperator log doesn't contain full ouput
How to configure Airflow email alert
Airflow Celery flower looks at the old config entry for mysql
Airflow: can a subdag be run inside another subdag?