airflow


How to retrieve default args in python callable


I need to be able to access default_args defined as part of DAG definition in a Python Operator, python_callable. Maybe it's my unfamiliartiy with python or airflow in general, but could someone guide on how to achieve this.
Following is a code sample of what am trying to achieve
default_args = {
'owner': 'airflow',
'depends_on_past': False,
'email': 'xyz#xyz.com',
'email_on_failure': 'xyz#xyz.com',
'email_on_retry': False,
'retries': 1,
'retry_delay': timedelta(minutes=5),
'start_date': datetime(2017, 5, 15, 23, 20),
'end_date': datetime(2017, 5, 16, 23, 45),
'touchfile_path': '/user/myname/touchfiles/',
}
dag = DAG(
'test',
default_args=default_args,
template_searchpath=['/Users/myname/Desktop/utils/airflow/resources'],
user_defined_macros=dict(SCHEMA_NAME='abc'),
#schedule_interval='*/2 * * * * ')
schedule_interval='#once')
def webhdfs_touchfile_create(ds, *args, **kwargs):
web_hdfs_hook = WebHDFSHook('webhdfs_default')
client = web_hdfs_hook.get_conn()
client.write("/user/myname/airflow_hdfs","stringToWrite")
pp.pprint(kwargs)
task1 = PythonOperator(
task_id='task1',
provide_context=True, #enabling this would allow to pass arguments automatically to your callable function
python_callable=webhdfs_touchfile_create,
templates_dict={'attr1': {{ default_args['touchfile_path'] }}},
dag=dag)
Since the template_dict for PythonOperator is the only attribute which jinja templating works, how can i retrieve the 'touchfile_path' paramter in there?

Related Links

Airflow dynamic DAG and Task Ids
Airflow too many connections as a default
Airflow SLA shows task that doesn't miss SLA
How exactly does the subDAG work in Airflow? What does it mean for a subDAG to be enabled?
Airflow parallelism
Airflow python client
Running dags with different frequency | Airflow
Fault Tolerance in Airflow
Scheduling AirfFlow DAG job
Airflow - Chart Query failing
Airflow - AD Profile group based access to DAGs
Airflow: changing the crontab time for a DAG in Airflow
execution_date in airflow: need to access as a variable
configuring Airflow to work with CeleryExecutor
How to work correctly airflow schedule_interval
Airflow - how to make EmailOperator html_content dynamic?

Categories

HOME
kendo-ui-angular2
primefaces
cxf
azure-eventhub
reference
postsharp
terminal
perl6
jshell
aggregate-functions
travis-ci-cli
qt4
samsung-gear-s2
select2
resharper
hapijs
orange
project-reactor
serenity-js
continuous-fourier
windows-10-mobile
request-uri
jms-topic
lines
ansible-inventory
campaign-monitor
julia-jump
aura-framework
datagridviewcombobox
binary-decision-diagram
spring-ide
spring-data-commons
hdpi
vertex
enet
sharpdevelop
gige-sdk
nsalert
breadcrumbs
portaudio
xcode7.1
git-cherry-pick
django-taggit
python-rq
mod
custom-build-step
datamodel
markerspiderfier
github-for-mac
lytro
updating
cloud-storage
joomla3.3
tooltwist
p4python
eggplant
opennebula
veeam
dockerpy
parallel-port
visual-studio-setup-proje
node-mongodb-native
kuka-krl
nativequery
logo-lang
xtify
ekevent
grunt-wiredep
google-voice
xcode5.1
libx264
netbiscuits
nude.js
clgeocoder
meego-harmattan
response-time
html5-animation
httpbrowsercapabilities

Resources

Database Users
RDBMS discuss
Database Dev&Adm
javascript
java
csharp
php
android
javascript
java
csharp
php
python
android
jquery
ruby
ios
html
Mobile App
Mobile App
Mobile App