airflow


How to retrieve default args in python callable


I need to be able to access default_args defined as part of DAG definition in a Python Operator, python_callable. Maybe it's my unfamiliartiy with python or airflow in general, but could someone guide on how to achieve this.
Following is a code sample of what am trying to achieve
default_args = {
'owner': 'airflow',
'depends_on_past': False,
'email': 'xyz#xyz.com',
'email_on_failure': 'xyz#xyz.com',
'email_on_retry': False,
'retries': 1,
'retry_delay': timedelta(minutes=5),
'start_date': datetime(2017, 5, 15, 23, 20),
'end_date': datetime(2017, 5, 16, 23, 45),
'touchfile_path': '/user/myname/touchfiles/',
}
dag = DAG(
'test',
default_args=default_args,
template_searchpath=['/Users/myname/Desktop/utils/airflow/resources'],
user_defined_macros=dict(SCHEMA_NAME='abc'),
#schedule_interval='*/2 * * * * ')
schedule_interval='#once')
def webhdfs_touchfile_create(ds, *args, **kwargs):
web_hdfs_hook = WebHDFSHook('webhdfs_default')
client = web_hdfs_hook.get_conn()
client.write("/user/myname/airflow_hdfs","stringToWrite")
pp.pprint(kwargs)
task1 = PythonOperator(
task_id='task1',
provide_context=True, #enabling this would allow to pass arguments automatically to your callable function
python_callable=webhdfs_touchfile_create,
templates_dict={'attr1': {{ default_args['touchfile_path'] }}},
dag=dag)
Since the template_dict for PythonOperator is the only attribute which jinja templating works, how can i retrieve the 'touchfile_path' paramter in there?

Related Links

Long running task blocking all dags in airflow
Example of using a parameter in Airflow?
Raw SQL and file transformation in airflow
Assigning tasks to specific machines with airflow
Removing Airflow task logs
Accessing the 'ds' variable in airflow
[airflow]hive_hook: can not connect to hive metastore
Airflow: New DAG is not found by webserver
Airflow scheduler marked deleted DAG as active
celery redis:airflow celery with redis broker :Error connecting to redis Int or String expected
Chaining together 2 Airflow macros
Run code before all Airflow DAGs
Refreshing dags without web server restart apache airflow
Airflow BashOperator log doesn't contain full ouput
How to configure Airflow email alert
Airflow Celery flower looks at the old config entry for mysql

Categories

HOME
classification
ns2
apache-storm
ebay
localhost
msbi
open-search-server
pyodbc
cracking
facebook-sharer
tracking
loss
openscad
jquery-easyui
loader
aiml
autologin
passbook
udeploy
vala
smali
matlab-gui
ng-tags-input
pushbullet
onclicklistener
procedural-generation
scripting-language
ipython-parallel
recreate
paper-trail-gem
spring-oauth2
qt-designer
ilog
gitosis
pyenv
motherboard
jibx
wicket-tester
mtm
wufoo
hierarchical-clustering
automapper-4
react.rb
audio-fingerprinting
fluent-nhibernate-mapping
wicketstuff
cardview
google-scholar
alarmmanager
symantec
tinybox2
netlink
mod
videoview
jquery-draggable
matlab-engine
bioconductor
webservices-client
php-gd
pagecontrol
log-analysis
firebase-tools
dockerpy
nsq
alpha-beta-pruning
cordova-3
uiblureffect
hotlinking
object-oriented-database
xtify
caroufredsel
outlook-object-model
coinbase-php
msbuild-task
setcookie
freeimage
linear-interpolation
orchestration
openafs
php-gettext
beep
winrt-async
sup
mediastreamsource
subdirectories
zend-acl
ffdshow
wtsapi32
jgrowl
ironpython-studio

Resources

Mobile Apps Dev
Database Users
javascript
java
csharp
php
android
MS Developer
developer works
python
ios
c
html
jquery
RDBMS discuss
Cloud Virtualization
Database Dev&Adm
javascript
java
csharp
php
python
android
jquery
ruby
ios
html
Mobile App
Mobile App
Mobile App