airflow


How to retrieve default args in python callable


I need to be able to access default_args defined as part of DAG definition in a Python Operator, python_callable. Maybe it's my unfamiliartiy with python or airflow in general, but could someone guide on how to achieve this.
Following is a code sample of what am trying to achieve
default_args = {
'owner': 'airflow',
'depends_on_past': False,
'email': 'xyz#xyz.com',
'email_on_failure': 'xyz#xyz.com',
'email_on_retry': False,
'retries': 1,
'retry_delay': timedelta(minutes=5),
'start_date': datetime(2017, 5, 15, 23, 20),
'end_date': datetime(2017, 5, 16, 23, 45),
'touchfile_path': '/user/myname/touchfiles/',
}
dag = DAG(
'test',
default_args=default_args,
template_searchpath=['/Users/myname/Desktop/utils/airflow/resources'],
user_defined_macros=dict(SCHEMA_NAME='abc'),
#schedule_interval='*/2 * * * * ')
schedule_interval='#once')
def webhdfs_touchfile_create(ds, *args, **kwargs):
web_hdfs_hook = WebHDFSHook('webhdfs_default')
client = web_hdfs_hook.get_conn()
client.write("/user/myname/airflow_hdfs","stringToWrite")
pp.pprint(kwargs)
task1 = PythonOperator(
task_id='task1',
provide_context=True, #enabling this would allow to pass arguments automatically to your callable function
python_callable=webhdfs_touchfile_create,
templates_dict={'attr1': {{ default_args['touchfile_path'] }}},
dag=dag)
Since the template_dict for PythonOperator is the only attribute which jinja templating works, how can i retrieve the 'touchfile_path' paramter in there?

Related Links

Long running task blocking all dags in airflow
Example of using a parameter in Airflow?
Raw SQL and file transformation in airflow
Assigning tasks to specific machines with airflow
Removing Airflow task logs
Accessing the 'ds' variable in airflow
[airflow]hive_hook: can not connect to hive metastore
Airflow: New DAG is not found by webserver
Airflow scheduler marked deleted DAG as active
celery redis:airflow celery with redis broker :Error connecting to redis Int or String expected
Chaining together 2 Airflow macros
Run code before all Airflow DAGs
Refreshing dags without web server restart apache airflow
Airflow BashOperator log doesn't contain full ouput
How to configure Airflow email alert
Airflow Celery flower looks at the old config entry for mysql

Categories

HOME
actionscript-3
repository
qemu
display
thermal-printer
yuv
gnuradio
genexus
webpack-dev-server
wkhtmltopdf
oms
ldap-query
guidewire
yui
dompdf
versioning
jquery-easyui
accordion
autologin
bug-reporting
smali
pushbullet
v4l2
ps
struts
deployment-descriptor
visual-studio-community
large-files
angular-schema-form
azure-table-storage
pypy
paper-trail-gem
ios9
apm
blockly
kubectl
oracle-nosql
availability
microstrategy
incapsula
geocomplete
android-databinding
hippocms
lex
usernotifications
server-side-swift
azure-web-roles
file-descriptor
autodesk-data-management
taiga
codepen
azure-data-catalog
google-chrome-storage
dcom
bioperl
ajv
sp
jtextarea
msiexec
alarmmanager
zsh-completion
networkstream
cosign-api
hotkeys
innertext
custom-build-step
music-player
transbase
redquerybuilder
edgar
tess4j
xml-simple
android-networking
skview
nexusdb
email-headers
carrier
jfilechooser
google-voice
ie-developer-tools
kendo-window
violation
vline
metapost
mail-gem
openafs
jquery-1.8
waveout
blackberry-eclipse-plugin
ios6-maps
systrace
overlays
netbeans7.0
excellibrary
activescaffold
table-valued-parameters
scala-designer
opengl-to-opengles

Resources

Encrypt Message



code
soft
python
ios
c
html
jquery
cloud
mobile