airflow


How to retrieve default args in python callable


I need to be able to access default_args defined as part of DAG definition in a Python Operator, python_callable. Maybe it's my unfamiliartiy with python or airflow in general, but could someone guide on how to achieve this.
Following is a code sample of what am trying to achieve
default_args = {
'owner': 'airflow',
'depends_on_past': False,
'email': 'xyz#xyz.com',
'email_on_failure': 'xyz#xyz.com',
'email_on_retry': False,
'retries': 1,
'retry_delay': timedelta(minutes=5),
'start_date': datetime(2017, 5, 15, 23, 20),
'end_date': datetime(2017, 5, 16, 23, 45),
'touchfile_path': '/user/myname/touchfiles/',
}
dag = DAG(
'test',
default_args=default_args,
template_searchpath=['/Users/myname/Desktop/utils/airflow/resources'],
user_defined_macros=dict(SCHEMA_NAME='abc'),
#schedule_interval='*/2 * * * * ')
schedule_interval='#once')
def webhdfs_touchfile_create(ds, *args, **kwargs):
web_hdfs_hook = WebHDFSHook('webhdfs_default')
client = web_hdfs_hook.get_conn()
client.write("/user/myname/airflow_hdfs","stringToWrite")
pp.pprint(kwargs)
task1 = PythonOperator(
task_id='task1',
provide_context=True, #enabling this would allow to pass arguments automatically to your callable function
python_callable=webhdfs_touchfile_create,
templates_dict={'attr1': {{ default_args['touchfile_path'] }}},
dag=dag)
Since the template_dict for PythonOperator is the only attribute which jinja templating works, how can i retrieve the 'touchfile_path' paramter in there?

Related Links

Run code before all Airflow DAGs
Refreshing dags without web server restart apache airflow
Airflow BashOperator log doesn't contain full ouput
How to configure Airflow email alert
Airflow Celery flower looks at the old config entry for mysql
Airflow: can a subdag be run inside another subdag?
Airflow: how to extend SubDagOperator?
Airflow 1.8.0 - execution_date doesn't have a default value exception
Schedule tasks on another computer with Airflow
Schedule airflow to run on weekdays
Airflow: dag_id could not be found
What is the difference between airflow trigger rule “all_done” and “all_success”?
Airflow and Docker Containers
Next Instance of Job Runs Before I Can Debug This Instance
Airflow - how to make a task only to be triggered after all its upstream tasks status have been known and at least one of them is successful
Airflow: running task t4 even if another has failed in such DAG: t1 >> t2 >> t3 >> t4

Categories

HOME
logging
repository
snappy
abc
ibm
ebay-api
conceptual
azure-database-mysql
spotfire
crm
google-docs-api
sugarcrm
glympse
foreach
android-webview
vsixmanifest
spring-data-solr
openbsd
google-search-appliance
websharper
virtualenv
expo
jinja2
background-image
openscad
connection-pooling
wai-aria
ppp
sha
sasl
large-files
azure-table-storage
jms-topic
jsonapi-resources
voyager
incapsula
mkdir
kepserverex
observer-pattern
icu
irr
rexx
firefox-developer-edition
libzip
cakephp-3.2
qtwebkit
opos
serialversionuid
extjs3
dotnet-cli
bioperl
htop
codesniffer
ios-keyboard-extension
sqlproj
directinput
turbolinks
mac-app-store
sprockets
android-xmlpullparser
python-rq
abstract
google-app-engine-php
overlapping
jcomponent
recode
taverna
notice
traminer
eaaccessory
tokudb
stereotype
photogrammetry
sicstus-prolog
asp.net-2.0
mri
pysvn
iphone-6
redmine-plugins
indic
visual-c#-express-2010
mvcgrid
episerver-7
yandex-api
mixradio
starling-framework
mongoid4
php-amqplib
jfilechooser
google-mirror-api
raddatepicker
windows-scheduler
junction-table
spawn
boost.build
teamsystem
project-files
file-processing
xhtmlrenderer
moss2007-security
noise-reduction
oaw
jgrowl
weak-typing
urlscan

Resources

Mobile Apps Dev
Database Users
javascript
java
csharp
php
android
MS Developer
developer works
python
ios
c
html
jquery
RDBMS discuss
Cloud Virtualization
Database Dev&Adm
javascript
java
csharp
php
python
android
jquery
ruby
ios
html
Mobile App
Mobile App
Mobile App