airflow


How to retrieve default args in python callable


I need to be able to access default_args defined as part of DAG definition in a Python Operator, python_callable. Maybe it's my unfamiliartiy with python or airflow in general, but could someone guide on how to achieve this.
Following is a code sample of what am trying to achieve
default_args = {
'owner': 'airflow',
'depends_on_past': False,
'email': 'xyz#xyz.com',
'email_on_failure': 'xyz#xyz.com',
'email_on_retry': False,
'retries': 1,
'retry_delay': timedelta(minutes=5),
'start_date': datetime(2017, 5, 15, 23, 20),
'end_date': datetime(2017, 5, 16, 23, 45),
'touchfile_path': '/user/myname/touchfiles/',
}
dag = DAG(
'test',
default_args=default_args,
template_searchpath=['/Users/myname/Desktop/utils/airflow/resources'],
user_defined_macros=dict(SCHEMA_NAME='abc'),
#schedule_interval='*/2 * * * * ')
schedule_interval='#once')
def webhdfs_touchfile_create(ds, *args, **kwargs):
web_hdfs_hook = WebHDFSHook('webhdfs_default')
client = web_hdfs_hook.get_conn()
client.write("/user/myname/airflow_hdfs","stringToWrite")
pp.pprint(kwargs)
task1 = PythonOperator(
task_id='task1',
provide_context=True, #enabling this would allow to pass arguments automatically to your callable function
python_callable=webhdfs_touchfile_create,
templates_dict={'attr1': {{ default_args['touchfile_path'] }}},
dag=dag)
Since the template_dict for PythonOperator is the only attribute which jinja templating works, how can i retrieve the 'touchfile_path' paramter in there?

Related Links

Aiflow exceeds pool slot limit
Airflow: How can i process callback from an external system inside a job
Call nested scripts in ariflow
Does Airflow restart affect current running jobs?
Airflow does not backfill latest run
Possible to set different executor for each Airflow DAG?
Airflow dynamic DAG and Task Ids
Airflow too many connections as a default
Airflow SLA shows task that doesn't miss SLA
How exactly does the subDAG work in Airflow? What does it mean for a subDAG to be enabled?
Airflow parallelism
Airflow python client
Running dags with different frequency | Airflow
Fault Tolerance in Airflow
Scheduling AirfFlow DAG job
Airflow - Chart Query failing

Categories

HOME
c#
azure-documentdb
stata
process
formatting
jquery-plugins
django-middleware
stdout
cryengine
where
psexec
virtualhost
normals
theory
ldap-query
osrm
watson-dialog
browsermob
fastreport
openscad
django-rest-auth
xforms
minimagick
xna
opengraph
smali
poisson
sha
libxml2
angular-schema-form
dynamics-ax-2012-r3
osgi-bundle
selenium3
drawrect
akamai
http-proxy
geocomplete
ponylang
flex3
strategy-pattern
anypoint-studio
yelp
const
framebuffer
queryover
ngrok
wicket-tester
socketcluster
susy
directadmin
taleo
google-cloud-shell
skype-bots
riemann
wink
richtext
typewriter
senti-wordnet
timex
group
vitamio
radar-chart
bluetooth-lowenergy-4.2
nonblocking
overlapping
gtkwave
gitk
edgar
jain-sip
jsoncpp
litedb
maven-archetype
strcpy
service-broker
sony-xperia
pysvn
launcher
javadb
redmine-plugins
msdropdown
hotlinking
starling-framework
netbeans-plugins
listpicker
jython-2.5
pyxplot
kademlia
proj4
junction-table
architectural-patterns
system.transactions
database-restore
rbm
gpl
ncml
databus
optimistic-concurrency
scriptmanager
rjs
flexbuilder
quazip
norton
lgpl
google-local-search
p4.net
dongle
newspeak
pkcs#7
pixel-bender
bootstrap-protocol
httpbrowsercapabilities
asp.net-1.1

Resources

Mobile Apps Dev
Database Users
javascript
java
csharp
php
android
MS Developer
developer works
python
ios
c
html
jquery
RDBMS discuss
Cloud Virtualization
Database Dev&Adm
javascript
java
csharp
php
python
android
jquery
ruby
ios
html
Mobile App
Mobile App
Mobile App