airflow


How to retrieve default args in python callable


I need to be able to access default_args defined as part of DAG definition in a Python Operator, python_callable. Maybe it's my unfamiliartiy with python or airflow in general, but could someone guide on how to achieve this.
Following is a code sample of what am trying to achieve
default_args = {
'owner': 'airflow',
'depends_on_past': False,
'email': 'xyz#xyz.com',
'email_on_failure': 'xyz#xyz.com',
'email_on_retry': False,
'retries': 1,
'retry_delay': timedelta(minutes=5),
'start_date': datetime(2017, 5, 15, 23, 20),
'end_date': datetime(2017, 5, 16, 23, 45),
'touchfile_path': '/user/myname/touchfiles/',
}
dag = DAG(
'test',
default_args=default_args,
template_searchpath=['/Users/myname/Desktop/utils/airflow/resources'],
user_defined_macros=dict(SCHEMA_NAME='abc'),
#schedule_interval='*/2 * * * * ')
schedule_interval='#once')
def webhdfs_touchfile_create(ds, *args, **kwargs):
web_hdfs_hook = WebHDFSHook('webhdfs_default')
client = web_hdfs_hook.get_conn()
client.write("/user/myname/airflow_hdfs","stringToWrite")
pp.pprint(kwargs)
task1 = PythonOperator(
task_id='task1',
provide_context=True, #enabling this would allow to pass arguments automatically to your callable function
python_callable=webhdfs_touchfile_create,
templates_dict={'attr1': {{ default_args['touchfile_path'] }}},
dag=dag)
Since the template_dict for PythonOperator is the only attribute which jinja templating works, how can i retrieve the 'touchfile_path' paramter in there?

Related Links

Example of using a parameter in Airflow?
Raw SQL and file transformation in airflow
Assigning tasks to specific machines with airflow
Removing Airflow task logs
Accessing the 'ds' variable in airflow
[airflow]hive_hook: can not connect to hive metastore
Airflow: New DAG is not found by webserver
Airflow scheduler marked deleted DAG as active
celery redis:airflow celery with redis broker :Error connecting to redis Int or String expected
Chaining together 2 Airflow macros
Run code before all Airflow DAGs
Refreshing dags without web server restart apache airflow
Airflow BashOperator log doesn't contain full ouput
How to configure Airflow email alert
Airflow Celery flower looks at the old config entry for mysql
Airflow: can a subdag be run inside another subdag?

Categories

HOME
c#
oauth-2.0
http
intellij-idea
display
abc
nltk
singleton
sms
perl6
aggregation-framework
markdown
obd-ii
phpexcel
oms
apache-httpclient-4.x
wowza
aws-sdk-cpp
virtualenv
oxid
nsoperation
smooth-scrolling
android-actionbar
audio-streaming
bug-reporting
cherrypy
polybase
combinations
typedef
poisson
datastax-enterprise
pixels
sasl
arp
andengine
jql
traitsui
kepserverex
matrix-multiplication
mybatis-generator
observer-pattern
ng-lightning
office-interop
azure-web-roles
socketcluster
directadmin
taleo
weld
hierarchical-clustering
confluent
nao-robot
pyopenssl
skype-bots
runtime-permissions
playframework-2.3
nservicebus6
grib
blackberry
ios-keyboard-extension
timex
addressbook
portaudio
glade
class-design
jira-agile
require-handlebars
tinybox2
patching
sbt-proguard
avaudiorecorder
skscene
javapackager
traminer
consul-template
autoscaling
avrcp
polarion
radgrid
connection-timeout
google-experiments
xmla
dojox.grid
kss
apache-pivot
go-gin
gnip
redmine-plugins
telerik-appbuilder
isl
rssi
farpoint-spread
ia-32
violation
wpa
forward-declaration
wpd
django-tinymce
esi
doxia
recess
hs-err
nstoolbar
file-storage

Resources

Encrypt Message



code
soft
python
ios
c
html
jquery
cloud
mobile