airflow


How to retrieve default args in python callable


I need to be able to access default_args defined as part of DAG definition in a Python Operator, python_callable. Maybe it's my unfamiliartiy with python or airflow in general, but could someone guide on how to achieve this.
Following is a code sample of what am trying to achieve
default_args = {
'owner': 'airflow',
'depends_on_past': False,
'email': 'xyz#xyz.com',
'email_on_failure': 'xyz#xyz.com',
'email_on_retry': False,
'retries': 1,
'retry_delay': timedelta(minutes=5),
'start_date': datetime(2017, 5, 15, 23, 20),
'end_date': datetime(2017, 5, 16, 23, 45),
'touchfile_path': '/user/myname/touchfiles/',
}
dag = DAG(
'test',
default_args=default_args,
template_searchpath=['/Users/myname/Desktop/utils/airflow/resources'],
user_defined_macros=dict(SCHEMA_NAME='abc'),
#schedule_interval='*/2 * * * * ')
schedule_interval='#once')
def webhdfs_touchfile_create(ds, *args, **kwargs):
web_hdfs_hook = WebHDFSHook('webhdfs_default')
client = web_hdfs_hook.get_conn()
client.write("/user/myname/airflow_hdfs","stringToWrite")
pp.pprint(kwargs)
task1 = PythonOperator(
task_id='task1',
provide_context=True, #enabling this would allow to pass arguments automatically to your callable function
python_callable=webhdfs_touchfile_create,
templates_dict={'attr1': {{ default_args['touchfile_path'] }}},
dag=dag)
Since the template_dict for PythonOperator is the only attribute which jinja templating works, how can i retrieve the 'touchfile_path' paramter in there?

Related Links

Airflow DAG run fails when task is up for retry
Airflow chart sharing
Airflow: how to delete a DAG?
Deploy procedure for new dags
Airflow webserver start only in debug mode
Airflow backfill clarification
Writing to Airflow Logs
airflow: 1.7: DAG not being updated in GUI
How can I get `ds` in SlackAPIPostOperator?
DAG started from airflow scheduler not finding values in airflow.cfg
How to create dependency between dynamically created tasks in Airflow
How to use AirFlow to run a list of python tasks?
How to avoid Airflow to backfill when using trigger_dag?
Airflow - mark a specific task_id of given dag_id and run_id as success or failure
Aiflow exceeds pool slot limit
Airflow: How can i process callback from an external system inside a job

Categories

HOME
spring-boot
botframework
ns2
pyspark
webvr
aix
azure-database-mysql
x509certificate
ms-application-insights
artifactory
angularjs-directive
glympse
android-webview
restheart
software-collections
segmentation-fault
google-my-business-api
olap
theory
resharper
opendaylight
jsfiddle
brightway
internet-explorer-7
civicrm
openoffice-basic
turn.js
monad-transformers
pushbullet
windowserror
dynamic-memory-allocation
request-uri
tikz
ng2-bootstrap
oracle-nosql
wit-ai
realloc
mongoid5
byobu
file-conversion
aspnetboilerplate
flume-ng
logan-square
synchronized
spring-ide
standard-error
hellosign
nls
vertex-shader
virtualenvwrapper
boost-bind
pebble-js
dday
magnet-uri
ruby-1.8.7
play-json
php-gd
redquerybuilder
lytro
bootstrap-tabs
android-filterable
azure-mobile-app
vspackage
vectordrawable
grails-3.0.9
genome
skview
veeam
yosemite
android-relativelayout
elixir-framework
android-jack-and-jill
imaplib
nop
mobile-robots
oc4j
xrandr
usersettings
jython-2.5
asp.net-4.0
dd4t
proj4
coinbase-php
simplemembership
blotter
feincms
netbiscuits
hypertable
google-email-audit-api
nsscrollview
eeprom
towerjs
ruby-1.9
cherokee
pex-and-moles
execjs
file-processing
html5-animation
cinder
large-data-volumes
numerical-computing
asp.net-1.1

Resources

Mobile Apps Dev
Database Users
javascript
java
csharp
php
android
MS Developer
developer works
python
ios
c
html
jquery
RDBMS discuss
Cloud Virtualization
Database Dev&Adm
javascript
java
csharp
php
python
android
jquery
ruby
ios
html
Mobile App
Mobile App
Mobile App