airflow


How to retrieve default args in python callable


I need to be able to access default_args defined as part of DAG definition in a Python Operator, python_callable. Maybe it's my unfamiliartiy with python or airflow in general, but could someone guide on how to achieve this.
Following is a code sample of what am trying to achieve
default_args = {
'owner': 'airflow',
'depends_on_past': False,
'email': 'xyz#xyz.com',
'email_on_failure': 'xyz#xyz.com',
'email_on_retry': False,
'retries': 1,
'retry_delay': timedelta(minutes=5),
'start_date': datetime(2017, 5, 15, 23, 20),
'end_date': datetime(2017, 5, 16, 23, 45),
'touchfile_path': '/user/myname/touchfiles/',
}
dag = DAG(
'test',
default_args=default_args,
template_searchpath=['/Users/myname/Desktop/utils/airflow/resources'],
user_defined_macros=dict(SCHEMA_NAME='abc'),
#schedule_interval='*/2 * * * * ')
schedule_interval='#once')
def webhdfs_touchfile_create(ds, *args, **kwargs):
web_hdfs_hook = WebHDFSHook('webhdfs_default')
client = web_hdfs_hook.get_conn()
client.write("/user/myname/airflow_hdfs","stringToWrite")
pp.pprint(kwargs)
task1 = PythonOperator(
task_id='task1',
provide_context=True, #enabling this would allow to pass arguments automatically to your callable function
python_callable=webhdfs_touchfile_create,
templates_dict={'attr1': {{ default_args['touchfile_path'] }}},
dag=dag)
Since the template_dict for PythonOperator is the only attribute which jinja templating works, how can i retrieve the 'touchfile_path' paramter in there?


Related Links

Modules in Airflow
AIrflow - Splitting DAG definition across multiple files
Airflow ignores resource pool flag when backfilling
Why is it recommended against using a dynamic start_date in Airflow?
Airflow Scheduler is slow by 1 interval
Airflow DAG run fails when task is up for retry
Airflow chart sharing
Airflow: how to delete a DAG?
Deploy procedure for new dags
Airflow webserver start only in debug mode
Airflow backfill clarification
Writing to Airflow Logs
airflow: 1.7: DAG not being updated in GUI
How can I get `ds` in SlackAPIPostOperator?
DAG started from airflow scheduler not finding values in airflow.cfg
How to create dependency between dynamically created tasks in Airflow

Categories

HOME
google-cloud-pubsub
twitter-bootstrap
azure-documentdb
salesforce
dronekit
ebay-api
google-docs-api
symfony-sonata
gnuradio
country-codes
pdf-generation
clip-path
ejb-3.0
google-my-business-api
aws-sdk-cpp
urllib2
email-attachments
ng2-charts
responsive
ringcentral
vivado-hls
hl7-fhir
screen-scraping
infinite-loop
apdu
android-xml
smart-device
launchd
restangular
osgi-bundle
jbossfuse
r-factor
ocl
mobile-development
overflow
textview
urhosharp
log4net-configuration
mkdir
hippocms
strategy-pattern
mongoid5
yelp
instances
jslider
susy
aura-framework
php-mongodb
test-kitchen
intervention
angular-fullstack
android-doze-and-standby
stanford-nlp-server
noise
e4
vertex
playframework-1.x
openpgp.js
sp
dmalloc
nsalert
ios-keyboard-extension
nls
recursive-query
blitz3d
indoor-positioning-system
visual-web-developer
x12
jira-agile
particle-system
avaudiorecorder
shtml
jcalendar
alchemy
yfiles
overlapping
android-contentresolver
notice
om
roslyn-code-analysis
joomla3.3
sparkcore
litedb
cakephp-2.6
glog
targetprocess
radgrid
crystal-reports-7
jsonschema2pojo
clearcanvas
parsefloat
ios-enterprise
rpt
visual-studio-setup-proje
code-smell
sqldatasource
infor-eam
nomachine
php-amqplib
nokogiri
extensibility
setcookie
zepto
invalidoperationexception
big-endian
ios6-maps
event-propagation
github-archive
helicontech
cherokee
n2
cryptolicensing
response-time
mass-emails
text-manipulation
data-formats
ffdshow
scalable





Mobile Apps Dev
Database Users
javascript
java
csharp
php
android
MS Developer
developer works
python
ios
c
html
jquery
RDBMS discuss
Cloud Virtualization
Database Dev&Adm