airflow


How to retrieve default args in python callable


I need to be able to access default_args defined as part of DAG definition in a Python Operator, python_callable. Maybe it's my unfamiliartiy with python or airflow in general, but could someone guide on how to achieve this.
Following is a code sample of what am trying to achieve
default_args = {
'owner': 'airflow',
'depends_on_past': False,
'email': 'xyz#xyz.com',
'email_on_failure': 'xyz#xyz.com',
'email_on_retry': False,
'retries': 1,
'retry_delay': timedelta(minutes=5),
'start_date': datetime(2017, 5, 15, 23, 20),
'end_date': datetime(2017, 5, 16, 23, 45),
'touchfile_path': '/user/myname/touchfiles/',
}
dag = DAG(
'test',
default_args=default_args,
template_searchpath=['/Users/myname/Desktop/utils/airflow/resources'],
user_defined_macros=dict(SCHEMA_NAME='abc'),
#schedule_interval='*/2 * * * * ')
schedule_interval='#once')
def webhdfs_touchfile_create(ds, *args, **kwargs):
web_hdfs_hook = WebHDFSHook('webhdfs_default')
client = web_hdfs_hook.get_conn()
client.write("/user/myname/airflow_hdfs","stringToWrite")
pp.pprint(kwargs)
task1 = PythonOperator(
task_id='task1',
provide_context=True, #enabling this would allow to pass arguments automatically to your callable function
python_callable=webhdfs_touchfile_create,
templates_dict={'attr1': {{ default_args['touchfile_path'] }}},
dag=dag)
Since the template_dict for PythonOperator is the only attribute which jinja templating works, how can i retrieve the 'touchfile_path' paramter in there?

Related Links

[airflow]hive_hook: can not connect to hive metastore
Airflow: New DAG is not found by webserver
Airflow scheduler marked deleted DAG as active
celery redis:airflow celery with redis broker :Error connecting to redis Int or String expected
Chaining together 2 Airflow macros
Run code before all Airflow DAGs
Refreshing dags without web server restart apache airflow
Airflow BashOperator log doesn't contain full ouput
How to configure Airflow email alert
Airflow Celery flower looks at the old config entry for mysql
Airflow: can a subdag be run inside another subdag?
Airflow: how to extend SubDagOperator?
Airflow 1.8.0 - execution_date doesn't have a default value exception
Schedule tasks on another computer with Airflow
Schedule airflow to run on weekdays
Airflow: dag_id could not be found

Categories

HOME
oop
ssms
stata
alasql
intellij-idea
postgresql-9.3
grpc
jquery-plugins
symfony-sonata
confluence-rest-api
terminal
css-selectors
country-codes
wavefront
promotions
ionic-native
pyodbc
box-api
ejb-3.0
olap
webseal
luis
nsoperation
greensock
jquery-easyui
hapijs
audio-streaming
aws-devicefarm
xna
ithit-webdav-server
priority-queue
marionette
angular-schema-form
sphinx4
hierarchy
selenium3
drawrect
overflow
login-required
directions
availability
voyager
mootools
gitosis
headphones
ponylang
javafxports
perf
rythm
const
rbac
glide-image-library
collation
text-extraction
spring-webflow-2
overhead
consolidation
deck.js
normalisation
mrunit
razorengine
treelist
periodic-processing
hana-sql-script
google-finance-api
magnet-uri
updating
adempiere
tess4j
tidesdk
valdr-bean-validation
strcpy
connection-timeout
optimistic-locking
aiff
javadb
frisby.js
xmlserializer
yosemite
disparity-mapping
android-relativelayout
satisfiability
enquire.js
nodeunit
iwork
adabas
8-puzzle
aspnet-compiler
select2-rails
windows-scheduler
database-restore
urlfetch
cryptarithmetic-puzzle
inflate
teamsystem
xcode4.6.3
pdf-reader
stl-algorithm
towerjs
maven-eclipse-plugin
soapexception
jqtransform
http-response-codes
azure-appfabric
miniport
web-widget
script-tag
onresize
jgrowl
weak-typing

Resources

Mobile Apps Dev
Database Users
javascript
java
csharp
php
android
MS Developer
developer works
python
ios
c
html
jquery
RDBMS discuss
Cloud Virtualization
Database Dev&Adm
javascript
java
csharp
php
python
android
jquery
ruby
ios
html