aggregate-functions


Howto aggregate on full data set in Spring Batch jobs?


I need to insert aggregation in my Spring Batch jobs.
But the aggregation step need to have the entire data set available.
In pure SQL, it's easy to code SQL aggregation requests : the full data set (as stored in database) is available.
But in Spring Batch jobs, everything is done in memory, and spread in chunked.
So howto deal with that kind of data strewing ?
Do you have any advice concerning the best practices to insert aggregation steps/processes ?
Thx a lot for your enlightments
You have Partitioning option in spring batch which can have StepExecutionAggregator, it has aggregate method which accepts list of StepContext of all partitioned steps.
We had i.e. integration with soap server where we first received list of something that needs to be processed, than we partitioned it to child steps and processed in parallel and after each child step finishes aggregator is invoked which can do stuff based on data in child step context.
It is good way if you have something in your data which can be good rule for partitioning (i.e. pull list of items from DB and process each item in parallel, save item data in step context, use aggregator and combine everything in each step context and do common operation on combined data).
Here is link to example with partitioning (there is no aggregation but you can add it to masterStep).

Related Links

incomplete results with aggregate (R)
How to do an aggregated OR on a column in Amazon Redshift
'Your query does not include the specified expression as part of an aggregate function' error for calculated fields
Creating percentile_cont as aggregate function in Vertica
Create Expression in Report Builder 3.0 Report to sum a column
SAP Hana - STRING_AGG issue with ORDER BY CLAUSE
rrdtool does not keep max
Tableau Aggregate Percentage
use aggregates (min, max, avg) in CONSTRUCT query [duplicate]
R: transposing and aggregating columns
SQL Query Aggregate Error
Usage of aggregate functions in MarkLogic
Querying customDimensions in Google Bigquery
NHibernate - QueryOver criteria appearing in Where instead in Having clause, error
Howto aggregate on full data set in Spring Batch jobs?
Creating an aggregate function fails

Categories

HOME
kendo-ui
redux
cluster-analysis
mysqli
aix
window.open
x86-64
magento-1.9.2.1
spss
symfony-sonata
snmp
powerquery
opacity
virtualhost
google-app-maker
apache-httpclient-4.x
resharper
jetbrains
logstash-configuration
openstack-horizon
multibinding
filehelpers
minimagick
restructuredtext
ejb-3.1
angular2-pipe
v4l2
pumping-lemma
apdu
pixels
marionette
resx
mobile-development
schedule
luhn
machine-code
ansible-inventory
dotnetnuke-module
byobu
mybatis-generator
wicket-tester
pyexcel
autodesk-data-management
datagridviewcombobox
nshttpurlresponse
f#-fake
multivalue
standard-error
fluent-nhibernate-mapping
multi-targeting
pyautogui
esprima
google-scholar
android-music-player
mavlink
instaparse
change-password
django-taggit
sbt-proguard
audiorecord
transaction-isolation
archiva
innertext
clique
jcalendar
nugetgallery
music-player
bluetooth-lowenergy-4.2
google-finance-api
android-facebook
play-json
random-sample
self-join
android-filterable
chrome-mobile
alpha-beta-pruning
clearcanvas
javacompiler
fscanf
rpt
telerik-appbuilder
nodeunit
sqldatasource
fastcgi-mono-server
windows-phone-store
ekevent
sygic
violation
record-locking
stringstream
azure-scheduler
wordbreaker
tasklist
openafs
certificate-revocation
nsscrollview
nude.js
unordered-set
scsf
jqzoom
mcpd
s60
xcdatamodel
mysql-pconnect
scala-designer
office-2003
moores-law
weak-typing

Resources

Mobile Apps Dev
Database Users
javascript
java
csharp
php
android
MS Developer
developer works
python
ios
c
html
jquery
RDBMS discuss
Cloud Virtualization
Database Dev&Adm
javascript
java
csharp
php
python
android
jquery
ruby
ios
html
Mobile App
Mobile App
Mobile App