aggregate-functions


Howto aggregate on full data set in Spring Batch jobs?


I need to insert aggregation in my Spring Batch jobs.
But the aggregation step need to have the entire data set available.
In pure SQL, it's easy to code SQL aggregation requests : the full data set (as stored in database) is available.
But in Spring Batch jobs, everything is done in memory, and spread in chunked.
So howto deal with that kind of data strewing ?
Do you have any advice concerning the best practices to insert aggregation steps/processes ?
Thx a lot for your enlightments
You have Partitioning option in spring batch which can have StepExecutionAggregator, it has aggregate method which accepts list of StepContext of all partitioned steps.
We had i.e. integration with soap server where we first received list of something that needs to be processed, than we partitioned it to child steps and processed in parallel and after each child step finishes aggregator is invoked which can do stuff based on data in child step context.
It is good way if you have something in your data which can be good rule for partitioning (i.e. pull list of items from DB and process each item in parallel, save item data in step context, use aggregator and combine everything in each step context and do common operation on combined data).
Here is link to example with partitioning (there is no aggregation but you can add it to masterStep).

Related Links

duplicate query on concatenated field
Aggregations With Geomesa
incomplete results with aggregate (R)
How to do an aggregated OR on a column in Amazon Redshift
'Your query does not include the specified expression as part of an aggregate function' error for calculated fields
Creating percentile_cont as aggregate function in Vertica
Create Expression in Report Builder 3.0 Report to sum a column
SAP Hana - STRING_AGG issue with ORDER BY CLAUSE
rrdtool does not keep max
Tableau Aggregate Percentage
use aggregates (min, max, avg) in CONSTRUCT query [duplicate]
R: transposing and aggregating columns
SQL Query Aggregate Error
Usage of aggregate functions in MarkLogic
Querying customDimensions in Google Bigquery
NHibernate - QueryOver criteria appearing in Where instead in Having clause, error

Categories

HOME
ftp
domain-driven-design
window.open
highmaps
crm
flowtype
doctrine2
segmentation-fault
ios-universal-links
theory
jpql
unreal-engine4
tmux
fastreport
react-select
box2d
jsfiddle
ibatis
ofbiz
alert
airbnb
django-rest-auth
summernote
computation-theory
minimagick
angular2-pipe
chartist.js
continuous-fourier
traveling-salesman
android-cursorloader
ballerinalang
workflow-foundation
emr
jbossfuse
paper-trail-gem
edsdk
keil
contextmenustrip
favorites
dom4j
ponylang
ragel
perf
rythm
ngrok
ng-lightning
fisheye
vertical-alignment
collation
ape-phylo
nshttpurlresponse
f#-fake
weld
reqif
spring-webflow-2
spring-ide
anonymous-function
opos
stream-socket-client
skype-bots
openpgp.js
sp
cc
ios-keyboard-extension
scrollmagic
alarmmanager
settext
class-design
normalisation
weblogic-maven-plugin
diameter-protocol
mrunit
audiorecord
jde
pyshark
jmap
periodic-processing
dcast
trendline
ooad
bluetooth-lowenergy-4.2
encode
syncano
firewire
vspackage
gcc-warning
asp.net-2.0
douglas-peucker
fiware-health
ios-enterprise
amzi-prolog
nspopupbutton
isl
image-preloader
object-oriented-database
mixradio
ojb
angularjs-select2
gpl
cron4j
netbiscuits
hypertable
waveout
django-admin-tools
win-prolog
motorola-droid
copyfile
backlight
jexcelapi
cinder
nstoolbar
authkit
wtsapi32
office-2003

Resources

Mobile Apps Dev
Database Users
javascript
java
csharp
php
android
MS Developer
developer works
python
ios
c
html
jquery
RDBMS discuss
Cloud Virtualization
Database Dev&Adm
javascript
java
csharp
php
python
android
jquery
ruby
ios
html
Mobile App
Mobile App
Mobile App