c#


OutOfMemoryException on MemoryStream writing


I have a little sample application I was working on trying to get some of the new .Net 4.0 Parallel Extensions going (they are very nice). I'm running into a (probably really stupid) problem with an OutOfMemoryException. My main app that I'm looking to plug this sample into reads some data and lots of files, does some processing on them, and then writes them out somewhere. I was running into some issues with the files getting bigger (possibly GB's) and was concerned about memory so I wanted to parallelize things which led me down this path.
Now the below code gets an OOME on smaller files and I think I'm just missing something. It will read in 10-15 files and write them out in parellel nicely, but then it chokes on the next one. It looks like it's read and written about 650MB. A second set of eyes would be appreciated.
I'm reading into a MemorySteam from the FileStream because that is what is needed for the main application and I'm just trying to replicate that to some degree. It reads data and files from all types of places and works on them as MemoryStreams.
This is using .Net 4.0 Beta 2, VS 2010.
namespace ParellelJob
{
class Program
{
BlockingCollection<FileHolder> serviceToSolutionShare;
static void Main(string[] args)
{
Program p = new Program();
p.serviceToSolutionShare = new BlockingCollection<FileHolder>();
ServiceStage svc = new ServiceStage(ref p.serviceToSolutionShare);
SolutionStage sol = new SolutionStage(ref p.serviceToSolutionShare);
var svcTask = Task.Factory.StartNew(() => svc.Execute());
var solTask = Task.Factory.StartNew(() => sol.Execute());
while (!solTask.IsCompleted)
{
}
}
}
class ServiceStage
{
BlockingCollection<FileHolder> outputCollection;
public ServiceStage(ref BlockingCollection<FileHolder> output)
{
outputCollection = output;
}
public void Execute()
{
var di = new DirectoryInfo(#"C:\temp\testfiles");
var files = di.GetFiles();
foreach (FileInfo fi in files)
{
using (var fs = new FileStream(fi.FullName, FileMode.Open, FileAccess.Read))
{
int b;
var ms = new MemoryStream();
while ((b = fs.ReadByte()) != -1)
{
ms.WriteByte((byte)b); //OutOfMemoryException Occurs Here
}
var f = new FileHolder();
f.filename = fi.Name;
f.contents = ms;
outputCollection.TryAdd(f);
}
}
outputCollection.CompleteAdding();
}
}
class SolutionStage
{
BlockingCollection<FileHolder> inputCollection;
public SolutionStage(ref BlockingCollection<FileHolder> input)
{
inputCollection = input;
}
public void Execute()
{
FileHolder current;
while (!inputCollection.IsCompleted)
{
if (inputCollection.TryTake(out current))
{
using (var fs = new FileStream(String.Format(#"c:\temp\parellel\{0}", current.filename), FileMode.OpenOrCreate, FileAccess.Write))
{
using (MemoryStream ms = (MemoryStream)current.contents)
{
ms.WriteTo(fs);
current.contents.Close();
}
}
}
}
}
}
class FileHolder
{
public string filename { get; set; }
public Stream contents { get; set; }
}
}
The main logic seems OK, but if that empty while-loop in main is literal then you are burning unnecesary CPU cycles. Better use solTask.Wait() instead.
But if individual files can run in Gigabytes, you still have the problem of holding at least 1 completely in memory, and usually 2 (1 being read, 1 being processed/written.
PS1: I just realized you don't pre-allocate the MemStream. That's bad, it will have to re-size very often for a big file, and that costs a lot of memory. Better use something like:
var ms = new MemoryStream(fs.Length);
And then, for big files, you have to consider the Large Object Heap (LOH). Are you sure you can't break a file up in segments and process them?
PS2: And you don't need the ref's on the constructor parameters, but that's not the problem.
Just looking through quickly, inside your ServiceStage.Execute method you have
var ms = new MemoryStream();
I don't see where you are closing ms out or have it in a using. You do have the using in the other class. That's one thing to check out.

Related Links

How to force a listbox to always have at least one item selected and highlighted?
Locking VSS progammatically
Allow null value when parsing file to datatable
C#: Replace a single / with a *single* \
How can i find what is missing at run time that makes my application crash?
Refresh ListView grouped items
Starting a windows service on remote machine in different domain
Double value not correct on Device
What would be the best and easiest way of getting currency rates within SharePoint (Custom Actions) or C# project
Decimal.Parse Issue
Route patterns vs individual routes
Why entity framework isn't implement identity map with unit of work?
How to wrap a thread unsafe class into one that is thread safe without wrapping each member function
Read console output while program is running c#
How to run a for loop after a for loop?
alert box from asp.net 3.5 Code-Behind

Categories

HOME
google-cloud-pubsub
optimization
ftp
domain-driven-design
aix
arguments
thermal-printer
grid
open-search-server
subprocess
dynamics-crm-2011
guidewire
desktop-application
comma
swift3.1
ofbiz
orange
computation-theory
xna
connectiq
smali
prepros
advertising
jira-zephyr
deployment-descriptor
marionette
large-files
drawrect
intermec
jql
spring-oauth2
angularjs-resource
wunderground
certbot
android-geofence
motherboard
armadillo
azure-web-roles
jquery-ui-slider
vpython
chrome-custom-tabs
mef
denodo
spring-cloud-bus
lucee
visualizer
px4
sharpdevelop
activesupport
multi-select
spring-cloud-aws
iban
scrollmagic
ldd
foreground
csc
android-xmlpullparser
skbio
wonderware
bioconductor
transbase
jain-sip
azure-mobile-app
sparkcore
clearquest
targetprocess
fileinputstream
tidesdk
jeet-grid
service-broker
pymol
pysvn
crystal-reports-7
fxmlloader
conditional-comments
ccmenuitem
mvcgrid
xpand
visual-studio-setup-proje
carrier
svg-android
grunt-contrib-imagemin
scidb
pechkin
octokit
cab
ia-32
wpd
stringstream
kademlia
proj4
qmainwindow
bho
doxia
jqueryi-ui-buttonset
code-organization
cherokee
parallel-python
lightopenid
moss2007-security
norton
linq-to-mysql
web-search
helios
dojox.gfx
atmega16
ffdshow
unauthorized

Resources

Mobile Apps Dev
Database Users
javascript
java
csharp
php
android
MS Developer
developer works
python
ios
c
html
jquery
RDBMS discuss
Cloud Virtualization
Database Dev&Adm
javascript
java
csharp
php
python
android
jquery
ruby
ios
html
Mobile App
Mobile App
Mobile App