c#


OutOfMemoryException on MemoryStream writing


I have a little sample application I was working on trying to get some of the new .Net 4.0 Parallel Extensions going (they are very nice). I'm running into a (probably really stupid) problem with an OutOfMemoryException. My main app that I'm looking to plug this sample into reads some data and lots of files, does some processing on them, and then writes them out somewhere. I was running into some issues with the files getting bigger (possibly GB's) and was concerned about memory so I wanted to parallelize things which led me down this path.
Now the below code gets an OOME on smaller files and I think I'm just missing something. It will read in 10-15 files and write them out in parellel nicely, but then it chokes on the next one. It looks like it's read and written about 650MB. A second set of eyes would be appreciated.
I'm reading into a MemorySteam from the FileStream because that is what is needed for the main application and I'm just trying to replicate that to some degree. It reads data and files from all types of places and works on them as MemoryStreams.
This is using .Net 4.0 Beta 2, VS 2010.
namespace ParellelJob
{
class Program
{
BlockingCollection<FileHolder> serviceToSolutionShare;
static void Main(string[] args)
{
Program p = new Program();
p.serviceToSolutionShare = new BlockingCollection<FileHolder>();
ServiceStage svc = new ServiceStage(ref p.serviceToSolutionShare);
SolutionStage sol = new SolutionStage(ref p.serviceToSolutionShare);
var svcTask = Task.Factory.StartNew(() => svc.Execute());
var solTask = Task.Factory.StartNew(() => sol.Execute());
while (!solTask.IsCompleted)
{
}
}
}
class ServiceStage
{
BlockingCollection<FileHolder> outputCollection;
public ServiceStage(ref BlockingCollection<FileHolder> output)
{
outputCollection = output;
}
public void Execute()
{
var di = new DirectoryInfo(#"C:\temp\testfiles");
var files = di.GetFiles();
foreach (FileInfo fi in files)
{
using (var fs = new FileStream(fi.FullName, FileMode.Open, FileAccess.Read))
{
int b;
var ms = new MemoryStream();
while ((b = fs.ReadByte()) != -1)
{
ms.WriteByte((byte)b); //OutOfMemoryException Occurs Here
}
var f = new FileHolder();
f.filename = fi.Name;
f.contents = ms;
outputCollection.TryAdd(f);
}
}
outputCollection.CompleteAdding();
}
}
class SolutionStage
{
BlockingCollection<FileHolder> inputCollection;
public SolutionStage(ref BlockingCollection<FileHolder> input)
{
inputCollection = input;
}
public void Execute()
{
FileHolder current;
while (!inputCollection.IsCompleted)
{
if (inputCollection.TryTake(out current))
{
using (var fs = new FileStream(String.Format(#"c:\temp\parellel\{0}", current.filename), FileMode.OpenOrCreate, FileAccess.Write))
{
using (MemoryStream ms = (MemoryStream)current.contents)
{
ms.WriteTo(fs);
current.contents.Close();
}
}
}
}
}
}
class FileHolder
{
public string filename { get; set; }
public Stream contents { get; set; }
}
}
The main logic seems OK, but if that empty while-loop in main is literal then you are burning unnecesary CPU cycles. Better use solTask.Wait() instead.
But if individual files can run in Gigabytes, you still have the problem of holding at least 1 completely in memory, and usually 2 (1 being read, 1 being processed/written.
PS1: I just realized you don't pre-allocate the MemStream. That's bad, it will have to re-size very often for a big file, and that costs a lot of memory. Better use something like:
var ms = new MemoryStream(fs.Length);
And then, for big files, you have to consider the Large Object Heap (LOH). Are you sure you can't break a file up in segments and process them?
PS2: And you don't need the ref's on the constructor parameters, but that's not the problem.
Just looking through quickly, inside your ServiceStage.Execute method you have
var ms = new MemoryStream();
I don't see where you are closing ms out or have it in a using. You do have the using in the other class. That's one thing to check out.

Related Links

WebDeploy (Package) from Visual Studio 2015 unexpected content
Why is Entity Framework's migrations Configuration.cs file located and executed automatically?
How to receive Authorization header in httpclient class
RazorEngine - Windows 10 Anniversary update 1607 - Could not load file or assembly System.Web.Razor
WPF ContextMenu: MenuItem Icon visibility binding error
failure to execute dotnet core site using framework 4.6.1 on azure
Unity2D - moving 2D object in a grid like, timed fashion
how to join in Linq dan display like tree in gridview
How to set Checkbox value in Word document using OpenXML
how to insert input type date to mysql using asp.net
Why isn't Text Binding Updating when it is empty?
Get Temp folder path or version of Dynamics NAV in C/AL?
Why can't you pass int[] array into a method that accept object[] array in C#? [duplicate]
C# -HttpWebRequest returns 400 Bad request when connecting to Report Server programmatically
Open delegate to IFormattable.ToString method in struct (Int32, DateTime)
SSRS Report Server sending report to web client using C#

Categories

HOME
alasql
mobile
webvr
ebay-api
highmaps
where
okhttp3
pandoc
opc-ua
powerquery
zabbix
vsixmanifest
software-collections
distinct
ibatis
beacon
caldav
gomobile
git-branch
ajaxcontroltoolkit
multipart
angular2-pipe
gnu-screen
priority-queue
deployment-descriptor
marionette
typeorm
federated
concrete5
schedule
opencsv
webspeech-api
imageprocessor
machine-code
angularjs-resource
javax
pyenv
perf
socketcluster
firefox-developer-edition
f#-fake
result
ocpjp
spring-ide
xmldom
e4
visio-2010
perldoc
r-commander
breadcrumbs
alarmmanager
turbolinks
optional-parameters
glade
dbi
class-design
nimbus
bytearray
avi
alchemy
methodology
event-tracking
serial-communication
database-project
random-sample
adempiere
cloud-storage
genome
jad
iphone-6
zoneminder
sat4j
serializer
step
cs193p
nvcc
nativequery
ardor3d
infinity.js
select2-rails
qmainwindow
windows-scheduler
bho
pageload
architectural-patterns
pymel
libx264
teamsystem
beep
unordered-set
parallel-python
funambol
cryptolicensing
quazip
system-analysis
time-management

Resources

Mobile Apps Dev
Database Users
javascript
java
csharp
php
android
MS Developer
developer works
python
ios
c
html
jquery
RDBMS discuss
Cloud Virtualization
Database Dev&Adm
javascript
java
csharp
php
python
android
jquery
ruby
ios
html
Mobile App
Mobile App
Mobile App