Tackling the OutOfMemoryException

Couple of months before I was quite struggling to solve one of my students problem. The problem is as follows

“The program need to read the file which stores data about many files and the file content itself.”

The program works like this

  1. Open the file.
  2. Read the metadata about each file i.e. name, size, createdon etc.
  3. Based on the length of each file, read the content of the file in one shot.
  4. Later on based on user interaction, uncompressed the selected file.
  5. Loading of the file earlier used to help it to uncompressed more faster as the content was in memory.

It works great in a normal scenario which was of size around 1 to 2 GB. But when he started stressing it out i.e a file of size sa 10GB it start failing.

We debugged the issue and found that there was a file of size 2 Gb which was trying to read and causing a OutOfMemoryException

byte[] fileContent = new byte[file.length];

Here the length was 2GB, which resulted in the said exception.

The following approach was then chosen to solve the issue

  1. Stop reading the file while reading the compressed file.
  2. This improved the performance of the app by almost 300-400%.
  3. The problem of uncompressing the file was still there.
  4. So instead of reading entire file in one chunk the file was read in many chunks.

The following is the code

remainingBytes = (long)FileLength;
bytesRead = (long)ChunkSize;
while (bytesRead > 0 && (Data = Reader.ReadBytes((int)bytesRead)) != null)
{
 	file.Write(Data, 0, (int)bytesRead);
	remainingBytes -= bytesRead;
	if (remainingBytes < bytesRead)
		bytesRead = remainingBytes;
}

The chunk size was kept configurable, so that it can be change without any code changes required.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: