Troubleshooting Server Crash

I was working on this project for almost 5 months and it was unfortunate and my bad luck that every time we thought we are done some issue or other would come and create a havoc for us. It was really a tough time and client was infact more tougher and demanding. So atlast when we resolved all issues somehow (hit n trial) and given the system for final QA to client. Within 5 minutes of testing the system crashed and we had no clue why it could have happen.

It become more troublesome for me when I received a mail from the external project manager address to my CEO. Suddenly there was lot of pressure on me and trust me it took 2 weeks to find what cause the crash. In those two weeks life was hell and I used every possible debugging tool available on web but all point to different things. There were no two logs providing me same results. Infact the system starts crashing because of using the tool. Lot of discussion went with me and my senior colleague to nail down the issue and atlast we were able to reproduce the issue and found that if we rapid click on all links on the left bar of the application it crashes the system. Trust me we were very happy. Atleast we have something to tell our client.

But now it was more tougher job how do we overcome this issue. We thought and thought and keep on thinking.  We started isolating the pages and found that one particular page is causing the issue. I didn’t have any answer why this page should crash the IIS. It was big question mark for me. The same weekend while travelling to mumbai I started thinking again and found that the only differnece between this and other page is MULTI THREADING. This page uses multi threading to render its content while others don’t use it.

The first thing on Monday morning I did was identify can we do without multi threading as it was important for performance. But as it was crashing I have no other option. We did the following

  1. Reduced the number of queries to database.
  2. Reduced to number of unnecessary iterations that create in memory objects from DB.
  3. Serialize the code that were using threads.

The results were great

  1. Crash problem got resolved.
  2. We achieved the same performance that was with Multi Threading.

It was great feeling.

Advertisements

One Response

  1. Loved it.
    Keep up the good work.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: