Troubleshooting Application Server

It was the same client that I discussed in the previous blog Troubleshooting Server Crash that we encountered another server crash related issue long back this issue occurred. The issue was our Application Server was crashing once sometimes twice in a day and the resolution was restart the Application Server. The Application Server was a Windows Service and the backend database was SQL Server 2000.

This problem almost run for months. We initially start speculating the problem to be related with number of database connection in the pool so as a first step we increased the number of database connection. But this didn’t helped us. So we started the usual process of instrumenting the application code and checking the log files. It took us sometime to come with a clear picture. The reason for the crash was identified. The reason was the Application Server used to send mails as part of some workflow process, so while sending the mail it used hung up and caused the database to go into a blocked state i.e. the tables used by that particular request’s database connection were locked. Slowly slowly all the database connection used to wait for the tables locks to be released.

There were two solution we could suggest

  1. Have a dedicated process for sending mails.
  2. Close the database connection before sending mails. So that tables are not locked even if there is an issue while sending mails.

After lot of thoughts the first solution was used for the following reasons

  1. Restart of application server is much more expensive then the restart of Mail Service.
  2. The logic of the program for some reason required the connection to be open while sending mails, so we could not close the database connection while sending mails.

After the changes were send to the production server we didn’t face the issue again as while writting the new mail service we created Watcher thread that will monitor the mail sending process. If the mail sender thread doesn’t return in a configured amount of time we killed that thread and start the mail sender again. this was quite useful as it really helped us to solve the issue.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: