This morning, a client called to tell me that their incoming e-mail had stopped over the weekend.  They could still send e-email, though. Probably like you, e-mail is their bread and butter life line to their customers and their vendors.  Maybe like you as well, they prefer to wait until their hair is on fire to do anything; active monitoring is not something they are interested in.

The server showed no reals signs of distress.  All services were up and running.  I could telnet on port 25 to Exchange.  The recent entries in the event log looked pretty normal.  My first symptom that something was amiss was that I could not connect to remote web workplace. IE did not error out in any way, it just sat for about 15 minutes waiting to connect.

As I was starting to dig deeper, one of the users forwarded me a NDR he received when he tried to send himself a message from his private account to the domain account.  The error code in the NDR was 452.4.3.1 insufficient system resources.

That proved to be the essential clue.  I looked at the drive where the Exchange data was stored, and it had only a bit over 5GB remaining.  The thin space was not so much caused by a growth in the Exchange stores but by some accumulation of junk in other folders.  Fortunately, there was a ton of empty space on a 1TB drive on the system.   I moved both Exchange and SharePoint data to that drive; user data was already there.

Since backups were only three hours stale, and they had not received incoming messages for more than 24 hours, I just skipped the new backup.  The moves went pretty quickly with no problems.  I am not sure I wouldn’t advise you to take another backup, just in case, but when the client is nervous…..

I hope you don’t suppose that solved the problem and starting incoming email.  It didn’t.

In order for everything to get cranked back up, some Exchange services had to be restarted.  I ended up restarting all of the services that were running, which was probably overkill, but instead of sorting out which were essential it seemed faster to do them all rather than continue to wait for incoming mail to show.  That feel of the client breathing down your neck…

Here are today’s lessons:

  1. With storage so cheap today, don’t be stingy. Give your Exchange and SharePoint data plenty of room to play.  If possible, devote a drive to Exchange.
  2. Monitor your drives occasionally. If you see a sudden decrease in available space, check to see if someone has started dumping stuff onto the drives (in this case, someone decided to use a server drive to do a full backup of a laptop.  When the person has admin privileges, what’s one to do?
  3. If you have smallish drives, go buy some new bigger ones. A 2TB drive on the commercial side is around $200 or less, and more standard ones under $100.
  4. Better still, invest in iSCSI targets. While you cannot use USB drives for Exchange data, you can with an iSCSI target.  You can provision a new drive in a few moments, attach it to your server (or anything else), and you are off and running.  Literally.

It’s true that an iSCSI target device is more expensive than USB or internal drives, but the increased versatility and flexibility is really worth it.  But let me calibrate more expensive for you.

I have had great success with a particular vendor, and their product line goes from a one drive unit up to an eight drive unit, and you can daisy-chain two of them together.  The software is essentially the same across all of the physical platforms, so you get all the benefits no matter what capacity you choose.  At the upper end, their eight drive units cost between $1400 and $1800 street price, depending on whether you choose rack mount or standalone and whether you get a first or second generation processor.  You then have to throw in the drives (a lot of companies resell them populated, but I prefer to buy my own drives and shop for the best deals; installing them is a trivial affair).

I am planning a post on iSCSI, so if you are not familiar with it, stay tuned for the details.  When I first encountered it, I thought this stuff is way too complex and esoteric and expensive.  It is not.  Especially if you have any thoughts about using virtual machines.  If I want to create a virtual machine, I can just create a new iSCSI target, attach it to the host, and then provision the machine on it.   But I will explain more later.

  1. Marco says:

    Thank you for this post. It saved my morning 🙂


  2. always a great thing to hear. Happy new year


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s