SBS 2011 Virtual Machine Lost

Posted: April 9, 2012 in Exchange 2010, Hyper-v, iSCSI, SBS 2011, Virtual Machine
Tags: , , ,

Summary

I guess it had to happen, and that is why we do backups and more.  But sometimes it is not quite enough and this was one of them.

I was logged onto a test server where I am creating a SharePoint application.  This server is a virtual W 2008 R2 connected via a VPN to a virtual SBS host. While I was doing some work, I noticed the VPN connection dropped.  Strange, I thought.  I remoted to the host machine for the SBS server and saw in Hyper-v Manager that SBS was stopped.  I checked on the drive where the .vhd files are stored and noticed that it was absent from the host server.  Oh, just a dropped iSCSI volume – something that happened a few times before (more about that later).  Simple enough, just reconnect.

Imagine my surprise when I couldn’t reconnect to the iSCSI target LUN!   I am going to skip over the details of why, for now, and get to the bigger point.  I was able to do a power cycle on the iSCSI box, and of course I could re-connect to the target.  Now, just restart the SBS virtual machine.

Horror Show Part I

The machine would not start.  Hyper-v reported a disk corruption and could not use the .vhd.  Well, that was a minor set-back.  I had a USB drive attached to the iSCSI box that did daily backups of the LUN.  I’ll just restore that to another iSCSI target I would create, just in case.

It took about an hour and lo and behold, a new and freshly restored LUN with the .vhd files.  I’ll attach it and start the virtual SBS 2011.

Horror Show Part II

You already knew, I bet, that it wouldn’t be that easy.  Diagnosis?  The LUN to USB is a copy and replace, so only one backup to restore.  That backup occurred AFTER a disk failed and was in rebuild on a RAID 6.  Doing the power cycle caused some data corruption, seemingly limited to the SBS 2011 .vhd.  Hyper-v failed to boot with a missing boot file.  Sigh.

Sounds like being up that creek.  There was no Windows Server backup of that volume.

But, we had installed AppAssure’s Replay backup. Unfortunately, some disk issues had caused it to fail and we were investigating the cause.  As it turns out, engineers were evaluating the logs that morning from some I/O stress testing we did.  They called to say the iSCSI target was generating lots of retrys, and a call to the vendor of he iSCSI box helped us determine bad sectors on a drive that went into rebuild that eventually died and …. you get the idea.

Even though Replay was disabled when this happened, it had created lots of backups during the previous two weeks. I picked the most recent backup of the volume and mounted it.  Sure enough, there was my precious .vhd file.  I coped it to a new location ( a lengthy process), modified the machine settings to point to the new .vhd, and fired it up.

Horror Show Starts to End

YES! It started fine.  Problem solved!

Of course not,  All services started but Exchange mailboxes were not accessible.  Neither the mailbox nor the public store properly mounted.  Here are some of the things I did to solve the problem:

  1. Ran eseutil.exe with /mh option.  This reported that the database was in a dirty shutdown mode.
  2. Ran eseutil .exe with /r option.  This ran successfully, but a subsequent eseutil /mh still reported dirty shutdown.
  3. Ran eseutil.exe with /p option.  Ran successfully, then eseutil /mh reported clean shutdown.

In order to run eseutil.exe with the various options, try this article.  To keep from typing long and tedious paths to the database and to the log file folder, stash them somewhere on your desktop and copy and paste them at the appropriate parts of the command line.  Remember these points:

  • Enclose paths and file names that contain spaces in double quotes.  I forgot and used single quotes and it couldn’t find the files.
  • Run cmd as administrator.
  • Set the path to the database location for convenience.

If you have not moved Exchange files, the default location  is “C:\Program Files\Microsoft\Exchange Server\V14\Mailbox\<mailbox name>”; otherwise it is exactly the same path but a different drive letter.  Get the mailbox name form Exchange Management Console Organization Configuration Mailbox.  NO .edb for the path name, please!

The log files will be in the original path on drive C: unless you manually moved them.

Public database is at “C:\Program Files\Microsoft\Exchange Server\V14\Mailbox\PublicFolderDatabase” or on another drive letter if moved and the log files on drive C:

For the /r option, the log prefix is by default E00 for the mailbox and E01 for public folders.

And yes, I had to repair each.  I repaired them at the same time using two open CMD windows.

It will take a long time if you have a sizable mailbox.  A nearly 30GB one took around 5 hours.

The End, Almost

Once they mounted, mail began to flow and all appeared well.  Except… several iPhones were no longer receiving email.  And ActiveSync was reporting errors on them.

Fixing this was easy, but it deserves a separate post.

Footnote

If you don’t know your locations precisely for eseutil, then rely on EMC.  Click on either the mailbox or public database that is unmounted, then chose Move Database Path from the actions pane.  In the first dialogue box that appears, the path to the database and the log files is there for you to copy and paste.  Remember you just want the database path or the path name unless you run eseutil from another path entirely.

When you run a repair with the /p option, you might loose bits of your mailbox.  I have done it before without loss, and this time the same thing.  But it is better than having nothing.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s