Archive for the ‘Exchange 2010’ Category


Introduction

Guess I am like the cobbler’s kids who didn’t get their shoes when everyone else did.  I am the last of my clients, save one, to migrate off SBS and onto Windows 2012 R2 Standard.  I thought you could benefit from some of the issues I ran into, and solved.

Here’s my scenario.  Have a Hyper-V server running SBS 2011 as a virtual machine.  Created a new virtual machine and installed Windows 2012 R2, did updates.  Unfortunately, it sat for several months while I finally got a few days to do he migration.  More about that later.

Long ago, I migrated email and SharePoint to Office 365, so I had disabled services and IIS application pools on SBS.  My starting point was to fire them up again in order to remove them.  I did not want AD to migrate with all of those extra objects.

Removing Exchange Server 2008 R2

This did not start off well.  When I launched EMC, it failed to connect to the SBS server.  I ended up putting it aside for a few days but came back to it.  I had tried to guess or remember which services should be started, but I seemed to have failed.  I also only started the Exchange-related application pools.  I easily tracked down an article describing which services start automatically, fired them up, and enabled those that should start manually.  BTW it describes all of SBS services.

That did the trick, and EMC successfully opened.  But I knew that in order to uninstall Exchange, I had to remove the mailboxes.  Fortunately, I had a small number of them.  And to make it easy, I used Exchange PowerShell commands to do this.

Get-Mailbox | Disable-Mailbox
Get-Mailbox -Archive | Disable-Mailbox -Archive
Get-Mailbox -Arbitration | Disable-Mailbox -Arbitration

First, please note that my scenario had with SBS – a single mailbox database and server.  That is why there are no qualifying parameters on the commands.

If your first thought is to use EMC to remove mailboxes, CAUTION!  That method removes both the mailbox AND the user from AD.  If you do want to remove some users and their mailboxes, do that but otherwise use Disable-Mailbox.  There is a Remove-Mailbox command but it also removes both user and mailbox.

So what the first command does is get a list of mailboxes which are piped to the next command.  The second does the same thing but gets the archive mailboxes.  You will then not be surprised the third command gets the arbitration ones.

I tried just getting archive and arbitration mailboxes, but the id names were too long to display in their entirety, so piping was essential.  And easier. And faster.

I then tried to uninstall Exchange but got two failures.  The first block came from Trend Micro Worry Free Advanced that it was using the database and the second was the Offline Address Book in Public Folders.  I uninstalled the messaging agent for Trend Micro but getting rid of the OAB was harder.

First I tried to simply delete the public database but it was not empty (I knew that from the OAB warnings).  I then tried to create a new, empty one to mount but then Exchange would not let me create a second one.  So under Tools, I chose Public Folders and expanded the tree and selected OAB then the firs entry and deleted that.  It was the only entry I could delete.

Luckily, I was then able to uninstall Exchange.

Removing SharePoint

Could not have been easier.  Just uninstall from Control Panel.

Back to Windows 2012 R2

I had already joined the domain and installed Active Directory Services, so I was ready to promote it to a domain controller by starting the configuration on that role.  Just added to the existing domain and it just worked.

I needed a few more roles on this new server, and then trouble started.  I tried to add both Remote Access and Windows Update at the same time and the installation failed.  Separately they also failed.  Again and again.

I cheeked for updates and found plenty and installed and rebooted.  Still no luck in adding roles or features.  I finally found this article which pointed me to a fix.  Note that I modified both policies that it refers to.  After the gpupdate, installation of roles worked fine.

I had not run into this issue before with 2012 R2, so I think it is related to both the GPO settings from SBS 2011 and that AD is at 2008 R2 levels and cannot be promoted until SBS is removed from the domain.

But this puts me well on my way to being where I want to be.

Now I just have to move Worry Free Advanced and get client machines set up under Windows Essentials role.


In my previous post, I listed the registry change to get sent email into the shared mailbox sent folder; otherwise it ends up in the sent folder of the primary email account.  Turns out the very same thing happens to deleted items, but there is also a registry key that will fix that.

  1. Close Outlook if it is running.
  2. From the search bar in Windows 10, type regedit and then click on the result to run.  In Windows 7 click Run and type regedit then click OK. Or from Windows 8 search, type regedit and run it.
  3. Navigate to HKEY_CURRENT_USER\Software\Microsoft\Office\16.0\Outlook\Options\General.Note Office 2013 uses 15.0, Office 2010 uses 14.0, 2007 12.0 instead of 15.0
  4. In the right hand pane, right-click and choose New DWORD.
  5. Type DelegateWastebasketStyle for the name and press enter.
  6. Right-click on the newly created entry and choose Modify.
  7. Enter 4 for the value.
  8. Exit regedit.
  9. Restart Outlook.

You are done.


What Are Shared Mailboxes

Although shared mailboxes are a feature of Exchange Server, I will focus more their use in Office 365.  A shared mailbox there is one which doesn’t require a specific user license but is created with access granted to other users how do have Exchange licenses. Additionally, they can be granted permissions as well.

Why use shared mailboxes and not just use a distribution list?  When you use a distribution list, incoming mail is forwarded to each person on the distribution list, and in general it is mixed with all the email for all addresses associated with that person.  In other words, mail from one particular address is not distinguished and separated.

A bigger problem can occur when sending replies or new emails. Although  inbound mail is sent to everyone on the distribution list, there is no guarantee that any replies, or new emails, will be copied to those recipients as well.  Keeping all of the sent and received emails together is at best challenging.

Shared mailboxes solve this problem.  It has its own set of standard folders such as inbox, sent, etc., and you can create custom folders.  If you have send permissions you can send from its email address.

So What Is the Problem?

Standard behavior for Outlook is to put all sent mail into the primary mailbox Sent folder, no matter what email address or mailbox you send from.  If you have all of the email related to the shared mailbox in its container, why would you want to mix sent items with your own?

There is a Solution

First, close Outlook if it is running.  While you can make the change while it is running, it needs to be restarted before it will work, so you might as well do it now.

From the search bar in Windows 10, type regedit and then click on the result to run.  In Windows 7 click Run and type regedit then click OK. Or from Windows 8 search, type regedit and run it.

Expand HKEY_CURRENT_USER and keep expanding until you have reached

HKEY_CURRENT_USER|Software\Microsoft\Office\16.0\Outlook\Preferences

Note this is for Office 2016.  If you are using Office 2013, change the 16.0 to 15.0; Office 2010, change the 16.0 to 14.0.

In the right hand pane, right-click and choose New->DWORD.   For the value type

DelegateSentItemsStyle

and press Enter.  Then right-click on that key when it shows up and click Modify.  In the Value data box, enter 1 and click OK.

Exit the registry editor.

Start Outlook and give it a try.  You will see sent items for the shared folder in its sent folder, not in your primary email sent folder.

 

 


Scenario

You have configured incoming e-mail for your SharePoint site (see my earlier post on how to do this) and have assigned an e-mail address to your list or library.  Incoming e-mails have been posting as you expected, then all of a sudden they don’t.  What’s up with that?

Things You Probably Check

If you are like me, here is what you might check for:

  1. In Central Administration, is inbound e-mail configured properly?
  2. Does the dropbox folder still exist, and it’s path match what you specified in #1 above?
  3. Are permissions on the dropbox folder correct?  The dropbox directory must have the following permissions assigned to it:
    • Network Service: Full Control
    • System: Full Control
    • Administrators: Full Control
  4. Is other incoming e-mail working, and if not, what is disrupting Exchange performance?
  5. Another thing I checked was the job status in SharePoint Central Administration.  I was looking for the Timer job for in-coming e-mail, and clicked to run it now.  No luck of course.

If you make it through all of those without discovering anything, the next thing to do is look in the dropbox folder.  I always use \inetpub\mailroot\drop, but your path may be different.

The Fix

When I browsed to that folder, I found a bunch of .eml files.  These were the attachments from incoming e-mail that were supposed to have landed in (in my case) a document library.  It’s as though they arrived at the airport but were being detained by customs agent, but no particular reason why.  And this post does not hint at why, either. But here’s how to spring them from confinement.

Simply run SharePoint configuration again.  Click through the default settings on the wizard and let it run.  I had the dropbox folder open, and I will leave it to you to imagine my pleasure at seeing, during Step 9 when the timer was being configured, to see all the waiting .eml files simply disappear.  Once the wizard completed, I opened the site and document library, and there they were, as expected.

Hope this helps and if anyone can help provide the cause, please share it with us.


The Remaining Problem to Be Solved

In my previous  post, I discussed the loss of a .vhd file for SBS 2011 and how I managed to retrieve an older copy and mount that.  However, I was not able to mount the Exchange mailbox and public folder databases.  I solved that problem by running eseutil.exe with the /p option and they mounted as expected.

What started to happen were critical and error events to the application log.  While most appeared to be transient and would likely cure themselves – they did – one did not.  Event ID 1008.  Here is an excerpt from the description:

Description:
An exception occurred and was handled by Exchange ActiveSync. This may have been caused by an outdated or corrupted Exchange ActiveSync device partnership. This can occur if a user tries to modify the same item from multiple computers. If this is the case, Exchange ActiveSync will re-create the partnership with the device. Items will be updated at the next synchronization.

URL=/Microsoft-Server-ActiveSync/default.eas?User=xxxxxx&DeviceId=Appl70111HXKA4S&DeviceType=iPhone&Cmd=SendMail
— Exception start —
Exception type: System.Web.HttpException
Exception message: An error occurred while communicating with the remote host. The error code is 0x800703E3.

Note in particular the line beginning with “URL=/”.  The user name of the mailbox owner appears and it is their phone that is not syncing properly.  In my particular case, the users could send email from their phones but not receive .  Of course in this instance, only two users were effected: the owners of the company!

What Didn’t Work

The first thing I had them do was to delete the Exchange email account from their iPhones and re-add it.  It was a real test to see if they could accomplish this on their own.  Happy to say they did, sad to say it re-added the account and folders but still didn’t allow the phones to receive email.

Time for harsher measures, so I opened Exchange Management Console and expanded the Recipient Configuration and select Mailbox.  Then I selected one of the users and from the Actions pane on the right, I selected Manage Mobile Phone.  The following wizard appeared:

EMC Recipient Mobile Phone

The wizard background didn’t appear pink, but you get the point.

Notice that the radio button “Remove mobile phone partnership” is selected.  I completed the wizard by clicking on the Remove button, and felt very pleased.  I allowed for about an hour to pass to have Exchange and ActiveSync re-establish the partnerships.  After three hours, I figured that trick was not going to happen.

What Did Work

The trick that did work was pretty straight forward.  Here’s what I did:

  1. Open EMC and expand Organization Configuration.
  2. In the right hand action pane, select New Mailbox Database.
  3. Complete the wizard to create a new mailbox database and mount it.
  4. In EMC, expand Recipient Configuration and select Mailbox.
  5. Select the user(s) whose phones are not syncing.
  6. In the right hand action pane, select New Local Move Request.
  7. Complete the wizard to move the mailboxes to the new database created in the previous steps.
  8. In the EMC navigation pane on the left, click on Move Request and follow the progress of the mailbox move(s).  NOTE: you can double-click on a mailbox and the properties will display the size of the mailbox and the percent complete.  It does not refresh so close it and re-display as needed.

Just as soon as the move(s) were complete, email flow to the phones was re-established.

Will ActiveSync Still Work if You Move Them Back?

I don’t have an answer for that yet. But I will have an occasion to try this by the end of this month.  Stay tuned.

 


Summary

I guess it had to happen, and that is why we do backups and more.  But sometimes it is not quite enough and this was one of them.

I was logged onto a test server where I am creating a SharePoint application.  This server is a virtual W 2008 R2 connected via a VPN to a virtual SBS host. While I was doing some work, I noticed the VPN connection dropped.  Strange, I thought.  I remoted to the host machine for the SBS server and saw in Hyper-v Manager that SBS was stopped.  I checked on the drive where the .vhd files are stored and noticed that it was absent from the host server.  Oh, just a dropped iSCSI volume – something that happened a few times before (more about that later).  Simple enough, just reconnect.

Imagine my surprise when I couldn’t reconnect to the iSCSI target LUN!   I am going to skip over the details of why, for now, and get to the bigger point.  I was able to do a power cycle on the iSCSI box, and of course I could re-connect to the target.  Now, just restart the SBS virtual machine.

Horror Show Part I

The machine would not start.  Hyper-v reported a disk corruption and could not use the .vhd.  Well, that was a minor set-back.  I had a USB drive attached to the iSCSI box that did daily backups of the LUN.  I’ll just restore that to another iSCSI target I would create, just in case.

It took about an hour and lo and behold, a new and freshly restored LUN with the .vhd files.  I’ll attach it and start the virtual SBS 2011.

Horror Show Part II

You already knew, I bet, that it wouldn’t be that easy.  Diagnosis?  The LUN to USB is a copy and replace, so only one backup to restore.  That backup occurred AFTER a disk failed and was in rebuild on a RAID 6.  Doing the power cycle caused some data corruption, seemingly limited to the SBS 2011 .vhd.  Hyper-v failed to boot with a missing boot file.  Sigh.

Sounds like being up that creek.  There was no Windows Server backup of that volume.

But, we had installed AppAssure’s Replay backup. Unfortunately, some disk issues had caused it to fail and we were investigating the cause.  As it turns out, engineers were evaluating the logs that morning from some I/O stress testing we did.  They called to say the iSCSI target was generating lots of retrys, and a call to the vendor of he iSCSI box helped us determine bad sectors on a drive that went into rebuild that eventually died and …. you get the idea.

Even though Replay was disabled when this happened, it had created lots of backups during the previous two weeks. I picked the most recent backup of the volume and mounted it.  Sure enough, there was my precious .vhd file.  I coped it to a new location ( a lengthy process), modified the machine settings to point to the new .vhd, and fired it up.

Horror Show Starts to End

YES! It started fine.  Problem solved!

Of course not,  All services started but Exchange mailboxes were not accessible.  Neither the mailbox nor the public store properly mounted.  Here are some of the things I did to solve the problem:

  1. Ran eseutil.exe with /mh option.  This reported that the database was in a dirty shutdown mode.
  2. Ran eseutil .exe with /r option.  This ran successfully, but a subsequent eseutil /mh still reported dirty shutdown.
  3. Ran eseutil.exe with /p option.  Ran successfully, then eseutil /mh reported clean shutdown.

In order to run eseutil.exe with the various options, try this article.  To keep from typing long and tedious paths to the database and to the log file folder, stash them somewhere on your desktop and copy and paste them at the appropriate parts of the command line.  Remember these points:

  • Enclose paths and file names that contain spaces in double quotes.  I forgot and used single quotes and it couldn’t find the files.
  • Run cmd as administrator.
  • Set the path to the database location for convenience.

If you have not moved Exchange files, the default location  is “C:\Program Files\Microsoft\Exchange Server\V14\Mailbox\<mailbox name>”; otherwise it is exactly the same path but a different drive letter.  Get the mailbox name form Exchange Management Console Organization Configuration Mailbox.  NO .edb for the path name, please!

The log files will be in the original path on drive C: unless you manually moved them.

Public database is at “C:\Program Files\Microsoft\Exchange Server\V14\Mailbox\PublicFolderDatabase” or on another drive letter if moved and the log files on drive C:

For the /r option, the log prefix is by default E00 for the mailbox and E01 for public folders.

And yes, I had to repair each.  I repaired them at the same time using two open CMD windows.

It will take a long time if you have a sizable mailbox.  A nearly 30GB one took around 5 hours.

The End, Almost

Once they mounted, mail began to flow and all appeared well.  Except… several iPhones were no longer receiving email.  And ActiveSync was reporting errors on them.

Fixing this was easy, but it deserves a separate post.

Footnote

If you don’t know your locations precisely for eseutil, then rely on EMC.  Click on either the mailbox or public database that is unmounted, then chose Move Database Path from the actions pane.  In the first dialogue box that appears, the path to the database and the log files is there for you to copy and paste.  Remember you just want the database path or the path name unless you run eseutil from another path entirely.

When you run a repair with the /p option, you might loose bits of your mailbox.  I have done it before without loss, and this time the same thing.  But it is better than having nothing.


yes, Virginia, there is a Santa Claus who can help you convert your physical SBS server to a virtual machine to run under Hyper-v.  But dear, they don’t tell you nearly enough, so pay careful attention to this.  Especially if you want to user the same server box to be your new Hyper-v host machine.

Preliminary Work

The assumption is that you have SBS installed on a physical machine.  You would like to use that physical machine as a Hyper-v host.  How can you convert SBS to a virtual machine?  And what does it take to host Hyper-v on that box?

If your SBS is 2008 or 2011, you needn’t worry about whether the server can support 64-bit because both of those are 65-bit operating systems.  If, however, your starting point is SBS 2003, you need to verify that it can run a 64-bit OS.  If your server is from around the time when SBS 2003 was first released, I suggest you look at a new server to save yourself a lot of frustration, time and eventually money.  You can purchase a lovely box from Dell, HP and others for about $1500.  Then  you can do a migration to SBS 2011 between two physical boxes.  See my post about setting up SBS 2011 as a virtual machine and do the migration to it.

Once you are convinced your current server box will support the Hyper-v host os, check on how much memory you you have.  I would recommend16GM as the minimum necessary for a successful system and as much more as you can add past that to accommodate the number of virtual machines you plan on hosting.

I have a few virtual machines for SBS running between 8GB and 16GB and all give more than satisfactory performance.  They run along with 4-8 other virtual machines (other 2008 R2 servers, Windows 7, and XP operating systems). More memory is better of course  but you may find the marginnal benefit of 1GB more memory to not show up in the other vm performance .

Now you have determined a proper machine to host Hyper-v with sufficient memory.  It is my very strong suggestion that you now purchase new disks for the OS hosting environment. Why? Because if something goes wrong in the conversion, you don’t want to have clobbered your original system disks to be able to go back to a physical machine.  I know the cost and complexity of new system disks will vary from installation to installation, but disk prices have dropped so much that the cost of even a pair of 500GB (RAID 1 scenario) is under $100.  What is your time for recovery worth:?

The last thing you will need is a disk that can house the .vhd created on the physical installation to be copied to the new os environment for the virtual machine.  An external USB drive is perfect for this, but an internal hard disk that can remain mounted when you create the new hosting environment works well, too.

Cleaning Up the System

Before you convert to a .vhd, do as much cleanup on your SBS system as possible.  Not only will this make the conversion faster, it will make it easier.  And in some cases, it can mean the difference between success and failure.  The conversion utility will convert up to 127GB, so if you have too much data on your system disk, it may not convert at all.

Here are some suggestions:

  1. Move your Exchange data files to another hard disk.  As a rule. I never leave them on the system volume.
  2. Do the same with your SharePoint data files.  Same rule for moving it to another disk.
  3. Move any user profiles to another disk.
  4. If you have any data files that are shared, move them as well and update the share.
  5. Empty the recycle bin.
  6. Delete any temporary files.

In short, anything that is on your system disk that is not essential to the system either delete or move.  Note: if files are small, weigh the inconvenience of moving them against the conversion impact.  A few kilobytes won’t make a difference but a few gigabytes might.

Disk2vhd Utility

You will need this free and downloadable utility to convert your system disk to a .vhd file (click to download it).  It is straightforward to use and can work with a system that is online.  I will share with you some of the tricks I have picked up to make it easier for you.

Disk2vhd works by taking a snapshot of the files to be converted.  What you don’t want to happen is have those files updated while the conversion is running, possibly causing data consistency issues.  Exchange is a case in point, so before you start the conversion, stop Exchange services.  Do the same for SharePoint and any other applications that might be updating files as they run,

If you have been following closely about the preparations, you will have detected my overall approach:  isolate the system to the c: drive, convert it to a .vhd, bring up the new os and create a virtual machine from it, and have all of the data files remain on media that are mounted onto the host operating system.

With that in mind, when you run the conversion utility, you will be just converting the system disk.

disk2vhd

In the above example, I have isolated Exchange, SharePoint and user data and even have a separate drive to store the .vhd file.  (Remember, I am a big iSCSI fan and this is a piece of cake to do).

Much  more important, however, is to note that I have selected not only drive C: but the System Reserved area as well.  If you don’t select the System Reserved area, the resulting .vhd will not be recognized as bootable.  When you try to attach it as a virtual drive and start the machine, you will get the following messages:

disk error has occurred
Ctrl+alt+del to continue

But of course there is nothing to continue to.

Choose your destination for the .vhd, then click Create.  It can take thirty minutes to over an hour, so entertain yourself while it runs.

The Hyper-v Host

Now shut down the SBS system, remove the system disk(s), insert the new disk(s) for the host os, and install the os.  If you have a license for Windows 2008 R2, use it by all means.  If you don’t own or want to pay for a license, you can download Windows 2008 Hyper-v for free.  Be aware that it has only a limited interface and that you will need to do enable remote execution of Hyper-v manager on it to be effective.  Still, it is free.

Install the os and launch Hp yer-v manager.  Create a new virtual machine following the wizard.  But instead of creating a new .vhd, instead point to the .vhd you created with the disk2vhd utility.

Before you start the virtual machine, adjust the settings as follows by highlighting that vm and clicking on settings.

  1. Memory – use dynamic setting.
  2. Processor – use the number of virtual processors that match the host cpu.
  3. Network adapter – you can use the same adapter as the host machine or an additional one.
  4. iSCSI Controller – Use this to add disk drives attached to the host system.  For example, if you have USB drives (for backup, e.g.) connect them to the virtual machine.  Same for data drives.  Note – if you are using iSCSI targets, the virtual machine can connect directly to them without adding them as hardware.

Now start the virtual machine.  What you will discover is that it doesn’t start right up like you would think.  Instead, it appears to be “hung” without the desktop appearing.  This could take from five to thirty minutes.  What is happening is that SBS is adding new devices to the system as they have been attached through the virtual interface.  When that is complete, you will get a message that new hardware has been added and you should reboot.

After the reboot, check the network settings to make sure they are correct.  You may have to run fix my network or connect to the internet wizard from the SBS console.  Check to make sure all services are properly started and that Exchange connected to the databases.

Your virtual SBS is now complete and you can add other virtual machines as necessary.