Windows Server 2008 R2, Exchange 2010 SP1 upgraded from Exchange 2003. I should note that this upgrade occurred long before this issue happened. The Exchange server is virtualized on VMWARE ESXi 5.0 Update 1.
Thursday, May 31st, at midnight I received an alert that the Datastore my Exchange Database sits on was at 90% capacity. This was odd because my Exchange Database had been right around 250GB for the last 9 months and the Datastore was 500GB in size. Somehow it was up to 480GB. After verifying there was no unusual activity, errors in the event logs, or backup snapshot issues, I doubled the size of the Datastore and went back to sleep. When I got into the office the next day at 7AM, the Database was up to nearly 600GB in size. Clearly something was not right.
Short version: Install Exchange 2010 SP2 and all the rollup packages. I ended up getting Microsoft support involved and this is a known issue for Exchange 2010 SP1 and is resolved with Exchange 2010 SP2 Rollup 1.
For no apparent reason my Exchange Database was increasing by 1-2 GB every 10 minutes. What started as a 250GB Mailbox Database ended up as a nearly 750GB Database in less than 36 hours. Multiple server reboots did nothing to resolve the problem. No users were experiencing issues and there were no errors or warnings of any kinds in the event logs.
Doing some research and reading online, it appeared the issue was possibly caused by a corrupt Mailbox and that moving the Mailboxes to a new Mailbox Database would resolve the issue. While doing the move, the “old” Mailbox Database continued to grow at the same size, but the “new” one did not. Everything was looking good I thought. However, 2 Mailboxes failed to move because of corrupt items. I moved them successfully by just increasing the number of corrupted items to allow to 10. No sooner had those 2 Mailboxes moved over than the “new” Mailbox Database started growing at the same rate. OK then. Back to the drawing board.
Doing some more research I found that there were Powershell commands that let you repair Mailboxes or the individual Mailbox Databases. I ran the following commands on the two Mailboxes that had corrupted items and failed the original move.
New-MailboxRepairRequest -Mailbox -CorruptionType SearchFolder,AggregateCounts,ProvisionedFolder,FolderView
New-MailboxRepairRequest -Mailbox -CorruptionType SearchFolder,AggregateCounts,ProvisionedFolder,FolderView -archive
After those two commands finished, and the event logs showed there was no corruption issues I ran the command against the entire Mailbox Database.
New-MailboxRepairRequest -Database -CorruptionType SearchFolder,AggregateCounts,ProvisionedFolder,FolderView
The event logs also showed that command hadn’t found any corruption either. So now what?
A post of mine on the Microsoft forums (http://social.technet.microsoft.com/Forums/en-US/exchangesvrgeneral/thread/7de2b043-a5e6-494c-9b2a-5fe2552fa21e/) had gotten some responses indicating that the Exchange User Monitor would help to figure out what user(s) were corrupted and causing the issue. I installed it, and ran it, but had zero idea what I was looking at and couldn’t find anything that was of much use in telling me how to use it. Sidenote, if you run it once and close it, apparently it requires a server reboot to run it again. Good times.
It was at this point, with the “new” Mailbox Database closing in 600GB and the overall size of the Datastore up to nearly 1.4TB at this point that I decided to get Microsoft support involved. While he couldn’t tell me why the Mailbox Repair Request didn’t find or fix the corrupted Mailboxes, he was positive the issue was fixed in SP2 Rollup Update 1, and it was. After installing both and rebooting the Exchange server after each installation the Mailbox Database immediately finished growing.
Next thing I did was to move the Mailboxes to another “new” Mailbox Database so that it was the proper size, and then delete the two “old” Mailbox Databases. Everything has been stable and back to normal ever since the updates were applied.