Replica Inconsistent Due to Inaccessible Files

Recently, one of the data sources backed up by our DPM server was becoming inconsistent and preventing backups from occurring. No matter how many times we ran a consistency check, the source remained inconsistent.

The DPM console was next to useless in assisting to determine the fault that was causing the source to be inconsistent. After a few days the inconsistency warning changed and gave a little further information:

“The replica of Volume E:\ on FILESERVER.COMPANY.LOCAL is inconsistent with the protected data source. All protection activities for data source will fail until the replica is synchronized with consistency check. You can recover data from existing recovery points, but new recovery points cannot be created until the replica is consistent.

For SharePoint farm, recovery points will continue getting created with the databases that are consistent. To backup inconsistent databases, run a consistency check on the farm. (ID 3106)

RECOMMENDED ACTION:

Review the failure errors for individual files from the log file \\?\Volume{ebad65e7-fcf2-13e4-962d-001ec9ee5638}\e3866ced-d582-263a-a326-a6aabdc8f361\FailedFilesLog.txt and take appropriate action. If some files fail consistently, you can exclude the folders containing these files by modifying the protection group or moving the files to another location.”

The first step in finding out what is in the logfile is mounting the volume:

mountvol x:\\?\Volume{ebad65e7-fcf2-13e4-962d-001ec9ee5638}

This gave me access to the volume via Windows Explorer using the x: drive (the drive letter is irrelevant, just use one that is spare). Opening the x: drive there is a folder, e3866ced-d582-263a-a326-a6aabdc8f361, within which should be the logfile. However, when attempting to open the folder, I received an “Access Denied” message.

The only way to get around that was to take ownership of the folder and apply permission that would allow me to traverse the folder to access the logfile. Once that was done, I could open the folder and see the logfile. Unfortunately, that was also “Access Denied”! Same process again, take ownership, apply permissions…

When I was able to access the logfile, it informed me that a large number of files were inaccessible to the DPM server. These were excluded from the Protection Group.

To prevent any issues with DPM accessing the volume again, I removed the permissions I had set on both the logfile and the folder and reassigned ownership to the SYSTEM account.

The Consistency Check was then restarted and completed successfully, allowing restoration point to be created again.

SCDPM 2012 R2 Exchange Mailbox Consistency Check

Recently ran into an issue where two of the mailbox databases on our Exchange server failed to create backups and failed consistency checks in System Center Data Protection Manager 2012 R2. It was a puzzling one, as the other two mailbox databases were happily creating recovery points.

The database generated an error that informed me that “Data consistency verification check failed for ****.edb of Exchange Mailbox Database **** on <email server>. (ID 30146 Details: Unknown error (0xfffffb4a) (0xFFFFFB4A))”. As a result of this we were unable to create any new recovery points due to the source being inconsistent.

Checking the logs on the email server itself revealed that every time a consistency check was requested, a VSS error was generated.

Error

It appears that an old unused profile was preventing the VSS service from running. To remove the profile, I used the following steps:

  1. Log on to the system by using an administrative user account other than the user account that is experiencing the problem.
  2. Back up all data in the current user’s profile folder if the profile folder still exists, and then delete the profile folder. By default, the profile resides in the following location:
    %SystemDrive%\Users\UserName
  3. Click Start, type regedit in the Start Search box, and then press ENTER.

    If you are prompted for an administrator password or for confirmation, type your password, or click Continue.

  4. Locate the following registry subkey:
    HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\ProfileList
  5. Under the ProfileList subkey, delete the subkey that is named SID.bak.
    Note: SID is a placeholder for the security identifier (SID) of the user account that is experiencing the problem. The SID.bak subkey should contain a ProfileImagePath registry entry that points to the original profile folder of the user account that is experiencing the problem.
  6. Exit Registry Editor.
  7. Log off the system.
  8. Log on to the system again.

Following this return to the DPM server and begin the consistency check for the affected database. The consistency check should now complete without error and a recovery point can now be created.