Replica Inconsistent Due to Inaccessible Files

Recently, one of the data sources backed up by our DPM server was becoming inconsistent and preventing backups from occurring. No matter how many times we ran a consistency check, the source remained inconsistent.

The DPM console was next to useless in assisting to determine the fault that was causing the source to be inconsistent. After a few days the inconsistency warning changed and gave a little further information:

“The replica of Volume E:\ on FILESERVER.COMPANY.LOCAL is inconsistent with the protected data source. All protection activities for data source will fail until the replica is synchronized with consistency check. You can recover data from existing recovery points, but new recovery points cannot be created until the replica is consistent.

For SharePoint farm, recovery points will continue getting created with the databases that are consistent. To backup inconsistent databases, run a consistency check on the farm. (ID 3106)

RECOMMENDED ACTION:

Review the failure errors for individual files from the log file \\?\Volume{ebad65e7-fcf2-13e4-962d-001ec9ee5638}\e3866ced-d582-263a-a326-a6aabdc8f361\FailedFilesLog.txt and take appropriate action. If some files fail consistently, you can exclude the folders containing these files by modifying the protection group or moving the files to another location.”

The first step in finding out what is in the logfile is mounting the volume:

mountvol x:\\?\Volume{ebad65e7-fcf2-13e4-962d-001ec9ee5638}

This gave me access to the volume via Windows Explorer using the x: drive (the drive letter is irrelevant, just use one that is spare). Opening the x: drive there is a folder, e3866ced-d582-263a-a326-a6aabdc8f361, within which should be the logfile. However, when attempting to open the folder, I received an “Access Denied” message.

The only way to get around that was to take ownership of the folder and apply permission that would allow me to traverse the folder to access the logfile. Once that was done, I could open the folder and see the logfile. Unfortunately, that was also “Access Denied”! Same process again, take ownership, apply permissions…

When I was able to access the logfile, it informed me that a large number of files were inaccessible to the DPM server. These were excluded from the Protection Group.

To prevent any issues with DPM accessing the volume again, I removed the permissions I had set on both the logfile and the folder and reassigned ownership to the SYSTEM account.

The Consistency Check was then restarted and completed successfully, allowing restoration point to be created again.

Migrating Azure Backup Scratch Space

When we set up our DPM server to use the Azure Backup Vault, the amount of data we were uploading was relatively small. Over time we have increased the amount that we backup to Azure and it led to us hitting a bit of an issue. We ran out of usable space on the drive that stored the DPM application causing it to fail and all online backups failing too!

When setting up the Azure Backup Agent, the recommendation is that the agent requires approximately a scratch disk of about 5% of that being backed up to Azure. This allows the agent to have storage in preparation for the upload. When we initially started using Azure with DPM, the 300GB we had available on the drive was more than adequate for this. Now it seems that this is inadequate and we needed more space for the scratch space.

There were two ways we could add additional space; we could add another drive and span across to it as a pair of dynamic disks, or we could add another drive with the required capacity to store the scratch disk. Both have pros and cons. By spanning, we could use the existing installation, but could run into the same issue again further down the line as our use of Azure backup grew. By adding the additional drive, we had a clean drive that wouldn’t prevent backups from occurring should the scratch space fill the disk, but we need to reinstall the agent. Or did we?

A little research into the issue showed that it was possible to migrate the scratch space without the need to reinstall.

  1. Stop the Azure backup agent by opening an elevated command prompt.                          net stop obengine
  2.  Once the service has stopped, locate the location of the scratch disk (usually “%programfiles%\Azure Recovery\Microsoft Azure Recovery Services Agent”) and copy the Scratch folder to its new location.
  3.  Once the copy has been completed, open the registry editor (regedit.exe). Go to “HKLM\SOFTWARE\Microsoft\Windows Azure Backup\Config” where you will find the key “ScratchLocation”. Modify this key to reflect the new location of the scratch folder.
  4. At an elevated command prompt, restart the Azure backup agent.                                      net start obengine

Once the engine has been restarted it will start to use the new location for the scratch disk.

SCDPM 2012 R2 Exchange Mailbox Consistency Check

Recently ran into an issue where two of the mailbox databases on our Exchange server failed to create backups and failed consistency checks in System Center Data Protection Manager 2012 R2. It was a puzzling one, as the other two mailbox databases were happily creating recovery points.

The database generated an error that informed me that “Data consistency verification check failed for ****.edb of Exchange Mailbox Database **** on <email server>. (ID 30146 Details: Unknown error (0xfffffb4a) (0xFFFFFB4A))”. As a result of this we were unable to create any new recovery points due to the source being inconsistent.

Checking the logs on the email server itself revealed that every time a consistency check was requested, a VSS error was generated.

Error

It appears that an old unused profile was preventing the VSS service from running. To remove the profile, I used the following steps:

  1. Log on to the system by using an administrative user account other than the user account that is experiencing the problem.
  2. Back up all data in the current user’s profile folder if the profile folder still exists, and then delete the profile folder. By default, the profile resides in the following location:
    %SystemDrive%\Users\UserName
  3. Click Start, type regedit in the Start Search box, and then press ENTER.

    If you are prompted for an administrator password or for confirmation, type your password, or click Continue.

  4. Locate the following registry subkey:
    HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\ProfileList
  5. Under the ProfileList subkey, delete the subkey that is named SID.bak.
    Note: SID is a placeholder for the security identifier (SID) of the user account that is experiencing the problem. The SID.bak subkey should contain a ProfileImagePath registry entry that points to the original profile folder of the user account that is experiencing the problem.
  6. Exit Registry Editor.
  7. Log off the system.
  8. Log on to the system again.

Following this return to the DPM server and begin the consistency check for the affected database. The consistency check should now complete without error and a recovery point can now be created.

 

 

 

Hello World!

In time honoured tradition, it seems fitting that a blog about IT starts with its first post using the phrase used by programmers the world over when they first learn to programme; Hello World!