Hi all,

We have a 3 node cluster using CSV.  DPM set to serial backups of the child servers.  All is working brilliantly, however on the odd occasion we have a event Volsnap 27 on the host Hyperv Server during a child backup.

event VolSnap 27

Any ideas why and how to fix?

Hi all,

I have a 3 node Hyper-V R2 cluster (Node majority model) with all MS updates installed, below is more information about them:

node1: Windows 2008 R2 Datacenter Core, Service Pack 1, Cluster Version 6.1.7601
node2: Windows 2008 R2 Datacenter Core, Service Pack 1, Cluster Version 6.1.7601
node3: Windows 2008 R2 Datacenter Core, Service Pack 1, Cluster Version 6.1.7601

These 3 nodes are connected to an Equallogic PS6010XV storage using 10Gbe cards (iSCSI SAN) like below:

node1,2,3 |----- 10 Gb Card 1 ----- Dell 8024F Switch 1 ----- |-----------------|
                 |                                                                          |   PS6010XV   |
                 |----- 10 Gb Card 2 ----- Dell 8024F Switch 2 ----- |-----------------|

All three nodes have Dell HitKit installed (HitKit provides hardware VSS snapshots)

C:\Users\administrator>vssadmin list providers
vssadmin 1.1 - Ferramenta de linha de comando administrativa de cópias de sombra
 de volume
(C) Copyright 2001-2005 Microsoft Corp.

Nome do provedor: 'Microsoft Software Shadow Copy provider 1.0'
   Tipo de provedor: Sistema
   Identificação do provedor: {b5946137-7b9f-4925-af80-51abd60b20d5}
   Versão: 1.0.0.7

Nome do provedor: 'Dell EqualLogic VSS HW Provider'
   Tipo de provedor: Hardware
   Identificação do provedor: {d4689bdf-7b60-4f6e-9afb-2d13c01b12ea}
   Versão: 3.4.2.5386

Microsoft DPM 2010 3.0.7707.0 (Rollup 1) is installed into another physical machine and connected to the 3 node cluster through another network (Public Network).

I've configured DPM to perform a host level backup using the hardware VSS provider using this linkHow to use Hardware VSS Writers for DPM Hyper-V Backup and configured DPM MaxAllowedParallellBackups param to 1 (don't know why DPM doesn't respect this parameter since I always see 3 simultaneous backups everytime, not only one).

I created a protection group for all virtual machines with an Express Full every day in the afternoon .... everything was running as expected except the fact of several 5121 events shown in cluster logs but searching on the forums I found that this event is pretty normal since IO becomes redirected for some time during a hardware VSS snapshot. I can see the snapshots being created from Equallogic console too, each with its own proprietary (node1, node2, node3).

What is really bothering me is that after some time the host level backup works Ok (several 5121 events) but suddenly the CSV volume crashes with a 1069 event followed by a 1034 event ... what put all the 13 virtual machines in a failure state :-(, below a screenshot of the problem:

To solve this problem I took the following steps:

1. I had to remove the storage from the cluster (Failover Cluster Manager) there was no way to put it online, it indicates failure.
2. I entered into each cluster node and through iscsicpl command recognized the volume again.
3. I entered into the cluster node that was the proprietary of the CSV disk and put it online.
4. After step 3 the storage was recognized by the cluster so I put the disk offline again.
5. I added the 1.5TB "non empty" volume back again in the cluster and put it into CSV.
6. I started all virtual machines.

I know that this shouldn't happen but I have no clue of where the problem could be ... maybe a KB for the cluster nodes, maybe a new version of Equallogic HitKit (Hardware VSS), maybe a storage firmware upgrade (it's relatively new).

Could someone help me to solve or spot this problem?

Thank you very much !!!
Marcos Hass Wakamatsu

We have a Hyper-V R2 cluster with DPM 2010 doing the VM backups at the host level. All VMs are backing up and restoring fine except two. On those VMs the following error occurs: (I changed the VM and server names)

Description:    The back up to tape job failed for the following reason: (ID 3311)

There are no significant events where he tells me to look, just the same VSS events as in a successful backup.

These 2 VMs do have an extra VHD with over 200GB attached. The second one was backing up fine until recently. Most VMs have dynamically expanding disks. These 2 have them.

Error details below.

Hi,

On event viewer indicated Event id 1584.

Description:-

A backup application initiated a VSS snapshot on Cluster Shared Volume '%1' ('%3') without properly preparing the volume for snapshot. This snapshot may be invalid and the backup may not be usable for restore operations. Please contact your backup application vendor to verify compatibility with Cluster Shared Volumes.

I have refer to http://technet.microsoft.com/en-us/library/ee830301(WS.10).aspx and request to confirm with backup solution vendor.

My environment is 2 node cluster running Hyper V R2 . Using DPM 2010 to backup VHD and Data.

DPM Backup is indicated OK (green).

I have patches both node with KB975354 and KB975921. Pls advise

I have a Dell server running Windows Server 2008 R2 and a Dell MD3200i storage array that uses CSVs to store Hyper-V virtual servers. On one of the virtual servers running on the Dell server, I have installed DPM 2010. Dell has said their current VSS Providers have issues with CSVs so I added the registry key on the Dell server to use the software VSS provider instead and also added the registry key to serialize backups. I have also updated the integration tools on the virtual server.

However, anytime I try to backup a virtual server using the Child Partition Snapshot in DPM, it always fails within 30 seconds and says replica inconsistent. On the Dell server, each time a backup is attempted, the following error gets logged in the Hyper-V VMMS Admin log (this is VMMS Error code 10102)

I have been trying with little luck to get DPM 2010 on it's feet backing up VMs on our cluster nodes. 

We have a cluster of (5) servers, running 2008 R2.  The cluster is running only Hyper-V VMs, distributed across the nodes.  The SAN is running Storage Server 2008 (R1), which is providing CSVs for the VMs to reside on. 

I have been trying to get the hardware VSS providers working in order to perform "best practice" backups of the VMs.   I do not want to take the performance hit of serialized backups. 

Here is what I have done so far, as far as modifying our environment to support this:

- I have added our storage server to the domain.  It was not previously.

- I created an account called 'sys_iscsi' in AD.   I added this account to the local administrators group of all cluster nodes, and the storage server.

- I installed iscsitargetclient.msi on all cluster nodes, using the above username and it's password as part of the setup process. 

I am not sure what I am missing, as from what I've studied from the relatively scant documentation I've found, that's about all there is supposed to be to it.  

On the DPM server, when I create a protection group, I target the cluster's object, and a hyper-v host contained underneath it.  However, the initial backup fails, with "Replica is inconsistent."  When I attempt to perform a consistency check, the job fails after 30-60 seconds, generating this error:

Failure occured while adding one or more of the volumes involved in the backup operation to snapshot set.  Please check the event log on [hyper-v VM that was targeted] to troubleshoot the issue.  (ID 30290 Details:  Internal Error Code:  0x80990A00)

What is strange is that the error is telling me to look on the event log of the VM that was targeted, even though the backup takes place at the cluster level, so the VM should not even be aware that it's being backed up.  Regardless, I checked the event logs on the target VM, and there was nothing there. 

All of the information I have found relating to this error is pertaining to third party SAN appliances, and not Windows Storage Server 2008, so any assistance I can get would be appreciated.

Hi,

I have a 2-node Hyper-V cluster with 1 CSV containing 5 VMs split across both nodes. I am receiving the below error when performing the child partition backups in DPM 2010.

Failed to prepare a Cluster Shared Volume (CSV) for backup as another backup using the same CSV is in progress. (ID 32612 Details: Back up is in progress. Please wait for backup completion before trying this operation again (0x8007173D))

I have completed the steps to serialize the backups in the Technet article, http://technet.microsoft.com/en-us/library/ff634192.aspx and have modified the Protection Group and restarted the DPM server.

Although the parallel backups is showing 1 in the registry and the datasourcegroups.xml has been created successfully, I am still receiving the error above.

Any help would be greatly appreciated.

Thanks,

Charlie.

Hi.

we are testing backing up csv volumes with dpm using hp hardware vss writers. when i backup the environment i see that the volumes are in redirect io while the snapshots are being generated and then brought online and after that the actual backup runs from the snapshots but i still get alot of error 5121 in the clusterlog should i still have these while using hardware snapshots? the disk is of course in rederict io mode during the snapshot phase so i guess it should be like that but it shouldnt be flaged as an error in that case?

I'm using DPM 2010 to backup our CSVs which reside on a HP P2000 G3 FC/iSCSI. CSVs on the Hyper-V cluster are accessed by FC only. iSCSI is used to connect the DPM server to the storage.

Log Name:      System
Source:        Microsoft-Windows-FailoverClustering
Date:          17.4.2011 12:39:47
Event ID:      5121
Task Category: Cluster Shared Volume
Level:         Error
Keywords:     
User:          SYSTEM
Computer:      server2.domain.local
Description:
Cluster Shared Volume 'Volume1' ('CSV10k') is no longer directly accessible from this cluster node. I/O access will be redirected to the storage device over the network through the node that owns the volume. This may result in degraded performance. If redirected access is turned on for this volume, please turn it off. If redirected access is turned off, please troubleshoot this node's connectivity to the storage device and I/O will resume to a healthy state once connectivity to the storage device is reestablished.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-Windows-FailoverClustering" Guid="{BAF908EA-3421-4CA9-9B84-6689B8C6F85F}" />
    <EventID>5121</EventID>
    <Version>0</Version>
    <Level>2</Level>
    <Task>38</Task>
    <Opcode>0</Opcode>
    <Keywords>0x8000000000000000</Keywords>
    <TimeCreated SystemTime="2011-04-17T10:39:47.521872700Z" />
    <EventRecordID>1197950</EventRecordID>
    <Correlation />
    <Execution ProcessID="2136" ThreadID="4360" />
    <Channel>System</Channel>
    <Computer>server2.domain.local</Computer>
    <Security UserID="S-1-5-18" />
  </System>
  <EventData>
    <Data Name="VolumeName">Volume1</Data>
    <Data Name="ResourceName">CSV10k</Data>
  </EventData>
</Event>
 
 
This happens during backup using Microsoft's Data Protection Manager 2010, which utilizes HP's VSS Hardware provider:
 
C:\Users\administrator>vssadmin list providers
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2005 Microsoft Corp.
 
Provider name: 'Microsoft Software Shadow Copy provider 1.0'
   Provider type: System
   Provider Id: {b5946137-7b9f-4925-af80-51abd60b20d5}
   Version: 1.0.0.7
 
Provider name: 'HP StorageWorks P2000/MSA2000 VSS Provider'
   Provider type: Hardware
   Provider Id: {bd04cbf9-212c-4553-9ea5-c5bfb05ccc8f}
   Version: 2.7.0.0
 

I'm in contact with HP and was told the storage (all latest firmware) has full support for CSV. Why am I even getting this error and is it something I should worry about. What additional steps can I take to resolve this. Is additional information, logs needed. What can I test, ask HP...

Hello,

I’m having issues with DPM backups on CSV volumes.

I’m trying to backup 2 or more VM concurrently on a Hyper-V cluster with 2 nodes.

I have the following problem: when I backup 2 VM that are on 2 different nodes, but on the same CSV,the Failover Clustering service restarts on one of the nodes.

Of course, this causes a failover to the remaining host and downtime for the VMs…

No issue if backed-up VMs are on the same node or on a different CSV.

The situation is very similar to the one described in the following KB :http://support.microsoft.com/kb/975354

But the hotfix is already installed on both nodes…

Environment :

·        Hyper-V OS : Windows 2008 Datacenter R2

·        SAN : EqualLogic PS6000

·        SAN Hardware Snapshot Provider : DELL EqualLogic VSS HW Provider - version 3.3.1.4944

·        DPM : DPM 2010 RTM

Below you will find the errors found on a host that presents the issue.

Event 5121 at 2:00:34 AM

Cluster Shared Volume 'Volume1' ('EQLPRD3') is no longer directly accessible from

this cluster node. I/O access will be redirected to the storage device over the network through the node that owns the volume. This may result in degraded performance. If redirected access is turned on for this volume, please turn it off. If redirected access is turned off, please troubleshoot this node's connectivity to the storage device and I/O will resume to a healthy state once connectivity to the storage device is reestablished.

Event 1038 at 2:01:44 AM

Ownership of cluster disk 'EQLPRD2' has been unexpectedly lost by this node. Run the Validate a Configuration wizard to check your storage configuration.

Event 1038 at 2:01:45 AM

Ownership of cluster disk 'Quorum' has been unexpectedly lost by this node. Run the Validate a Configuration wizard to check your storage configuration.

Event 4201 at 2:01:46 AM

Isatap interface isatap.{55DFCEFD-ED71-4C57-9277-8FBD5219D184} is no longer active.

Event 7031 at 2:01:51 AM

The Cluster Service service terminated unexpectedly.  It has done this 1 time(s).  The following corrective action will be taken in 60000 milliseconds: Restart the service.

Has anyone seen and solved this problem or is it a normal behavior (which I doubt) ?

Thanks for the help.

Since applying security updates to our Hyper-V server (2008R2 Ent. Core), DPM 2010 has been unable to create recovery points of any of the Hyper-V guests.

DPM encountered a retryable VSS error. (ID 30112 Details: VssError:The writer experienced a transient error.  If the backup process is retried, the error may not reoccur. (0x800423F3))

I also see in the Hyper-V server System event logs:

Log Name:      Microsoft-Windows-Hyper-V-VMMS-Admin
Source:        Microsoft-Windows-Hyper-V-VMMS
Date:          1/18/2011 2:02:02 AM
Event ID:      10111
Task Category: None
Level:         Error
Keywords:     
User:          SYSTEM
Computer:      server.lan.com
Description:
Failed to mount the virtual hard disk '\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy24\hyper-v...

When I do a vssadmin list writers on the Hyper-V server is shows that the Hyper-V VSS Writer is in a stable state, but it did have a retryable error:

Writer name: 'Microsoft Hyper-V VSS Writer'

  Writer Id: {66841cd4-6ded-4f4b-8f17-fd23f8ddc3de}

  Writer Instance Id: {de88412e-6663-4486-ac77-3781a8556a5f}

  State: [1] Stable

  Last error: Retryable error

 I tried rebooting our Hyper-V server which cleared the Hyper-V VSS writer 'Retryable error'.  But once DPM ran it's scheduled recovery points, the 'Retryable error' returned, same errors as before.  I upgraded SQL 2008 to SP2 which updated the Microsoft SQL Server VSS Writer, but that didn’t help either when I attempt to retry a recovery point on one of our Hyper-V guests.  I then applied hotfix KB 2250444 as I was also seeing the MSDPM error (see below) addressed in this hotfix… but the problem still persists. 

I am able to use Windows Backup within the Hyper-V guest and create snapshots from Hyper-V, which leads me to believe that this is a DPM issue.

Log Name:      Application
Source:        MSDPM
Date:          1/19/2011 9:26:26 AM
Event ID:      945
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      server.lan.com

Description:The description for Event ID 945 from source MSDPM cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

Any ideas would be greatly appreciated.... Thanks!

I have a two node Hyper-V cluster running Windows Server 2008 R2 SP1 and am using DPM 2010 for backup. Very similar to the provided fix for R2 pre-SP1http://support.microsoft.com/kb/982210 my SYSTEM registry hive keeps growing and growing.

I receive (only during backup):

Log Name:      System
Source:        Microsoft-Windows-FilterManager
Date:          16.6.2011 19:04:57
Event ID:      3
Task Category: None
Level:         Error
Keywords:     
User:          SYSTEM
Computer:      server2.domain.local
Description:
Filter Manager failed to attach to volume '\Device\Harddisk9\DR82'.  This volume will be unavailable for filtering until a reboot.  The final status was 0xc03a001c.

It appears I'm not the only one with this issue (unfortunatelly the hotfix doesn't work on SP1 and the registy entries provided as a solution in the second thread don't work in my case):
http://social.technet.microsoft.com/Forums/en-US/windowsbackup/thread/c5e8add1-c3e9-46c1-917f-2694c820acf0/
http://int.social.technet.microsoft.com/Forums/en-GB/windowsbackup/thread/2a434be6-9b61-4f7d-a74b-b7707d30c655

locations are mentioned in KB :

• If you view the registry hive by using Registry Editor, the following registry entries are logged repeatedly. For example, the following registry entries are logged repeatedly 10000 times or more.
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum\SCSI\Disk&Ven_Msft&Prod_Virtual_Disk – this doesn't seem excesive on my servers.

HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Enum\STORAGE\Volume\{f1030a29-1024-11df-b7a9-001b214f4bef}#0000000006500000 – huge issues (please take a look at: http://imageshack.us/photo/my-images/808/sshypervnode.png/).

" – this doesn't seem excesive on my servers.

I also have a large number of entries under the two keys under: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum\MPIO and HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Enum\MPIO

Is Microsoft aware of this issue, is a bug open, can I for the time being do anything to stop the growth?

I have recently setup my DPM server to backup my failover cluster via this linkhttp://technet.microsoft.com/en-us/library/ff634192.aspx &http://support.microsoft.com/kb/2473194.  I made these modifications because I was having VM backups fail because of a VSS error "Failed to prepare a Cluster Shared Volume (CSV) for backup as another backup using the same CSV is in progress"  

After making these changes - My VMs now randomly stop when being backed up.  I have a 3 node cluster running Server 2008 R2 NON SP1.  Any help is greatly appreciated.

Zach

I am using all of the technologies in the title in my environment. Seems that after installing the VSS hardware provider, If I attempt to synchronize or create a replica of mulitple VMs on the same cluster shared volume, something somewhere is going crazy and kicking a node out of the cluster for a few seconds. Almost seems as though there is contention for the CSV and one of the nodes is getting the boot. My questions are below...

1. What is the recommended method to setting up protection groups of VMs when using clustered shared volumes?

    single protection group for all CSVs assuming i have two 500 GB CSVs or a protection group for each of my CSVs

2. Shoud I be using the redirected access feature with the CSVs? If I turn it off what is the impact?

Thanks in advance.

Chris

Hi,

Having been backing up various VMs on a Hyper-V box for months without any problems, I'm suddenly receiving this error:

"DPM failed to fixup VHD parent locator post backup for Microsoft Hyper-V\Backup Using Child Partition ShapShot\Exchange on hyperv.domain.dom.  Your recovery point is valid, but you will be unable to perform item-level recoveries using this recovery point. (ID 3130)".

It says I can safely dismiss the alert if I don't need item level restores but, if I do, then I need to install the Hyper-V role on the DPM server.  Why do I need to do this all of a sudden?

The one thing that strikes me as more than a coincidence is that I installed Update Rollup 4 on the Exchange 2010 server to which this alert relates at the end of last week and this is the first backup schedule that's executed since then.  Has the rollup changed something in Exchange that now requires Hyper-V to be installed on DPM?

Hello.

I'm getting an error on one of the protected server in DPM 2010.

Created one protected group and this group contains 4 servers and one of the server is getting an error.

Error:
The replica of Microsoft virtual server 2005 online backup\******001 on Server*** is inconsistent with the protected data source.All protection activitites for data source will fail until the replica is synchronized with consistency check.You can recover data from existing recovery points,but new recovery points cannot be created until the replica is consistent.

For sharedpoint farm, recovery points will continue getting created with the databases that are consistent.To backup inconsistent databases run a consistency check on the farm (ID 3106)

DPM encountered a retryable VSS error.(ID:30112 Details:VssError: The writer experienced a transient error.If the backup process is retried, the error may not reoccur.(0x800423F3)

Guest server details:

Operating system: Windows 2003 with SP2
Application: Exchange server 2003

My Virtual Host server setup is as followed.
Windows 2003 R2 SP2 and KB940349
VS2005 R2 SP1 with all latest MS-updates
VM Additions 13.820 installed on the guests

And i have run the VSSadmin list writers commond on the virtual Host server and result follows

Writer Name:'Microsoft virtual Server 2005 Writer'
Writer ID:{76afb926-87ad-4a20-cdc69412ddfc}
Writer Instanace Id:{b06a18ac-7ff8-4058-863d-59e1109e114804}
State:[8] Failed
last error:Retryable error

Also i have run the vssadmin list writers commond on guest virtual machine (i.e. issue server)

Writer name: 'System Writer'
   Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}
   Writer Instance Id: {bdd43fdf-ec14-4fa7-bb15-bdd1159382d4}
   State: [7] Failed
   Last error: No error

Writer name: 'Microsoft Exchange Writer'
   Writer Id: {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}
   Writer Instance Id: {76652728-884c-4228-a228-92796f665116}
   State: [7] Failed
   Last error: Retryable error

Writer name: 'COM+ REGDB Writer'
   Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}
   Writer Instance Id: {30d4a78b-f937-4acf-9c86-f1e3365d93ba}
   State: [1] Stable
   Last error: No error

Writer name: 'Event Log Writer'
   Writer Id: {eee8c692-67ed-4250-8d86-390603070d00}
   Writer Instance Id: {8aead84f-70b9-4224-acd5-ef569c232ae9}
   State: [1] Stable
   Last error: No error

Writer name: 'WMI Writer'
   Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
   Writer Instance Id: {f7d0b39c-90e4-49b0-9b28-5c57fbba286c}
   State: [7] Failed
   Last error: No error

Writer name: 'FRS Writer'
   Writer Id: {d76f5a28-3092-4589-ba48-2958fb88ce29}
   Writer Instance Id: {531f50d2-558c-4a81-905e-9f04544da7a4}
   State: [7] Failed
   Last error: No error

Writer name: 'NTDS'
   Writer Id: {b2014c9e-8711-4c5c-a5a9-3cf384484757}
   Writer Instance Id: {7b47775b-fdc4-4ae6-b531-c617e1c73c38}
   State: [7] Failed
   Last error: No error

Writer name: 'IIS Metabase Writer'
   Writer Id: {59b1f0cf-90ef-465f-9609-6ca8b2938366}
   Writer Instance Id: {7d144325-4980-431c-979c-364b36e4afbf}
   State: [7] Failed
   Last error: No error

Writer name: 'Registry Writer'
   Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}
   Writer Instance Id: {ac6f2515-eea0-4e2b-9c05-9f66264ca541}
   State: [1] Stable
   Last error: No error

Writer name: 'MSDEWriter'
   Writer Id: {f8544ac1-0611-4fa5-b04b-f7ee00b03277}
   Writer Instance Id: {3b623bd2-da87-48a3-a4ef-9c8c8b69ac39}
   State: [1] Stable
   Last error: No error

Please help me and how i can fix this issue?

Thanks,
Sri

I'm trying to backup virtual machines hosted on a 4 nodes Windows Server 2008 R2 Hyper-V cluster. The cluster has two cluster shared volumes. One is on Netapp iSCSI storage, the other one on the Dell MD3200. If I try to backup machines on the MD3200 CSV they are not snapped using the hardware provider.

Looking further into this issue with vssadmin,  diskshadow and a simple NTFS volume on the MD3200:

C:\Program Files (x86)\Dell\MD Storage Software>vssadmin list providers
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2005 Microsoft Corp.

Provider name: 'SmVssProvider'
 Provider type: Hardware
 Provider Id: {24e4e8a3-69cb-4370-8b7b-c5276cd49765}
 Version: 10.70.08

Provider name: 'Microsoft Software Shadow Copy provider 1.0'
 Provider type: System
 Provider Id: {b5946137-7b9f-4925-af80-51abd60b20d5}
 Version: 1.0.0.7

C:\Program Files (x86)\Dell\MD Storage Software>diskshadow
Microsoft DiskShadow version 1.0
Copyright (C) 2007 Microsoft Corporation
On computer: FCHV-01A, 16.11.2010 16:19:09


DISKSHADOW> set context persistent

DISKSHADOW> add volume e: provider {24e4e8a3-69cb-4370-8b7b-c5276cd49765}
The registered provider does not support shadow copy of this volume.

Not even snapping this simple volume is possible!

Dell claims it's the backup software and DPM 2010 is not supported. But at least the above test with diskshadow should work in my opinion....

I have configured the HW provider according to Dell's document "Data Protection in a Hyper-V Environment" but to no avail. Now I'm stuck, spent money on the array and the snapshot license for it and it doesn't even do the basic stuff.

Hi All,

First of all, I know it's a beta and these are the perils of being an early adopter, but I've got a serious problem.

I've upgraded our production Hyper-V cluster to Server 2012. The setup is a 4 node cluster running CSVs on an ISCSI SAN with MPIO via dual gigabit Ethernet networks. The SAN storage is provided byOpen-E DSS7 and replicated to another server in a different building.

Post the upgrade everything about the cluster seemed stable and to work as expected - live migrations etc all working. I then turned my attention to backups, and I discovered that Server 2012 wasn't supported by DPM. Fortunately there is a beta of DPM 2012 SP1 which adds support for Server 2012, unfortunately there is no upgrade path from the beta to RTM of SP1. Not wanting to upgrade our production DPM server to a beta, I installed a copy of DPM 2012 SP1 beta on a VM to provide a stopgap backup solution for VM level backups of certain machines that couldn't be backed up in other ways. I realise that running the backup server on the same cluster / SAN as the stuff that's being backed up is an odd thing to do, but this at least serves to provide snapshots, SAN replication provides resilience, and like I say, this is a stopgap.

Then I started noticing problems. First symptom was that on starting / rebooting VMs, sometimes other VMs would hang for perhaps 30s - 2m, people would start complaining that SharePoint had gone unresponsive etc. However, they would come back to life in a minute or two.On a couple of occasions we came in in the morning to find a number of VMs off or paused (backups ran overnight). Both of these problems occurred only when the DPM server was turned on. I thought the issue might be general load on the SAN, having both the backup server and the machines being backed up living on the same CSV / hardware. I moved the DPM server to a different ISCSI box and put on aggressive throttling (200Mbps) to try to reduce load, but the problem continues.

The event logs on the Hyper-V cluster suggest I/O timeouts to the SAN at the times of the backups. Lot's of event ID 1069, 1205, 1146, 1230,  (various cluster resources failed). The interesting one I think is 5120 Cluster Shared Volume 'Volume5' ('VOLUME NAME') is no longer available on this node because of 'STATUS_CLUSTER_CSV_AUTO_PAUSE_ERROR(c0130021)'. All I/O will temporarily be queued until a path to the volume is reestablished.

Is anyone else using SP1 beta to successfully backup a 2012 Hyper-V cluster? 

Is anyone seeing the same problem?

Is it likely that this is a problem with SP1 beta, will it be fixed at RTM?

Any suggestions for a stopgap solution?

I think I might try setting up a test physical DPM server to check the issue isn't in someway related to the fact that the DPM server sits on the same cluster it's backing up. I'm also happy to consider the problem could lie elsewhere i.e. with the SAN storage (this was upgraded from v6 to v7 at the same time as the 2012 upgrade, but as soon as I tell the vendor that the problem relates to running a beta of DPM they will be pointing fingers at that.

Thanks,

Tim