DPM 2016 – UR2 – MMC console crashes – event ID 999

Once you are running DPM 2016 UR2 – you may experience console crashes (services unaffected) with event ID’s 999:

The description for Event ID 999 from source MSDPM cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had to be saved with the event.

The following information was included with the event:

An unexpected error caused a failure for process 'mmc'.  Restart the DPM process 'mmc'.
Problem Details:

19009/05/2017 13:29:28DpmThreadPool.cs163TrueNullReferenceExceptionObject reference not set to an instance of an object.System.NullReferenceException: Object reference not set to an instance of an object.

   at Microsoft.Internal.EnterpriseStorage.Dls.UI.ObjectModel.Inquiry.MultiDatasourceSizeInquiry.ResultUniquePath(InquiryPath junctionlessPath, InquiryPath startNodeRelativePath, String serverName)

   at Microsoft.Internal.EnterpriseStorage.Dls.UI.ObjectModel.Inquiry.MultiDatasourceSizeInquiry.ProcessResults(InquiryTaskInformation taskInfo)

   at Microsoft.Internal.EnterpriseStorage.Dls.UI.ObjectModel.Inquiry.MultiDatasourceSizeInquiry.OnInquiryResultsArrival(InquiryResult result)

   at Microsoft.Internal.EnterpriseStorage.Dls.UI.ObjectModel.Inquiry.MultiDatasourceSizeInquiry.rornTaskDef_TaskProgress(Object sender, EventMatchedEventArgs eventEntry)

   at Microsoft.Internal.EnterpriseStorage.Dls.EngineUICommon.RORN.RornTaskDef.Task_TaskProgress(Object sender, EventMatchedEventArgs e)

   at Microsoft.Internal.EnterpriseStorage.Dls.EventManagement.BaseEventFilter.Send(EventMatchedEventArgs args)

   at Microsoft.Internal.EnterpriseStorage.Dls.EventManagement.EventManager.PollingThreadProc(Object data)

   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)

   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)

   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)

   at System.Threading.ThreadHelper.ThreadStart(Object obj)

the message resource is present but the message is not found in the string/message table

These occur when adding secondary protection (DPM to DPM) to an existing Protection Group, as soon as you reach the storage calculation screen – bang, MMC crashes.

The issue resides with objectmodel.dll (in the BIN directory) – a replacement from MS was supplied and immediately resolved the issue.

If this is the case, currently the fix exists as a private hotfix only and must be obtained by logging a call with MS support.

I have asked if it will be made public and will update the blog when I hear back.

Update 10 May 2017

Premier Support have advised that the fixed DLL will be part of a future UR pack, but could not confirm if it would be UR3 or 4. So my advice is you test your environment with a few machines and see if you encounter the error – if you do, contact support and request the fixed DLL before you proceed into full production use.

P2V Windows 2008 R1 SP2 – Still going on 2017…..

Those of us in Enterprise IT environments are plagued with ancient OS’s and the complete lack of desire to migrate to the latest OS/SQL from business colleagues forces us to live with unpaletable solutions.

Anyway…. with just about supported OS’s lurking on some seriously old hardware, I’ve had to P2V some really old stuff to Hyper-V.

The P2V tool disappeared from SCVMM 2012 and took a path of its own – it’s available here: MVMC v3.0

Download it to a fresh VM – 2012 R2 will do nicely. Make sure to install BITS compact server:

add-windowsfeature BITS-Compact-Server

On your physical server (in my example case, an old dell, 2008 R1, SQL 2008- with FC cards, drives attached to NetApp) stop and disable all application/role services – SQL, backups etc… so nothing important tries to launch on first boot.

Uninstall any dell OMSA apps, leave anything physical hardware related for now, as you need your SAN attached drives online for the migration.

If you want to reboot the server now for a clean run, go for it. Start the MVMC console and point it at your target, capture all your drives and point it at a suitable Hyper-V host for the VM to be created.

*Make sure to have over 2 x the required disk space for the VM to be created. This will allow for MVMC to create dynamically expanding VHD files and for you to convert them to VHDX if you desire…

For me – this section completed quite quickly, then the fun begins. There are a few articles out there referencing this – some say convert to VMware first etc… but we want to avoid multiple hops and want it to go straight to Hyper-V, so – boot your VM with the ISO revelavnt to the installed OS. Open recovery console.

If you are worried about making mistakes on the next bit – then at this point take a copy of the VHD for the C Drive you have created, so you can recover quickly if you make a mistake.

run:

Bcdedit /enum

Check for any missing references and correct where necessary – for example:

bcdedit /set {bootmgr} device partition=C:
bcdedit /set {default} device partition=C:
bcdedit /set {default} osdevice partition=C:

Open regedit, open the SYSTEM hive, expand expand CurrentControlSet1\Services – find the below services and set the start value as below:

Aliide = 3

Amdide =3

Atapi = 0

Cmdide = 3

iaStorV = 3

intelide = 0

msahci = 3

pciide = 3

viaide = 3

Also – find anything SAS related and set the value to 4 (LSI, LSI-SAS,Mega and so on….) This instructs the OS to load IDE drivers on boot, as your horrible gen-1 VM is booting from IDE only – your physical was likely to be using a PERC raid card (SCSI/SAS).

Restart and you should avoid any STOP: 0x0000007B errors.

Attach the VMguest.iso to the VM and upgrade the integration services asap. It can be found on Hyper-V 2012 R2 here: c:\windows\system32\vmguest.iso

Once booted – set about removing any NetApp MPIO/software, any thing hardware related.

Check your drives and letters are as they should be. Hyper-V VM is created with a DVD drive and it may well have popped in with letter E/F causing one of your drives not to have the correct letter online. Sort those issues out, then restart to complete the clean up.

Once you are happy, set your services back to auto/manual and fire up your services.

Don’t forget, MVMC has only created a VM with the config in the default location, so you may want to carry out a storage migration to get all your files together in the right place, especially if you intend to make the VM highly available in your Hyper-V cluster.

SCVMM Cluster to Cluster Live Migrations – SMB Storage – NetApp SVM

In an ideal world we’d be using SoFS. But, as we have already thrown investment into NetApp, then we are going to use it – or – be thrown out the door.

There are many good articles about shared nothing migration, migrating from host to host – but not many cover those of us living with multiple clusters, or using other SMB providers.

Leading you here may be SCVMM error codes:

Error 12700 – 0x80072746, 0x80070005 – forcibly closed by the remote host, or General access denied error for your Run_As account to the SMB file path.

Our Example setup is this:

HyperVCluster1 – DomainA, multi node, storage is on SMB served up by a NetApp SVM dedicated for Hyper-V workloads.

HyperVCluster2 – DomainA, multi node, same storage access as cluster 1.

So – we already have the same file shares, assigned in SCVMM to both clusters, servers have all the access they require, as does the VMM run_as account that the clusters are using for file operations.

If you are reading this, then you already have looked at Kerberos delegation, started assigning every node CIFS & Microsoft virtual system migration service delegations to every other node – STOP NOW!

In SCVMM – make sure every cluster node is set to ‘Use Kerberos’ This prevents the tedious requirement of setting a web of host to host delegations.

Kerberos

Set this using the VMM PowerShell by running:

VMM-kerberos

Once done, set up a Kerberos delegation to the AD object name of your NetApp SVM SMB server.

Example file share: \\NetappSMB3\Tier1_01

Add a Kerberos delegation for CIFS (ugh – why does this term still linger everywhere) for DomainA\NetappSMB3 on all your hyper V cluster nodes that have access to SMB.

You can use the SCVMM & AD PowerShell modules to set this quickly using:

Delegation

As soon as this is in place, you will be free to live migrate running VMs between clusters that have access to the same file system.