Hyper-Vine

VMM 2016 UR 2 on Windows 2016 – Storage Management Bug

Recent installs of VMM 2016 have shown nice improvements over 2012, especially a much needed performance boost with storage operations through SMI-S.

Our latest SMI-S provider from NetApp (for ontap 9.1) in combination with VMM 2016 – seems to be light years ahead of 2012 R2. It’s responsive and carries out tasks in a tenth of the time that VMM 2012 took with ontap 8.

All issues with the SMI-S provider going unresponsive have been found to be due to a little service running in windows:

SMI-S

This little chap seems to randomly keel over – with system and application logs revealing nothing as to why.

Log Name: Application
Source: Application Error
Date: 28/04/2017 17:44:20
Event ID: 1000
Task Category: (100)
Level: Error
Keywords: Classic
User: N/A
Computer:
Description:
Faulting application name: svchost.exe_MSStrgSvc, version: 10.0.14393.0, time stamp: 0x57899b1c
Faulting module name: concrete.dll, version: 10.0.14393.0, time stamp: 0x57899a8c
Exception code: 0xc0000005
Fault offset: 0x0000000000002eb0
Faulting process ID: 0xe38
Faulting application start time: 0x01d2c03e31b6c56d
Faulting application path: C:\Windows\system32\svchost.exe
Faulting module path: c:\windows\system32\concrete.dll
Report ID: 1bc955fe-6a3b-469b-8c0c-de7992f3858d
Faulting package full name:
Faulting package-relative application ID:

I have logged it with premier support – they know about it, but a fix isn’t available as yet. Frustrating, but not the end of the world – just start the service again, wait 2 mins, then refresh the provider in SCVMM – all is good!

As soon as a fix is available – I will post an update here.

Updated:

WinCXE is planning of hotfix for Windows 10 version 1607. The KB article ID is going to be 4019472.

This should be available towards the end of May 2017.

DPM 2016 – Modern Backup Storage (MBS) Volumes

An excellent feature of 2016 and not to be missed. Gone is the LDM limitation and the volume mess of 2012. This is by far the best improvement on the table for DPM.

I followed some guides to test – mounting 64tb of backup storage as drive letters to windows makes it a target for temporary storage – VERY IRRITATING INDEED

You will quickly find your DPM volume has attracted itself as a temporary dumping ground for hotfixes and patches. This is not acceptable.

To avoid this – simply create yourself a folder on C: – in this case ‘MountPoints’, create yourself two empty folders – then mount your backup storage in there – rather than assigning a drive letter.

mount-points

Open DPM, refresh disks and click add – you will see the two volumes available for use.

Windows wont see them in it’s drive letter searches and thusly – they will remain clutter free and available for DPM use only!

As long as you make sure they have a volume label – SCOM will happily monitor the health of drives mounted in this manner.

Update – Aug 2017

Currently under UR2, we experience errors pertaining to DPM being unable to mount certain files under the mount points. It’s a known issue and current workaround is to also add a drive letter to the disks (can be added along side the mount point, no need to change anything else).

This should be resolved in further updates.

DPM 2016 – Installation – SQL 2016 SP1

All,

Just a quick snippet from Technet as it’s easily overlooked. When installing DPM 2016 RTM, you need to make sure that SQL 2016 does NOT have SP1 installed.

Once DPM is installed and you have applied UR2, you can patch SQL to SP1.

Don’t forget to enable browser service, named pipes (for Library sharing) and clear all pending reboots…

Enjoy.

DPM 2016 – UR2 – MMC console crashes – event ID 999

Once you are running DPM 2016 UR2 – you may experience console crashes (services unaffected) with event ID’s 999:

The description for Event ID 999 from source MSDPM cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had to be saved with the event.

The following information was included with the event:

An unexpected error caused a failure for process 'mmc'.  Restart the DPM process 'mmc'.
Problem Details:

19009/05/2017 13:29:28DpmThreadPool.cs163TrueNullReferenceExceptionObject reference not set to an instance of an object.System.NullReferenceException: Object reference not set to an instance of an object.

   at Microsoft.Internal.EnterpriseStorage.Dls.UI.ObjectModel.Inquiry.MultiDatasourceSizeInquiry.ResultUniquePath(InquiryPath junctionlessPath, InquiryPath startNodeRelativePath, String serverName)

   at Microsoft.Internal.EnterpriseStorage.Dls.UI.ObjectModel.Inquiry.MultiDatasourceSizeInquiry.ProcessResults(InquiryTaskInformation taskInfo)

   at Microsoft.Internal.EnterpriseStorage.Dls.UI.ObjectModel.Inquiry.MultiDatasourceSizeInquiry.OnInquiryResultsArrival(InquiryResult result)

   at Microsoft.Internal.EnterpriseStorage.Dls.UI.ObjectModel.Inquiry.MultiDatasourceSizeInquiry.rornTaskDef_TaskProgress(Object sender, EventMatchedEventArgs eventEntry)

   at Microsoft.Internal.EnterpriseStorage.Dls.EngineUICommon.RORN.RornTaskDef.Task_TaskProgress(Object sender, EventMatchedEventArgs e)

   at Microsoft.Internal.EnterpriseStorage.Dls.EventManagement.BaseEventFilter.Send(EventMatchedEventArgs args)

   at Microsoft.Internal.EnterpriseStorage.Dls.EventManagement.EventManager.PollingThreadProc(Object data)

   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)

   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)

   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)

   at System.Threading.ThreadHelper.ThreadStart(Object obj)

the message resource is present but the message is not found in the string/message table

These occur when adding secondary protection (DPM to DPM) to an existing Protection Group, as soon as you reach the storage calculation screen – bang, MMC crashes.

The issue resides with objectmodel.dll (in the BIN directory) – a replacement from MS was supplied and immediately resolved the issue.

If this is the case, currently the fix exists as a private hotfix only and must be obtained by logging a call with MS support.

I have asked if it will be made public and will update the blog when I hear back.

Update 10 May 2017

Premier Support have advised that the fixed DLL will be part of a future UR pack, but could not confirm if it would be UR3 or 4. So my advice is you test your environment with a few machines and see if you encounter the error – if you do, contact support and request the fixed DLL before you proceed into full production use.

P2V Windows 2008 R1 SP2 – Still going on 2017…..

Those of us in Enterprise IT environments are plagued with ancient OS’s and the complete lack of desire to migrate to the latest OS/SQL from business colleagues forces us to live with unpaletable solutions.

Anyway…. with just about supported OS’s lurking on some seriously old hardware, I’ve had to P2V some really old stuff to Hyper-V.

The P2V tool disappeared from SCVMM 2012 and took a path of its own – it’s available here: MVMC v3.0

Download it to a fresh VM – 2012 R2 will do nicely. Make sure to install BITS compact server:

add-windowsfeature BITS-Compact-Server

On your physical server (in my example case, an old dell, 2008 R1, SQL 2008- with FC cards, drives attached to NetApp) stop and disable all application/role services – SQL, backups etc… so nothing important tries to launch on first boot.

Uninstall any dell OMSA apps, leave anything physical hardware related for now, as you need your SAN attached drives online for the migration.

If you want to reboot the server now for a clean run, go for it. Start the MVMC console and point it at your target, capture all your drives and point it at a suitable Hyper-V host for the VM to be created.

*Make sure to have over 2 x the required disk space for the VM to be created. This will allow for MVMC to create dynamically expanding VHD files and for you to convert them to VHDX if you desire…

For me – this section completed quite quickly, then the fun begins. There are a few articles out there referencing this – some say convert to VMware first etc… but we want to avoid multiple hops and want it to go straight to Hyper-V, so – boot your VM with the ISO revelavnt to the installed OS. Open recovery console.

If you are worried about making mistakes on the next bit – then at this point take a copy of the VHD for the C Drive you have created, so you can recover quickly if you make a mistake.

run:

Bcdedit /enum

Check for any missing references and correct where necessary – for example:

bcdedit /set {bootmgr} device partition=C:
bcdedit /set {default} device partition=C:
bcdedit /set {default} osdevice partition=C:

Open regedit, open the SYSTEM hive, expand expand CurrentControlSet1\Services – find the below services and set the start value as below:

Aliide = 3

Amdide =3

Atapi = 0

Cmdide = 3

iaStorV = 3

intelide = 0

msahci = 3

pciide = 3

viaide = 3

Also – find anything SAS related and set the value to 4 (LSI, LSI-SAS,Mega and so on….) This instructs the OS to load IDE drivers on boot, as your horrible gen-1 VM is booting from IDE only – your physical was likely to be using a PERC raid card (SCSI/SAS).

Restart and you should avoid any STOP: 0x0000007B errors.

Attach the VMguest.iso to the VM and upgrade the integration services asap. It can be found on Hyper-V 2012 R2 here: c:\windows\system32\vmguest.iso

Once booted – set about removing any NetApp MPIO/software, any thing hardware related.

Check your drives and letters are as they should be. Hyper-V VM is created with a DVD drive and it may well have popped in with letter E/F causing one of your drives not to have the correct letter online. Sort those issues out, then restart to complete the clean up.

Once you are happy, set your services back to auto/manual and fire up your services.

Don’t forget, MVMC has only created a VM with the config in the default location, so you may want to carry out a storage migration to get all your files together in the right place, especially if you intend to make the VM highly available in your Hyper-V cluster.

SCVMM Cluster to Cluster Live Migrations – SMB Storage – NetApp SVM

In an ideal world we’d be using SoFS. But, as we have already thrown investment into NetApp, then we are going to use it – or – be thrown out the door.

There are many good articles about shared nothing migration, migrating from host to host – but not many cover those of us living with multiple clusters, or using other SMB providers.

Leading you here may be SCVMM error codes:

Error 12700 – 0x80072746, 0x80070005 – forcibly closed by the remote host, or General access denied error for your Run_As account to the SMB file path.

Our Example setup is this:

HyperVCluster1 – DomainA, multi node, storage is on SMB served up by a NetApp SVM dedicated for Hyper-V workloads.

HyperVCluster2 – DomainA, multi node, same storage access as cluster 1.

So – we already have the same file shares, assigned in SCVMM to both clusters, servers have all the access they require, as does the VMM run_as account that the clusters are using for file operations.

If you are reading this, then you already have looked at Kerberos delegation, started assigning every node CIFS & Microsoft virtual system migration service delegations to every other node – STOP NOW!

In SCVMM – make sure every cluster node is set to ‘Use Kerberos’ This prevents the tedious requirement of setting a web of host to host delegations.

Kerberos

Set this using the VMM PowerShell by running:

VMM-kerberos

Once done, set up a Kerberos delegation to the AD object name of your NetApp SVM SMB server.

Example file share: \\NetappSMB3\Tier1_01

Add a Kerberos delegation for CIFS (ugh – why does this term still linger everywhere) for DomainA\NetappSMB3 on all your hyper V cluster nodes that have access to SMB.

You can use the SCVMM & AD PowerShell modules to set this quickly using:

Delegation

As soon as this is in place, you will be free to live migrate running VMs between clusters that have access to the same file system.