VMM 2016 UR 2 on Windows 2016 – Storage Management Bug

Recent installs of VMM 2016 have shown nice improvements over 2012, especially a much needed performance boost with storage operations through SMI-S.

Our latest SMI-S provider from NetApp (for ontap 9.1) in combination with VMM 2016 – seems to be light years ahead of 2012 R2. It’s responsive and carries out tasks in a tenth of the time that VMM 2012 took with ontap 8.

All issues with the SMI-S provider going unresponsive have been found to be due to a little service running in windows:

SMI-S

This little chap seems to randomly keel over – with system and application logs revealing nothing as to why.

  • Log Name: Application
    Source: Application Error
    Date: 28/04/2017 17:44:20
    Event ID: 1000
    Task Category: (100)
    Level: Error
    Keywords: Classic
    User: N/A
    Computer:
    Description:
    Faulting application name: svchost.exe_MSStrgSvc, version: 10.0.14393.0, time stamp: 0x57899b1c
    Faulting module name: concrete.dll, version: 10.0.14393.0, time stamp: 0x57899a8c
    Exception code: 0xc0000005
    Fault offset: 0x0000000000002eb0
    Faulting process ID: 0xe38
    Faulting application start time: 0x01d2c03e31b6c56d
    Faulting application path: C:\Windows\system32\svchost.exe
    Faulting module path: c:\windows\system32\concrete.dll
    Report ID: 1bc955fe-6a3b-469b-8c0c-de7992f3858d
    Faulting package full name:
    Faulting package-relative application ID:

I have logged it with premier support – they know about it, but a fix isn’t available as yet. Frustrating, but not the end of the world – just start the service again, wait 2 mins, then refresh the provider in SCVMM – all is good!

As soon as a fix is available – I will post an update here.

Updated:

  • WinCXE is planning of hotfix for Windows 10 version 1607. The KB article ID is going to be 4019472.

This should be available towards the end of May 2017.

P2V Windows 2008 R1 SP2 – Still going on 2017…..

Those of us in Enterprise IT environments are plagued with ancient OS’s and the complete lack of desire to migrate to the latest OS/SQL from business colleagues forces us to live with unpaletable solutions.

Anyway…. with just about supported OS’s lurking on some seriously old hardware, I’ve had to P2V some really old stuff to Hyper-V.

The P2V tool disappeared from SCVMM 2012 and took a path of its own – it’s available here: MVMC v3.0

Download it to a fresh VM – 2012 R2 will do nicely. Make sure to install BITS compact server:

add-windowsfeature BITS-Compact-Server

On your physical server (in my example case, an old dell, 2008 R1, SQL 2008- with FC cards, drives attached to NetApp) stop and disable all application/role services – SQL, backups etc… so nothing important tries to launch on first boot.

Uninstall any dell OMSA apps, leave anything physical hardware related for now, as you need your SAN attached drives online for the migration.

If you want to reboot the server now for a clean run, go for it. Start the MVMC console and point it at your target, capture all your drives and point it at a suitable Hyper-V host for the VM to be created.

*Make sure to have over 2 x the required disk space for the VM to be created. This will allow for MVMC to create dynamically expanding VHD files and for you to convert them to VHDX if you desire…

For me – this section completed quite quickly, then the fun begins. There are a few articles out there referencing this – some say convert to VMware first etc… but we want to avoid multiple hops and want it to go straight to Hyper-V, so – boot your VM with the ISO revelavnt to the installed OS. Open recovery console.

If you are worried about making mistakes on the next bit – then at this point take a copy of the VHD for the C Drive you have created, so you can recover quickly if you make a mistake.

run:

Bcdedit /enum

Check for any missing references and correct where necessary – for example:

bcdedit /set {bootmgr} device partition=C:
bcdedit /set {default} device partition=C:
bcdedit /set {default} osdevice partition=C:

Open regedit, open the SYSTEM hive, expand expand CurrentControlSet1\Services – find the below services and set the start value as below:

Aliide = 3

Amdide =3

Atapi = 0

Cmdide = 3

iaStorV = 3

intelide = 0

msahci = 3

pciide = 3

viaide = 3

Also – find anything SAS related and set the value to 4 (LSI, LSI-SAS,Mega and so on….) This instructs the OS to load IDE drivers on boot, as your horrible gen-1 VM is booting from IDE only – your physical was likely to be using a PERC raid card (SCSI/SAS).

Restart and you should avoid any STOP: 0x0000007B errors.

Attach the VMguest.iso to the VM and upgrade the integration services asap. It can be found on Hyper-V 2012 R2 here: c:\windows\system32\vmguest.iso

Once booted – set about removing any NetApp MPIO/software, any thing hardware related.

Check your drives and letters are as they should be. Hyper-V VM is created with a DVD drive and it may well have popped in with letter E/F causing one of your drives not to have the correct letter online. Sort those issues out, then restart to complete the clean up.

Once you are happy, set your services back to auto/manual and fire up your services.

Don’t forget, MVMC has only created a VM with the config in the default location, so you may want to carry out a storage migration to get all your files together in the right place, especially if you intend to make the VM highly available in your Hyper-V cluster.

SCVMM Cluster to Cluster Live Migrations – SMB Storage – NetApp SVM

In an ideal world we’d be using SoFS. But, as we have already thrown investment into NetApp, then we are going to use it – or – be thrown out the door.

There are many good articles about shared nothing migration, migrating from host to host – but not many cover those of us living with multiple clusters, or using other SMB providers.

Leading you here may be SCVMM error codes:

Error 12700 – 0x80072746, 0x80070005 – forcibly closed by the remote host, or General access denied error for your Run_As account to the SMB file path.

Our Example setup is this:

HyperVCluster1 – DomainA, multi node, storage is on SMB served up by a NetApp SVM dedicated for Hyper-V workloads.

HyperVCluster2 – DomainA, multi node, same storage access as cluster 1.

So – we already have the same file shares, assigned in SCVMM to both clusters, servers have all the access they require, as does the VMM run_as account that the clusters are using for file operations.

If you are reading this, then you already have looked at Kerberos delegation, started assigning every node CIFS & Microsoft virtual system migration service delegations to every other node – STOP NOW!

In SCVMM – make sure every cluster node is set to ‘Use Kerberos’ This prevents the tedious requirement of setting a web of host to host delegations.

Kerberos

Set this using the VMM PowerShell by running:

VMM-kerberos

Once done, set up a Kerberos delegation to the AD object name of your NetApp SVM SMB server.

Example file share: \\NetappSMB3\Tier1_01

Add a Kerberos delegation for CIFS (ugh – why does this term still linger everywhere) for DomainA\NetappSMB3 on all your hyper V cluster nodes that have access to SMB.

You can use the SCVMM & AD PowerShell modules to set this quickly using:

Delegation

As soon as this is in place, you will be free to live migrate running VMs between clusters that have access to the same file system.