System Center 2016 UR3

It’s out now –

https://support.microsoft.com/en-hk/help/4020906/update-rollup-3-for-system-center-2016

A lot of good VMM fixes in there – which I will be testing soon. Bulk host agent update script is in Charbel’s blog here: https://charbelnemnom.com/2017/05/update-rollup-3-for-system-center-2016-is-now-available-sysctr-systemcenter-scvmm/

Details of SCOM fixes in Kevin’s blog here: http://kevingreeneitblog.blogspot.co.uk/2017/05/scom-2016-update-rollup-3-ur3-now.html

I’m a little disappointed to see DPM missed an update in UR3. VMware support is still missing from 2016 – but all will be forgiven if this turns up in UR4 along with fixes for woes experienced with UR2 currently:

Tape Library Sharing – 2012 OS cannot remove TL sharing & re-establishing 2016 OS TL required a manual DB cleanout (with Premier Support).

Console Crashing on PG alteration – requires DLL from MS (see my previous posts)

Mount points, whilst supported for the storage (see my other posts) uncover a known issue with DPM mouting the VHDX files for Modern backup Storage – the workaround for this is to add a drive letter to the storage.

If you dont urgently supported SQL 2016 backups / sharepoint 2016 protection from DPM, I would serously condider sticking to UR1 for now.

Roll on UR4! 🙂

 

 

DPM 2016 agent installations – Making your life easier with SCCM

Take the pain away from manual deployment – grab the agent and put it into SCCM. The command lines for agent install (2016 UR2) are:

DPMAgentInstaller_KB3209593_AMD64.exe /q /IAcceptEULA

DPMAgentInstaller_KB3209593.exe /q /IAcceptEULA (for x86)

Just make sure all the agent pre-reqs are in place (WMF 4.0 for 2008 R2 etc…) and make the detection of those a pre-req for the SCCM deployment.

If you know what DPM server you are going to protect with – simply add the server name to the install above – that will open the ports and make the agent ready to be attached.

If you dont just yet – then run a second SCCM task to call a batch file running the setdpmserver.exe (in the DPM agent Bin directory) to configure the agent.

Run an “Application deployment type compliance details” report in SCCM, using your target collections, application, deployment type and status of “Success” to generate a CSV file of the installed agents.

Take the computer name column in excel, append your domain name (using concatenate) and put the resulting list into a .txt file (no headings or any other info required)

Open the DPM console – select Install, Attach Agents, click add from file and point to your txt file.

Output from SCCM report, manipulate and import in ~10 mins saving many hours of manual config.

Job Done!

VMM 2016 UR 2 on Windows 2016 – Storage Management Bug

Recent installs of VMM 2016 have shown nice improvements over 2012, especially a much needed performance boost with storage operations through SMI-S.

Our latest SMI-S provider from NetApp (for ontap 9.1) in combination with VMM 2016 – seems to be light years ahead of 2012 R2. It’s responsive and carries out tasks in a tenth of the time that VMM 2012 took with ontap 8.

All issues with the SMI-S provider going unresponsive have been found to be due to a little service running in windows:

SMI-S

This little chap seems to randomly keel over – with system and application logs revealing nothing as to why.

  • Log Name: Application
    Source: Application Error
    Date: 28/04/2017 17:44:20
    Event ID: 1000
    Task Category: (100)
    Level: Error
    Keywords: Classic
    User: N/A
    Computer:
    Description:
    Faulting application name: svchost.exe_MSStrgSvc, version: 10.0.14393.0, time stamp: 0x57899b1c
    Faulting module name: concrete.dll, version: 10.0.14393.0, time stamp: 0x57899a8c
    Exception code: 0xc0000005
    Fault offset: 0x0000000000002eb0
    Faulting process ID: 0xe38
    Faulting application start time: 0x01d2c03e31b6c56d
    Faulting application path: C:\Windows\system32\svchost.exe
    Faulting module path: c:\windows\system32\concrete.dll
    Report ID: 1bc955fe-6a3b-469b-8c0c-de7992f3858d
    Faulting package full name:
    Faulting package-relative application ID:

I have logged it with premier support – they know about it, but a fix isn’t available as yet. Frustrating, but not the end of the world – just start the service again, wait 2 mins, then refresh the provider in SCVMM – all is good!

As soon as a fix is available – I will post an update here.

Updated:

  • WinCXE is planning of hotfix for Windows 10 version 1607. The KB article ID is going to be 4019472.

This should be available towards the end of May 2017.

DPM 2016 – Modern Backup Storage (MBS) Volumes

An excellent feature of 2016 and not to be missed. Gone is the LDM limitation and the volume mess of 2012. This is by far the best improvement on the table for DPM.

I followed some guides to test – mounting 64tb of backup storage as drive letters to windows makes it a target for temporary storage – VERY IRRITATING INDEED

You will quickly find your DPM volume has attracted itself as a temporary dumping ground for hotfixes and patches. This is not acceptable.

To avoid this – simply create yourself a folder on C: – in this case ‘MountPoints’, create yourself two empty folders – then mount your backup storage in there – rather than assigning a drive letter.

mount-points

Open DPM, refresh disks and click add – you will see the two volumes available for use.

Windows wont see them in it’s drive letter searches and thusly – they will remain clutter free and available for DPM use only!

As long as you make sure they have a volume label – SCOM will happily monitor the health of drives mounted in this manner.

DPM 2016 – Installation – SQL 2016 SP1

All,

Just a quick snippet from Technet as it’s easily overlooked. When installing DPM 2016 RTM, you need to make sure that SQL 2016 does NOT have SP1 installed.

Once DPM is installed and you have applied UR2, you can patch SQL to SP1.

Don’t forget to enable browser service, named pipes (for Library sharing) and clear all pending reboots…

Enjoy.

DPM 2016 – UR2 – MMC console crashes – event ID 999

Once you are running DPM 2016 UR2 – you may experience console crashes (services unaffected) with event ID’s 999:

The description for Event ID 999 from source MSDPM cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had to be saved with the event.

The following information was included with the event:

An unexpected error caused a failure for process 'mmc'.  Restart the DPM process 'mmc'.
Problem Details:

19009/05/2017 13:29:28DpmThreadPool.cs163TrueNullReferenceExceptionObject reference not set to an instance of an object.System.NullReferenceException: Object reference not set to an instance of an object.

   at Microsoft.Internal.EnterpriseStorage.Dls.UI.ObjectModel.Inquiry.MultiDatasourceSizeInquiry.ResultUniquePath(InquiryPath junctionlessPath, InquiryPath startNodeRelativePath, String serverName)

   at Microsoft.Internal.EnterpriseStorage.Dls.UI.ObjectModel.Inquiry.MultiDatasourceSizeInquiry.ProcessResults(InquiryTaskInformation taskInfo)

   at Microsoft.Internal.EnterpriseStorage.Dls.UI.ObjectModel.Inquiry.MultiDatasourceSizeInquiry.OnInquiryResultsArrival(InquiryResult result)

   at Microsoft.Internal.EnterpriseStorage.Dls.UI.ObjectModel.Inquiry.MultiDatasourceSizeInquiry.rornTaskDef_TaskProgress(Object sender, EventMatchedEventArgs eventEntry)

   at Microsoft.Internal.EnterpriseStorage.Dls.EngineUICommon.RORN.RornTaskDef.Task_TaskProgress(Object sender, EventMatchedEventArgs e)

   at Microsoft.Internal.EnterpriseStorage.Dls.EventManagement.BaseEventFilter.Send(EventMatchedEventArgs args)

   at Microsoft.Internal.EnterpriseStorage.Dls.EventManagement.EventManager.PollingThreadProc(Object data)

   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)

   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)

   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)

   at System.Threading.ThreadHelper.ThreadStart(Object obj)

the message resource is present but the message is not found in the string/message table

These occur when adding secondary protection (DPM to DPM) to an existing Protection Group, as soon as you reach the storage calculation screen – bang, MMC crashes.

The issue resides with objectmodel.dll (in the BIN directory) – a replacement from MS was supplied and immediately resolved the issue.

If this is the case, currently the fix exists as a private hotfix only and must be obtained by logging a call with MS support.

I have asked if it will be made public and will update the blog when I hear back.

Update 10 May 2017

Premier Support have advised that the fixed DLL will be part of a future UR pack, but could not confirm if it would be UR3 or 4. So my advice is you test your environment with a few machines and see if you encounter the error – if you do, contact support and request the fixed DLL before you proceed into full production use.

P2V Windows 2008 R1 SP2 – Still going on 2017…..

Those of us in Enterprise IT environments are plagued with ancient OS’s and the complete lack of desire to migrate to the latest OS/SQL from business colleagues forces us to live with unpaletable solutions.

Anyway…. with just about supported OS’s lurking on some seriously old hardware, I’ve had to P2V some really old stuff to Hyper-V.

The P2V tool disappeared from SCVMM 2012 and took a path of its own – it’s available here: MVMC v3.0

Download it to a fresh VM – 2012 R2 will do nicely. Make sure to install BITS compact server:

add-windowsfeature BITS-Compact-Server

On your physical server (in my example case, an old dell, 2008 R1, SQL 2008- with FC cards, drives attached to NetApp) stop and disable all application/role services – SQL, backups etc… so nothing important tries to launch on first boot.

Uninstall any dell OMSA apps, leave anything physical hardware related for now, as you need your SAN attached drives online for the migration.

If you want to reboot the server now for a clean run, go for it. Start the MVMC console and point it at your target, capture all your drives and point it at a suitable Hyper-V host for the VM to be created.

*Make sure to have over 2 x the required disk space for the VM to be created. This will allow for MVMC to create dynamically expanding VHD files and for you to convert them to VHDX if you desire…

For me – this section completed quite quickly, then the fun begins. There are a few articles out there referencing this – some say convert to VMware first etc… but we want to avoid multiple hops and want it to go straight to Hyper-V, so – boot your VM with the ISO revelavnt to the installed OS. Open recovery console.

If you are worried about making mistakes on the next bit – then at this point take a copy of the VHD for the C Drive you have created, so you can recover quickly if you make a mistake.

run:

Bcdedit /enum

Check for any missing references and correct where necessary – for example:

bcdedit /set {bootmgr} device partition=C:
bcdedit /set {default} device partition=C:
bcdedit /set {default} osdevice partition=C:

Open regedit, open the SYSTEM hive, expand expand CurrentControlSet1\Services – find the below services and set the start value as below:

Aliide = 3

Amdide =3

Atapi = 0

Cmdide = 3

iaStorV = 3

intelide = 0

msahci = 3

pciide = 3

viaide = 3

Also – find anything SAS related and set the value to 4 (LSI, LSI-SAS,Mega and so on….) This instructs the OS to load IDE drivers on boot, as your horrible gen-1 VM is booting from IDE only – your physical was likely to be using a PERC raid card (SCSI/SAS).

Restart and you should avoid any STOP: 0x0000007B errors.

Attach the VMguest.iso to the VM and upgrade the integration services asap. It can be found on Hyper-V 2012 R2 here: c:\windows\system32\vmguest.iso

Once booted – set about removing any NetApp MPIO/software, any thing hardware related.

Check your drives and letters are as they should be. Hyper-V VM is created with a DVD drive and it may well have popped in with letter E/F causing one of your drives not to have the correct letter online. Sort those issues out, then restart to complete the clean up.

Once you are happy, set your services back to auto/manual and fire up your services.

Don’t forget, MVMC has only created a VM with the config in the default location, so you may want to carry out a storage migration to get all your files together in the right place, especially if you intend to make the VM highly available in your Hyper-V cluster.