Moving Shared VHDX files between Host clusters

Shared VHDX files in Hyper-V are useful for VM guest clusters – SQL, file servers and so forth. However, when moving from cluster to cluster, you need to plan for an outage in order to move the shared disks.

For me it makes the most sense to let SCVMM move the VM’s, in order to do this, I need to disconnect the shared VHDX files and reconnect them after the move. This is cleaner than offline copying/importing and trying to clean up VM config manually afterwards.

Example is below:

#Set Variables for VM cluster nodes
$currenthost='Host1'
$currenthost2='Host2'
$newhost='Host3'
$newhost2='Host4'
$vmnode1='VMname1'
$vmnode2='VMname2'
$storage='\ClusterStorage\myvolume\folder'
$newstorage='\ClusterStorage\mynewvolume\folder'

#Verify locations for shared disks in SCVMM - alter controller locations below

#Disconnect Disks from VM Node1
Remove-VMHardDiskDrive -computername $currenthost -vmname $VMnode1 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 2 
Remove-VMHardDiskDrive -computername $currenthost -vmname $VMnode1 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 3 
Remove-VMHardDiskDrive -computername $currenthost -vmname $VMnode1 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 4 
Remove-VMHardDiskDrive -computername $currenthost -vmname $VMnode1 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 5

#Disconnect Disks from VM Node2
Remove-VMHardDiskDrive -computername $currenthost2 -vmname $VMnode2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 2
Remove-VMHardDiskDrive -computername $currenthost2 -vmname $VMnode2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 3 
Remove-VMHardDiskDrive -computername $currenthost2 -vmname $VMnode2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 4 
Remove-VMHardDiskDrive -computername $currenthost2 -vmname $VMnode2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 5

#Open SCVMM, refresh VMs. Start storage migration (cluster to cluster in SCVMM)

#Copy Clustered disks from location A to B
Copy-Item \\$currenthost\c$\$storage\*.* -Destination \\$newhost\c$\$newstorage\ -Force

#Once Copy completed - reconnect clustered disks from new location

#Add Disks to VM Node1
Add-VMHardDiskDrive -computername $newhost -vmname $VMnode1 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 2 -Path c:\$newstorage\shared_disk_1.vhdx -SupportPersistentReservations
Add-VMHardDiskDrive -computername $newhost -vmname $VMnode1 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 3 -Path c:\$newstorage\shared_disk_2.vhdx -SupportPersistentReservations
Add-VMHardDiskDrive -computername $newhost -vmname $VMnode1 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 4 -Path c:\$newstorage\shared_disk_3.vhdx -SupportPersistentReservations
Add-VMHardDiskDrive -computername $newhost -vmname $VMnode1 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 5 -Path c:\$newstorage\shared_disk_4.vhdx -SupportPersistentReservations
 
#Add Disks to VM Node2
Add-VMHardDiskDrive -computername $newhost2 -vmname $VMnode2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 2 -Path c:\$newstorage\shared_disk_1.vhdx -SupportPersistentReservations
Add-VMHardDiskDrive -computername $newhost2 -vmname $VMnode2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 3 -Path c:\$newstorage\shared_disk_2.vhdx -SupportPersistentReservations
Add-VMHardDiskDrive -computername $newhost2 -vmname $VMnode2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 4 -Path c:\$newstorage\shared_disk_3.vhdx -SupportPersistentReservations
Add-VMHardDiskDrive -computername $newhost2 -vmname $VMnode2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 5 -Path c:\$newstorage\shared_disk_4.vhdx -SupportPersistentReservations

#Power on VMs
Advertisements

Hyper-V and HP c7000 Flex 10D, some thoughts…

I like the good old HP chassis, a reliable piece of kit and relatively straightforward to administer and maintain. On the host & networking side, networks are provided by modules in the rear of the chassis, allocated over the chassis fabric to each blade.

In essence, 2 x rear chassis Flex 10 modules (each with 8 x ports – FCoE, 10gbE or 1gbE SFP+ modules) provide resiliency to the server blades in the front of the chassis. Upstream you can have LACP connections for maximum throughput. Links are aggregated in shared uplink sets (SUS) and you can specify allowed VLAN tags across these uplink sets, which can then be dragged and dropped in server profiles to allow blade connectivity.

Each Flex module can provide 4 VF’s to a server (Virtual Functions). These can be 3 x Ethernet + 1 x FCoE or 4 Ethernet and so on. You provide this via a server profile in the Virtual connect manager.

As these are mirrored for availability, you don’t have 8 Ethernet to play with – you have 4 pairs. So, how best to utilise these connections? Answer: it depends.

4. If you have FCoE or iSCSI storage, you are immediately going to lose 1 pair of connections to facilitate this, leaving you with 3 pairs.

3. Now you are left with choices to make in regard to splitting up your traffic. These decisions will be driven by the back end network infrastructure connecting to the flex10’s. If you have physically separate networks for Internet facing traffic (IF/DMZ) – then you will have at least 1 pair of ports on the Flex side connected to this and you will therefore be forced to lose another pair on the host side.

2. and 1. You need host management, Live Migration, Heartbeat and Trusted/Non internet traffic for your VMs – from 2 pairs of NICs – So however you go about this, you will likely need to use LBFO teams & vSwitches (2012 R2) or SET (2016) over your pairs, carving out vEthernets for Management, LM & Heartbeat and VM networks.

Just be cautious of how you allocate bandwidth to the host network functions. Remember that internode CSV traffic is vital to maintaining storage connectivity and you don’t want your design to allow an out of control VM to consume all the available bandwidth, choking the host connectivity.

Likewise, a live migration needs to be fast, but not to the detriment of your workloads. Test, find the balance, tune and then prepare for distribution.

More reading of QoS in Switch Embedded Teaming is here….

 

 

Hyper-V – Don’t forget these simple administrative points! Part 2.

Continued from Part 1 – here.

8. DO use SCCM/WSUS for patching

For heavens sake, DON’T have firewall rules in your environment allowing all servers to talk outbound to the internet. WHEN they infiltrate your environment, you’ll have given them an easy exit with your data.

Block all internet bound traffic from your servers and monitor the blocking. Allow only key services and send them all via a proxy – by exception! Get WSUS to collect the patches you need and distribute them internally in a safe and controlled manner.

9. DO use VLANS on your Hyper-V switches – Get VMM to do the donkey work

Networks & IP pools in VMM are a wonderful thing. Set a VLAN, Static MAC and Static IP’s – all handled on build automatically. Make the most of this feature to keep tabs on your infrastructure. There’s no need to go back and forth to your IPAM tools for VM deployments.

Removing native networks and DHCP prevents a poorly configured VM from gaining access to anything it shouldn’t be talking to.

10. DON’T Mix and match CPU/Memory types in a cluster

Whilst it’s entirely possible to do so, you’ll end up compromising on performance of your VM’s. Everything will require CPU compatibility mode enabled (VM shutdown required to enable).

You will also be adding complexity to cluster capacity issues. Draining a host with 1tb of ram across a selection of 256gb hosts will be possible (provided no single VMs exist with memory assigned greater than 256gb), but sustaining node failures will become a complex calculation, rather than being able to tell at a glance.

If you are purchasing new nodes to expand your environment, then group them together and create a new cluster. Utilise live storage migrations between clusters – or – if using SMB3 storage, just simply live migrate the VM between clusters.

11. SQL per CPU host licencing on Hyper-V – beware!

Example: You have a Hyper-V cluster with 10 nodes, 2 processors per node (20 processors total) and you decide you want to use 2 nodes for SQL enterprise licensing on a per host CPU basis (4 CPU licences),

In short, a VM that has a licence covered by a host could potentially exist on any of the 10 nodes in your cluster. It does not matter about the settings on the VM defining the host it will reside on, it could be powered on, on any of the ten hosts and as such it is likely you would be seen to have fallen short of the required licences.

You would be best off removing 3 nodes from your existing cluster and creating a separate Hyper-V cluster dedicated for the SQL workloads and licences. Two nodes of running VM’s with a single empty host (which allows for a node failure/maintenance). This clearly shows your used hosts and your hot standby for a failure.

If in doubt – contact a licencing specialist company and get it checked and double checked. You do not want a bill in the post for the 8 remaining nodes 🙂

12. DO configure everything in PowerShell and DO retain it elsewhere

There’s many ways to automate the build of your hosts, SCVMM/SCCM or third party products, even LUN cloning (if boot from SAN). However, if you are manually building for any reason at all, then prepare the config and retain it afterwards. If the worst occurs, you can be back up and running in no time at all.

13. DO have a separate VLAN and address range for your host networks and DO keep them in step!

Make life easier for yourself. If you have 10 nodes, get 10 concurrent IP addresses from each of the ranges you may require (Management, Cluster-HB, Live Migration etc…) and make them the same last octet. Example:

Node1:

  • Mgmt: 10.0.1.11
  • ClusterHB: 172.10.0.11
  • Live Migration: 192.168.8.11

Node2:

  • Mgmt: 10.0.1.12
  • ClusterHB: 172.10.0.12
  • Live Migration: 192.168.8.12

And so on…. this will make your life easier in the long run. Keep these subnets away from any other use – so when you grow, you have more concurrent IP’s available.

If you ever need to revisit and reconfigure anything remotely via powershell, complexity will be reduced.

14. DO standardise your naming convention and use scripting/deployment tools!

As per the above points, name your network adaptors the same on every node – example:

  • MGMT-Team-NIC1, MGMT-Team-NIC2 –> MGMT-Team
  • LM-Team-NIC1, LM-Team-NI2 –> LM-Team
  • SMB-NIC1, SMB-NIC2
  • iSCSI-NIC1, iSCSI-NIC2

etc…

Again, when using PowerShell to build/configure it is easy to maintain convention and will save you countless config headaches over time.

Hyper-V – Don’t forget these simple administrative points! Part 1.

All,

Thought i’d put together a quick guide of often forgotten/overlooked configuration items, which may seem to be benign, but can pop up at exactly the wrong time and bite you in the proverbial. Most of these config items I have seen over the years and still run in to them from time to time!

1. DO configure all your Hyper-V hosts as CORE

There is absolutely no good reason to install GUI/Desktop Experience. If you are still not sure on your powershell commands, then use ‘SCONFIG’ to get an initial IP address in, firewall rules open for remote management, join the domain.

As the old adage goes – practice makes perfect. None of us started out knowing powershell off the top of our heads, so roll up your sleeves and give it a try. Once you have used it for a while it’ll become second nature.

2. DO run AV on your Hyper-V hosts

The exclusions are well known – published on technet for all to use – HERE  There is nothing worse these days, than making it easy for malware to spread. Be mindful of lateral movement in your organisation – think of malware intrusion not as an ‘IF’ but as a ‘WHEN’. Plan ahead and don’t make things easy for them.

3. DO turn on local firewalls

Again, group policy makes it easy for domain machines to have common rule sets applied to them. Don’t open the gates, for the sake of a few mins configuration work

4. DO plan your Hyper-V deployment ahead of time

Bad configuration is often the cause of bad planning. Even if you are only deploying a few hosts in a small cluster, how you approach this should be exactly the same as if you were planning for 32 hosts. Draw up your design according to the requirements. Even if the requirements are on the slim side, document your process and decision making.

Try to think of scale, if the company grows, would it be easy to add another few hosts later on?

5. DO connect a pair of 1gb onboard NICs – for dedicated managament use

You’ve invested in 10GbE – you have a pair of NICs and you are going to use them. There’s usually 2 x 1gb NICs onboard, doing nothing. Configure them as your OS main management network. Don’t put a vSwitch on top, keep it simple and easy to reach. If you do encounter any vLAN/vSwitch/vAnything issues down the line – you’ll be glad you can still manage your server – without the need for clunky iLO/DRAC sessions.

6. DO check your cluster metrics!

It’s pretty smart, it’ll auto assign the networks, but it’s not infallable. Get powershell open and check them HERE. It might be an old article – but it is still relevant. Make sure your CSV traffic is going where you want it to go, in the event of a redirect – you dont want it going over the wrong network.

Finally…..

7. DO make the most of your Microsoft Licences!

So many people overlook system center, believing it to be a set of expensive management tools. If you have datacenter licensing, software assurance and the right CALs then you are set to use the full suite of products, plus some of the MDOP goodies that have been written for you.

https://www.microsoft.com/en-gb/cloud-platform/system-center-pricing

I’m sure you are all familiar with the components now. Quite simply, Hyper-V without VMM is like toast without butter – dry and boring!

Take your time planning system center and the investment will pay dividends, alternatively, get a consultant in to do the leg work for you! 🙂 You can contact me on the site.

Part 2 here…

 

 

 

System Center 2016 UR3

It’s out now –

https://support.microsoft.com/en-hk/help/4020906/update-rollup-3-for-system-center-2016

A lot of good VMM fixes in there – which I will be testing soon. Bulk host agent update script is in Charbel’s blog here: https://charbelnemnom.com/2017/05/update-rollup-3-for-system-center-2016-is-now-available-sysctr-systemcenter-scvmm/

Details of SCOM fixes in Kevin’s blog here: http://kevingreeneitblog.blogspot.co.uk/2017/05/scom-2016-update-rollup-3-ur3-now.html

I’m a little disappointed to see DPM missed an update in UR3. VMware support is still missing from 2016 – but all will be forgiven if this turns up in UR4 along with fixes for woes experienced with UR2 currently:

Tape Library Sharing – 2012 OS cannot remove TL sharing & re-establishing 2016 OS TL required a manual DB cleanout (with Premier Support).

Console Crashing on PG alteration – requires DLL from MS (see my previous posts)

Mount points, whilst supported for the storage (see my other posts) uncover a known issue with DPM mouting the VHDX files for Modern backup Storage – the workaround for this is to add a drive letter to the storage.

If you don’t urgently need supported SQL 2016 backups / SharePoint 2016 protection from DPM, I would seriously consider sticking to UR1 for now.

Roll on UR4! 🙂

 

 

VMM 2016 UR 2 on Windows 2016 – Storage Management Bug

Recent installs of VMM 2016 have shown nice improvements over 2012, especially a much needed performance boost with storage operations through SMI-S.

Our latest SMI-S provider from NetApp (for ontap 9.1) in combination with VMM 2016 – seems to be light years ahead of 2012 R2. It’s responsive and carries out tasks in a tenth of the time that VMM 2012 took with ontap 8.

All issues with the SMI-S provider going unresponsive have been found to be due to a little service running in windows:

SMI-S

This little chap seems to randomly keel over – with system and application logs revealing nothing as to why.

  • Log Name: Application
    Source: Application Error
    Date: 28/04/2017 17:44:20
    Event ID: 1000
    Task Category: (100)
    Level: Error
    Keywords: Classic
    User: N/A
    Computer:
    Description:
    Faulting application name: svchost.exe_MSStrgSvc, version: 10.0.14393.0, time stamp: 0x57899b1c
    Faulting module name: concrete.dll, version: 10.0.14393.0, time stamp: 0x57899a8c
    Exception code: 0xc0000005
    Fault offset: 0x0000000000002eb0
    Faulting process ID: 0xe38
    Faulting application start time: 0x01d2c03e31b6c56d
    Faulting application path: C:\Windows\system32\svchost.exe
    Faulting module path: c:\windows\system32\concrete.dll
    Report ID: 1bc955fe-6a3b-469b-8c0c-de7992f3858d
    Faulting package full name:
    Faulting package-relative application ID:

I have logged it with premier support – they know about it, but a fix isn’t available as yet. Frustrating, but not the end of the world – just start the service again, wait 2 mins, then refresh the provider in SCVMM – all is good!

As soon as a fix is available – I will post an update here.

Updated:

  • WinCXE is planning of hotfix for Windows 10 version 1607. The KB article ID is going to be 4019472.

This should be available towards the end of May 2017.

SCVMM Cluster to Cluster Live Migrations – SMB Storage – NetApp SVM

In an ideal world we’d be using SoFS. But, as we have already thrown investment into NetApp, then we are going to use it – or – be thrown out the door.

There are many good articles about shared nothing migration, migrating from host to host – but not many cover those of us living with multiple clusters, or using other SMB providers.

Leading you here may be SCVMM error codes:

Error 12700 – 0x80072746, 0x80070005 – forcibly closed by the remote host, or General access denied error for your Run_As account to the SMB file path.

Our Example setup is this:

HyperVCluster1 – DomainA, multi node, storage is on SMB served up by a NetApp SVM dedicated for Hyper-V workloads.

HyperVCluster2 – DomainA, multi node, same storage access as cluster 1.

So – we already have the same file shares, assigned in SCVMM to both clusters, servers have all the access they require, as does the VMM run_as account that the clusters are using for file operations.

If you are reading this, then you already have looked at Kerberos delegation, started assigning every node CIFS & Microsoft virtual system migration service delegations to every other node – STOP NOW!

In SCVMM – make sure every cluster node is set to ‘Use Kerberos’ This prevents the tedious requirement of setting a web of host to host delegations.

Kerberos

Set this using the VMM PowerShell by running:

VMM-kerberos

Once done, set up a Kerberos delegation to the AD object name of your NetApp SVM SMB server.

Example file share: \\NetappSMB3\Tier1_01

Add a Kerberos delegation for CIFS (ugh – why does this term still linger everywhere) for DomainA\NetappSMB3 on all your hyper V cluster nodes that have access to SMB.

You can use the SCVMM & AD PowerShell modules to set this quickly using:

Delegation

As soon as this is in place, you will be free to live migrate running VMs between clusters that have access to the same file system.