Removing All Universal Objects using PowerNSX and PowerShell Scripting

A requirement for a transit NSX manager appliance before it can assume the standalone role.

Cross-vCenter NSX is an awesome feature. Introduced back with NSX 6.2, it breaks down barriers and allows the spanning of logical networks well beyond what was possible before. Adding and synchronizing additional secondary NSX managers and their associated vCenter Servers is a relatively simple process and generally just works. But what about moving a secondary back to a standalone manager? Not quite so simple, unfortunately.

The process should be straight forward – disconnect all the VMs from the universal logical switches, make the secondary manager a standalone and then go through the documented removal process.

From a high level, that’s correct, but you’ll be stopped quickly in your tracks by the issue I outlined in NSX Troubleshooting Scenario 7. Before a secondary NSX manager can become a standalone, all universal objects must be removed from it.

uniremove-1

Now, assuming Cross-VC NSX will still be used and there will be other secondary managers that still exist after this one is removed, we don’t want to completely remove all universal objects as they’ll still be used by the primary and other secondaries.

In this situation, the process to get a secondary NSX manager back to a standalone would look something like this:

  1. Remove all VMs from universal logical switches at the secondary location.
  2. Use the ‘Remove Secondary Manager’ option from the ‘Installation and Upgrade’ section in the vSphere Client. This will change the secondary’s role to ‘Transit’ and effectively stops all universal synchronization with the primary.
  3. Remove all universal objects from the new ‘Transit’ NSX manager. These universal objects were originally all created on the primary manager and synchronized to this one while it was a secondary.
  4. Once all universal objects have been removed from the ‘Transit’ manager, it’s role can be changed to ‘Standalone’.

Much of the confusion surrounding this process revolves around a transitional NSX manager role type aptly named ‘Transit’. When a manager assumes the ‘Transit’ role, it is effectively disconnected from the primary and all universal synchronization to it stops. Even though it won’t synchronize, all the universal objects are preserved. This is done because Cross-VC NSX is designed to allow any of the secondary NSX managers to assume the primary role if necessary.

uniremove-2

Continue reading “Removing All Universal Objects using PowerNSX and PowerShell Scripting”

Home Lab Power Automation – Part 3

In part 2, I shared the PowerCLI scripting I used to power on my entire lab environment in the correct order. In this final installment, I’ll take you through the scripting used to power everything down. Although you may think the process is just the reverse of what I covered in part 2, you’ll see there were some other things to consider and different approaches required.

Step 1 – Shutting Down Compute Cluster VMs

To begin the process, I’d need to shut down all VMs in the compute-a cluster. None of the VMs there are essential for running the lab, so they can be safely stopped at any time. I was able to do this by connecting to vCenter with PowerCLI and then using a ‘foreach’ loop to gracefully shut down any VMs in the ‘Powered On’ state.

"Connecting to vCenter Server ..." |timestamp
Connect-VIServer -Server 172.16.1.15 -User administrator@vsphere.local -Password "VMware9("

"Shutting down all VMs in compute-a ..." |timestamp
$vmlista = Get-VM -Location compute-a | where{$_.PowerState -eq 'PoweredOn'}
foreach ($vm in $vmlista)
    {
    Shutdown-VMGuest -VM $vm -Confirm:$false | Format-List -Property VM, State
    }

The above scripting ensures the VMs start shutting down, but it doesn’t tell me that they completed the process. After this is run, it’s likely that one or more VMs may still be online. Before I can proceed, I need to check that they’re all are in a ‘Powered Off’ state.

"Waiting for all VMs in compute-a to shut down ..." |timestamp
do
{
    "The following VM(s) are still powered on:"|timestamp
    $pendingvmsa = (Get-VM -Location compute-a | where{$_.PowerState -eq 'PoweredOn'})
    $pendingvmsa | Format-List -Property Name, PowerState
    sleep 1
} until($pendingvmsa -eq $null)
"All VMs in compute-a are powered off ..."|timestamp

A ‘do until’ loop does the trick here. I simply populate the list of all powered on VMs into the $pendingvmsa variable and print that list. After a one second delay, the loop continues until the $pendingvmsa variable is null. When it’s null, I know all of the VMs are powered off and I can safely continue.

Continue reading “Home Lab Power Automation – Part 3”

Home Lab Power Automation – Part 2

In part 1, I shared some of the tools I’d use to execute the power on and shutdown tasks in my lab. Today, let’s have a look at my startup PowerCLI script.

A Test-Connection Cmdlet Replacement

As I started working on the scripts, I needed a way to determine if hosts and devices were accessible on the network. Unfortunately, the Test-Connection cmdlet was not available in the Linux PowerShell core release. It uses the Windows network stack to do its thing, so it may be a while before an equivalent gets ported to Linux. As an alternative, I created a simple python script that achieves the same overall result called pinghost.py. You can find more detail on how it works in a post I did a few months back here.

The script is very straightforward. You specify up to three space separated IP addresses or host names as command line arguments, and the script will send one ICMP echo request to each of the hosts. Depending on the response, it will output either ‘is responding’ or ‘is not responding’. Below is an example:

pi@raspberrypi:~/scripts $ python pinghost.py vc.lab.local 172.16.10.67 172.16.10.20
vc.lab.local is not responding
172.16.10.67 is responding
172.16.10.20 is not responding

Then using this script, I could create sleep loops in PowerShell to wait for one or more devices to become responsive before proceeding.

Adding Timestamps to Script Output

As I created the scripts, I wanted to record the date/time of each event and output displayed. In a sense, I wanted it to look like a log that could be written to a file and referred to later if needed. To do this, I found a simple PowerShell filter that could be piped to each command I ran:

#PowerShell filter to add date/timestamps
filter timestamp {"$(Get-Date -Format G): $_"}

Step 1 – Power On the Switch

Powering up the switch requires the use of the tplink_smartplug.py python script that I discussed in part 1. The general idea here is to instruct the smart plug to set its relay to a state of ‘1’. This brings the switch to life. I then get into a ‘do sleep’ loop in PowerCLI until the Raspberry Pi is able to ping the management interface of the switch. More specifically, it will wait until the pinghost.py script returns a string of “is responding”. If that string isn’t received, it’ll wait two seconds, and then try again.

"Powering up the 10G switch ..." |timestamp
/home/pi/scripts/tplink-smartplug-master/tplink_smartplug.py -t 192.168.1.199 -c on |timestamp

"Waiting for 10G switch to boot ..." |timestamp
do
{
$pingresult = python ~/scripts/pinghost.py 172.16.1.1 |timestamp
$pingresult
sleep 2
} until($pingresult -like '*is responding*')

When run, the output looks similar to the following:

Continue reading “Home Lab Power Automation – Part 2”

Home Lab Power Automation – Part 1

My home lab has grown substantially over the last few years and with it, so did power consumption, heat and noise. I selected power efficient parts where possible, but even with 500-600W power usage, 24/7 operation adds up. I can certainly notice the extra cost on my hydro bill, but it’s not just the financial impact – it’s also the environmental impact I’m concerned about.

The bottom line is that I have no reason to run the lab 24/7. I generally use it an hour or two each day – sometimes more, sometimes less. But there are also stretches where I won’t use it for several days like on weekends or when I’m busy with other things.

I found myself manually shutting down the lab when I knew I wouldn’t be using it and then manually powering everything back up. As you can imagine, this was quite a process. Everything had to be powered on or shut down in a very specific order to avoid problems. I’d also need to be standing in front of the equipment for part of this process as some equipment didn’t have remote power-on capability. Because of the work and time involved, I’d inevitably just leave it powered on for much longer stretches than I needed to.

It wasn’t until I added a 10Gbps Quanta LB6M switch to the lab that I realized I needed to do something. It’s not a quiet or energy efficient switch, consuming an average of 120W at idle.

Continue reading “Home Lab Power Automation – Part 1”

Creating a Python Replacement for the Test-Connection cmdlet.

I’ve been working on a PowerShell script to automatically power up and shut down all devices in my home lab. Both in the interest of saving on hydro and to reduce it’s environmental impact. To do this, I’ve been using a low-power Raspberry Pi 3 B+, which will stay powered on to orchestrate these and other scripted activities.

One great PowerShell cmdlet that comes in handy is Test-Connection. Although it can be used for numerous network testing purposes, it’s most commonly used to send an ICMP echo request to a host to see if it responds or not. The cmdlet responds with a simple true or false response, which makes it very handy for scripting purposes. I was really hoping to use this cmdlet because it makes it easy to determine if a device is still online or not and if it’s okay to move on to the next phase of powering things on or shutting things down. For example, I don’t want to power on my management ESXi host until the freenas SAN is online.

The problem I ran into was that not all PowerShell cmdlets have been ported over to PowerShell Core for Linux. Test-Connection is unfortunately one of them due to the way it uses the Microsoft Windows network stack to function. Looking at the GitHub PR, it seems they are getting close to porting it over, but I decided to try my hand at creating a python script with similar basic functionality.

I should note that I’m not a programmer by any stretch of the imagination and it’s been many years since I’ve done any serious coding. This is actually the first bit of python code I’ve written, so I’m sure there are many more efficient ways to achieve what I’ve done here:

import os
import sys

# If no arguments parsed, display the greeting:
if len(sys.argv) == 1:
    print("vswitchzero.com ICMP response script. Specify up to 3 hosts to ping separated by spaces.")
    print("example: python ./pinghost.py 172.16.10.15 192.168.1.1 vc.lab.local")

# If one arg parsed, ping a single host
if len(sys.argv) == 2:
    host1 = sys.argv[1]
    response = os.system("ping -c 1 " + host1 + "> /dev/null")
    if response == 0:
        print host1, 'is responding'
    else:
        print host1, 'is not responding'

# If two args parsed, ping two hosts
if len(sys.argv) == 3:
    host1 = sys.argv[1]
    host2 = sys.argv[2]
    response1 = os.system("ping -c 1 " + host1 + "> /dev/null")
    if response1 == 0:
        print host1, 'is responding'
    else:
        print host1, 'is not responding'
    response2 = os.system("ping -c 1 " + host2 + "> /dev/null")
    if response2 == 0:
        print host2, 'is responding'
    else:
        print host2, 'is not responding'

# If three args parsed, ping three hosts
if len(sys.argv) == 4:
    host1 = sys.argv[1]
    host2 = sys.argv[2]
    host3 = sys.argv[3]
    response1 = os.system("ping -c 1 " + host1 + "> /dev/null")
    if response1 == 0:
        print host1, 'is responding'
    else:
        print host1, 'is not responding'
    response2 = os.system("ping -c 1 " + host2 + "> /dev/null")
    if response2 == 0:
        print host2, 'is responding'
    else:
        print host2, 'is not responding'
 response3 = os.system("ping -c 1 " + host3 + "> /dev/null")
    if response3 == 0:
        print host3, 'is responding'
    else:
        print host3, 'is not responding'

# If more than three args parsed, display an error
if len(sys.argv) > 4:
    print('vswitchzero.com ICMP response script. Specify up to 3 hosts to ping separated by spaces.')
    print('example: python ./pinghost.py 172.16.10.15 192.168.1.1 vc.lab.local')
    print(' ')
    print('ERROR: Too many arguments specified. Use 1-3 IPs or hostnames only. Each should be separated by a space')

Continue reading “Creating a Python Replacement for the Test-Connection cmdlet.”