Home Lab Power Automation – Part 3

In part 2, I shared the PowerCLI scripting I used to power on my entire lab environment in the correct order. In this final installment, I’ll take you through the scripting used to power everything down. Although you may think the process is just the reverse of what I covered in part 2, you’ll see there were some other things to consider and different approaches required.

Step 1 – Shutting Down Compute Cluster VMs

To begin the process, I’d need to shut down all VMs in the compute-a cluster. None of the VMs there are essential for running the lab, so they can be safely stopped at any time. I was able to do this by connecting to vCenter with PowerCLI and then using a ‘foreach’ loop to gracefully shut down any VMs in the ‘Powered On’ state.

"Connecting to vCenter Server ..." |timestamp
Connect-VIServer -Server 172.16.1.15 -User administrator@vsphere.local -Password "VMware9("

"Shutting down all VMs in compute-a ..." |timestamp
$vmlista = Get-VM -Location compute-a | where{$_.PowerState -eq 'PoweredOn'}
foreach ($vm in $vmlista)
    {
    Shutdown-VMGuest -VM $vm -Confirm:$false | Format-List -Property VM, State
    }

The above scripting ensures the VMs start shutting down, but it doesn’t tell me that they completed the process. After this is run, it’s likely that one or more VMs may still be online. Before I can proceed, I need to check that they’re all are in a ‘Powered Off’ state.

"Waiting for all VMs in compute-a to shut down ..." |timestamp
do
{
    "The following VM(s) are still powered on:"|timestamp
    $pendingvmsa = (Get-VM -Location compute-a | where{$_.PowerState -eq 'PoweredOn'})
    $pendingvmsa | Format-List -Property Name, PowerState
    sleep 1
} until($pendingvmsa -eq $null)
"All VMs in compute-a are powered off ..."|timestamp

A ‘do until’ loop does the trick here. I simply populate the list of all powered on VMs into the $pendingvmsa variable and print that list. After a one second delay, the loop continues until the $pendingvmsa variable is null. When it’s null, I know all of the VMs are powered off and I can safely continue.

Step 2 – Shutting Down Compute ESXi Hosts

Now that all of the VMs in compute-a are powered down, I can shut down the ESXi hosts.

"Shutting down all ESXi hosts in compute-a ..."|timestamp

Get-VMHost -Location compute-a | Stop-VMHost -Confirm:$false -Force

The above PowerCLI command retrieves a list of all ESXi hosts in the compute-a cluster and then shuts them down without requiring confirmation. I don’t need to wait for these hosts to shut down before proceeding as they’ll eventually get there and they are not holding anything else up.

After this, I disconnect PowerCLI from vCenter. The remaining PowerCLI tasks will be done directly to the management host.

"Disconnecting from VC ..."|timestamp
Disconnect-VIServer -Server * -Force -Confirm:$false

Step 3 – Shutting Down Management VMs

This next step will include the shutting down of all management VMs – including vCenter – so I’ll need to connect directly to the management ESXi host instead:

"Connecting to esx-m1 ..."|timestamp
Connect-VIServer -Server 172.16.1.20 -User root -Password "VMware1!"

Next, I’ll shut down all the VMs in the management cluster and get into a ‘do until’ loop to wait for them to finish. This is pretty much the same process I used for the compute-a cluster earlier.

"Shutting down all guests on esx-m1 ..."|timestamp
$vmlist0 = Get-VM | where{$_.PowerState -eq 'PoweredOn'}
foreach ($vm0 in $vmlist0)
    {
    Shutdown-VMGuest -VM $vm0 -Confirm:$false | Format-List -Property VM, State
    }
"Waiting for all guests to shut down ..."|timestamp
do
{
    "The following VM(s) are still powered on:"|timestamp
    $pendingvms0 = (Get-VM | where{$_.PowerState -eq 'PoweredOn'})
    $pendingvms0 | Format-List -Property Name, PowerState
    sleep 10
} until($pendingvms0 -eq $null)
"VMs are all powered off..."|timestamp

The vCenter Server appliance is the longest to gracefully shut down for some reason. In my lab, it consistently takes about 4-5 minutes.

Step 4 – Shutting Down the Management ESXi Host

After all the management VMs – including vCenter – are down, I can safely shut down the esx-m1 management host. To do this, I use the PowerCLI cmdlet Stop-VMHost, followed by a ‘do until’ loop to ensure it’s no longer accessible before I proceed. I talk more about the pinghost.py script I created in part 2.

"Shutting down esx-m1 ..."|timestamp
Stop-VMHost -Server 172.16.1.20 -Confirm:$false -Force

"Waiting for esx-m1 to power off ..."|timestamp
do
{
    $pingresult = python ~/scripts/pinghost.py 172.16.1.20 |timestamp
    $pingresult
    sleep 5
} until($pingresult -like '*not responding*')

The host will usually power down in about 30-60 seconds or so.

Step 5 – Shutting Down Shared Storage

I’ll admit that I took the easy way out in this step. I was originally planning to script the shutdown via SSH, but it turned out it wasn’t necessary. The FreeNAS BSD based OS is ACPI compliant and initiates a graceful shutdown of the system when it detects a ‘momentary press’ of the power button. I could replicate this momentary press using an IPMI command:

"Issuing a soft shutdown of freenas ..."|timestamp
ipmitool -I lanplus -H 172.16.1.67 -U root -P "VMware1!" power soft

"Waiting for freenas to power off ..."|timestamp
do
{
    $pingresult = python ~/scripts/pinghost.py 172.16.1.17 |timestamp
    $pingresult
    sleep 5
} until($pingresult -like '*not responding*')

Using the ‘soft’ option with ipmitool is key to making this work. In my testing, it’s been successful 100% of the time and achieves a proper shutdown of the system.

Step 6 – Shutting Off the Switch

And finally, I power down the hot and noisy Quanta LB6M. This is done with the same tplink_smartplug.py python script I discussed in part 1.

"Shutting off the outlet to the 10G switch ..."|timestamp
/home/pi/scripts/tplink-smartplug-master/tplink_smartplug.py -t 192.168.5.199 -c off

Issuing the ‘off’ command with -c sets the relay to ‘0’ and cuts power to the switch. I just have to remember to do a ‘write mem’ whenever making changes to the switch config so it’s not lost when I power it off.

I’m not sure if powering on and off an enterprise grade switch like this is good for it in the long term. These switches are generally intended to run 24/7. So far, in the six months or so I’ve been doing this, I haven’t had any issues. I suppose time will tell.

Conclusion

And there you have it! Below is the average amount of time it takes to power everything on and shut it down:

Power Up: ~15-18 minutes

Power Down: ~8-9 minutes

It was a great experience creating these scripts and I’ve already noticed a very positive change in my hydro bill as a result. I’m not sure how much I’ve saved, but my bills are considerably lower. It’s also a good feeling knowing that I’m able to reduce my carbon footprint yet still have access to an essential tool I use daily.

Have some suggestions on how I could improve these scripts? Please leave a comment below or reach out to me on Twitter (@vswitchzero)

Home Lab Power Automation Series:

Part 1 – The Tools
Part 2 – Powering Everything On
Part 3 – Shutting Everything Down