Cross-vCenter NSX is an awesome feature. Introduced back with NSX 6.2, it breaks down barriers and allows the spanning of logical networks well beyond what was possible before. Adding and synchronizing additional secondary NSX managers and their associated vCenter Servers is a relatively simple process and generally just works. But what about moving a secondary back to a standalone manager? Not quite so simple, unfortunately.
The process should be straight forward – disconnect all the VMs from the universal logical switches, make the secondary manager a standalone and then go through the documented removal process.
From a high level, that’s correct, but you’ll be stopped quickly in your tracks by the issue I outlined in NSX Troubleshooting Scenario 7. Before a secondary NSX manager can become a standalone, all universal objects must be removed from it.
Now, assuming Cross-VC NSX will still be used and there will be other secondary managers that still exist after this one is removed, we don’t want to completely remove all universal objects as they’ll still be used by the primary and other secondaries.
In this situation, the process to get a secondary NSX manager back to a standalone would look something like this:
- Remove all VMs from universal logical switches at the secondary location.
- Use the ‘Remove Secondary Manager’ option from the ‘Installation and Upgrade’ section in the vSphere Client. This will change the secondary’s role to ‘Transit’ and effectively stops all universal synchronization with the primary.
- Remove all universal objects from the new ‘Transit’ NSX manager. These universal objects were originally all created on the primary manager and synchronized to this one while it was a secondary.
- Once all universal objects have been removed from the ‘Transit’ manager, it’s role can be changed to ‘Standalone’.
Much of the confusion surrounding this process revolves around a transitional NSX manager role type aptly named ‘Transit’. When a manager assumes the ‘Transit’ role, it is effectively disconnected from the primary and all universal synchronization to it stops. Even though it won’t synchronize, all the universal objects are preserved. This is done because Cross-VC NSX is designed to allow any of the secondary NSX managers to assume the primary role if necessary.
Admittedly, NSX should be smart enough to just nuke all the universal objects when making a transit manager a standalone manager, but this is unfortunately not the case yet. I’ll be pushing to ensure this gets changed in a future release, but for now, we must clean these up manually.
If you just had a handful of objects, this would not be a very daunting task. As you can see in the solution to NSX Troubleshooting Scenario 7, it is quite manageable in small scale deployments. But what if you had hundreds of universal logical switches, universal firewall rules, universal DLRs, IP sets and security groups? Deleting these from the NSX vSphere Client UI would not be a reasonable ask.
Automating the Process with PowerNSX
I’ve run into this scenario several times now in large customer deployments and decided to create a script to automate this process. Thankfully, PowerShell scripting and PowerNSX – and the NSX REST APIs it utilizes – can make quick work of this process.
You can download the script here.
The script does the following:
- Displays a warning message about taking backups prior to executing the script.
- As a safety precaution, it checks to ensure you’re connected to a ‘Transit’ manager before proceeding. The script will end if you are connected to a primary/secondary or standalone manager.
- Runs several ‘Get’ cmdlets to obtain a total count of all universal objects in the environment.
- Asks you if you want to proceed to remove all objects.
- Removes all universal objects of all types and in the correct order. The script will report how many objects were removed along the way.
- When finished, it will display a new count of all objects for comparison purposes.
Here are the contents of the script:
# Universal Object Removal Script (removealluniversal.ps1) v1.1
# https://vswitchzero.com
# This script will remove all universal objects from an NSX manager. To be used only with Transit role managers.
# First, display a warning and ensure user is connected to the correct manager.
cls
echo " "
Write-Host -foregroundcolor yellow "This script is designed to remove all universal objects from an NSX manager in the 'Transit' role so that it can become a 'Standalone' manager. Please ensure that you are connected to the correct 'Transit' NSX Manager and associated vCenter Server in PowerCLI/PowerNSX before proceeding. Please do not proceed unless you have taken a backup of both the Primary and Transit NSX managers. No warranty or guarantees are provided with this script. Although it has been tested to function correctly, use it at your own risk."
echo " "
Write-Host -NoNewLine "Are you sure you have taken a backup and wish to proceed? [y/n]:"
$YNresponse = read-host
if ( $YNresponse -ne "y" ) {exit}
# Checking to ensure the connected NSX manager is in the Transit role
echo " "
echo "Checking to ensure the connected NSX Manager is in the 'Transit' role..."
Get-NsxManagerRole
If ((Get-NsxManagerRole).role -eq "TRANSIT") {
echo " "
echo "Confirmed role is 'Transit'. Proceeding..."}
Else {
echo " "
echo "The currently connected NSX Manager is not in the 'Transit' role. Are you sure you are connected to the correct NSX Manager?"
exit
}
# Getting counts of all universal objects in the environment:
echo " "
echo "Getting counts of all Universal Objects associated with this NSX Manager. This may take a minute..."
$countTZ = Get-NsxTransportZone | where{$_.isUniversal -eq "true"}|measure
$countSID = Get-NsxSegmentIdRange |where{$_.isUniversal -eq "true"}|measure
$countUDLR = Get-NsxLogicalRouter |where{$_.isUniversal -eq "true"}|measure
$countULS = Get-NsxLogicalSwitch | where{$_.isUniversal -eq "true"}|measure
$countFWSEC = Get-NsxFirewallSection | where{$_.managedBy -eq "universalroot-0"} |measure
$countFWSVG = Get-NsxServiceGroup -scopeid universalroot-0 |measure
$countFWSV = Get-NsxService -UniversalOnly |measure
$countFWSG = Get-NsxSecurityGroup -scopeid universalroot-0 |measure
$countIPS = Get-NsxIpSet -scopeid universalroot-0 |measure
$countUST = Get-NsxSecurityTag | where{$_.isUniversal -eq "true"}|measure
echo "Done."
echo " "
echo "Total count of all Universal Object types:"
echo " "
# Displays counts of all object types:
echo "Universal Transport Zones: $($countTZ.count)"
echo "Universal Segment ID Ranges: $($countSID.count)"
echo "Universal Logical Routers: $($countUDLR.count)"
echo "Universal Logical Switches: $($countULS.count)"
echo "Universal Firewall Sections: $($countFWSEC.count)"
echo "Universal Firewall Service Groups: $($countFWSVG.count)"
echo "Universal Firewall Services: $($countFWSV.count)"
echo "Universal Firewall Security Groups: $($countFWSG.count)"
echo "Universal IP Sets: $($countIPS.count)"
echo "Universal Security Tags: $($countUST.count)"
# Confirm before proceeding
echo " "
Write-Host -NoNewLine "Remove all universal objects from this NSX Manager? [y/n]:"
$YNresponse = read-host
if ( $YNresponse -ne "y" ) {exit}
# Start removing all of the object types and report the number successfully removed.
echo "Removing all universal objects. This may take several minutes..."
echo " "
echo "Removing Universal Logical Routers..."
Get-NsxLogicalRouter |where{$_.isUniversal -eq "true"}| Remove-NsxLogicalRouter -Confirm:$false
$afterUDLR = Get-NsxLogicalRouter |where{$_.isUniversal -eq "true"}|measure
$removedUDLR = $countUDLR.count - $afterUDLR.count
echo "$($removedUDLR) object(s) successfully removed."
echo "Removing Universal Logical Switches..."
Get-NsxLogicalSwitch | where{$_.isUniversal -eq "true"}| Remove-NsxLogicalSwitch -Confirm:$false
$afterULS = Get-NsxLogicalSwitch | where{$_.isUniversal -eq "true"}|measure
$removedULS = $countULS.count - $afterULS.count
echo "$($removedULS) object(s) successfully removed."
echo "Removing Universal Firewall Sections..."
Get-NsxFirewallSection | ? {$_.managedBy -eq "universalroot-0"} | Remove-NsxFirewallSection -confirm:$false -force
$afterFWSEC = Get-NsxFirewallSection | where{$_.managedBy -eq "universalroot-0"} |measure
$removedFWSEC = $countFWSEC.count - $afterFWSEC.count
echo "$($removedFWSEC) object(s) successfully removed."
echo "Removing Universal Firewall Service Groups..."
Get-NsxServiceGroup -scopeid universalroot-0 | Remove-nsxservicegroup -confirm:$false -ErrorAction SilentlyContinue
$afterFWSVG = Get-NsxServiceGroup -scopeid universalroot-0 |measure
$removedFWSVG = $countFWSVG.count - $afterFWSVG.count
echo "$($removedFWSVG) object(s) successfully removed."
echo "$($afterFWSVG.count) object(s) not removed. These are likely read-only service groups and can be ignored."
echo "Removing Universal Firewall Services..."
Get-NsxService -UniversalOnly | Remove-NsxService -confirm:$false -ErrorAction SilentlyContinue
$afterFWSV = Get-NsxService -UniversalOnly |measure
$removedFWSV = $countFWSV.count - $afterFWSV.count
echo "$($removedFWSV) object(s) successfully removed."
echo "$($afterFWSV.count) object(s) not removed. These are likely read-only service groups and can be ignored."
echo "Removing Universal Firewall Security Groups..."
Get-NsxSecurityGroup -scopeid universalroot-0 | Remove-NsxSecurityGroup -confirm:$false
$afterFWSG = Get-NsxSecurityGroup -scopeid universalroot-0 |measure
$removedFWSG = $countFWSG.count - $afterFWSG.count
echo "$($removedFWSG) object(s) successfully removed."
echo "Removing Universal IP Sets..."
Get-NsxIpSet -scopeid universalroot-0 | Remove-NsxIpSet -confirm:$false
$afterIPS = Get-NsxIpSet -scopeid universalroot-0 |measure
$removedIPS = $countIPS.count - $afterIPS.count
echo "$($removedIPS) object(s) successfully removed."
echo "Removing Universal Security Tags..."
Get-NsxSecurityTag | where{$_.isUniversal -eq "true"}|Remove-NsxSecurityTag -Confirm:$false
$afterUST = Get-NsxSecurityTag | where{$_.isUniversal -eq "true"}|measure
$removedUST = $countUST.count - $afterUST.count
echo "$($removedUST) object(s) successfully removed."
echo "Removing Universal Transport Zones..."
Get-NsxTransportZone | where{$_.isUniversal -eq "true"}| Remove-NsxTransportZone -Confirm:$false
$afterTZ = Get-NsxTransportZone | where{$_.isUniversal -eq "true"}|measure
$removedTZ = $countTZ.count - $afterTZ.count
echo "$($removedTZ) object(s) successfully removed."
echo "Removing Universal Segment ID Ranges..."
Get-NsxSegmentIdRange |where{$_.isUniversal -eq "true"}|Remove-NsxSegmentIdRange -Confirm:$false
$afterSID = Get-NsxSegmentIdRange |where{$_.isUniversal -eq "true"}|measure
$removedSID = $countSID.count - $afterSID.count
echo "$($removedSID) object(s) successfully removed."
#Finish up
echo " "
echo "Finished!"
echo " "
echo "Total object counts after removal attempt:"
echo " "
echo "Universal Transport Zones: $($afterTZ.count)"
echo "Universal Segment ID Ranges: $($afterSID.count)"
echo "Universal Logical Routers: $($afterUDLR.count)"
echo "Universal Logical Switches: $($afterULS.count)"
echo "Universal Firewall Sections: $($afterFWSEC.count)"
echo "Universal Firewall Service Groups: $($afterFWSVG.count)"
echo "Universal Firewall Services: $($afterFWSV.count)"
echo "Universal Firewall Security Groups: $($afterFWSG.count)"
echo "Universal IP Sets: $($afterIPS.count)"
echo "Universal Security Tags: $($afterUST.count)"
echo " "
echo "Note: It is expected to see non-zero counts for both Universal Service Groups and Services as there are internal read-only objects that can't be removed. Only user-created objects will be removed using this script. If all other counts are zero, you can now attempt to change the NSX Manager's role from Transit to Standalone."
Below is an example run of the script in my lab environment. I have a transit NSX manager (10.61.0.40) ready for cleanup.
First, I must connect to the Transit NSX manager, and it’s associated vCenter Server. PowerNSX now recommends using the -vCenterServer parameter to automatically connect to the right NSX Manager, so I’ve been using that method:
PS C:\scratch> Connect-NsxServer -vCenterServer vc2.vswitchzero.net Version : 6.4.1 BuildNumber : 8599035 Credential : System.Management.Automation.PSCredential Server : 10.61.0.40 Port : 443 Protocol : https UriPrefix : ValidateCertificate : False VIConnection : vc2.vswitchzero.net DebugLogging : False DebugLogfile : C:\Users\MIKE~1.VSW\AppData\Local\Temp\PowerNSXLog-administrator@vsphere.local@-2018_11_16_11_01_00.log
I can now launch the script. I’m first greeted with a warning/disclaimer. Please do take the time to read it and ensure you have taken backups:
This script is designed to remove all universal objects from an NSX manager in the 'Transit' role so that it can become a 'Standalone' manager. Please ensure that you are connected to the correct 'Transit' NSX Manager and associated vCenter Server in PowerCLI/PowerNSX before proceeding. Please do not proceed unless you have taken a backup of both the Primary and Transit NSX managers. No warranty or guarantees are provided with this script. Although it has been tested to function correctly, use it at your own risk. Are you sure you have taken a backup and wish to proceed? [y/n]:
In my case, it confirmed that I’m connected to a ‘Transit’ manager and allows me to proceed. As you can see, there are a number of universal objects of each type on this manager:
Checking to ensure the connected NSX Manager is in the 'Transit' role... role ---- TRANSIT Confirmed role is 'Transit'. Proceeding... Getting counts of all Universal Objects associated with this NSX Manager. This may take a minute... Done. Total count of all Universal Object types: Universal Transport Zones: 1 Universal Segment ID Ranges: 1 Universal Logical Routers: 1 Universal Logical Switches: 3 Universal Firewall Sections: 1 Universal Firewall Service Groups: 42 Universal Firewall Services: 499 Universal Firewall Security Groups: 2 Universal IP Sets: 2 Universal Security Tags: 1 Remove all universal objects from this NSX Manager? [y/n]:
After answering ‘y’ the script will remove each object type in the correct order:
A banner at the top will display some output as objects are removed. Depending on how many objects exist, this process could take some time. It is normal to see some vCenter Server tasks get kicked off, including the removal of portgroups as the universal switches are removed.
You’ll notice that many universal services and service groups will not be removed. This is because the script parses through read-only built-in objects that can’t be deleted. These can be safely ignored and will not prevent the manager from becoming a standalone.
Once finished, you’ll get an updated tally of remaining objects. If the script ran correctly, you should hopefully see only a number of services and service groups. Everything else should be zero:
Finished! Total object counts after removal attempt: Universal Transport Zones: 0 Universal Segment ID Ranges: 0 Universal Logical Routers: 0 Universal Logical Switches: 0 Universal Firewall Sections: 0 Universal Firewall Service Groups: 41 Universal Firewall Services: 496 Universal Firewall Security Groups: 0 Universal IP Sets: 0 Universal Security Tags: 0 Note: It is expected to see non-zero counts for both Universal Service Groups and Services as there are internal read-only objects that cannot be removed. Only user-created objects will be removed using this script. If all other counts are zero, you can now attempt to change the NSX Manager's role from Transit to Standalone.
Conclusion
PowerNSX is a powerful tool and can make very quick work out of daunting tasks. I’m hoping that built-in functionality will render this script unnecessary in the future, but until then, I hope it can help others in the community.
Have any questions? Please feel free to leave a comment below or reach out to me on Twitter (@vswitchzero).
vswitchzero 