Welcome to the first installment of a new series of NSX-T troubleshooting scenarios. Thanks to everyone who took the time to comment on the first half of the scenario. Today I’ll be performing some troubleshooting and will show how I came to the solution.
Please see the first half for more detail on the problem symptoms and some scoping.
As we saw in the first half, the installation of the NSX-T VIBs were failing with the following error:
At first glance, it looked as if the NSX-T VIBs, or an older version of them were already installed. Taking a closer look at the actual VIB names, however, was very telling. The ‘esx-nsxv’ in the name denotes that these belong to NSX for vSphere.
Logging in to host esx-a3 via SSH and checking for installed VIBs with ‘nsx’ in the name came back with the following:
[root@esx-a3:~] esxcli software vib list |grep nsx esx-nsxv 6.5.0-0.0.8590012 VMware VMwareCertified 2018-08-31
Indeed, the NSX-V VIBs are still installed. Having a look at the environment, we saw that all other traces of NSX-V were gone – the manager, controllers, vmkernel ports, portgroups and Web Client plugin were missing. Only these lingering VIBs were not removed from these three hosts for some reason. It’s important to properly remove NSX to prevent issues like this from occurring.
Removing the NSX-V VIBs
The first order of business was to put the host in maintenance mode. I didn’t have any running VMs created yet, so I just went ahead and put all three in maintenance mode:
Once that was done, I could remove the VIBs using the following esxcli software vib command: