Tag Archives: NSX Manager

Check NSX 6.2.x Compatibility Before Upgrading to 6.3.5!

Unlike previous 6.3.x releases, 6.3.5 has some new upgrade minimum version compatibility requirements. This is not only true from a vSphere perspective, but also for the version of NSX 6.2.x you are running. If you are running an older 6.2.0, 6.2.1 or 6.2.2 release of NSX, you’ll need to upgrade to at least 6.2.4 before taking the big step up to 6.3.5. VMware has just updated the NSX Upgrade Matrix to reflect this requirement:

622upg635-1

Screenshot taken from the VMware Interoperability Matrix site.

I expect that VMware will update the 6.3.5 release notes and release a new KB article very shortly. I’ll provide some more detail when that is out. In the meantime, please be sure to heed the version requirements or you will most likely run into problems.

Thankfully there aren’t too many customers still running these old releases of 6.2.x, but if you have already attempted the upgrade and hit problems, you’ll need to roll back. If you took a cold-snapshot of the manager or a clone, you can roll back that way. Otherwise, you’ll need to deploy the original 6.2.x OVA again and restore your FTP backup.

** Edit 11/29/2017: VMware has just updated the NSX 6.3.5 release notes to include mention of the minimum version requirements. The following statement was added:

Important: If you are upgrading NSX 6.2.0, 6.2.1, or 6.2.2 to NSX 6.3.5, you must complete a workaround before starting the upgrade. See VMware Knowledge Base article 000051624 for details.

VMware calls it a “workaround” but it’s basically just upgrading to an interim version before going to 6.3.5. In KB 000051624, VMware recommends going to 6.2.9 as that workflow has been tested. I.e. upgrading from 6.2.0 to 6.2.9, and then to 6.3.5. On a positive note, you only need to upgrade your NSX Manager to 6.2.9, no other components need to be upgraded before proceeding on to 6.3.5.

If you attempt an upgrade from 6.2.2 or older releases, my understanding is that the upgrade will appear to be completed successfully, but your configuration will be missing. VMware calls out the remediation steps of rolling back to the previous version should you run into this issue.

NSX Engineering Mode ‘root shell’ Access Now Available to Customers

In an interesting move, VMware has released public KB 2149630 on September 29th, providing information on how to access the root shell of the NSX Manager appliance.

If you’ve been on an NSX support call with VMware dealing with a complex issue, you may have seen your support engineer drop into a special shell called ‘Engineering Mode’. This is sometimes also referred to as ‘Tech Support Mode’. Regardless of the name used, this is basically a root bash shell on the underlying Linux based appliance. From here, system configuration files and scripts as well as most normal Linux functions can be accessed.

Normally, when you open a console or SSH session to NSX manager, you are dropped into a restricted ‘admin’ shell with a hierarchical system of commands like Cisco’s IOS. For the majority of what an administrator needs to do, this is sufficient. It’s only in more complex cases – especially when dealing with issues in the Postgres DB – or issues with the underling OS that this may be required.

There are several important statements and disclaimers that VMware makes in this KB article that I want to outline below:

“Important: Do not make any changes to the underlying system without the help of VMware Technical Support. All such changes are not supported and as a result, your system may no longer be supportable by GSS.”

In NSX 6.3.2 and later, you’ll also be greeted by the following disclaimer:

“Engineering Mode: The authorized NSX Manager system administrator is requesting a shell which is able to perform lower level unix commands/diagnostics and make changes to the appliance. VMware asks that you do so only in conjunction with a support call to prevent breaking your virtual infrastructure. Please enter the shell diagnostics string before proceeding.Type Exit to return to the NSX shell. Type y to continue:”

And finally, you’ll want to ensure you have a full backup of NSX Manager should anything need to be modified:

VMware recommends to take full backup of the system before performing any changes after logging into the Tech Support Mode.

Although it is very useful to take a ‘read only’ view at some things in the root shell, making any changes is not supported without getting direct assistance from VMware support.

A few people have asked whether or not making the root shell password public is a security issue, but the important point to remember is that you cannot even get to a position where you can enter the shell unless you are already logged in as an NSX enterprise administrator level account. For example, the built-in ‘admin’ account. For anyone concerned about this, VMware does allow the root password to be changed. It’s just critical that this password not be lost in case VMware support requires access to the root shell for troubleshooting purposes. More information on this can be found in KB 2149630.

To be honest, I’m a bit torn on this development. As someone who does backline support, I know what kind of damage that can be done from the root shell – even with the best intentions. But at the same time, I see this as empowering. It gives customers additional tools to troubleshoot and it also provides some transparency into how NSX Manager works rather than shielding it behind a restricted shell. I think that overall, the benefits outweigh the risks and this was a positive move for VMware.

When I think back to VI 3.5 and vSphere 4.0 when ESXi was shiny and new, VMware initially took a similar stance. You had to go so far as to type ‘UNSUPPORTED’ into the console to access a shell. Today, everyone has unrestricted root access to the hypervisor. The same holds true for the vCenter appliance – the potential for destruction is no different.

I’d welcome any comments or thoughts. Please share them below!

Completely Removing NSX

Admittedly, removing NSX from an environment was not my first choice of topics to cover, but I have found that the process is often misunderstood and done improperly. NSX isn’t just a few virtual machine appliances that can be deleted – there are hooks into numerous vCenter objects, your ESXi hosts and vCenter Server itself. To save yourself from some grief and a lot of manual cleanup, the removal must be done properly.

Thankfully, VMware does provide some high level instructions to follow in the public documentation. You’ll find these public docs for NSX 6.2.x and 6.3.x respectively here and here.

There are many reasons that someone may wish to remove NSX from a vSphere environment – maybe you’ve installed an evaluation copy to run a proof of concept or just want to start fresh again in your lab environment.  In my case I need to completely remove NSX 6.2.5 and install an older version of NSX for some version-specific testing in my home lab.

From a high level, the process should look something like this:

  1. Remove all VMs from Logical Switches.
  2. Remove NSX Edges and Distributed Logical Routers.
  3. Remove all Logical Switches.
  4. Uninstall NSX from all ESXi hosts in prepared clusters.
  5. Delete any Transport Zones.
  6. Delete the NSX Manager and NSX Controller appliances.
  7. Remove the NSX Manager hooks into vCenter, including the plugin/extension.
  8. Cleaning up the vSphere Web Client leftovers on the vCenter Server.

Continue reading