Cisco nenic Driver Issue During NSX Upgrades

The nenic driver versions prior to 1.0.11.0 may cause an outage during NSX upgrades.

If you are planning an NSX upgrade in a Cisco UCS environment, pay close attention to your ‘nenic’ driver version before you begin. The nenic driver is the new native driver replacement for the older vmklinux enic driver. It’s used exclusively for the Cisco VIC adapters found in UCS systems and is now the default in vSphere 6.5 and 6.7.

We’ve seen several instances now where Cisco VIC adapters can go link-down in an error state during NSX VIB upgrades. It doesn’t appear to matter what version of NSX is being upgraded from/to, but the common denominator is an older nenic driver version. This seems to be reproducible with nenic driver version 1.0.0.2 and possibly others. Version 1.0.11.0 and later appear to correct this problem. At the time of writing, 1.0.26.0 is the latest version available.

You can obtain your current nenic driver and firmware version using the following command:

# esxcli network nic get -n vmnicX

Before you upgrade your drivers, be sure to reach out to Cisco to ensure your firmware is also at the recommended release version. Quite often vendors have a recommended driver/firmware combination for maximum stability and performance.

I expect a KB article and an update to the NSX release notes to be made public soon but wanted to ensure this information got out there as soon as possible.

NSX 6.4.0 Upgrade Compatibility

Thinking about upgrading to NSX 6.4.0? As I discussed in my recent Ten Tips for a Successful NSX Upgrade post, it’s always a good idea to do your research before upgrading. Along with reading the release notes, checking the VMware compatibility Matrix is essential.

VMware just updated some of the compatibility matrices to include information about 6.4.0. Here are the relevant Links:

From an NSX upgrade path perspective, you’ll be happy to learn that any current build of NSX 6.2.x or 6.3.x should be fine. At the time of writing, this would be 6.2.9 and earlier as well as 6.3.5 and earlier.

640upg-0
NSX upgrade compatibility – screenshot from 1/17/2018.

On a positive note, VMware required a workaround to be done for some older 6.2.x builds to go to 6.3.5, but this is no longer required for 6.4.0. The underling issue that required this has been resolved.

From a vCenter and ESXi 6.0 and 6.5 perspective, the requirements for NSX 6.4.0 remain largely unchanged from late 6.3.x releases. What you’ll immediately notice is that NSX 6.4.0 is not supported with vSphere 5.5. If you are running vSphere 5.5, you’ll need to get to at least 6.0 U2 before considering NSX 6.4.0.

From the NSX 6.4.0 release notes:

Supported: 6.0 Update 2, 6.0 Update 3
Recommended: 6.0 Update 3. vSphere 6.0 Update 3 resolves the issue of duplicate VTEPs in ESXi hosts after rebooting vCenter server. See VMware Knowledge Base article 2144605 for more information.

Supported: 6.5a, 6.5 Update 1
Recommended: 6.5 Update 1. vSphere 6.5 Update 1 resolves the issue of EAM failing with OutOfMemory. See VMware Knowledge Base Article 2135378 for more information.

Note: vSphere 5.5 is not supported with NSX 6.4.0.

It doesn’t appear that the matrix has been updated yet for other VMware products that interact with NSX, such as vCloud Director.

Before rushing out to upgrade to NSX 6.4.0, be sure to check for compatibility – especially if you are using any third party products. It may be some time before other vendors certify their products for 6.4.0.

Stay tuned for a closer look at some of the new NSX 6.4.0 features!

 

Ten Tips for a Successful NSX Upgrade

As you’ve probably noticed, VMware is regularly releasing new NSX versions and updates to introduce new features and to improve stability and scalability. Eventually, you’ll find yourself in a situation where you’ll either want or need to upgrade. Maybe you want to take advantage of some new features, encountered a problem or your version isn’t supported any more. Whatever the reason, and whatever the version, here are ten tips that will help to ensure your upgrade is successful!

Tip 1 – Check The Compatibility Matrix

Before getting started, you’ll want to thoroughly check the compatibility of your target NSX version. That doesn’t just mean checking if you can upgrade from version X to version Y, but rather to check everything that interacts with NSX in the environment.

Start with the NSX Upgrade Path found at the VMware Interoperability Matrices page. There you may be surprised to find that there are several versions of NSX that are not a feasible upgrade path. For example, you can’t upgrade from NSX 6.2.8 to 6.3.2, nor can you upgrade from 6.2.6 to 6.3.1.

nsxupgtips-1
Taken from the NSX Upgrade Path on VMware’s website.

Once you’ve confirmed that your target version is supported in the upgrade path, you’ll want to look at the Interoperability Matrix to ensure products like vSphere and Cloud Director are compatible. Again, there are several incompatible releases that you may not expect. For example, NSX 6.3.3 and later releases aren’t compatible with vCenter Server 6.0 U1 and older, but are compatible with all releases of 5.5. Another example is the initial release of vSphere 6.5. Only 6.5a or later can be used with any version of NSX 6.3.x.

Continue reading “Ten Tips for a Successful NSX Upgrade”

Check NSX 6.2.x Compatibility Before Upgrading to 6.3.5!

Unlike previous 6.3.x releases, 6.3.5 has some new upgrade minimum version compatibility requirements. This is not only true from a vSphere perspective, but also for the version of NSX 6.2.x you are running. If you are running an older 6.2.0, 6.2.1 or 6.2.2 release of NSX, you’ll need to upgrade to at least 6.2.4 before taking the big step up to 6.3.5. VMware has just updated the NSX Upgrade Matrix to reflect this requirement:

622upg635-1
Screenshot taken from the VMware Interoperability Matrix site.

I expect that VMware will update the 6.3.5 release notes and release a new KB article very shortly. I’ll provide some more detail when that is out. In the meantime, please be sure to heed the version requirements or you will most likely run into problems.

Thankfully there aren’t too many customers still running these old releases of 6.2.x, but if you have already attempted the upgrade and hit problems, you’ll need to roll back. If you took a cold-snapshot of the manager or a clone, you can roll back that way. Otherwise, you’ll need to deploy the original 6.2.x OVA again and restore your FTP backup.

** Edit 11/29/2017: VMware has just updated the NSX 6.3.5 release notes to include mention of the minimum version requirements. The following statement was added:

Important: If you are upgrading NSX 6.2.0, 6.2.1, or 6.2.2 to NSX 6.3.5, you must complete a workaround before starting the upgrade. See VMware Knowledge Base article 000051624 for details.

VMware calls it a “workaround” but it’s basically just upgrading to an interim version before going to 6.3.5. In KB 000051624, VMware recommends going to 6.2.9 as that workflow has been tested. I.e. upgrading from 6.2.0 to 6.2.9, and then to 6.3.5. On a positive note, you only need to upgrade your NSX Manager to 6.2.9, no other components need to be upgraded before proceeding on to 6.3.5.

If you attempt an upgrade from 6.2.2 or older releases, my understanding is that the upgrade will appear to be completed successfully, but your configuration will be missing. VMware calls out the remediation steps of rolling back to the previous version should you run into this issue.

VUM Challenges During vCenter 6.5 Upgrade

After procrastinating for a while, I finally started the upgrade process in my home lab to go from vSphere 6.0 to 6.5. The PSC upgrade was smooth, but I hit a roadblock when I started the upgrade process on the vCenter Server appliance.

After going through some of the first steps in the process, I ran into the following error when trying to connect to the source appliance.

vumupgrade-1

The exact text of the error reads:

“Unable to retrieve the migration assistant extension on source vCenter Server. Make sure migration assistant is running on the VUM server.”

I had forgotten that I even had Update Manager deployed. Because my lab is small, I generally applied updates manually to my hosts via the CLI. What I do remember, however, is being frustrated that I had to deploy a full-scale Windows VM to run the Update Manager service.

Continue reading “VUM Challenges During vCenter 6.5 Upgrade”