Missing NSX vdrPort and Auto Deploy

If you are running Auto Deploy and noticed your VMs didn’t have connectivity after a host reboot or upgrade, you may have run into the problem described in VMware KB 52903. I’ve seen this a few times now with different customers and thought a PSA may be in order. You can find all the key details in the KB, but I thought I’d add some extra context here to help anyone who may want more information.

I recently helped to author VMware KB 52903, which has just been made public. Essentially, it describes a race condition causing a host to come up without its vdrPort connected to the distributed switch. The vdrPort is an important component on an ESXi host that funnels traffic to/from the NSX DLR module. If this port isn’t connected, traffic can’t make it to the DLR for east/west routing on that host. Technically, VMs in the same logical switches will be able to communicate across hosts, but none of the VMs on this impacted host will be able to route.

The Problem

The race condition occurs when the DVS registration of the host occurs too late in the boot process. Normally, the distributed switch should be initialized and registered long before the vdrPort gets connected. In some situations, however, DVS registration can be late. Obviously, if the host isn’t yet initialized/registered with the distributed switch, any attempt to connect something to it will fail. And this is exactly what happens.

Using the log lines from KB 52903 as an example, we can see that the host attempts to add the vdrPort to the distributed switch at 23:44:19:

2018-02-08T23:44:19.431Z error netcpa[3FFEDA29700] [Originator@6876 sub=Default] Failed to add vdr port on dvs 96 ff 2c 50 0e 5d ed 4a-e0 15 b3 36 90 19 41 5d, Not found

The reason the operation fails is because the DVS switch with the UUID specified is not found from the perspective of this host. It simply hasn’t been initialized yet. A few moments later, the DVS is finally ready for use on the host. Notice the time stamps – you can see the registration of the DVS about 9 seconds later:

2018-02-08T23:44:28.389Z info hostd[4F540B70] [Originator@6876 sub=Hostsvc.DvsTracker] Registered Dvs 96 ff 2c 50 0e 5d ed 4a-e0 15 b3 36 90 19 41 5d

The above message can be found in /var/log/hostd.log.

Although technically this problem could occur to any ESXi host with a long delay in DVS registration, it seems to be much more likely to occur with auto-deploy stateless hosts. Because these hosts have many bootup activities relating to host profiles and the distributed switch, it seems that the conditions for this are more ideal.

In my experience, this possibility of hitting this issue can vary greatly. There are many factors, but the size of the environment as well as vCenter performance can play a role. One customer I’ve worked with seems to have about a one in five chance of running into this after a reboot. Another customer seems to hit it almost every time. And of course, there are many auto-deploy customers I’ve worked with who have never hit this issue.

Identifying an Impacted Host

Unfortunately, this issue isn’t easily idetifyable unless you know what you are looking for. Hosts that come up without a vdrPort do not report any kind of health check error in the UI. From the perspective of ‘host preparation’ as well as communication channel health, the host will look good.

To see if you’ve hit this problem, you’ll need to run net-vdr -C -l from the ESXi command line after the host has booted up or look for the logging examples mentioned earlier. Below is an example from my lab showing a missing vdrPort:

[root@esx-a1:~] net-vdr -C -l

Host locale Id: 4226cdee-1dda-9ff9-9e2a-8fdd64facd35

Connection Information:

DvsName VdrPort NumLifs VdrVmac
------- ------- ------- -------

You’ll also notice that the vdrPort isn’t listed when you run esxtop and view the ‘n’ screen.

The Workaround

Thankfully, the workaround to get a host in this condition up again is quite simple. All you need to do is restart the control plane agent on the host. Restarting netcpad will cause the host to check on its vdrPort and when it realizes it’s not there, will re-connect it. Because the host is on the DVS at this point, it’ll work 100% of the time. This can be done from the ESXi command line:

[root@esx-a1:~] /etc/init.d/netcpad restart
watchdog-netcpaMonitor: Terminating watchdog process with PID 19773228
netCP agent service monitor is stopped
watchdog-netcpa: Terminating watchdog process with PID 19773192
Memory reservation released for netcpa
netCP agent service is stopped
Memory reservation set for netcpa
Reload security domains
netCP agent service starts
netCP agent service monitor is started

You can then check on the vdrPort using the following net-vdr command:

[root@esx-a1:~] net-vdr -C -l

Host locale Id: 4226cdee-1dda-9ff9-9e2a-8fdd64facd35

Connection Information:
DvsName VdrPort NumLifs VdrVmac
------- ------- ------- -------
dvs-compute-a vdrPort 8 02:50:56:56:44:52
 Vdr Switch Port: 33554456
 Teaming Policy: Default Teaming
 Uplink : Uplink 1(33554436): 00:50:56:f9:75:f4(Team member)
 Uplink : Uplink 2(33554434): 00:50:56:f9:a8:33(Team member)

Stats : Pkt Dropped Pkt Replaced Pkt Skipped
 Input : 0 0 98039311
 Output : 71020 2429 68459936

Notice it’s listed above with a MAC address beginning with OUI 02:50:56 and a valid switch port number this time.

Final Words

Now obviously, the important thing here is to recognize that this problem has occurred before you move any production workload to the host. I.e. you need to check that the host’s vdrPort is connected before taking it out of maintenance mode.

Another thing you can do is create an alert or report in Log Insight to watch for failed vdrPort creation based on the log entries described earlier. The VMware engineering team is working on a proper fix for this issue that will be released in a future release of NSX, but until then it’s worth doing a quick check on rebooted hosts.

I hope this helps to provide some additional context to KB 52903. If there are any questions, please feel free to comment below or reach out to me on Twitter (@vswitchzero)


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s