USB Passthrough and vMotion

I was recently speaking with someone about power management in a home lab environment. Their plan was to use USB passthrough to connect a UPS to a virtual machine in a vSphere cluster. From there, they could use PowerCLI scripting to gracefully power off the environment if the UPS battery got too low. This sounded like a wise plan.

Their concern was that the VM would need to be pinned to the host where the USB cable was connected and that vMotion would not be possible. To their pleasant surprise, I told them that support for vMotion of VMs with USB passthrough had been added at some point in the past and it was no longer a limitation.

When I started looking more into this feature, however, I discovered that this was not a new addition at all. In fact, this has been supported ever since USB passthrough was introduced in vSphere 4 over seven years ago. Have a look at the vSphere Administration Guide for vSphere 4 on page 105 for more information.

I had done some work with remote serial devices in the past, but I’ve never been in a situation where I needed to vMotion a VM with a USB device attached. It’s time to finally take this functionality for a test drive.

Testing it Out

I’ll talk more about how this works in the next section, but I basically just connected a 16GB USB stick to host esx-a1 where a VM called win-a1 resides.

I then added a new ‘Host USB Device’ to the VM. Notice that I checked off the ‘vMotion support’ option, which is required for this to work.

Once that was done, I couldn’t access the USB stick from within the guest for some reason, but a quick reboot resolved that issue and I was ready to proceed.

Next, I migrated the VM from host esx-a1 to another host in the cluster called esx-a2. As you can see, there were no compatibility warnings and the migration went smoothly.

And voila, even after moving to host esx-a2, the virtual machine can still access the USB device connected to host esx-a1. Pretty nifty!

How it Works

So how does a virtual machine communicate with a USB device if it’s not physically connected to the host it’s on? Think USB over TCP/IP. It is not exactly a ratified standard per-say, but that’s exactly what VMware is doing behind the scenes.

As you can see above, a virtual machine on host esx-a2 can still access a passed through USB device connected to host esx-a1. It does this by allowing host esx-a2 to relay all of it’s USB commands to the services on host esx-a1. This communication channel is maintained between the two ESXi hosts over TCP port 902. As you are probably aware, TCP 902 is the primary management service communication port used by ESXi.

As you can see below, a TCP connection is maintained by the ‘remoteDeviceConnect’ service on the ESXi host:

[root@esx-a1:/tmp] esxcli network ip connection list |grep 902
tcp 0 78 172.16.10.21:51628 172.16.10.22:902 ESTABLISHED 22891592 newreno remoteDeviceConnect
tcp 0 0 172.16.10.21:902 172.16.10.100:50119 ESTABLISHED 66460 newreno busybox
tcp 0 0 0.0.0.0:902 0.0.0.0:0 LISTEN 66460 newreno busybox

I also did a quick packet capture to see if there was anything interesting to be found. I was curious to see if there was anything human readable transmitted, so I created a 1MB text file and transferred it off the USB stick on the win-a1 VM. I also navigated around the folder structure from Windows Explorer.

Interestingly, the transfer of a 1MB text document resulted in only about 46KB of data and nothing human readable in the captures. This tells me that any data over this channel is compressed. I didn’t see anything resembling SSL or any other encryption standard, however.

Limitations

As with any passthrough feature, there will be limitations. Thankfully, vMotion isn’t one of them. You can find a full list of limitations in the vSphere docs.

Here are a few notable ones:

1. The VM with the USB device configured for vMotion cannot be suspended or powered off.
2. When a suspended or powered off machine is turned back on, the USB device can only be reconnected from the host where it’s physically connected.
3. DPM is not supported because it may power off the host that’s physically connected to the USB device.
4. ESXi hosts must be able to communicate with each other using TCP port 902 on the network used for management (vmk0).

Performance Impact

Obviously, having to relay USB commands over the network will have some cost associated with it. From a bandwidth perspective, USB 2.0 with it’s 480Mbps maximum speed shouldn’t have any issue over gigabit ethernet. If you are using XHCI and USB 3.0, however, you’ll need 10Gbps networking to be able to move the 5Gbps theoretical maximum. Of course, this is only if you are looking for maximum performance – it’ll work just fine on slower network links as well.

Assuming you are not network bandwidth limited, there can be some performance impact due to extra latency as well. The total delay will be the sum of all latency along the path to the remote host. This includes any latency on the USB bus, the network latency between hosts as well as any additional processing work needed to make this feature work.

To test this out, I decided to run a couple of storage performance tests on the connected USB stick. I used ATTO disk benchmark with a 1024KB transfer size. I repeated each test three times and then got the average for reads and writes. I first ran it on host esx-a1, and then again on esx-a2 to see the impact:

Keep in mind that this is a very cheap USB 2.0 mass storage device, so I wasn’t expecting good numbers to begin with. All I was looking to illustrate here is that there is a small performance penalty when relaying USB commands over the network. Even with a USB stick that can’t do better than 5MB/s, we see about 5-10% performance penalty. I’d guess that this is likely due to the extra latency.

Again, I doubt very many enterprises would use USB based storage in this manner. And if so, it’s unlikely that a 5% performance penalty would be of any concern.

Conclusion

I was happy to finally get a chance to play around with this cool – albeit old – vSphere feature. Being able to migrate around VMs with USB devices attached can have some enterprise use cases. The first that comes to mind would be a legacy application that requires a USB license dongle. Enabling the dongle for vMotion would allow DRS to migrate the VM around in the cluster without losing access to it. Another would be a UPS management cable as I mentioned earlier.

Do you use USB passthrough in a production environment? I’d be curious to hear of any other use cases. Please feel free to comment below, or reach out to me on Twitter (@vswitchzero).