FreeNAS 9.10 Lab Build – Part 3

In Part 2 of my FreeNAS 9.10 build series, I installed my newly flashed Dell PERC H200 LSI 9211-8i card into my primary management host to try out FreeNAS as a VM with PCI passthrough.

After opening the case to do some rewiring I was totally shocked at how hot the little aluminum heatsink on the card was. This thing wasn’t just hot – it was scalding hot. Hot enough to invoke the subconscious and instinctive reaction pull your finger away from the heatsink. If I had to guess, I’d say that heatsink was at least 90’C or hotter.

Although it was surprising, I had to remind myself that this adapter is not a consumer grade card and does have an airflow requirement for it to run reliably. Dell’s H200 user guide says very little about the cooling requirements, unfortunately, but some of the Dell PowerEdge servers that support the H200 classify it as a ‘25W TDP’ card. That may not sound like a lot of thermal output, but when you consider most of that is concentrated on an area the size of a dime, it’s a lot to dissipate.

In most rackmount type cases, a minimum amount of front-to-back airflow is usually directed across all of the PCI-Express slots, but in my large Phanteks Enthoo Pro, there happens to be very little in this area with the default fan configuration.

After perusing around online, I could see that this was not an uncommon problem among SAS 2008 based adapters – including the popular IBM M1015 – and that in some circumstances the card could even overheat under heavy load. Considering how hot mine was while sitting idle, I can only imagine.

Even though I’ll only be using this card in a lab, I do want to ensure it runs reliably and lives a long life. It seemed that there were a few possible solutions I could pursue:

  1. Add a small 40mm fan to the heatsink.
  2. Replace the tiny heatsink with something larger.
  3. Find a way to increase the airflow in this area of the case.

The third option appealed to me most – mainly because I hate small fans. They usually have an annoying ‘whine’ to them as they need to spin at a higher RPM and they can be pretty unreliable. I’d also expect the width of the fan to block the adjoining PCI-Express slot as well.

So after taking a look through my spare parts, I came across an old Noctua 92mm PWM fan collecting dust. Although I think Noctua’s marketing is a bit over the top, I have been using their fans for many years and can attest to their high quality and quiet operation.

After MacGyver’ing a couple of thumbscrews and metal brackets I could get the fan into the perfect position. Also, because it’s a PWM modulated fan, it spins down with the rest of the system fans and is pretty much inaudible at under 1000RPM.

Even though there feels like there is barely any airflow coming from the Noctua NF-B9 fan at reduced RPM, it’s enough to dissipate the hot air from around the heatsink fins and the heatsink is now only warm to the touch! It really did make a huge difference.

Problem solved. Hopefully whatever case I ultimately use for my FreeNAS build will not have these sorts of airflow dead spots, but at least there could be a simple solution.

FreeNAS 9.10 Lab Build Series:

Part 1 – Defining the requirements and flashing the Dell PERC H200 SAS card.
Part 2 – FreeNAS and VMware PCI passthrough testing.
Part 3 – Cooling the toasty Dell PERC H200.
Part 4 – A close look at the Dell PowerEdge T110.
Part 5 – Completing the hardware build.

Beacon Probing Deep Dive

Today I’ll be looking at a feature I’ve wanted to examine for some time – Beacon Probing. I hope to take a fresh look at this often misunderstood feature, explore the pros, cons, quirks and take a bit of a technical deep-dive into its inner workings.

According to the vSphere Networking Guide, we see that Beacon Probing is one of two available NIC failure detection mechanisms. Whenever we’re dealing with a team of two or more NICs, ESXi must be able to tell when a network link is no longer functional so that it can fail-over all VMs or kernel ports to the remaining NICs in the team.

Beacon Probing

Beacon probing takes network failure detection to the next level. As you’ve probably already guessed, it does not rely on NIC link-state to detect a failure. Let’s have a look at the definition of Beacon Probing in the vSphere 6.0 Network guide on page 92:

“[Beacon Probing] sends out and listens for beacon probes on all NICs in the team and uses this information, in addition to link status, to determine link failure.”

This statement sums up the feature very succinctly, but obviously there is a lot more going on behind the scenes. How do these beacons work? How often are they sent out? Are they broadcast or unicast frames? What do they look like? How do they work when multiple VLANs are trunked across a single link? What are the potential problems when using beacon probing?

Today, we’re going to answer these questions and hopefully give you a much better look at how beacon probing actually works.

Continue reading

FreeNAS 9.10 Lab Build – Part 2

In Part 1 of this series, I discussed building a proper FreeNAS server and prepared a Dell PERC H200 by flashing it to an LSI 9211-8i in IT mode. But while I was looking around for suitable hardware for the build, I decided to try something that I’ve wanted to do for a long time – PCI passthrough.

This would give me an opportunity to tinker with vt-d passthrough and put my freshly flashed Dell PERC H200 through it’s paces.

Why Not VMDK Disks?

As mentioned in Part 1 of this series, FreeNAS makes use of ZFS, which is much more than just a filesystem. It combines the functionality of a logical volume manager and an advanced filesystem providing a whole slew of features including redundancy and data integrity. For it to do this effectively – and safely – ZFS needs direct access to SATA or SAS drives. We want ZFS to manage all aspects of the drives and the storage pool and should remove all layers of abstraction between the FreeNAS OS and the drives themselves.

As you probably know, FreeNAS works well enough as a virtual machine for lab purposes. After all, ignoring what’s in between, ones and zeros still make it to from FreeNAS to the disks. That said, using a virtual SCSI adapter and VMDK disks certainly does not qualify as ‘direct access’. In fact, the data path would be packed with layers of abstraction and would look something like this:

Physical Disk > < SATA/SAS HBA > < ESXi Hypervisor HBA driver > < VMFS 5 Filesystem > < VMDK virtual disk > < Virtual SCSI Adapter > < FreeNAS SCSI driver > < FreeNAS/FreeBSD

In contrast, a physical FreeNAS server would look more like:

Physical Disk > < SATA/SAS HBA > < FreeNAS HBA driver > < FreeNAS/FreeBSD


Continue reading

FreeNAS 9.10 Lab Build – Part 1

Since the early iterations of my home lab, I’ve been using FreeNAS for shared storage. Admittedly, I use only a fraction of what this powerful FreeBSD based solution is cable of, but I’ve always found it to be very reliable and it has met my needs.

Over the years, I have used it in several capacities, including as a virtual machine, on consumer grade hardware and more recently as a virtual machine with PCI pass-through of SAS and network devices.

After the recent overhaul of my home lab, I decided that it would be a good time to build a ‘semi-proper’ FreeNAS box that feels more like a permanent fixture. My goal was to bring something online that was:

  • Inexpensive.
  • Relatively small.
  • Using proper server-grade hardware with error correcting (ECC) memory.
  • Enough PCI-E expandability for a SAS card as well as a 10Gb NIC or Fiberchannel HBA in the future.
  • Somewhat power-efficient and quiet.
  • Preferably a Xeon based system and not an Atom or Avaton.

Some of the points above very rarely go hand-in-hand; like “inexpensive” and “Xeon” or “Proper server-grade” and “quiet”. This likely meant I’d be required to do a custom build from scratch.

The SAS Card – Dell PERC H200

Many would argue that a good storage build is focused around a good storage controller card. I had originally wanted to buy an LSI 9211-8i or an IBM M1015 card, but I came across a great deal on a Dell PERC H200 on eBay. It cost me only $25 CDN, and was an as-is, untested part so I took a chance on it. To my pleasant surprise, the card worked just fine.


Although this is a Dell branded part, the PERC H200 is essentially just a SAS 2008 based LSI 9211-8i adapter. The most notable difference is the location of the SFF-8087 ports on the top of the card, instead of the end.

Continue reading

Completely Removing NSX

Admittedly, removing NSX from an environment was not my first choice of topics to cover, but I have found that the process is often misunderstood and done improperly. NSX isn’t just a few virtual machine appliances that can be deleted – there are hooks into numerous vCenter objects, your ESXi hosts and vCenter Server itself. To save yourself from some grief and a lot of manual cleanup, the removal must be done properly.

Thankfully, VMware does provide some high level instructions to follow in the public documentation. You’ll find these public docs for NSX 6.2.x and 6.3.x respectively here and here.

There are many reasons that someone may wish to remove NSX from a vSphere environment – maybe you’ve installed an evaluation copy to run a proof of concept or just want to start fresh again in your lab environment.  In my case I need to completely remove NSX 6.2.5 and install an older version of NSX for some version-specific testing in my home lab.

From a high level, the process should look something like this:

  1. Remove all VMs from Logical Switches.
  2. Remove NSX Edges and Distributed Logical Routers.
  3. Remove all Logical Switches.
  4. Uninstall NSX from all ESXi hosts in prepared clusters.
  5. Delete any Transport Zones.
  6. Delete the NSX Manager and NSX Controller appliances.
  7. Remove the NSX Manager hooks into vCenter, including the plugin/extension.
  8. Cleaning up the vSphere Web Client leftovers on the vCenter Server.

Continue reading

Suppressing ESXi Shell and SSH Warnings

Are you tired of seeing SSH and Shell warnings on your ESXi hosts? If you are at all like me, it’s maddening to see yellow warnings and banners on hosts in the vCenter Server inventory – especially when it’s for something as simple as the ESXi Shell and SSH service being enabled.

Granted, what’s a minor annoyance in a lab environment might be a warning that’s taken seriously in a locked down production environment. In these sorts of environments, administrators will need to enable/disable SSH and Shell access on an as-needed basis. Without the alarms and banners, services may be left turned on accidentally.


ESXi Shell and SSH warning banners

Nobody likes warning banners on the summary page 😦

If you are using vSphere 6.0 or later, there is a nifty new ‘Suppress Warning’ option in the vSphere Web client. It can be found on the summary page of an ESXi host with an ESXi Shell or SSH warning currently triggered.

As you can see in the above screenshot, there are separate alerts for both the ESXi Shell and for SSH as well as an option to ‘Suppress Warning’ on each. Although it may appear that each can be suppressed independently, clicking one of the ‘Suppress Warning’ links will disable both ESXi Shell and SSH warnings on the host.

Continue reading