FreeNAS 9.10 Lab Build – Part 1

Since the early iterations of my home lab, I’ve been using FreeNAS for shared storage. Admittedly, I use only a fraction of what this powerful FreeBSD based solution is cable of, but I’ve always found it to be very reliable and it has met my needs.

Over the years, I have used it in several capacities, including as a virtual machine, on consumer grade hardware and more recently as a virtual machine with PCI pass-through of SAS and network devices.

After the recent overhaul of my home lab, I decided that it would be a good time to build a ‘semi-proper’ FreeNAS box that feels more like a permanent fixture. My goal was to bring something online that was:

• Inexpensive.
• Relatively small.
• Using proper server-grade hardware with error correcting (ECC) memory.
• Enough PCI-E expandability for a SAS card as well as a 10Gb NIC or Fiberchannel HBA in the future.
• Somewhat power-efficient and quiet.
• Preferably a Xeon based system and not an Atom or Avaton.

Some of the points above very rarely go hand-in-hand; like “inexpensive” and “Xeon” or “Proper server-grade” and “quiet”. This likely meant I’d be required to do a custom build from scratch.

The SAS Card – Dell PERC H200

Many would argue that a good storage build is focused around a good storage controller card. I had originally wanted to buy an LSI 9211-8i or an IBM M1015 card, but I came across a great deal on a Dell PERC H200 on eBay. It cost me only $25 CDN, and was an as-is, untested part so I took a chance on it. To my pleasant surprise, the card worked just fine.

Although this is a Dell branded part, the PERC H200 is essentially just a SAS 2008 based LSI 9211-8i adapter. The most notable difference is the location of the SFF-8087 ports on the top of the card, instead of the end.

The PERC H200 may be functional with the Dell firmware, but one of the key requirements for FreeNAS is to have a non-RAID HBA that allows FreeNAS direct access to the attached drives. Most RAID adapters will build a logical ‘container’ or ‘virtual disk’ of sorts that is exposed to the OS, and will not allow direct access to the disks. These cards usually mask S.M.A.R.T data as well that FreeNAS would use to determine disk health. The bottom line is that the caching, redundancy and features provided by Hardware RAID will simply conflict with the caching, optimization, redundancy and features provided by ZFS. This is actually listed as the first point in the FreeNAS worst practices guide, and I’d recommend reading that if you’d like to learn more.

So that said, I needed to find a way to get my Dell PERC H200 RAID adapter to behave like an LSI 9211-8i in IT (initiator-target) mode. Thankfully there are plenty of great guides on the subject – maybe not as many as there are for the more popular IBM M1015 cards – but they are definitely out there. The best one I found was the updated guide on the techmattr site.

Have a look through the guide for detailed steps. I found it easy to follow and there were links to all of the required software. From a high level, the process was as follows:

1. Create a bootable USB DOS disk with the required software on it, including the sas2flash tool.
2. Get the current SAS address of the card and write it down.
3. To begin, an empty BIOS is written to the card and then the card’s flash is wiped. I’m not clear if this is a requirement but I did it anyway and it was successful.
4. I then flashed the ‘HBA’ version of the Dell H200 firmware named 6GBPSAS.fw and rebooted.
5. Next, the P20 version of the LSI 9211-8i IT mode firmware is flashed onto the card and the server is rebooted again.
6. Finally, I wrote the appropriate SAS address back onto the card and it was fully functional as an LSI 9211-8i in IT mode.

For completeness sake, I did a few extra steps that were not required. The steps above did not include a card BIOS that loads at startup, nor does it include the updated UEFI boot ROM that can also be flashed to the card.

To include the boot ROM, it’s just a simple matter of including the -b switch during the flash process, so to flash only the IT firmware, you’d run:

sas2flsh.exe -o -f 2118it.bin

If you want to flash the P20 IT firmware as well as the updated boot ROM back onto the card, you’d use:

sas2flash.efi -o -f 2118it.bin -b mptsas2.rom

It’s not necessary to flash the BIOS back onto the card as it’ll function just fine as an HBA without one. In fact, your boot times will improve quite a bit as you won’t have to wait for the normal initialization to occur.

Although there isn’t much that can be configured in the boot ROM of a SAS HBA compared to a RAID card, I do like to be able to look at the SAS topology and ensure the drives are detected outside of FreeBSD or any other operating system on the box.

The Cables

Unlike the usual onboard AHCI SATA, SAS cards usually include one of several port types. In the case of the Dell PERC H200 – as well as the LSI 9211-8i – there are two SFF-8087 ports.

There are indeed SFF-8087 to SFF-8087 cables for connections to server back-planes and drive cages etc but as I’ll be using regular SATA drives, i’ll need a special SFF-8087 to SATA breakout cable. Each SFF-8087 connector on the card includes four SAS ‘lanes’ or channels that can be broken out to individual devices.

I wound up buying a couple of the cheapest 3ft breakout cables from Newegg that I could find. I think they were about $12 each. They looked well made, but to be honest, I wish I spent a few extra dollars for the regular plastic insulated type as opposed to these partially sleeved ones. I found the sleeved portion of the cable too stiff and the individual exposed cables too thin and delicate feeling. But despite this, they did work just fine.

So with a great 6Gbps SAS card ready to go, I’ll continue my hunt for some appropriate hardware. In the mean time, stay tuned for Part 2 where I’ll test out the card in a PCI passthrough configuration.

FreeNAS 9.10 Lab Build Series:

Part 1 – Defining the requirements and flashing the Dell PERC H200 SAS card.
Part 2 – FreeNAS and VMware PCI passthrough testing.
Part 3 – Cooling the toasty Dell PERC H200.
Part 4 – A close look at the Dell PowerEdge T110.
Part 5 – Completing the hardware build.