Category Archives: FreeNAS

Using FreeNAS for NSX FTP Backups

FreeNAS is a very powerful storage solution and is quite popular with those running vSphere and NSX home labs. I recently built a new FreeNAS 9.10 system and wanted to share some of my experiences getting NSX FTP backups going.

To get this configured, I found the FTP section of the FreeNAS 9.10 documentation to be very useful. I’d definitely recommend giving it a read through as well.

Before Getting Started

Before enabling the FTP service in FreeNAS, you’ll want to decide where to put your NSX backups. In theory, you can dump them in any of your volumes or datasets but you may want to set aside a specific amount of storage space for them. To do this in my lab, I created a dedicated dataset with a 60GB quota for FTP purposes. I like to separate it out to ensure nothing else competes with the backups and the amount of space available is predictable.

FreeNASNSXbackups-1

If you plan to use FTP for more than just NSX, it would be a good idea to create a subdirectory in the dataset or other location you want them to reside. In my case, I created a directory called ‘NSX’ in the dataset:

[root@freenas] ~# cd /mnt/vol1/dataset-ftp
[root@freenas] /mnt/vol1/dataset-ftp# mkdir NSX
[root@freenas] /mnt/vol1/dataset-ftp# ls -lha
total 2
drwxr-xr-x 3 root wheel 3B Sep 7 09:23 ./
drwxr-xr-x 5 root wheel 5B Sep 5 16:13 ../
drwxr-xr-x 2 root wheel 2B Sep 7 09:23 NSX/
[root@freenas] /mnt/vol1/dataset-ftp#

Setting Permissions

One step that is often missed during FreeNAS FTP configuration is to set the appropriate permissions. The proftpd service in FreeNAS uses the built in ftp user account. If that user does not have the appropriate permissions to the location you intend to use, backups will not write successfully.

Since I used a dedicated dataset for FTP called dataset-ftp, I can easily set permissions recursively for this location from the UI:

FreeNASNSXbackups-2

As shown above, we want to set both the owner user and group to ftp. Because I created the NSX directory within the dataset, I’ll be setting permission recursively as well.

If I log into FreeNAS via SSH or console again, I can confirm that this worked because the dataset-ftp mount is now owned by ftp as is the NSX subdirectory within.

[root@freenas] /mnt/vol1# ls -lha
total 14
drwxrwxr-x  5 root  wheel     5B Sep  5 16:13 ./
drwxr-xr-x  4 root  wheel   192B Sep  5 16:09 ../
drwxr-xr-x  3 ftp   ftp       3B Sep  7 09:23 dataset-ftp/
drwxrwxr-x  5 root  wheel    13B Jul 29 16:23 dataset-static/
drwxrwxr-x  2 root  wheel     2B Sep  5 16:13 dataset-tftp/
[root@freenas] /mnt/vol1# ls -lha dataset-ftp
total 2
drwxr-xr-x  3 ftp   ftp       3B Sep  7 09:23 ./
drwxrwxr-x  5 root  wheel     5B Sep  5 16:13 ../
drwxr-xr-x  2 ftp   ftp       2B Sep  7 13:46 NSX/

The Easy Option – Anonymous FTP Access

Setting up anonymous FTP access requires the least amount of effort and is usually sufficient for home lab purposes. I would strongly discourage the use of anonymous access in a production or security sensitive environment as anyone on the network can access the FTP directory configured.

First, configure FTP under services in the FreeNAS UI:

FreeNASNSXbackups-4

As you’d obviously expect, the ‘Allow Anonymous Login’ option needs to be checked off in order for anonymous FTP to work. The ‘Allow Local Users Login’ option should be unchecked if you don’t want to use authentication. It’s also important to select the ‘Path’ to the FTP root directory you wish to use. In my example above, any anonymous FTP logins will go directly into the NSX subdirectory I created earlier.

If you want to use FTP for more than just NSX backups, you can make the path the root of the dataset and NSX can be configured to use a specific subdirectory within as I’ll show later.

Once that’s done, you can enable the FTP service. It’ll be off by default:

FreeNASNSXbackups-5

Now we can do some basic tests to ensure FTP is functional. You can use an FTP client like FileZilla if you like, but I’m just going to use the good old Windows FTP command line utility. First, let’s make sure we can login anonymously:

C:\Users\mike.LAB\Desktop>ftp freenas.lab.local
Connected to freenas.lab.local.
220 ProFTPD 1.3.5a Server (freenas.lab.local FTP Server) [::ffff:172.16.10.17]
User (freenas.lab.local:(none)): anonymous
331 Anonymous login ok, send your complete email address as your password
Password:
230 Anonymous access granted, restrictions apply

A return status of 230 is what we’re looking for here and this seems to work fine. Keep in mind that it technically doesn’t matter what password you enter for the anonymous username. You can just hit enter, but I usually just re-enter the username. It’s not necessary to enter anything that resembles an email address.

Next, let’s make sure we have permission to write to this location. I’ll do an FTP ‘PUT’ of a small text file:

ftp> bin
200 Type set to I
ftp> put C:\Users\mike.LAB\Desktop\test.txt
200 PORT command successful
150 Opening BINARY mode data connection for test.txt
226 Transfer complete
ftp: 14 bytes sent in 0.00Seconds 14000.00Kbytes/sec.
ftp> dir
200 PORT command successful
150 Opening ASCII mode data connection for file list
-rw-r----- 1 ftp ftp 14 Sep 7 13:56 test.txt
226 Transfer complete
ftp: 65 bytes received in 0.01Seconds 10.83Kbytes/sec.
ftp>

As seen above, the file was written successfully with a 226 return code. The last step I’d recommend doing before configuring NSX is to confirm the relative path after login from the FTP server’s perspective. Because I stayed in the FTP root directory, it simply lists a forward slash as shown below:

ftp> pwd
257 "/" is the current directory
ftp>

NSX expects this path as you’ll see shortly. Now that we know anonymous FTP is working, we can configure the FTP server from the NSX appliance UI:

FreeNASNSXbackups-3

As you can see above, I’ve entered ‘anonymous’ as the user name, and entered the same as the password string. The backup directory is the location you want NSX to write backups to. If you had a specific directory you wanted to use within the FTP root directory that was configured, you could enter it here. For example, /backups. As mentioned earlier, my FTP root directory is the NSX directory so it’s not necessary in my case.

Two other pieces of information are mandatory in NSX – the filename prefix and the pass phrase. The filename prefix is just that – a string that is appended to the beginning of the filename. It usually makes sense to identify the environment or NSX manager by name here. This is especially important if you have multiple NSX managers all backing up to one location. The pass phrase is a password used to encrypt the backup binary file generated. Be sure not to lose this or you will not be able to restore your backups.

After hitting OK, we can then do a quick backup to ensure it can connect and write to the location configured.

FreeNASNSXbackups-6

If everything was successful, you should then see your file listed in the backup history pane at the bottom of the view:

FreeNASNSXbackups-7

FTP User Authentication

Anonymous FTP may be sufficient for most home lab purposes, but there are several advantages to configuring users and authentication. FTP by nature transmits in plain text and is not secure, but adding authentication provides a bit more control over who can access the backups and allows the direction of users to specific FTP locations. This can be useful if you plan to use your FTP server for more than just NSX.

Before we begin, let’s create a user in FreeNAS that we’ll use for NSX backups:

FreeNASNSXbackups-8

Some of the key things you’ll need to ensure is that the user’s primary group is the built-in ftp group used by proftpd and that the user’s home directory is where you want them to land after log in. In my example above, I’m creating a user called nsxftpuser with a home directory of the FTP root directory I configured earlier.

Keep in mind that by default FreeNAS will create a new home directory hence the wording “Create Home Directory In:”. I expect the home directory to actually be /mnt/vol1/dataset-ftp/nsxftpuser and not /mnt/vol1/dataset-ftp/.

Next, we need to modify the FTP settings slightly:

FreeNASNSXbackups-9

Since we want to use local user authentication, we need to check ‘Allow Local User Login’. I’ve also unchecked ‘Allow Anonymous Login’ to ensure only authenticated users can now login.

In order to test that we’re dumped into the user’s home directory after login, I changed the FTP default path one level back to the root of the dataset.

As a last step, it’s necessary to stop and start the FTP service again for the changes to take effect.

Before we test this new user, let’s double check that the home directory is located where we want it:

[root@freenas] /mnt/vol1/dataset-ftp# ls -lha
total 18
drwxr-xr-x  4 ftp         ftp       4B Sep  7 14:42 ./
drwxrwxr-x  5 root        wheel     5B Sep  5 16:13 ../
drwxr-xr-x  2 ftp         ftp       5B Sep  7 14:43 NSX/
drwxr-xr-x  2 nsxftpuser  ftp      10B Sep  7 14:42 nsxftpuser/

As you can see above, we now have a home directory matching the username in the FTP root location.

Now let’s try to log in using the nsxftpuser account:

C:\Users\mike.LAB\Desktop>ftp freenas.lab.local
Connected to freenas.lab.local.
220 ProFTPD 1.3.5a Server (freenas.lab.local FTP Server) [::ffff:172.16.10.17]
User (freenas.lab.local:(none)): nsxftpuser
331 Password required for nsxftpuser
Password:
230-Welcome to FreeNAS FTP Server
230 User nsxftpuser logged in
ftp>

So far so good, now let’s PUT a file to ensure we have write access to this location:

ftp> bin
200 Type set to I
ftp> put C:\Users\mike.LAB\Desktop\test.txt
200 PORT command successful
150 Opening BINARY mode data connection for test.txt
226 Transfer complete
ftp: 14 bytes sent in 0.00Seconds 14000.00Kbytes/sec.
ftp> dir
200 PORT command successful
150 Opening ASCII mode data connection for file list
-rw-r----- 1 nsxftpuser ftp 14 Sep 7 14:45 test.txt
226 Transfer complete
ftp: 67 bytes received in 0.00Seconds 22.33Kbytes/sec.

Success! The last thing we need to do is modify the NSX configuration slightly to use the new user account:

FreeNASNSXbackups-10

And sure enough, the backup was successful at 18:47 GMT:

FreeNASNSXbackups-11

If I look at the files from the FreeNAS SSH session, I can see both the encrypted backup binary and metadata properties file located in the user’s home directory:

[root@freenas] /mnt/vol1/dataset-ftp/nsxftpuser# ls -lha lab*
-rw-r-----  1 nsxftpuser  ftp   2.4M Sep  7 14:47 lab18_47_33_Thu07Sep2017
-rw-r-----  1 nsxftpuser  ftp   227B Sep  7 14:47 lab18_47_33_Thu07Sep2017.backupproperties

Scheduling Backups

Once we know NSX backups are functional, it’s a good idea to get them going on a schedule.

An important consideration to keep in mind when deciding when to schedule is when your vCenter backups are done. Because NSX relies heavily upon the state of the vCenter Server inventory and objects, it’s a good idea to try to schedule your backups at around the same time. That way, if you ever need to restore, you’ll have vCenter and NSX objects in sync as closely as possible.

FreeNASNSXbackups-12

In my lab, I have it backing up every night at midnight, but depending on how dynamic your environment is, you may want to do it more frequently.

Another important point to note is that NSX Manager doesn’t handle large numbers of backups very well in the backup directory. The UI will throw a warning once you get up to 100 backups and eventually you’ll get a slow or non-responsive UI in the Backup and Restore section. To get around this, you can manually archive older backups to another location outside of the FTP root directory or create a script to move older files to another location.

The only piece that I haven’t gotten to work with FreeNAS yet is SFTP encrypted backups using TLS. Once I get that going well, I’ll hopefully write up another post on the topic.

Thanks for reading! If you have any questions please leave a comment below.

FreeNAS Power Consumption and ACPI

As I alluded to in part 4 and part 5 of my recent ‘FreeNAS 9.10 Lab Build’ series, I could achieve better power consumption figures after enabling deeper C states. I first noticed this a year or two ago when I had re-purposed a Shuttle SH67H3 cube PC for use with FreeNAS. I was familiar with the expected power draw of this system when using it with other operating systems in the past, but with FreeNAS installed, it seemed higher than it should have been.

After doing some digging on the subject, I came across a thread on the FreeNAS forums describing the default ACPI C-state used and information on how to modify it.

A Bit of Background on ACPI C-States

What are often referred to as ‘C-States’ are basically levels of CPU power savings defined by the ACPI (Advanced Configuration and Power Interface) standard. This standard defines states for other devices in the system as well – not just CPUs – but all states prefixed by a ‘C’ refer to CPU power states.

The states supported by a system will often vary depending on the age of the system and the CPU manufacturer, but most modern Intel based systems will support ACPI states C0, C1, C2 and C3. Some older systems only supported C1 and a lower power state called C1E.

C0 is essentially the state where the CPU is completely awake, operating at its full frequency and performance potential and actually executing instructions. The higher the ‘C’ value, the deeper into power savings or sleep modes the CPU can go.

All ACPI compliant systems must support another state called C1. In the C1 state, the CPU is basically at idle and isn’t executing any instructions. The key requirement for the C1 state is that the CPU must be able to return to the C0 state to execute instructions immediately without any latency or delay. Because of this requirement, there are only so many power saving tweaks that the CPU can implement.

This is where the C2 state comes in, also known as ‘Stop-Clock’. In this ACPI idle state, additional power savings features can be used. This is where modern Intel processors shine. They can do all sorts of power saving wizardry, like turning off unused portions of the CPU or even entire idle CPU cores. Because there can be a very slight delay in returning to the C0 state when using these these features, they cannot be implemented when limited to ACPI C1.

Enabling Deeper ACPI C-States

By default, FreeNAS has ACPI configured to use only the C1 power state. Presumably, this is to guarantee maximum performance and prevent any quirks with older CPUs switching between power states.

If maximum performance and performance consistency is desired over power savings – as is often the case in critical production environments – leaving the default at C1 is probably a wise choice. This choice also becomes much less important if your system is heavily utilized and spends very little time at idle anyway. But if you are like me, running a home lab, idle power consumption is an important consideration.

From an SSH or console prompt, you can determine the detected and supported C-States by querying the relevant ACPI sysctls:

[root@freenas] ~# sysctl -a | grep cx_
hw.acpi.cpu.cx_lowest: C1
dev.cpu.3.cx_usage: 100.00% 0.00% last 19us
dev.cpu.3.cx_lowest: C1
dev.cpu.3.cx_supported: C1/1/1 C2/3/96
dev.cpu.2.cx_usage: 100.00% 0.00% last 136us
dev.cpu.2.cx_lowest: C1
dev.cpu.2.cx_supported: C1/1/1 C2/3/96
dev.cpu.1.cx_usage: 100.00% 0.00% last 2006us
dev.cpu.1.cx_lowest: C1
dev.cpu.1.cx_supported: C1/1/1 C2/3/96
dev.cpu.0.cx_usage: 100.00% 0.00% last 2101us
dev.cpu.0.cx_lowest: C1
dev.cpu.0.cx_supported: C1/1/1 C2/3/96

As you can see above, my Xeon X3430 supports C1, C2 and C3 states. The cx_usage value appears to report how much time the processor spends in that particular state and the cx_lowest reports the deepest allowed state.

In the C1 state, my system is idling at about 95W total power consumption.

I had originally suspected that Intel SpeedStep technology (a feature that provides dynamic frequency and voltage scaling at idle) was not functioning in ACPI C1, but that doesn’t seem to be the case. My CPU’s normal (non turbo-boost) frequency is 2.4GHz. If SpeedStep is functional, I’d expect it to use one of the following lower power states as defined in the following sysctl:

[root@freenas] ~# sysctl -a | grep dev.cpu.0.freq_levels
dev.cpu.0.freq_levels: 2395/95000 2394/95000 2261/78000 2128/63000 1995/57000 1862/46000 1729/36000 1596/32000 1463/25000 1330/19000 1197/17000

As seen above, the processor should be able to scale down to 1197MHz at idle. Even with the powerd daemon stopped, you can still use the powerd command line tool to see what the current CPU frequency is as well as any changes as load increases. Using powerd with the -v verbose option, we can see that the processor frequency does indeed jump up and down in the ACPI C1 state and stabilizes at 1197MHz when idle:

[root@freenas] ~# powerd -v
<snip>
changing clock speed from 1463 MHz to 1330 MHz
load   4%, current freq 1330 MHz ( 9), wanted freq 1263 MHz
load   0%, current freq 1330 MHz ( 9), wanted freq 1223 MHz
load   0%, current freq 1330 MHz ( 9), wanted freq 1197 MHz
changing clock speed from 1330 MHz to 1197 MHz
load  26%, current freq 1197 MHz (10), wanted freq 1197 MHz
load   3%, current freq 1197 MHz (10), wanted freq 1197 MHz
load   0%, current freq 1197 MHz (10), wanted freq 1197 MHz
load   4%, current freq 1197 MHz (10), wanted freq 1197 MHz

You can change the lowest allowed ACPI state by using the following command. In this example, I will allow the system to use more advanced power saving features by setting cx_lowest to C2:

[root@freenas] ~# sysctl hw.acpi.cpu.cx_lowest=C2
hw.acpi.cpu.cx_lowest: C1 -> C2

After making this change, the system power consumption immediately dropped down to about 75W. That’s more than 25% – not bad!

Now if we repeat the previous command, we can see some different reporting:

[root@freenas] ~# sysctl -a | grep cx_
hw.acpi.cpu.cx_lowest: C2
dev.cpu.3.cx_usage: 0.00% 100.00% last 19695us
dev.cpu.3.cx_lowest: C2
dev.cpu.3.cx_supported: C1/1/1 C2/3/96
dev.cpu.2.cx_usage: 5.12% 94.87% last 23us
dev.cpu.2.cx_lowest: C2
dev.cpu.2.cx_supported: C1/1/1 C2/3/96
dev.cpu.1.cx_usage: 1.87% 98.12% last 255us
dev.cpu.1.cx_lowest: C2
dev.cpu.1.cx_supported: C1/1/1 C2/3/96
dev.cpu.0.cx_usage: 1.28% 98.71% last 1200us
dev.cpu.0.cx_lowest: C2
dev.cpu.0.cx_supported: C1/1/1 C2/3/96

Part of the reason the power savings was so significant is because the system is spending over 95% of its time at idle in the C2 state.

Making it Stick

One thing you’ll notice is that after rebooting, this change will revert back to the default again. Since this is a sysctl, you’d think this could just be added as a system tunable parameter in the UI. I tried this to no avail.

After doing some digging, I found a bug reported on this. It appears that this is a known problem due to other rc.d scripts interfering with the cx_lowest sysctl.

I took a look in the /etc/rc.d directory and found a script called power_profile. It does indeed appear to be overwriting the ACPI lowest state at bootup:

[root@freenas] ~# cat /etc/rc.d/power_profile
<snip>
# Set the various sysctls based on the profile's values.
node="hw.acpi.cpu.cx_lowest"
highest_value="C1"
lowest_value="Cmax"
eval value=\$${profile}_cx_lowest
sysctl_set
<snip>

I could probably get by this issue by modifying the startup scripts, but the better solution is to simply add the required command to the list of post-init scripts. This can be done from the UI in the following location:

freenas-acpi-1

They key thing is to ensure is that the command is run ‘post init’. When that is done, the setting sticks and is applied after the power_profile script.

Hopefully this could save you a few dollars on your power bill as well!

FreeNAS 9.10 Lab Build – Part 5

In part 4 of this series, I took a look at my new used Dell PowerEdge T110 and talked about the pros and cons about using this type of machine. Today, I’ll be installing the drives and completing the build.

freenas5-3

To begin, I installed my drives into the server’s normal 3.5″ mounting locations. I had a few challenges here, but I was very thankful that I kept the Dell branded SATA cable that came with the PERC H200 card. It’s totally meant for this machine and keeps the wiring organized. In all of the custom builds I’ve done, it’s really tough to get clean SATA-power wiring because of the close proximity of the drives. Because of this, there is often too much pressure on the connectors and I’ve even had issues with them coming loose. This Dell breakout cable is very flexible and because of the built-in power connector, makes for a very clean and secure wiring job.

Unfortunately, I didn’t need to buy the two SATA breakout cables that I discussed in part 1 of the series, but at least I’ve got extras if I ever want to add more drives to the system.

freenas5-2

To get the two 2.5″ SATA drives installed, I used a couple of Kingston brand 2.5″ to 3.5″ adapters. They were a perfect fit for the drive caddies I bought off of ebay, but they interfered with the Dell SATA brakout cable connector. To get around this, I just loosened the screws securing the SSDs and the cable connector to get a couple of extra millimeters of clearance for the connector. Eventually, I hope to add a hotswap enclosure to one of the 5.25″ drive bays for 2.5″ drives, but for now this will have to do.

Continue reading

FreeNAS 9.10 Lab Build – Part 4

On my daily perusal through my favorite eBay seller’s inventory, I came across a well used and somewhat scratched up Dell T110 tower server. It probably didn’t garner a lot of interest with its meager 1GB of RAM, lack of hard drives and rough appearance. Despite this, the seller’s asking price of $99 wasn’t bad when you consider that is had a Xeon X3430 quad core processor and was tested and functional.

Since I already had a working Perc H200 card – an optional and supported card in the Dell T110 – as well as a pair of 4GB 2Rx8 ECC DIMMs collecting dust, this box was suddenly appealing to me.

The next day, I saw the price drop to $75, and almost pulled trigger on it until I saw the hefty shipping cost. I didn’t think it would last long at that price, but decided to make an offer of $63 to offset the shipping a bit and was pleased to find that the seller accepted.

It needed a lot of TLC when it arrived. I spent a fair amount of time removing caked-on dust from the fan assembly and also spent some quality time with a can of compressed air.

Continue reading

FreeNAS 9.10 Lab Build – Part 3

In Part 2 of my FreeNAS 9.10 build series, I installed my newly flashed Dell PERC H200 LSI 9211-8i card into my primary management host to try out FreeNAS as a VM with PCI passthrough.

After opening the case to do some rewiring I was totally shocked at how hot the little aluminum heatsink on the card was. This thing wasn’t just hot – it was scalding hot. Hot enough to invoke the subconscious and instinctive reaction pull your finger away from the heatsink. If I had to guess, I’d say that heatsink was at least 90’C or hotter.

Although it was surprising, I had to remind myself that this adapter is not a consumer grade card and does have an airflow requirement for it to run reliably. Dell’s H200 user guide says very little about the cooling requirements, unfortunately, but some of the Dell PowerEdge servers that support the H200 classify it as a ‘25W TDP’ card. That may not sound like a lot of thermal output, but when you consider most of that is concentrated on an area the size of a dime, it’s a lot to dissipate.

In most rackmount type cases, a minimum amount of front-to-back airflow is usually directed across all of the PCI-Express slots, but in my large Phanteks Enthoo Pro, there happens to be very little in this area with the default fan configuration.

After perusing around online, I could see that this was not an uncommon problem among SAS 2008 based adapters – including the popular IBM M1015 – and that in some circumstances the card could even overheat under heavy load. Considering how hot mine was while sitting idle, I can only imagine.

Even though I’ll only be using this card in a lab, I do want to ensure it runs reliably and lives a long life. It seemed that there were a few possible solutions I could pursue:

  1. Add a small 40mm fan to the heatsink.
  2. Replace the tiny heatsink with something larger.
  3. Find a way to increase the airflow in this area of the case.

The third option appealed to me most – mainly because I hate small fans. They usually have an annoying ‘whine’ to them as they need to spin at a higher RPM and they can be pretty unreliable. I’d also expect the width of the fan to block the adjoining PCI-Express slot as well.

So after taking a look through my spare parts, I came across an old Noctua 92mm PWM fan collecting dust. Although I think Noctua’s marketing is a bit over the top, I have been using their fans for many years and can attest to their high quality and quiet operation.

After MacGyver’ing a couple of thumbscrews and metal brackets I could get the fan into the perfect position. Also, because it’s a PWM modulated fan, it spins down with the rest of the system fans and is pretty much inaudible at under 1000RPM.

Even though there feels like there is barely any airflow coming from the Noctua NF-B9 fan at reduced RPM, it’s enough to dissipate the hot air from around the heatsink fins and the heatsink is now only warm to the touch! It really did make a huge difference.

Problem solved. Hopefully whatever case I ultimately use for my FreeNAS build will not have these sorts of airflow dead spots, but at least there could be a simple solution.

FreeNAS 9.10 Lab Build Series:

Part 1 – Defining the requirements and flashing the Dell PERC H200 SAS card.
Part 2 – FreeNAS and VMware PCI passthrough testing.
Part 3 – Cooling the toasty Dell PERC H200.
Part 4 – A close look at the Dell PowerEdge T110.
Part 5 – Completing the hardware build.

FreeNAS 9.10 Lab Build – Part 2

In Part 1 of this series, I discussed building a proper FreeNAS server and prepared a Dell PERC H200 by flashing it to an LSI 9211-8i in IT mode. But while I was looking around for suitable hardware for the build, I decided to try something that I’ve wanted to do for a long time – PCI passthrough.

This would give me an opportunity to tinker with vt-d passthrough and put my freshly flashed Dell PERC H200 through its paces.

Why Not VMDK Disks?

As mentioned in Part 1 of this series, FreeNAS makes use of ZFS, which is much more than just a filesystem. It combines the functionality of a logical volume manager and an advanced filesystem providing a whole slew of features including redundancy and data integrity. For it to do this effectively – and safely – ZFS needs direct access to SATA or SAS drives. We want ZFS to manage all aspects of the drives and the storage pool and should remove all layers of abstraction between the FreeNAS OS and the drives themselves.

As you probably know, FreeNAS works well enough as a virtual machine for lab purposes. After all, ignoring what’s in between, ones and zeros still make it to from FreeNAS to the disks. That said, using a virtual SCSI adapter and VMDK disks certainly does not qualify as ‘direct access’. In fact, the data path would be packed with layers of abstraction and would look something like this:

Physical Disk > < SATA/SAS HBA > < ESXi Hypervisor HBA driver > < VMFS 5 Filesystem > < VMDK virtual disk > < Virtual SCSI Adapter > < FreeNAS SCSI driver > < FreeNAS/FreeBSD

In contrast, a physical FreeNAS server would look more like:

Physical Disk > < SATA/SAS HBA > < FreeNAS HBA driver > < FreeNAS/FreeBSD

 

Continue reading

FreeNAS 9.10 Lab Build – Part 1

Since the early iterations of my home lab, I’ve been using FreeNAS for shared storage. Admittedly, I use only a fraction of what this powerful FreeBSD based solution is cable of, but I’ve always found it to be very reliable and it has met my needs.

Over the years, I have used it in several capacities, including as a virtual machine, on consumer grade hardware and more recently as a virtual machine with PCI pass-through of SAS and network devices.

After the recent overhaul of my home lab, I decided that it would be a good time to build a ‘semi-proper’ FreeNAS box that feels more like a permanent fixture. My goal was to bring something online that was:

  • Inexpensive.
  • Relatively small.
  • Using proper server-grade hardware with error correcting (ECC) memory.
  • Enough PCI-E expandability for a SAS card as well as a 10Gb NIC or Fiberchannel HBA in the future.
  • Somewhat power-efficient and quiet.
  • Preferably a Xeon based system and not an Atom or Avaton.

Some of the points above very rarely go hand-in-hand; like “inexpensive” and “Xeon” or “Proper server-grade” and “quiet”. This likely meant I’d be required to do a custom build from scratch.

The SAS Card – Dell PERC H200

Many would argue that a good storage build is focused around a good storage controller card. I had originally wanted to buy an LSI 9211-8i or an IBM M1015 card, but I came across a great deal on a Dell PERC H200 on eBay. It cost me only $25 CDN, and was an as-is, untested part so I took a chance on it. To my pleasant surprise, the card worked just fine.

IMG_0334

Although this is a Dell branded part, the PERC H200 is essentially just a SAS 2008 based LSI 9211-8i adapter. The most notable difference is the location of the SFF-8087 ports on the top of the card, instead of the end.

Continue reading