Noctua NH-U9DX i4 and NH-D9DX i4 3U Heatsink Review

There are many different approaches you can take when building a VMware home lab, but I always prefer to do custom-build tower systems. This allows me to select the components I want – the fans, heatsinks, PSU – to keep the noise to a minimum. Even though I keep the lab in the basement, I really don’t like to hear it running. I don’t need the density that a rack provides, and I’m quite happy to have my nearly silent towers humming along on a wire shelf instead. Not to mention that lower noise usually equates to lower power consumption and of course, it keeps my wife happy as well – the most critical metric of all.

I finally got around to upgrading my three ancient compute nodes recently. I have been using the included Dynatron R13 1U copper heatsinks that came with the motherboards I bought on eBay. They work, and keep the CPUs relatively cool, but as you can imagine, noise was not a key consideration in their design. They are as compact and as efficient as possible at only an inch tall. At idle, the Supermicro X9SRL-F keeps them at a low RPM, but put any load on the systems and you’re quickly at 7500RPM and the noise is pretty unbearable. No problem for a datacenter, but not for a home lab.

Today I’ll be taking a bit of a departure from my regular posts and will be doing an in-depth review of two high-end Noctua heatsinks. Noctua was kind enough to send me a review sample of not one, but two of their Xeon heatsinks – the NH-U9DX i4 and the NH-D9DX i4 3U.

Noctua

noctua_200x200pxNoctua is an Austrian company well known for their low noise fans and high-end heatsinks. I’ve been using Noctua heatsinks for ages. In fact, I reviewed some of their original heatsinks and fans many years ago when I used to write hardware reviews. This included their original NH-U12P, the NH-C12P and the smaller NH-U9B. Back then, I praised them for their high-quality construction, near silent operation, excellent mounting hardware and most importantly – excellent cooling performance. That was over ten years ago, and it seems that Noctua is still very well respected for all the same reasons today.

Their gear has always been pricey compared to the competition, but when it comes to Noctua, you get what you pay for.

Square vs Narrow ILM LGA 2011

One of the challenges I had with my new/used Supermicro X9SRL-F boards is that they don’t use the common ‘square’ LGA2011 mounting pattern. Instead, they use what’s referred to as ‘Narrow ILM’ that allows the socket to be more rectangular in shape. This allows more real estate for memory slots and other components on the board. Because of this, I was severely limited in my heatsink choices. Most of what’s available out there for Narrow ILM are rack mount heatsinks similar to what I’m hoping to remove. The majority of the consumer-grade stuff for LGA2011 simply won’t fit.

noctuai4-24

Here you can see the Dynatron R13 narrow ILM heatsink mounted:

Continue reading “Noctua NH-U9DX i4 and NH-D9DX i4 3U Heatsink Review”

Low-Voltage DDR3 – Performance, Heat and Power

When I built my new compute nodes, I chose PC3L-12800R DIMMs that could run in both standard 1.5V and 1.35V low-voltage modes. What I didn’t realize, however, is that Intel’s specification for 1.35V registered memory on Socket R based systems limits low-voltage operation to 1333MHz. The Supermicro X9SRL-F boards I’m using enforce this, despite the modules being capable of the full 1600MHz at 1.35V.

This meant I could run the modules at 1333MHz and enjoy the power/heat reduction that goes with 1.35V operation, or I could force the modules to run at 1600MHz at the ‘standard’ 1.5 volts. As I considered different configuration options, a lot of questions came to mind. Today I’ll be taking an in-depth look at DDR3 memory bandwidth, latency, power consumption and heat. I hope to answer the following questions in this post:

  1. What is the bandwidth difference between 1333MHz with tighter 9-9-9 timings versus 1600MHz at looser 11-11-11 timings?
  2. What is the latency difference between 1333MHz with tighter 9-9-9 timings versus 1600MHz at looser 11-11-11 timings?
  3. Just how much power savings can be realized at 1.35V versus 1.5V?
  4. How much cooler will my DIMMs run at 1.35V versus 1.5V?
  5. What about overclocking my DIMMs to 1867MHz at 12-12-12 timings?
  6. How does memory bandwidth and latency fair with fewer than 4 DIMMs populated on the SandyBridge-EP platform?

A Note On Overclocking

Although there are IvyBridge-EP CPUs that support higher 1866MHz DIMM operation, my SandyBridge-EP based E5-2670s and PC3L-12800 DIMMs do not officially support this. The Supermicro X9SRL-F allows the forcing of 1866MHz at 12-12-12 timings even though the DIMMs don’t have this speed in their SPD tables. Despite being server-grade hardware, this is an ‘overclock’ plain and simple.

I would never consider doing this in a production environment, but in a lab for educational purposes – it was worth a shot. Thankfully, to my surprise, the system appears completely stable at 1866MHz and 1.5V. Because the DIMMs can technically run at 1600MHz with a lower 1.35V, it would seem logical that at 1.5V they’d have additional headroom available. This certainly appears to be true in my case – even with a mixture of three different brands of PC3L-12800R.

With my overclocking success, I decided to include results for 1866MHz operation at 1.5V to see if it’s worth pushing the DIMMs beyond their rated specification.

Bandwidth and Latency

Back in my days as an avid overclocker, increasing memory frequency would generally provide better overall performance than tightening the CAS, RAS and other timings of the modules. Obviously doing both was ideal but if you had to choose between higher frequency and tighter timings, you’d usually come out ahead by sacrificing tighter timings for higher frequencies.

I fully expected the 1600MHz memory running at 11-11-11 timings to be quicker overall than 1333MHz at 9-9-9, but the question was just how much quicker. To test, I booted one of my X9SRL-F systems from a USB stick running Lubuntu 18.04 and used Intel’s handy ‘Memory Latency Checker’ (MLC) tool. MLC allows you to get several performance metrics including peak bandwidth, latency as well as various cross-socket NUMA measurements that can be handy for multi-socket processor systems. I’ll only be looking at a few metrics:

  1. Peak Read Bandwidth
  2. 1:1 Reads/Writes Bandwidth
  3. Idle Latency

memspeed-1

As I expected, despite the looser timings, memory bandwidth increases substantially as the frequency increases. About a 12% boost is obtained going to 1600MHz at 11-11-11 timings. An additional 10% was obtained by overclocking the modules to 1866MHz. Mixed reads and writes get a pretty proportional boost in all three tests.

Continue reading “Low-Voltage DDR3 – Performance, Heat and Power”

Moving a FreeNAS ZFS Pool from a Physical Server to a VM using RDMs

I’ve been using FreeNAS for several years now for both block and NFS storage in my home lab with great success. For more information on my most recent FreeNAS build, you can check out the series here.

Although I’ve been quite pleased with this setup, I had to repurpose the SSDs in the box and had yet another USB boot device failure. This meant I had to reinstall FreeNAS and left me with just a single ZFS pool with a pair of 2TB mechanical drives. It just didn’t feel right to have a full system up and running for just a pair of 2TB drives when I could run them just fine in my management ESXi host. Not to mention the fact that I’ve got 224GB of RAM available there to provide for a much larger L1 ARC cache.

In part 2 of my FreeNAS build series, I looked at using VT-d to passthrough a proper LSI SAS HBA to a VM. This is really the best possible virtual FreeNAS configuration as it bypasses all of the hypervisor’s storage stack and grants direct access to the HBA and drives. I considered using this setup, but I didn’t think it was worth the extra power consumption and cooling needed for the toasty PERC H200 card I’ve been using. Since I wanted to preserve all data on the drives, RDMs seemed to be the next logical solution. This isn’t as ‘pure’ as the VT-d solution, but it still gives the VM full block access to the drives in the system. At any rate, it was worth a try!

Disclaimer: If you are using ZFS and FreeNAS for production purposes or for any critical data that you care about, using a proper physical setup is important. I wouldn’t recommend virtualizing FreeNAS or any other ZFS based storage system for anything but testing or lab purposes.

What I hoped to do was the following:

  1. Take the 2x2TB Western Digital hard drives out of the Dell T110.
  2. Re-install the 2x2TB drives in my Intel S2600 management host on the integrated SATA controller.
  3. Create a new FreeNAS virtual machine.
  4. Add the two drives to the VM as virtual mode RDMs.
  5. Import the existing ZFS volume that is striped across these two drives in FreeNAS.
  6. Re-create the iSCSI target and NFS shares and have access to all existing data in the pool! (assuming all goes well).

Creating a new FreeNAS VM

Once I got the two drives installed in my Intel S2600 management host, I created a new VM and got the FreeNAS OS installed. Below is the virtual hardware configuration I used:

Guest OS type: Other, FreeBSD 64-bit
CPUs: 2x vCPUs
Memory: 16GB (a minimum of 8GB is required)
Hard Disk: 16GB (for the FreeNAS OS boot device, a minimum of 8GB is required)
New SCSI controller: LSI Logic SAS
Network adapter type: VMXNET3
CD/DVD Drive: Mount the FreeNAS 11.2 ISO from a datastore

You’ll notice that some of the options I selected are not defaults for FreeBSD based VMs. This includes the LSI SAS adapter, and the VMware VMXNET3 NIC. LSI Parallel is the default for FreeBSD VMs, but the SAS adapter works well with all recent BSD builds. The same holds true for the VMXNET3 adapter, which has many benefits over the emulated E1000 adapter type.

Continue reading “Moving a FreeNAS ZFS Pool from a Physical Server to a VM using RDMs”

Home Lab Power Automation – Part 3

In part 2, I shared the PowerCLI scripting I used to power on my entire lab environment in the correct order. In this final installment, I’ll take you through the scripting used to power everything down. Although you may think the process is just the reverse of what I covered in part 2, you’ll see there were some other things to consider and different approaches required.

Step 1 – Shutting Down Compute Cluster VMs

To begin the process, I’d need to shut down all VMs in the compute-a cluster. None of the VMs there are essential for running the lab, so they can be safely stopped at any time. I was able to do this by connecting to vCenter with PowerCLI and then using a ‘foreach’ loop to gracefully shut down any VMs in the ‘Powered On’ state.

"Connecting to vCenter Server ..." |timestamp
Connect-VIServer -Server 172.16.1.15 -User administrator@vsphere.local -Password "VMware9("

"Shutting down all VMs in compute-a ..." |timestamp
$vmlista = Get-VM -Location compute-a | where{$_.PowerState -eq 'PoweredOn'}
foreach ($vm in $vmlista)
    {
    Shutdown-VMGuest -VM $vm -Confirm:$false | Format-List -Property VM, State
    }

The above scripting ensures the VMs start shutting down, but it doesn’t tell me that they completed the process. After this is run, it’s likely that one or more VMs may still be online. Before I can proceed, I need to check that they’re all are in a ‘Powered Off’ state.

"Waiting for all VMs in compute-a to shut down ..." |timestamp
do
{
    "The following VM(s) are still powered on:"|timestamp
    $pendingvmsa = (Get-VM -Location compute-a | where{$_.PowerState -eq 'PoweredOn'})
    $pendingvmsa | Format-List -Property Name, PowerState
    sleep 1
} until($pendingvmsa -eq $null)
"All VMs in compute-a are powered off ..."|timestamp

A ‘do until’ loop does the trick here. I simply populate the list of all powered on VMs into the $pendingvmsa variable and print that list. After a one second delay, the loop continues until the $pendingvmsa variable is null. When it’s null, I know all of the VMs are powered off and I can safely continue.

Continue reading “Home Lab Power Automation – Part 3”

Home Lab Power Automation – Part 2

In part 1, I shared some of the tools I’d use to execute the power on and shutdown tasks in my lab. Today, let’s have a look at my startup PowerCLI script.

A Test-Connection Cmdlet Replacement

As I started working on the scripts, I needed a way to determine if hosts and devices were accessible on the network. Unfortunately, the Test-Connection cmdlet was not available in the Linux PowerShell core release. It uses the Windows network stack to do its thing, so it may be a while before an equivalent gets ported to Linux. As an alternative, I created a simple python script that achieves the same overall result called pinghost.py. You can find more detail on how it works in a post I did a few months back here.

The script is very straightforward. You specify up to three space separated IP addresses or host names as command line arguments, and the script will send one ICMP echo request to each of the hosts. Depending on the response, it will output either ‘is responding’ or ‘is not responding’. Below is an example:

pi@raspberrypi:~/scripts $ python pinghost.py vc.lab.local 172.16.10.67 172.16.10.20
vc.lab.local is not responding
172.16.10.67 is responding
172.16.10.20 is not responding

Then using this script, I could create sleep loops in PowerShell to wait for one or more devices to become responsive before proceeding.

Adding Timestamps to Script Output

As I created the scripts, I wanted to record the date/time of each event and output displayed. In a sense, I wanted it to look like a log that could be written to a file and referred to later if needed. To do this, I found a simple PowerShell filter that could be piped to each command I ran:

#PowerShell filter to add date/timestamps
filter timestamp {"$(Get-Date -Format G): $_"}

Step 1 – Power On the Switch

Powering up the switch requires the use of the tplink_smartplug.py python script that I discussed in part 1. The general idea here is to instruct the smart plug to set its relay to a state of ‘1’. This brings the switch to life. I then get into a ‘do sleep’ loop in PowerCLI until the Raspberry Pi is able to ping the management interface of the switch. More specifically, it will wait until the pinghost.py script returns a string of “is responding”. If that string isn’t received, it’ll wait two seconds, and then try again.

"Powering up the 10G switch ..." |timestamp
/home/pi/scripts/tplink-smartplug-master/tplink_smartplug.py -t 192.168.1.199 -c on |timestamp

"Waiting for 10G switch to boot ..." |timestamp
do
{
$pingresult = python ~/scripts/pinghost.py 172.16.1.1 |timestamp
$pingresult
sleep 2
} until($pingresult -like '*is responding*')

When run, the output looks similar to the following:

Continue reading “Home Lab Power Automation – Part 2”

Home Lab Power Automation – Part 1

My home lab has grown substantially over the last few years and with it, so did power consumption, heat and noise. I selected power efficient parts where possible, but even with 500-600W power usage, 24/7 operation adds up. I can certainly notice the extra cost on my hydro bill, but it’s not just the financial impact – it’s also the environmental impact I’m concerned about.

The bottom line is that I have no reason to run the lab 24/7. I generally use it an hour or two each day – sometimes more, sometimes less. But there are also stretches where I won’t use it for several days like on weekends or when I’m busy with other things.

I found myself manually shutting down the lab when I knew I wouldn’t be using it and then manually powering everything back up. As you can imagine, this was quite a process. Everything had to be powered on or shut down in a very specific order to avoid problems. I’d also need to be standing in front of the equipment for part of this process as some equipment didn’t have remote power-on capability. Because of the work and time involved, I’d inevitably just leave it powered on for much longer stretches than I needed to.

It wasn’t until I added a 10Gbps Quanta LB6M switch to the lab that I realized I needed to do something. It’s not a quiet or energy efficient switch, consuming an average of 120W at idle.

Continue reading “Home Lab Power Automation – Part 1”

Home Lab

One of the most important tools I use day-to-day is my lab. Although I’m fortunate to have access to some shared lab resources at VMware, I still choose to maintain a dedicated home lab. I like to have the freedom to build it up, tear it down and configure it in any way I see fit.

I’ve had a few people ask me about my home lab recently, so I wanted to take a moment to share my setup. I’m not going to go too much into how I use the lab, or the software side of things but will stay focused on the hardware for now.

My Goals

I’ve had several iterations of home lab over the years, but my most recent overhaul was done about two years ago in 2016. At that time, I had several goals in mind:

  1. To keep cost low. I chose mainly EOL, second hand hardware that was relatively inexpensive. I often looked for the ‘sweet spot’ to get the best performance for the dollar.
  2. To use server/workstation grade hardware wherever possible. I’ve had some mixed experiences with consumer grade equipment and prefer having IPMI and being able to run large amounts of registered ECC memory.
  3. Low noise. I really didn’t like the noise and heat generated by rackmount gear and tried to stick with custom-build server systems wherever possible.
  4. Power efficiency. Building custom machines with simple cooling systems allowed me to keep power consumption down. I also didn’t see the point of running the lab 24/7 and chose to automate power on and power off activities.
  5. Sized right. Although more RAM and compute power is always desirable, I tried to keep things reasonably sized to keep costs and power consumption down. I wanted to be able to have some flexibility, but would try to keep VMs sized smaller and power down what I didn’t need.

The Lab

homelabv1

I’ll get more into each component, but here’s a summary:

  • 1x Management Node (2x Xeon E5-2670, 96GB RAM)
  • 3x Compute Nodes (Xeon X3440, 16GB RAM)
  • 1x FreeNAS Server (Dell T110, Xeon 3430, 8GB RAM)
  • 1x Raspberry Pi 3 Model B (Automation and remote access)
  • Quanta LB6M 24 port 10Gbps Switch (24x SFP+ ports)
  • D-link DGS-1210-16 Managed Switch (16x copper ports, 4x SFP)
  • Cyber Power PFCLCD1500 UPS system

All of the equipment sits comfortably in a wire shelf/rack in a corner of my unfinished basement. Here it can stay nice and cool and the noise it generates doesn’t bother anyone.

Continue reading “Home Lab”

Creating a Python Replacement for the Test-Connection cmdlet.

I’ve been working on a PowerShell script to automatically power up and shut down all devices in my home lab. Both in the interest of saving on hydro and to reduce it’s environmental impact. To do this, I’ve been using a low-power Raspberry Pi 3 B+, which will stay powered on to orchestrate these and other scripted activities.

One great PowerShell cmdlet that comes in handy is Test-Connection. Although it can be used for numerous network testing purposes, it’s most commonly used to send an ICMP echo request to a host to see if it responds or not. The cmdlet responds with a simple true or false response, which makes it very handy for scripting purposes. I was really hoping to use this cmdlet because it makes it easy to determine if a device is still online or not and if it’s okay to move on to the next phase of powering things on or shutting things down. For example, I don’t want to power on my management ESXi host until the freenas SAN is online.

The problem I ran into was that not all PowerShell cmdlets have been ported over to PowerShell Core for Linux. Test-Connection is unfortunately one of them due to the way it uses the Microsoft Windows network stack to function. Looking at the GitHub PR, it seems they are getting close to porting it over, but I decided to try my hand at creating a python script with similar basic functionality.

I should note that I’m not a programmer by any stretch of the imagination and it’s been many years since I’ve done any serious coding. This is actually the first bit of python code I’ve written, so I’m sure there are many more efficient ways to achieve what I’ve done here:

import os
import sys

# If no arguments parsed, display the greeting:
if len(sys.argv) == 1:
    print("vswitchzero.com ICMP response script. Specify up to 3 hosts to ping separated by spaces.")
    print("example: python ./pinghost.py 172.16.10.15 192.168.1.1 vc.lab.local")

# If one arg parsed, ping a single host
if len(sys.argv) == 2:
    host1 = sys.argv[1]
    response = os.system("ping -c 1 " + host1 + "> /dev/null")
    if response == 0:
        print host1, 'is responding'
    else:
        print host1, 'is not responding'

# If two args parsed, ping two hosts
if len(sys.argv) == 3:
    host1 = sys.argv[1]
    host2 = sys.argv[2]
    response1 = os.system("ping -c 1 " + host1 + "> /dev/null")
    if response1 == 0:
        print host1, 'is responding'
    else:
        print host1, 'is not responding'
    response2 = os.system("ping -c 1 " + host2 + "> /dev/null")
    if response2 == 0:
        print host2, 'is responding'
    else:
        print host2, 'is not responding'

# If three args parsed, ping three hosts
if len(sys.argv) == 4:
    host1 = sys.argv[1]
    host2 = sys.argv[2]
    host3 = sys.argv[3]
    response1 = os.system("ping -c 1 " + host1 + "> /dev/null")
    if response1 == 0:
        print host1, 'is responding'
    else:
        print host1, 'is not responding'
    response2 = os.system("ping -c 1 " + host2 + "> /dev/null")
    if response2 == 0:
        print host2, 'is responding'
    else:
        print host2, 'is not responding'
 response3 = os.system("ping -c 1 " + host3 + "> /dev/null")
    if response3 == 0:
        print host3, 'is responding'
    else:
        print host3, 'is not responding'

# If more than three args parsed, display an error
if len(sys.argv) > 4:
    print('vswitchzero.com ICMP response script. Specify up to 3 hosts to ping separated by spaces.')
    print('example: python ./pinghost.py 172.16.10.15 192.168.1.1 vc.lab.local')
    print(' ')
    print('ERROR: Too many arguments specified. Use 1-3 IPs or hostnames only. Each should be separated by a space')

Continue reading “Creating a Python Replacement for the Test-Connection cmdlet.”