I’ve recently been working on a scripting project to orchestrate the power up and power down of my entire lab environment. As part of this, I’ve been using IPMI commands to power on physical servers in the correct order and at the correct time.
As I discussed in my recent FreeNAS Build Series, I’ve been using a Dell T110 tower server for storage purposes in my lab. Being an entry level server, the T110 has a very trimmed down iDRAC BMC (Baseboard Management Controller) that doesn’t have a dedicated NIC or a web based management page. Despite it’s limitations, I can still use the IPMI protocol to gather information and to run simple tasks, like powering it on and off.
pi@raspberrypi:~ $ ipmitool -I lanplus -H 172.16.10.67 -U root -P "vmware" lan print 1 Set in Progress : Set Complete Auth Type Support : NONE MD2 MD5 PASSWORD Auth Type Enable : Callback : MD2 MD5 : User : MD2 MD5 : Operator : MD2 MD5 : Admin : MD2 MD5 : OEM : IP Address Source : Static Address IP Address : 172.16.10.67 Subnet Mask : 255.255.255.0 MAC Address : b8:ac:6f:92:0b:e9 SNMP Community String : public IP Header : TTL=0x40 Flags=0x40 Precedence=0x00 TOS=0x10 Default Gateway IP : 172.16.10.1 Default Gateway MAC : 00:00:00:00:00:00 Backup Gateway IP : 0.0.0.0 Backup Gateway MAC : 00:00:00:00:00:00 802.1q VLAN ID : Disabled 802.1q VLAN Priority : 0
Above you can see a simple IPMI query using the ipmitool application available for Linux and other operating systems. In this example, I’m pulling the network configuration of the BMC.
I hadn’t had a need to use IPMI with the Dell T110 until recently, but was surprised to see that the BMC was not responding to ping. Thinking the BMC was just hung up, I did a cold power cycle and double checked the configuration. After several frustrating reboots, it became clear to me that FreeNAS/FreeBSD was not playing nicely with the Dell BMC. It appeared that it would work just fine until the FreeBSD kernel used by FreeNAS started loading. As soon as the bge driver claimed the BCM5722 card, the BMC couldn’t be accessed over the network.
To make things even more frustrating, it would not recover when the machine was shut down or rebooted. I could only get the BMC on the network again after doing a cold power-cycle of the server, or after going into the BMC configuration, changing something, and rebooting.
After doing some digging, I came across a thread on the FreeBSD forum that described my symptoms exactly. I’m not the only one who has run into this issue with Dell BMCs and shared Broadcom adapters in FreeBSD. This thread then led me to FreeBSD bug 196944 regarding a regression in the Broadcom bge driver. It looks like this has actually been broken for some time – all the way back to FreeBSD 9.2 – and is still a problem in 11.1 as well.
A few people were able to work around this issue by recompiling the kernel with the Broadcom driver from back in FreeBSD 9.1. I really didn’t feel comfortable doing this level of tinkering with FreeNAS – especially since any subsequent FreeNAS patches would likely just break it again.
Thankfully, someone in comment 6 of the bug describes a potential workaround that involves nothing more than enabling the PXE bootrom of the onboard Broadcom adapter in the BIOS. This was reported as having mixed results on varying models of Dell servers, but I was willing to give it a try. After changing my onboard NIC from ‘Enabled’ to ‘Enabled with PXE’ in the BIOS, the problem disappeared!
If you have this problem – give it a shot. It’s a simple workaround and the only down side is the extra 2-3 seconds at boot up.
vswitchzero 