The machine came with the BMC installed and based on the documentation (found here we connected the RJ45 jack closer to the middle of the system to our "IPMI network" (a switch with one other machine attached to it). We then updated the firmware as described in "IV. BMC Firmware Refresh". We then installed their "Graceful Power Control agent" software as described in section V., and encountered one problem: the agent installer script modifies rc.local to make sure the relevant IPMI kernel modules get loaded, and installs and starts two services (that are also set to start in rc.local) called smagent and smmsg; smmsg relies on some gnome libs to start, so the installer threw an error when trying to start smmsg, so we need to disable smmsg? (I'm waiting for confirmation from a SuperMicro engineer that smmsg isn't necessary) Note, at this point, they've appended a call to smmsg to the end of /etc/profile, so when logging into this machine, one gets an error each time. I hoped that I could continue without having smmsg working.
At this point, I installed IPMIview following their instructions. I then followed their instructions under "VI. Mangaging an IPMI-enabled System for the first time". In the OS on the managed machine, I set the IP for that interface (eth0) to be 192.168.1.2 with a mask of 255.255.255.0, and contrary to their instructions the default route was not set (as our cluster nodes don't route out). I started the application, and following their instructions it worked! I was able to connect to the machine. Note, smagent appears to be "listening" and bridging the IPMI traffic that ISN'T getting to the BMC. I configured the BMC's IP info as described in the instructions, and the BMC reset itself. After the BMC rebooted (note, the system didn't reboot, just the BMC), I was able to connect to it again. I then did a "graceful shutdown". The machine powered itself off (which "logout now" messages getting echoed to consoles). And, after soft power down, I pressed the "Power Up" button on the gui, and the node powered itself up! WooHoo!
Next up, SOL (no, that means Serial Over LAN)...