This page contains useful command for the ipmitool. ipmitool is installed on computenode01. This node can access the IPMI interfaces of all nodes in the cluster.

You need to know the following information to connect to an IPMI interface.

IPMI_IP=10.1.0.105 # computenode05
PASSWORD="" # not shown here for security purposes

Get a serial console. This console is only useful during boot time and can access the boot configuration menu and the bios. Also you can see if the PXE boot process works.

ipmitool -I lanplus -H $IPMI_IP -U ADMIN -P $PASSWORD sol set     volatile-bit-rate 19.2
ipmitool -I lanplus -H $IPMI_IP -U ADMIN -P $PASSWORD sol set non-volatile-bit-rate 19.2
ipmitool -I lanplus -H $IPMI_IP -U ADMIN -P $PASSWORD sol activate

To get linux kernel output and a login shell through the IPMI sol console you need to edit the syslinux or grub configuration and add the following parameters to the kernel:

vmlinuz <other params ...> xencons=ttyS1 console=tty console=ttyS1,19200n8 lowres

• Note: on some machines the console needs to be set to ttyS0 instead.
• Note: computenode01 - computenode04 and storagenode use a different IPMI card and have different login credentials

controlnode   ttyS1
networknode   ttyS1
storagenode   ?
sdncontrol    ?
computenode01 ?
computenode02 ttyS1 and access over ssh to ipmi-card 
computenode03 ?
computenode04 ?
computenode05 ttyS0 (no bitrate config)
computenode06 ?
computenode07 ?
computenode08 ?
computenode09 ?
computenode10 ?
computenode11 ?
computenode12 ?
computenode13 ?
computenode14 ?
computenode15 ?
computenode16 ttyS0 (no bitrate config)
sapXX ttyS0,9600n8 (username: root, ssh->start /HOST/console, type "ESC+(" to stop)

You can check a node's power status and change it. Useful if you need to reboot a node if you locked yourself out of ssh with a bad iptables configuration.

ipmitool -I lanplus -H $IPMI_IP -U ADMIN -P $PASSWORD chassis power status
ipmitool -I lanplus -H $IPMI_IP -U ADMIN -P $PASSWORD chassis power off
ipmitool -I lanplus -H $IPMI_IP -U ADMIN -P $PASSWORD chassis power on

You can change the boot device if you want to switch from PXE boot to disk and vice versa. It is also possible to enter the BIOS setup on boot.

ipmitool -I lanplus -H $IPMI_IP -U ADMIN -P $PASSWORD chassis bootdev pxe
ipmitool -I lanplus -H $IPMI_IP -U ADMIN -P $PASSWORD chassis bootdev disk
ipmitool -I lanplus -H $IPMI_IP -U ADMIN -P $PASSWORD chassis bootdev bios

You can list additional IPMI commands and possible parameters if you provide insufficient parameters for the tool.

ipmitool -I lanplus -H $IPMI_IP -U ADMIN -P $PASSWORD
ipmitool -I lanplus -H $IPMI_IP -U ADMIN -P $PASSWORD chassis
ipmitool -I lanplus -H $IPMI_IP -U ADMIN -P $PASSWORD chassis bootdev

You can show and change the IPMI card lan interface configuration. DONT FUCK THIS UP. Note: Setting the IPMI IP will terminate the IPMI connection and confuse ipmitool, but the change will be made.

ipmitool -I lanplus -H $IPMI_IP -U ADMIN -P $PASSWORD lan print 1
ipmitool -I lanplus -H $IPMI_IP -U ADMIN -P $PASSWORD lan set 1 ipaddr $NEW_IPMI_IP

You can also use the web interface to access sensor readings and change settings that are vendor specific and not (easily) accessible with the commandline tool.

IPMI on computenode01 - computenode04

ssh Administrator@10.1.0.101
# enter password and wait 10 seconds (it's slow)
# you get a "-> " prompt

# power on
start /admin1/system1

# power off
stop /admin1/system1

# power off (forced)
stop -f /admin1/system1

# disconnect any existing sessions
set /admin1/system1/textredirectsvc1/textredirectsap1 EnabledState=6

# connect to console ("ESCstop" (no newline) to quit)
start /admin1/system1/textredirectsvc1

# command list
help

# quit ssh
exit

extract sensor data

VAL=$(ipmitool sdr get "PS1 Input Power" |grep "Sensor Reading" | cut -d ':' -f2 | cut -d ' ' -f2)

This value can be then monitored in ganglia:

gmetric -n 'ipmi_input_power-PS1' -v $VAL -t int32 -u 'Watts'