pxe boot

PXE boot nodes over the network

TL;DR version

follow the following 3 guides (in order) to get a working PXE boot setup. Watch out for pitfalls like iptables and user permission problems though.

http://www.server-world.info/en/note?os=CentOS_6&p=dhcp

http://www.server-world.info/en/note?os=CentOS_6&p=pxe&f=1

http://www.server-world.info/en/note?os=CentOS_6&p=pxe&f=4

the long version

we start a DHCP server on the storagenode that provides PXE boot information (TFTP protocol) to each diskless node. The information is stored on the storagenode as well. The boot information then boots a linux kernel and ramfs. The root filesystem is accessed by the kernel via NFS and is also stored on the storagenode for each individual diskless computenode. DHCP, TFTP and NFS need to be installed and configured. IPTABLES needs new rules to allow DHCP, TFTP and NFS to pass through the firewall. The boot information and filesystem needs be created for each diskless computenode.

do this once

install prerequisites

yum -y install dhcp syslinux xinetd tftp-server dracut-network

configure IPTABLES to allow incomming access to DHCP, TFTP and NFS

# DHCP
iptables -I INPUT 1 -s 10.1.1.0/24 -m state --state NEW,RELATED,ESTABLISHED -p udp --dport 67    -j ACCEPT
iptables -I INPUT 1 -s 10.1.1.0/24 -m state --state NEW,RELATED,ESTABLISHED -p udp --dport 68    -j ACCEPT
# TFTP
iptables -I INPUT 1 -s 10.1.1.0/24 -m state --state NEW,RELATED,ESTABLISHED -p udp --dport 69    -j ACCEPT
# NFS
# FIXME NFS does not yet work
iptables -I INPUT 1 -s 10.1.1.0/24 -m state --state NEW,RELATED,ESTABLISHED -p udp --dport 111   -j ACCEPT
iptables -I INPUT 1 -s 10.1.1.0/24 -m state --state NEW,RELATED,ESTABLISHED -p tcp --dport 111   -j ACCEPT
iptables -I INPUT 1 -s 10.1.1.0/24 -m state --state NEW,RELATED,ESTABLISHED -p tcp --dport 2049  -j ACCEPT
iptables -I INPUT 1 -s 10.1.1.0/24 -m state --state NEW,RELATED,ESTABLISHED -p tcp --dport 32803 -j ACCEPT
iptables -I INPUT 1 -s 10.1.1.0/24 -m state --state NEW,RELATED,ESTABLISHED -p udp --dport 32769 -j ACCEPT
iptables -I INPUT 1 -s 10.1.1.0/24 -m state --state NEW,RELATED,ESTABLISHED -p tcp --dport 892   -j ACCEPT
iptables -I INPUT 1 -s 10.1.1.0/24 -m state --state NEW,RELATED,ESTABLISHED -p udp --dport 892   -j ACCEPT
iptables -I INPUT 1 -s 10.1.1.0/24 -m state --state NEW,RELATED,ESTABLISHED -p tcp --dport 875   -j ACCEPT
iptables -I INPUT 1 -s 10.1.1.0/24 -m state --state NEW,RELATED,ESTABLISHED -p udp --dport 875   -j ACCEPT
iptables -I INPUT 1 -s 10.1.1.0/24 -m state --state NEW,RELATED,ESTABLISHED -p tcp --dport 662   -j ACCEPT
iptables -I INPUT 1 -s 10.1.1.0/24 -m state --state NEW,RELATED,ESTABLISHED -p udp --dport 662   -j ACCEPT

prepare directories for all diskless nodes

mkdir -p /var/lib/tftpboot/computenode08/centos6/root/
mkdir -p /var/lib/tftpboot/computenode08/pxelinux.cfg
...

configure "/etc/dhcp/dhcpd.conf" to include PXE boot information

# Uni Ulm DNS server
option domain-name-servers 134.60.1.111;

# default lease time
default-lease-time 600;

# max lease time
max-lease-time 7200;

# this DHCP server to be declared valid
authoritative;

# specify network address and subnet mask
subnet 10.1.1.0 netmask 255.255.255.0 {
	# specify broadcast address
	option broadcast-address 10.1.1.255;

	# specify default gateway
	option routers 10.1.1.11;

	# add fixed hosts here	
	host computenode08 {
		# MAC that distinguishes the computenode
		hardware ethernet 00:1E:67:39:84:03;
		fixed-address 10.1.1.108;
		option host-name "computenode08";
		ddns-hostname "computenode08";
		# server that hosts the TFTP boot information
		next-server 10.1.1.12;
		# path from the TFTP root to the bootstrap kernel
		filename "computenode08/pxelinux.0";
	}

	# add more hosts here
	...
}

start / restart the DHCP server and run it on boot

service dhcpd restart
chkconfig dhcpd on

configure TFTP configuration in "/etc/xinetd.d/tftp"

service tftp
{
	socket_type		= dgram
	protocol		= udp
	wait			= yes
	user			= root
	server			= /usr/sbin/in.tftpd
	server_args		= -s /var/lib/tftpboot
	disable			= no
	per_source		= 11
	cps			= 100 2
	flags			= IPv4
}

start / restart TFTP server and run it on boot

service xinetd restart 
chkconfig xinetd on

do this for every diskless computenode

prepare root filesystem for server

yum -y --releasever=6.5 groupinstall "Base" "Server Platform" --installroot=/var/lib/tftpboot/computenode08/centos6/root/
yum -y install openssh-server openssh-clients --installroot=/var/lib/tftpboot/computenode08/centos6/root/

set root password hash in "/var/lib/tftpboot/computenode08/centos6/root/etc/shadow" from current hash

...
root:$6$xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx:14937:0:99999:7:::
...

set mount points in "/var/lib/tftpboot/computenode08/centos6/root/etc/fstab"

none    /tmp        tmpfs   defaults   0 0
tmpfs   /dev/shm    tmpfs   defaults   0 0
sysfs   /sys        sysfs   defaults   0 0
proc    /proc       proc    defaults   0 0

make sure networking is enabled

echo "NETWORKING=yes" >> /var/lib/tftpboot/computenode08/centos6/root/etc/sysconfig/network

edit interface definitions for both interfaces in /var/lib/tftpboot/computenode08/centos6/root/etc/sysconfig/network-scripts/ifcfg-ethX (create files if necessary)

DEVICE="ethX"
BOOTPROTO="dhcp"
IPV6INIT="no"
NM_CONTROLLED="no"
ONBOOT="yes"
TYPE="Ethernet"

copy misc. config files you want or need into the node's fs

cp /root/.bashrc /root/.bash_profile /var/lib/tftpboot/computenode08/centos6/root/root/

generate kernel and ramfs

cp /boot/vmlinuz-2.6.32-431.29.2.el6.x86_64 /var/lib/tftpboot/computenode08/centos6/vmlinuz
dracut /var/lib/tftpboot/computenode08/centos6/initramfs.img 2.6.32-431.29.2.el6.x86_64
chmod 644 /var/lib/tftpboot/computenode08/centos6/initramfs.img

prepare bootstrap information

cp /usr/share/syslinux/pxelinux.0 /var/lib/tftpboot/computenode08/
touch /var/lib/tftpboot/computenode08/pxelinux.cfg/default

edit boot entry in "/var/lib/tftpboot/computenode08/pxelinux.cfg/default"

# create new
default centos6

label centos6
  kernel centos6/vmlinuz
  append initrd=centos6/initramfs.img root=nfs:10.1.1.12:/var/lib/tftpboot/computenode08/centos6/root rw selinux=0 lowres ip=dhcp

add entry to NFS exports in "/etc/exports"

/var/lib/tftpboot/computenode08/centos6/root 10.1.1.0/24(rw,sync,no_root_squash,no_all_squash)

start / restart NFS and run it on boot

service rpcbind restart
service nfslock restart
service nfs restart
chkconfig rpcbind on
chkconfig nfslock on
chkconfig nfs on

boot computenode with ipmitool and take a look at the console to make sure everything works

ipmitool -H 10.1.0.108 -I lanplus -U ADMIN -P ADMIN chassis power on
ipmitool -H 10.1.0.108 -I lanplus -U ADMIN -P ADMIN sol activate

wait 2 minutes and check if you can reach the node with ssh

ssh root@10.1.1.108

previous attempt

saved computenode12 MAC addresses in a txt file
powered down node 12 and set boot to pxe
powered up node 12 and connected to serial console
MAC of PXE boot attempt matches saved MAC (this is good)
node 12 did not receive DHCP from storagenode (this is bad)
conclusion: dnsmasq is not yet working properly
note: restored boot order on node 12 and started it again (no problems)

previous attempt

use ipmitool to control boot device, power status and get a serial terminal to the node
node does not receive DHCP information :(

previous attempt

installed dnsmasq on storagenode (yes storage)
downloaded CentOS 6.5 netboot image to storagenode:/root
mounted image to storagenode:/mnt/liveiso
configured dnsmasq to act as dhcp server with address space 10.1.1.: .8-.10, .13-16
configured dnsmasq to provide network boot information that is pointing to /mnt/liveiso
started computenode08 via IMPI
waited for something awesome to happen
pinged around for new hosts
stopped computenode08 again

resources

pxe boot

PXE boot nodes over the network

TL;DR version

the long version

do this once

do this for every diskless computenode

previous attempt

previous attempt

previous attempt

resources

ipmitool (awesome)

using dnsmasq to set up PXE boot

dnsmasq manpage, as it could be used to solve the dhcp issues

arch based guide for setting up PXE boot

creating a live CentOS iso

PXE boot specification (very large document)