PXE boot nodes over the network
TL;DR version
follow the following 3 guides (in order) to get a working PXE boot setup. Watch out for pitfalls like iptables and user permission problems though.
http://www.server-world.info/en/note?os=CentOS_6&p=dhcp
http://www.server-world.info/en/note?os=CentOS_6&p=pxe&f=1
http://www.server-world.info/en/note?os=CentOS_6&p=pxe&f=4
the long version
we start a DHCP server on the storagenode that provides PXE boot information (TFTP protocol) to each diskless node. The information is stored on the storagenode as well. The boot information then boots a linux kernel and ramfs. The root filesystem is accessed by the kernel via NFS and is also stored on the storagenode for each individual diskless computenode. DHCP, TFTP and NFS need to be installed and configured. IPTABLES needs new rules to allow DHCP, TFTP and NFS to pass through the firewall. The boot information and filesystem needs be created for each diskless computenode.
do this once
- install prerequisites
yum -y install dhcp syslinux xinetd tftp-server dracut-network
- configure IPTABLES to allow incomming access to DHCP, TFTP and NFS
# DHCP
iptables -I INPUT 1 -s 10.1.1.0/24 -m state --state NEW,RELATED,ESTABLISHED -p udp --dport 67 -j ACCEPT
iptables -I INPUT 1 -s 10.1.1.0/24 -m state --state NEW,RELATED,ESTABLISHED -p udp --dport 68 -j ACCEPT
# TFTP
iptables -I INPUT 1 -s 10.1.1.0/24 -m state --state NEW,RELATED,ESTABLISHED -p udp --dport 69 -j ACCEPT
# NFS
# FIXME NFS does not yet work
iptables -I INPUT 1 -s 10.1.1.0/24 -m state --state NEW,RELATED,ESTABLISHED -p udp --dport 111 -j ACCEPT
iptables -I INPUT 1 -s 10.1.1.0/24 -m state --state NEW,RELATED,ESTABLISHED -p tcp --dport 111 -j ACCEPT
iptables -I INPUT 1 -s 10.1.1.0/24 -m state --state NEW,RELATED,ESTABLISHED -p tcp --dport 2049 -j ACCEPT
iptables -I INPUT 1 -s 10.1.1.0/24 -m state --state NEW,RELATED,ESTABLISHED -p tcp --dport 32803 -j ACCEPT
iptables -I INPUT 1 -s 10.1.1.0/24 -m state --state NEW,RELATED,ESTABLISHED -p udp --dport 32769 -j ACCEPT
iptables -I INPUT 1 -s 10.1.1.0/24 -m state --state NEW,RELATED,ESTABLISHED -p tcp --dport 892 -j ACCEPT
iptables -I INPUT 1 -s 10.1.1.0/24 -m state --state NEW,RELATED,ESTABLISHED -p udp --dport 892 -j ACCEPT
iptables -I INPUT 1 -s 10.1.1.0/24 -m state --state NEW,RELATED,ESTABLISHED -p tcp --dport 875 -j ACCEPT
iptables -I INPUT 1 -s 10.1.1.0/24 -m state --state NEW,RELATED,ESTABLISHED -p udp --dport 875 -j ACCEPT
iptables -I INPUT 1 -s 10.1.1.0/24 -m state --state NEW,RELATED,ESTABLISHED -p tcp --dport 662 -j ACCEPT
iptables -I INPUT 1 -s 10.1.1.0/24 -m state --state NEW,RELATED,ESTABLISHED -p udp --dport 662 -j ACCEPT
- prepare directories for all diskless nodes
mkdir -p /var/lib/tftpboot/computenode08/centos6/root/
mkdir -p /var/lib/tftpboot/computenode08/pxelinux.cfg
...
- configure "/etc/dhcp/dhcpd.conf" to include PXE boot information
# Uni Ulm DNS server
option domain-name-servers 134.60.1.111;
# default lease time
default-lease-time 600;
# max lease time
max-lease-time 7200;
# this DHCP server to be declared valid
authoritative;
# specify network address and subnet mask
subnet 10.1.1.0 netmask 255.255.255.0 {
# specify broadcast address
option broadcast-address 10.1.1.255;
# specify default gateway
option routers 10.1.1.11;
# add fixed hosts here
host computenode08 {
# MAC that distinguishes the computenode
hardware ethernet 00:1E:67:39:84:03;
fixed-address 10.1.1.108;
option host-name "computenode08";
ddns-hostname "computenode08";
# server that hosts the TFTP boot information
next-server 10.1.1.12;
# path from the TFTP root to the bootstrap kernel
filename "computenode08/pxelinux.0";
}
# add more hosts here
...
}
- start / restart the DHCP server and run it on boot
service dhcpd restart
chkconfig dhcpd on
- configure TFTP configuration in "/etc/xinetd.d/tftp"
service tftp
{
socket_type = dgram
protocol = udp
wait = yes
user = root
server = /usr/sbin/in.tftpd
server_args = -s /var/lib/tftpboot
disable = no
per_source = 11
cps = 100 2
flags = IPv4
}
- start / restart TFTP server and run it on boot
service xinetd restart
chkconfig xinetd on
do this for every diskless computenode
- prepare root filesystem for server
yum -y --releasever=6.5 groupinstall "Base" "Server Platform" --installroot=/var/lib/tftpboot/computenode08/centos6/root/
yum -y install openssh-server openssh-clients --installroot=/var/lib/tftpboot/computenode08/centos6/root/
- set root password hash in "/var/lib/tftpboot/computenode08/centos6/root/etc/shadow" from current hash
...
root:$6$xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx:14937:0:99999:7:::
...
- set mount points in "/var/lib/tftpboot/computenode08/centos6/root/etc/fstab"
none /tmp tmpfs defaults 0 0
tmpfs /dev/shm tmpfs defaults 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
- make sure networking is enabled
echo "NETWORKING=yes" >> /var/lib/tftpboot/computenode08/centos6/root/etc/sysconfig/network
- edit interface definitions for both interfaces in /var/lib/tftpboot/computenode08/centos6/root/etc/sysconfig/network-scripts/ifcfg-ethX (create files if necessary)
DEVICE="ethX"
BOOTPROTO="dhcp"
IPV6INIT="no"
NM_CONTROLLED="no"
ONBOOT="yes"
TYPE="Ethernet"
- copy misc. config files you want or need into the node's fs
cp /root/.bashrc /root/.bash_profile /var/lib/tftpboot/computenode08/centos6/root/root/
- generate kernel and ramfs
cp /boot/vmlinuz-2.6.32-431.29.2.el6.x86_64 /var/lib/tftpboot/computenode08/centos6/vmlinuz
dracut /var/lib/tftpboot/computenode08/centos6/initramfs.img 2.6.32-431.29.2.el6.x86_64
chmod 644 /var/lib/tftpboot/computenode08/centos6/initramfs.img
- prepare bootstrap information
cp /usr/share/syslinux/pxelinux.0 /var/lib/tftpboot/computenode08/
touch /var/lib/tftpboot/computenode08/pxelinux.cfg/default
- edit boot entry in "/var/lib/tftpboot/computenode08/pxelinux.cfg/default"
# create new
default centos6
label centos6
kernel centos6/vmlinuz
append initrd=centos6/initramfs.img root=nfs:10.1.1.12:/var/lib/tftpboot/computenode08/centos6/root rw selinux=0 lowres ip=dhcp
- add entry to NFS exports in "/etc/exports"
/var/lib/tftpboot/computenode08/centos6/root 10.1.1.0/24(rw,sync,no_root_squash,no_all_squash)
- start / restart NFS and run it on boot
service rpcbind restart
service nfslock restart
service nfs restart
chkconfig rpcbind on
chkconfig nfslock on
chkconfig nfs on
- boot computenode with ipmitool and take a look at the console to make sure everything works
ipmitool -H 10.1.0.108 -I lanplus -U ADMIN -P ADMIN chassis power on
ipmitool -H 10.1.0.108 -I lanplus -U ADMIN -P ADMIN sol activate
- wait 2 minutes and check if you can reach the node with ssh
ssh root@10.1.1.108
previous attempt
- saved computenode12 MAC addresses in a txt file
- powered down node 12 and set boot to pxe
- powered up node 12 and connected to serial console
- MAC of PXE boot attempt matches saved MAC (this is good)
- node 12 did not receive DHCP from storagenode (this is bad)
- conclusion: dnsmasq is not yet working properly
- note: restored boot order on node 12 and started it again (no problems)
previous attempt
- use ipmitool to control boot device, power status and get a serial terminal to the node
- node does not receive DHCP information :(
previous attempt
- installed dnsmasq on storagenode (yes storage)
- downloaded CentOS 6.5 netboot image to storagenode:/root
- mounted image to storagenode:/mnt/liveiso
- configured dnsmasq to act as dhcp server with address space 10.1.1.: .8-.10, .13-16
- configured dnsmasq to provide network boot information that is pointing to /mnt/liveiso
- started computenode08 via IMPI
- waited for something awesome to happen
- pinged around for new hosts
- stopped computenode08 again
resources
ipmitool (awesome)
https://validation.linaro.org/static/docs/ipmi-pxe-deploy.html
using dnsmasq to set up PXE boot
http://serverfault.com/questions/415857/can-isc-dhcpd-operate-as-a-proxy-dhcp-server-for-pxe-boot
dnsmasq manpage, as it could be used to solve the dhcp issues
http://www.thekelleys.org.uk/dnsmasq/docs/dnsmasq-man.html
arch based guide for setting up PXE boot
https://wiki.archlinux.org/index.php/PXE
creating a live CentOS iso
https://projects.centos.org/trac/livecd/wiki/PxeBoot
PXE boot specification (very large document)
http://download.intel.com/design/archives/wfm/downloads/pxespec.pdf