ryanj

Alright, this blog is as much about me as a DJ as it is about what’s going on in my life, so in following with the open-source “I help you if you help me” movement, I have a post that’s a how-to for getting a PXE server running with support for Clonezilla and ESX (vSphere) 4.  (For those of you who read this on facebook, it gets auto-uploaded there after I post it on my main site at blog.thewulph.com – that’s why you’re seeing it). Also keeping in mind that the steps presented could be modified for just about anything except a Windows boot – but there’s plenty of resources for THAT out there as well (if you’re interested, look into WinPE – it’s designed for Windows booting over PXE).

Ok, so why PXE?  The simple answer is that in my course development for SAIT, I have 8 HP blade servers that need to have ESX 4 installed on them.  But wait, there’s more! Each blade needs to have 6 different copies of ESX on it (because there’s 6 different lab groups).  But wait, there’s more! The con-ed students will also have access to those blades (although for what we don’t yet know) and we’ll likely have to reimage them at some point – that’s what Clonezilla is for.  But wait, there’s more! Because they’re blades, they have only a dongle out of which the connection points for each server connect (and there’s only one dongle).  Each dongle has 2 USB ports, a video port, a network port, and a serial port (yes, I could use HP’s iLo, but I don’t have it set up yet).  The problem is that the little CDROM that I have to install things off of needs 2 USB ports – so where do I plug in my keyboard??? (Yes, I could get a powered hub, but that’s INCREDIBLY stupid when you’re in the datacenter) – that’s what PXE booting ESX is for. (*update* – this isn’t quite how I’m doing it anymore, but the PXE boot is still very useful so I still implemented it.)

Anyway, history aside, let’s get started.

This process has a few steps:

1. Get the base OS installed on our PXE server, if it isn’t already
2. Set up the base OS for our application
3. Install the necessary software
4. Set up the tftp daemon
5. Put SYSLINUX in the right place
6. Set up the PXE boot menu
7. Get the software to be PXE booted in the right place and set up NFS
8. Set up your DHCP server to hand out the right information

---

1. Get the base OS installed for the PXE server.

I used a Debian Lenny (5) machine, which is my Linux distro of choice.  I installed it into a virtual machine, and gave the machine 256mb of RAM (you could give it more if you like), a 4gig main drive and a 300gig secondary drive. I mounted /var on the second drive, the rest were all mounted on the 4 gig drive.

I always use the Debian netinstall disc (since I want only a minimal system anyway).  I’m not going to go through installing it, because there’s tons of good guides out there for that.  You can install however you like, however I strongly recommend that you install only a minimal system (uncheck EVERYTHING (yes, everything, including “Standard System” when the task selector menu comes up – that’s the one that asks if you’d like to have a web server, etc. installed).

I’m also not going to go over the basics of PXE, as there’s plenty of good resources out there for that too. If you’re looking at this, chances are you already know what it does and why you want it.

Once you’ve got everything installed (with a GOOD password, right?), update/upgrade the system:

apt-get update
apt-get upgrade

---

2. Set up the Base OS:

Pretty much the only thing we have to do is make sure that the system has a static IP that’s valid for our network. In my case, this happens to be 192.168.2.109 /24. Note that there’s plenty of good guides out there (as always) for setting up the network, so I won’t go over the steps.

Note: You could also take the time to set up sudo (which would have to be installed first) and the sudoers file properly for a non-root-named account. This would be a VERY GOOD IDEA (as would preventing root logins from SSH) but I’ll leave it to you to do if you care about the security of your system.

---

3. Install the necessary software

Now, we’ll install the software we’ll need for the whole thing, including ssh server, dhcp server, tftp server, and nfs server:

apt-get install ssh dhcp3-server tftpd-hpa openbsd-inetd nfs-kernel-server nfs-common portmap

---

4. Set up the tftp daemon
Make sure that the tftp daemon is running:

netstat -uap | grep tftp

And if you see something like this:

udp 0 0 *:tftp *:* 2242/inetd

You’re in business. If not, check the /etc/inetd.conf file and make sure the following lines exist:

[...]
#:BOOT: TFTP service is provided primarily for booting. Most sites
# run this only on machines acting as "boot servers."
tftp dgram udp wait root /usr/sbin/in.tftpd /usr/sbin/in.tftpd -s /var/lib/tftpboot
[...]
(Note that the line beginning with “tftp” and ending in “/var/lib/tftpboot” should be ONE LINE. Excuse the wordpress formatting please…)

If not, add it and restart the inetd service:

/etc/init.d/openbsd-inetd restart

And you should be in business.

---

5. Put SYSLINUX in the right place

SYSLINUX is essentially the files we need to give the system you’re booting off the network a basic operating system so that we can have access to the keyboard, screen, and network. Think of it like a boot diskette (in fact, you could build a boot diskette using syslinux).

Make sure you have the /var/lib/tftpboot directory:

mkdir /var/lib/tftpboot

This will be the tftp “root”. When something gets requested via tftp, it MUST be in this directory or a subdirectory. Everything related to tftp is relative to here.

Get SYSLINUX. We really only need 3 files from the package, but it’s better to download the whole thing:

cd
wget http://www.kernel.org/pub/linux/utils/boot/syslinux/syslinux-3.82.tar.gz
tar -xvzf syslinux-3.82.tar.gz


As you can see, I used version 3.82 of syslinux. You may need to adjust the link if that’s not available anymore.

We’re going to copy the required files into specific places in the /var/lib/tftpboot directory. pxelinux.0 must be in the tftpboot root (aka /var/lib/tftpboot), but the other 2 we’ll put in a different directory (/var/lib/tftpboot/pxelinux) for cleanliness’ sake.

mkdir /var/lib/tftpboot/pxelinux
cd
cp syslinux-3.82/core/pxelinux.0 /var/lib/tftpboot
cp syslinux-3.82/com32/menu/menu.c32 /var/lib/tftpboot/pxelinux
cp syslinux-3.82/com32/modules/chain.c32 /var/lib/tftpboot/pxelinux


For the sake of completeness, what does each file do?

Well, the pxelinux.0 file is basically the boot loader. It provides a very minimal operating system so that the user can interact with the system to choose what they want to do. menu.c32 is an application that makes pretty menus, and chain.c32 is a chainloader that allows you to add the option to “Boot from first hard disk”, among MANY other (very powerful) things.

---

6. Set up the PXE boot menu

Whew, we’re getting there. Just a few more steps.

Next, we’ll set up the menu for the boot options.

First, set up the directory structure required:

mkdir /var/lib/tftpboot/pxelinux.cfg

And then edit the /var/lib/tftpboot/pxelinux.cfg/default file, which contains the actual boot menu definitions.

nano /var/lib/tftpboot/pxelinux.cfg/default

Here’s what mine looks like:


# Default boot option to use
DEFAULT pxelinux/menu.c32
# Prompt user for selection
PROMPT 0
# Set a timeout of 5 seconds
TIMEOUT 50
# when it times out, run the boot-disk section
ONTIMEOUT boot-disk
# Menu Configuration
MENU WIDTH 80
MENU MARGIN 10
MENU PASSWORDMARGIN 3
MENU ROWS 12
MENU TABMSGROW 18
MENU CMDLINEROW 18
MENU ENDROW 24
MENU PASSWORDROW 11
MENU TIMEOUTROW 20
MENU TITLE H403 Lab PXE Boot
# menu definitions
# Return to Main Menu
LABEL MainMenu
MENU DEFAULT
MENU LABEL ^Main Menu
KERNEL pxelinux/menu.c32
#ESX/Vsphere 4
LABEL esx-install
MENU LABEL ^Install ESX 4.0
KERNEL esx/vmlinuz
APPEND initrd=esx/initrd.img vmkopts=debugLogToSerial:1 mem=512M askmedia
#Clonezilla
LABEL clonezilla
MENU LABEL Clonezilla Live
KERNEL clonezilla/vmlinuz1
APPEND initrd=clonezilla/initrd1.img boot=live union=aufs noswap noprompt vga=788 ocs_live_keymap="NONE" ocs_lang="en_US.UTF-8" fetch=tftp://192.168.2.109/clonezilla/filesystem.squashfs
#boot first hard drive
LABEL boot-disk
MENU LABEL ^Boot First Hard Disk
COM32 pxelinux/chain.c32
APPEND hd0 0

Note that any APPEND lines should all be on one line. Excuse WP formatting again please…

I leave it to you to read up on the format of the menu.c32 default file, but there’s a few things I will point out:

• Although it’s not obvious, there are paths in the DEFAULT, KERNEL, APPEND and COM32 lines. They are all relative to /var/lib/tftpboot; for example, esx/vmlinuz (from the ESX KERNEL defintion) is actually /var/lib/tftpboot/esx/vmlinuz. We’ll be placing these files and setting this up in the next steps.
• Options are all specific to what is being booted. Do not take these options as being relevant for anything other than what’s presented here. Most things will be similar, and will have some documentation available out there if you’re network booting, as it seems to be a common thing
• The ONTIMEOUT directive seems to be in 10ths of a second. i.e. 50 is actually 5 seconds.

---

7a. Get the software to be PXE booted in the right place

Obviously, we need the software we are going to boot. We’re going to place this in the /var/lib/tftpboot “root” we have created. You COULD put all the software in this “root” directory if you like; but since we’re booting more than one option of software, I’ve made some subdirectories; notably esx and clonezilla:

cd /var/lib/tftpboot
mkdir esx
mkdir clonezilla

Grab Clonezilla from http://sourceforge.net/projects/clonezilla/files/clonezilla_live_stable/clonezilla-live-1.2.2-31.zip/download; ESX you’ll have to have for yourself.

Now, we move the appropriate files from each piece of software into the appropriate subdirectory.
In the case of Clonezilla, we need: live/vmlinuz1, live/initrd1.img, and live/filesystem.squashfs from the Clonezilla zip file. Put these inside /var/lib/tftpboot/clonezilla. These are all the files you need to run Clonezilla over the network.

For ESX, you’ll need to copy vmlinuz and initrd.img from the /isolinux directory on the ESX DVD into /var/lib/tftpboot/esx.

That’s it for the basics. Clonezilla is ready to run; however you’ll probably need a place to store your images on the network. Clonezilla can write to nearly anything (including USB attached hard drives/flash drives, NTFS (Windows) network shares, and my favorite: NFS shares), so we’ll set up an NFS share on the network that is designed for storing all sorts of things.

---

7b. Set up NFS

NFS is NOT required for MOST PXE environments. In this case, it is, as my NFS repository will hold the RPMs from the ESX install as well as the images for my blades. It seems to be a good thing to set up (at least in Linux environments – if you were PXE booting Windows, you might set up SAMBA in place of (or as well as) NFS.

You can skip this section if you don’t need a network-available repository for source files (such as RPMs). You might also look into setting up an HTTP or FTP server to do the same; most products give you the option to use one of those as well.

Our server will also be holding the disk images of the various servers from Clonezilla, so we’ll set it up to do that as well.

Create the directories required:

mkdir /var/nfs
mkdir /var/nfs/esx4
mkdir /var/nfs/images

Open up your /etc/exports file and add the following 2 lines:

/var/nfs/esx4 *(rw,no_root_squash)
/var/nfs/images *(rw,no_root_squash)

Restart your NFS daemons:

/etc/init.d/nfs-common restart
/etc/init.d/nfs-kernel-server restart

Verify that your directories are exported properly:

exportfs

If you see /var/nfs/esx4 and /var/nfs/images in there exported as , you’re in business.

---

7c. Put the ESX RPMs in the NFS repository

I only include this because it “gotcha’d” me when I did it; I didn’t copy the whole directory and so ended up with failed ESX installs. Unfortunately the ESX installer wasn’t intelligent enough (in graphical mode) to tell me what was going on; a colleague switched into text mode console and that’s where we discovered the error.

Anyway, the lesson is that you need to make sure you copy EVERYTHING from the ESX DVD (excluding the isolinux directory), maintaining the directory structure, into the /var/nfs/esx4 directory. Without ALL of the files (although it seems as though the most important are the packages.xml and VMWare directories), it won’t boot right and will give you NO indication as to why it failed.

---

8. Set up DHCP to hand out the right information

You have two options for DHCP: one, use your current DHCP server and modify it to hand out the PXE booting information, or two, use the one we’ll set up here.

The beauty of using the dhcpd3 service in linux (as opposed to the admittedly MUCH easier to use DHCP server service in Windows Server, for example) is that you have the ability to define groups of hosts that will boot via PXE. This is kinda cool; you can exclude (or include) a series of hosts from ever getting the PXE boot menu. For example, you could have JUST your lab subnet boot via PXE (say, for deployment purposes) and no one else would ever see the prompt. While you can do this in Windows, the Linux version even allows you to be as selective as MAC groups – for example, anything with the first 6 characters of the MAC belonging to an Intel NIC should PXE boot; anything else shouldn’t, etc – the dhcpd3 server is VERY powerful.

I’m not going to go into that, since there’s plenty of really good info out there already on the subject, and I didn’t use it on my server (although I think I will, to prevent VM’s in the network from accidentally PXE booting something they’re not supposed to).

---

8a. Linux DHCP

Since we installed dhcpd3-server as part of the software installs above, all we need to do is define a scope within /etc/dhcp3/dhcpd.conf:


option domain-name-servers 192.168.2.201;
option domain-name "lab.local";
default-lease-time 86400;
max-lease-time 604800;
authoritative;

subnet 192.168.2.0 netmask 255.255.255.0 {
range 192.168.2.1 192.168.2.99;
filename "pxelinux.0";
next-server 192.168.2.109;
option subnet-mask 255.255.255.0;
option routers 192.168.2.254;
}


Most of the file is self-explanatory, but the important lines are the filename "pxelinux.0"; and next-server 192.168.2.109; lines; they define the boot loader you’re going to use (filename) as well as where to get it (next-server – the IP address of your PXE server). Adjust IP addresses and anything else as necessary.

Restart dhcpd to ensure the new configuration is active:

/etc/init.d/dhcp3-server restart

And you should be good to go.

---

8b. Windows DHCP

Windows DHCP is simpler, but the option is not obvious. Open up your dhcp service, find the scope you’re interested in (or use the “server options”) and open up the scope options.

The options you’re interested in are option 066: Boot Server Host Name (put the IP of your PXE server here) and 067: Bootfile Name (put pxelinux.0 here).

Done!

---

Whew. You should now have a fully functional PXE booting server! Good luck!

*Note* I obviously didn’t run through the steps of ACTUALLY booting over the network. I’ll assume you are all smart enough to figure that one out on your own for your particular hardware.

The beauty of the PXE system that I’ve set up is that if you don’t select something within 5 seconds, it skips out and boots the first hard disk rather than just sitting there stupidly. You might have to modify the boot-disk entry in /var/lib/tftpboot/pxelinux.cfg/default to point somewhere else if you don’t want that to happen.

---

-r-

Credit where it’s due:
I borrowed from lots of sources for this, so I’ll attempt to credit them all (and fail miserably).
http://www.howtoforge.com/setting-up-a-pxe-install-server-for-multiple-linux-distributions-on-debian-lenny
http://www.howtoforge.com/nfs-server-and-client-debian-etch
http://www.vmware.com/pdf/vsphere4/r40/vsp_40_esx_vc_installation_guide.pdf