Diskless Beowulf cluster

The idea for this cluster is to have a modification of a standard Linux distribution and to avoid having a complete copy of the operating system for each node stored on the file server.  Instead, for each client, the file server has separate copies of the directories needed for startup which are sent out at the initial startup and a unique copy of /usr is mounted later in the startup process.  (In fact, some older hard drives are used as local swap and /tmp.) 

Hardware configuration

ItemQuantity
MSI K8D motherboard 8
AMD Opteron 240 CPU 16
ATP PC2700 512 MByte Registered ECC Memory32
Tripp-Lite SmartPro 3000VA UPS1
D-Link DGS-1008T gigabit ethernet switch2
Enermax EGS465AX-VE PFC power supply8
Seagate Barracuda 40 Gbyte hard drive4
Generic mid-tower ATX case8
Enermax case fan8
The total cost for the system was about $16,000.  We also purchased 6 monitors and 8 data acquisition cards since the computers will serve as workstations in the freshman physics lab.  Funds for the system were provided by Geneva College and NSF grant PHY-0200060. 

Network

The network configuration consists of two parallel Gigabit Ethernets.  Currently, one network is used to serve the operating system and files and the other network is used for running parallel code.  We would have liked to try channel bonding but that seems to impractical for our setup. 

Operating system configuration

Initially, the operating system was GinGin64, a 64-bit version of RedHat 8.  Advice given in Diskless Linux by Chip Coldwell and Computer Network Booting by Colby Kraybill, et al. were most helpful.  Set-up details are recorded in a rather sketchy text file. 

During August 2004, we updated the system to Fedora Core 2 for x86-64.  In the following, instructions are given for a new install of the operating system and for recovery from a hard drive failure.  In both cases, one one starts by a standard install of the operating system on a new hard drive.  These are the options chosen during the installation:

On reboot:

Recovery from a disk failure

If you are recovering from a hard drive failure, you can now finish installation by untarring the needed files from the backup disk.  These were created by bdump.  See info tar for background on incremental backups.

New install

For a new install, you need to modify a number of files: Then lots of little stuff: To serve the kernel out to the clients, we will use Syslinux.  Download Syslinux from www.kernel.org. 

File system for the first client

We will first set up the file system for the first client barak.  
mkdir /export; mkdir /export/barak 
cd /export/barak  # work in this directory!
For now on, we assume this is the working directory. 
mkdir usr;  mkdir home
mkdir usr/bin; mkdir tmp; mkdir proc
mkdir usr45
cp -a /dev . ; cp -a /etc .  ; cp -a /var . 
cp -a /bin . ; cp -a /sbin . ; cp -a /root .
cp -a /mnt . ; cp -a /lib .  ; cp -a /lib64 .
chomod a+w tmp # Else X doesn't start!
echo "ServerName jael" >> /etc/cups/client.conf
Turn off ip-forwarding in etc/sysctl.conf.  Remove any unneeded kernel modules from lib/modules.  The startup script /etc/rc.d/rc.sysinit uses /bin/bash.  However, /bin/bash needs two /user/bin files to run: 
cp -a /usr/bin/id usr/bin
cp -a /usr/bin/dircolors usr/bin
The following init scripts were modified: see the older instructions for more information.  Turn off extra services nfs, tftp, dhcpd on the clients by switching the root /usr/sbin/chroot /export/barak and using /sbin/chkconfig ... off.  You can use chroot to verify that the client filesystem has been set up properly. 

Kernel for diskless nodes

The diskless nodes will need a custom kernel.  I used kernel version 2.6.7.  Download the kernel from www.kernel.org and put in /usr/src/linux-2.6.7-diskless.  One can start with the Fedora config file: copy the appropriate config in /usr/src/linux*/configs to /usr/src/linux-2.6.7-diskless/.config.  Start up the configuration script and set: 
Processor type and features
  2 CPUs
Device Drivers
  Block devices
    N Compaq SMART (compile error)
  Networking support
    Network device support
      Ethernet (100 Mbit)
        Y 3COM 3C590 etc. support
      Ethernet (1000 Mbit)
        Y Broadcom Tigon 3 support
      Wireless LAN
        N Wireless LAN (otherwise, a module compile error)
      WAN 
        N WAN (otherwise, module compile error)
    Networking options
      N  IP: multicasting (cruft)
      N  IP: advanced router (cruft)
      Y  IP: kernel level autoconfiguration
      Y  IP: DHCP support
  Character devices
    I2C support
      M I2C support
      M I2C PCF 8584 interfaces
      M I2C device interface
  Turned off sound support (due to a module compile error)
File systems
  Y ext3 filesystem support
  Network file systems
    Y NFS file system support
    Y Root file system
Make using make -j4.  Now we can install the kernel and the modules: 
make -j 4 modules_install INSTALL_MOD_PATH=/export/barak
cp System.map /export/barak/System.map
cp arch/x86_64/boot/bzImage /tftpboot/vmlinuz

Copy files for other clients

The file system has now been set up for the client barak.  It is relatively easy to setup the file system for the other clients.  In the /export directory, copy over files for each node.  When copying over, one needs to modify the files etc/fstab and etc/sysconfig/network-scripts/ifcfg-eth?.  The files /etc/dhcpd.conf and /etc/hosts should include entries for each client node. 
Brett van de Sande,
Homepage: http://www.geneva.edu/~bvds and
E-mail: bvds@pitt.edu
Valid HTML 4.01!