The Value Cluster (Part 3): Packages and Testing

Article Index

Sun Grid Engine

Since we share kronos, we also chose to install a queuing/scheduling system to avoid stepping on each other's toes. We chose to install Sun Grid Engine. We needed to patch SGE to work with Fedora Core 2, so you can download a newer version from the Kronos download page.

Before we install SGE, we need to install korn-shell (ksh) since SGE requires it. You can grab a fairly recent copy off the web, your Fedora CDs, or simply use yum.

                                                                       
# yum install pdksh

Next install the Grid Engine package that you downloaded from Cluster Monkey:

                                                                          
# rpm -i gridengine-5.3p6-8.caos.i386.rpm

If you are not using FC2, you may want to rebuild the RPM. Source RPMs are available on the Kronos download page as well.

The installation of SGE will print out a bunch of diagnostic code. SGE requires some further customization before it can be used. Fortunately, it includes a script to handle this procedure. Move to the /sge directory and enter the following command.

# ./inst_sge -m -fast
You can select the default answers for most of the questions. When you come to the "Adding Grid Engine hosts" you will be required to supply a file or manually enter the names of the hosts. It is probably convenient to create a list of nodes in file for this purpose (open another terminal to make the list). There is one thing to consider at this point, however. If you want to include the master node as an execution host (meaning it will be used to run programs) then include it in the list (use the name kronos). If you do not, then the list should contain only the seven node names (kronos00-kronos06). After entering your nodes, there are a few more questions to answer (default answers will suffice).

SGE is large package with plenty of documentation. Consult the Resources Sidebar for more information. There is also a graphical font end called qmon that you can use to change some of the settings and add nodes.

Installing SGE on the Nodes

The previous section only installs SGE on the master node (the hard part). We also have to install a small piece of SGE on the compute nodes. However, like many things, it requires some RPMs to be installed first.

SGE on the compute nodes requires the strings command installed on the nodes. This program comes with the package binutils. You can use wget to grab the rpm. {mosgoogle right}

# wget http://download.fedora.redhat.com\
            /pub/fedora/linux/core/2/i386/os/Fedora\
            /RPMS/binutils-2.15.90.0.3-5.i386.rpm

A couple of notes are in order. First, we are not using Yum because we do not want to install this on the master node (it may be installed already). So we want to download the rpm from the Fedora site. Second, check the download site. Fedora Core 2 was about to switch over to the Fedora-Legacy project (which means the URL may change) as this article was being written.

Next we need to install the rpms for binutils and Grid Engine into the Kronos VNFS. Execution nodes use a separate rpm for Grid Engine as they do not need all the files on the head node. Grab the gridengine-node rpm from the Kronos download page. There are a couple of ways to do this, but we're going to use the --root option of the rpm command to install it in the correct location.

# rpm -i --root /vnfs/kronos/ binutils-2.15.90.0.3-5.i386.rpm
# rpm -i --root /vnfs/kronos/ gridengine-node-5.3p6-7.caos.i386.rpm
Note that we have used the --root option for rpm to define a new root for the install path. You can also administer (add/delete/query) the VNFS rpms by using the --root option as well. We have bintuils and SGE for the nodes installed in the Kronos VNFS (getting closer!). To save space on the nodes, the SGE rpm that is installed on the nodes doesn't include the basic SGE commands that we might need. So, we're going to NFS export the directory /sge to the compute nodes.

First, go to the file, /etc/exports on the master node and add the following line to the end of the file.

/sge   10.0.0.0/255.255.255.0(rw,no_root_squash)
Then enter the command exportfs -r which will reexport all of the directories listed in /etc/exports. Next, go to the file /etc/warewulf/vnfs/fstab and uncomment the following line.
#%{sharedfs ipaddr}:/sge  /sge  nfs rw,bg,rsize=8192,wsize=8192 0 0
To uncomment it you just remove the "#" character. We also need to slightly modify what files that get excluded when the VNFS is built (it makes the ramdisk image smaller). This task is accomplished by using a "VNFS excludes" file. We are using the excludes-aggressive to keep out ramdisk size as small as possible. In the file /etc/warewulf/vnfs/excludes-aggressive add the following line.
+ usr/lib/libbfd-2.15.90.0.3.so
After, this line
usr/lib/libpopt.[^a]*
Finally, we're ready to rebuild our Kronos VNFS! As before, enter the commands to rebuild the VNFS, reboot, and distribute the user accounts.

Final SGE Configuration

After the compute nodes come back up (you can use wwtop to check this) and you have used the wwnode.sync command to propagate the accounts to the nodes, go to the /sge/utils directory on the master node. Run the following command.

Sidebar Two: What No Ganglia ?
Ganglia is often associated with administering a cluster. While Ganglia is a great package, it can be a bit resource intensive. We prefer to use lightweight "top like" tools to check on the cluster status. Both wwtop and userstat are great tools to watch the work queue and loads on the cluster with minimum overhead. But, don't worry, we'll provide instruction on how to install Ganglia in the future -- or maybe something better.
# ./install_cluster.sh kronos00 kronos01 \
   kronos02 kronos03 kronos04 kronos05 \
   kronos06
This command will configure SGE to allow kronos00 to konos06 to act as "execution" hosts that will run actual jobs. You can check to see SGE finds all you nodes by running qhost and inspecting the output. If you are not seeing your nodes, check the value cluster web page and the SGE documentation for more information. If you want to allow the master node to also run jobs, then you need to run the following command on the master node.
# ./install_execd -fast -auto
You will also need to install the rsh-server rpm on the master to allow an rsh connections from the nodes. We will address this below.

Install UserStat

If you have every used a batch scheduler, there is usually some kind of qstat (queue status) command. Usually these utilities are command line programs that provide a snap shot of the queue. To make this kind of information more accessible, there is a nice "top like" utility for viewing SGE queue status and processor loads. If you download the userstat rpm from the a href="http://www.clustermonkey.net/download/kronos/">Kronos download page, you only need to install it on the master node (rpm userstat-1.0-2.i386.rpm).

While you are at it, you may wish to download the sge-tests.tgz tar file from the Kronos download page (in the src directory). This file contains some simple test scripts for SGE. Extract the file in a working directory and consult the README.

Userstat is helpful in two ways. First as shown in Figure Two, it provides (in the top of the window) a list of jobs in the queue (running and waiting). The lower window shows the load and memory usage for the cluster. If you highlight a running job in the top window and press enter, only those nodes that are used by the job are shown in the lower window. An example of this this display is shown in Figure Three. Userstat is designed for both administrators and users. Indeed, it provides a very low overhead way for users to watch their jobs.

userstat screen
Figure Two: Example userstat output for whole cluster

Userstat screen
Figure Three: Example userstat output for a single job

    Search

    Login And Newsletter

    Create an account to access exclusive content, comment on articles, and receive our newsletters.

    Feedburner

    Share The Bananas


    Creative Commons License
    ©2005-2012 Copyright Seagrove LLC, Some rights reserved. Except where otherwise noted, this site is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 2.5 License. The Cluster Monkey Logo and Monkey Character are Trademarks of Seagrove LLC.