• 1. Introduction
• 2. The Anatomy of a Batch Script
• 2.1. Specify Your Shell
• 2.2. Required PBS Directives
• 2.2.1. Number of Cores
• 2.2.2. How Long to Run
• 2.2.3. Which Queue to Run In
• 2.2.4. Your Project ID
• 2.3. The Execution Block
• 3. Submitting Your Job
• 4. Simple Batch Script Example
• 5. Job Management Commands
• 6. Optional PBS Directives
• 6.1. Job Identification Directives
• 6.2. Job Environment Directives
• 6.3. Reporting Directives
• 6.3.1. Redirecting Stdout and Stderr
• 6.3.2. Setting up E-mail Alerts
• 6.4. Job Dependency Directives
• 7. Environment Variables
• 7.1. PBS Environment Variables
• 7.2. Other Important Environment Variables
• 8. Example Scripts
• 8.1. MPI Script
• 8.2. MPI Script (accessing more memory per process)
• 8.3. OpenMP Script
• 8.4. SHMEM Script
• 8.5. Hybrid MPI/OpenMP Script

1. Introductionto top

On large-scale computers, many users must share available resources. Because of this, you cannot just log on to one of these systems, upload your programs, and start running them. Essentially, your programs (called batch jobs) have to "get in line" and wait their turn, and there is more than one of these lines (called queues) for them to wait in. Some queues have a higher priority than others (like the express checkout at the grocery store). The queues available to you are determined by the projects that you are involved with.

The jobs in the queues are managed and controlled by a batch queuing system, without which, users could overload systems, resulting in tremendous performance degradation. The queuing system will run your job as soon as it can while still honoring the following:

At Navy DSRC, we use the PBS Professional queuing system. The PBS module should be loaded automatically for you at login, allowing you access to the PBS commands.

2. The Anatomy of a Batch Scriptto top

A batch script is simply a small text file that can be created with a text editor such as vi or notepad. You may create your own from scratch, or start with one of the sample batch scripts available in $SAMPLES_HOME. Although the specifics of a batch script will differ slightly from system to system, a basic set of components are always required, and a few components are just always good ideas. The basic components of a batch script must appear in the following order:

Note: PBS does not handle ^M characters well. Scripts created on a MS Windows system should be transferred to the HPC systems in ASCII mode, or else use dos2unix to convert the file before use.

2.1. Specify Your Shell

First of all, remember that your batch script is a script. It's a good idea to specify which shell your script is written in. Unless you specify otherwise, PBS will use your default login shell to run your script. To tell PBS which shell to use, start your script with a line similar to the following, where shell is either bash, sh, ksh, csh, or tcsh:

#!/bin/shell

2.2. Required PBS Directives

The next block of your script will tell PBS about the resources that your job needs by including PBS directives. These directives are actually a special form of comment, beginning with "#PBS". As you might suspect, the # character tells the shell to ignore the line, but PBS reads these directives and uses them to set various values. IMPORTANT!! All PBS directives MUST come before the first line of executable code in your script, otherwise they will be ignored.

Every script must include directives for the following:

PBS also provides additional optional directives. These are discussed in Optional PBS Directives, below.

Note: While it is technically possible to specify PBS directives via the command line, we strongly suggest that you include them in your batch script instead. Doing so minimizes the opportunity for typing errors, allows you to see what you submitted later, and allows us to more easily assist you if something goes wrong.

2.2.1. Number of Cores

Before PBS can schedule your job, it needs to know how many cores are required. However, the number of cores that you request is not necessarily the number that you will get. On our systems, PBS cannot allocate less than a complete node. So, for example, if you request 1 core, your request will be rounded up to the nearest complete node. You should keep this in mind, because your project's allocation will be charged for the full node, not just 1 core. Because of this, it is wise to make sure that your job utilizes all of the resources of the full node when it runs. Otherwise, the extra cores are wasted.

To specify the number of cores, include the following directive, where N is the number of cores you are requesting:

#PBS -l mppwidth=N

2.2.2. How Long to Run

Next, PBS needs to know how long your job will run. For this, you will have to make an estimate. There are three things to keep in mind.

To specify how long your job will run, include the following directive:

#PBS -l walltime=HHH:MM:SS

2.2.3. Which Queue to Run In

Now, PBS needs to know which queue you want your job to run in. Your options here are determined by your project. Most users only have access to the debug, standard, and background queues. Other queues exist, but access to these queues is restricted to projects that have been granted special privileges due to urgency or importance, and they will not be discussed here. As their names suggest, the standard and debug queues should be used for normal day-to-day and debugging jobs. The background queue, however, is a bit special because although it has the lowest priority, jobs that run in this queue are not charged against your project allocation. Users may choose to run in the background queue for several reasons:

To see the list of queues available on the system, type the "show_queues" command. To specify the queue you want your job to run in, include the following directive:

#PBS -q queue_name

2.2.4. Your Project ID

PBS now needs to know which project ID to charge for your job. You can use the show_usage command to find the projects that are available to you and their associated project IDs. In the show_usage output, project IDs appear in the column labeled "Subproject." Note: Users with access to multiple projects should remember that the project they specify may limit their choice of queues.

To specify the Project ID for your job, include the following directive:

#PBS -A Project_ID

2.3. The Execution Block

Once the PBS directives have been supplied, the execution block may begin. This is the section of your script that contains the actual work to be done. A well written execution block will generally contain the following stages:

3. Submitting Your Jobto top

Once your batch script is complete, you will need to submit it to PBS for execution using the qsub command. For example, if you have saved your script into a text file named run.pbs, you would type "qsub run.pbs".

Occasionally you may want to supply one or more directives directly on the qsub command line. Directives supplied in this way override the same directives if they are already included in your script. The syntax to supply directives on the command line is the same as within a script except that #PBS is not used. For example:

qsub -l walltime=HHH:MM:SS run.pbs

4. Simple Batch Script Exampleto top

The batch script below contains all of the required directives and common script components discussed above.

#!/bin/bash -l ##Specify your shell
## Required PBS Directives --------------------------------------
#PBS -A NavyS12345ABC
#PBS -q standard
#PBS -l mppwidth=64
#PBS -l walltime=12:00:00
##Optional PBS Directives --------------------------------------
#PBS -j oe
#PBS -V
#PBS -S /bin/bash
#
## Execution Block ----------------------------------------------
# Environment Setup
# cd to your scratch directory in /work
cd $WORKDIR
#
# create a job-specific subdirectory based on JOBID and cd to it
JOBID=`echo $PBS_JOBID | cut -d '.' -f 1`
if [ ! -d $JOBID ]; then
  mkdir -p $JOBID
fi
cd $JOBID
#
# Launching
# copy executable from $HOME and submit it
cp $HOME/my_prog.exe .
aprun -n 64 ./my_prog.exe > my_prog.out
#
# Clean up
# archive your results
# Using the "here document" syntax, create a job script
# for archiving your data.
cd $WORKDIR
rm -f archive_job
cat > archive_job <<END
#!/bin/bash -l
#PBS -l walltime=06:00:00
#PBS -q transfer
#PBS -A Project_ID
#PBS -j oe
#PBS -S /bin/bash
cd $WORKDIR/$JOBID
rsh $ARCHIVE_HOST mkdir $ARCHIVE_HOME/$JOBID
rcp -r $JOBID $ARCHIVE_HOST:$ARCHIVE_HOME/
rsh $ARCHIVE_HOST ls -l $ARCHIVE_HOME/$JOBID
# Remove scratch directory from the file system.
rm -rf $JOBID
END

# Submit the archive job script.
qsub archive_job

5. Job Management Commandsto top

The table below contains commands for managing your jobs in PBS.

6. Optional PBS Directivesto top

In addition to the required directives mentioned above, PBS has many other directives, but most users will only use a few of them. Some of the more useful directives are listed below.

6.1. Job Identification Directives

Job identification directives allow you to identify characteristics of your jobs. These directives are voluntary, but strongly encouraged. The following table contains some useful job identification directives.

-l application - allows you to identify the application being used by your job. To use this directive, add a line in the following form to your batch script:

#PBS -l application=application_name
or
qsub -l application=application_name

-N - allows you to designate a name for your job. In addition to being easier to remember than a numeric job ID, the PBS environment variable, $PBS_JOBNAME, inherits this value and can be used instead of the job ID to create job-specific output directories. For example:

#PBS -N job_20
or
qsub -N job_20...

6.2. Job Environment Directives

Job environment directives allow you to control the environment in which your script will operate. The following table contains a few useful job environment directives.

-I - allows you to request an interactive batch shell. Within that shell, you can perform normal Unix commands, including launching parallel jobs. To use "-I", append it to the end of your qsub request. For example:

qsub -A NAVYS12345ABC -q debug -l mppwidth=64 -l walltime=01:00:00 -I

-V - tells PBS to export all of the environment variables from your login environment into your batch environment. For example:

#PBS -V
or
qsub -V ...

-v - tells PBS to export specific environment variables from your login environment into your batch environment. For example:

#PBS -v DISPLAY
or
#PBS -v my_variable=my_value
or
qsub -v DISPLAY ...
or
qsub -v my_variable=my_value ...

6.3. Reporting Directives

Reporting directives allow you to control what happens to standard output and standard error messages generated by your script. They also allow you to specify e-mail options to be executed at the beginning and end of your job.

6.3.1. Redirecting Stdout and Stderr

By default, messages written to stdout and stderr are captured for you in files named x.ojob_id and x.ejob_id, respectively, where x is either the name of the script or the name specified with the "-N" directive, and job_id is the ID of the job. If you want to change this behavior, the "-o" and "-e" directives allow you to redirect stdout and stderr messages to different named files. The "-j" directive allows you to combine stdout and stderr into the same file.

6.3.2. Setting up E-mail Alerts

Many users want to be notified when their jobs begin and end. The "-mb" and "-me" directives make this possible.

For example:

#PBS -m be
#PBS -M joesmith@gmail.com,joe.smith@us.army.mil

6.4. Job Dependency Directives

Job dependency directives allow you to specify dependencies that your job may have on other jobs. These directives will generally take the following form:

#PBS -W depend=dependency_expression

where dependency_expression is a comma-delimited list of one or more dependencies, and each dependency is of the form:

type:jobid[:jobid...]

where type is one of the directives listed below, and jobid[:jobid...] is a list of one or more job IDs that your job is dependent upon.

For example, run a job after completion (success or failure) of job ID 1234:

#PBS -W depend=afterany:1234

For example, run a job after successful completion of job ID 1234:

#PBS -W depend=afterok:1234

For more information about job dependencies, see the qsub man page.

7. Environment Variablesto top

7.1. PBS Environment Variables

While there are many PBS environment variables, you only need to know a few important ones to get started using PBS. The table below lists the most important PBS environment variables and how you might generally use them.

The following additional PBS variables may be useful to some users.

7.2. Other Important Environment Variables

In addition to the PBS environment variables, the table below lists a few other variables which are not generally required, but may be important depending on your job.

8. Example Scriptsto top

8.1. MPI Script

The following script is for a 128 core MPI job running for 20 hours in the standard queue.

#!/bin/bash -l
## Required Directives ------------------------------------
#PBS -l mppwidth=128
#PBS -l walltime=20:00:00
#PBS -q standard
#PBS -A Project_ID
#
## Optional Directives ------------------------------------
#PBS -N testjob
#PBS -j oe
#PBS -m be
#PBS -V
#PBS -S /bin/bash
#
## Execution Block ----------------------------------------
# cd to your scratch directory in /work
cd $WORKDIR
#
# create a job-specific subdirectory based on JOBID and cd to it
JOBID=`echo $PBS_JOBID | cut -d '.' -f 1`
if [ ! -d $JOBID ]; then
  mkdir -p $JOBID
fi
cd $JOBID
#
# stage input data from archive
rcp $ARCHIVE_HOST:$ARCHIVE_HOME/my_data_dir/"*.dat" .
#
# copy the executable from $HOME
cp $HOME/my_prog.exe .
#
## Launching ----------------------------------------------
aprun -n 128 ./my_prog.exe > my_prog.out
#
## Cleanup ------------------------------------------------
# archive your results
# Using the "here document" syntax, create a job script
# for archiving your data.
cd $WORKDIR
rm -f archive_job
cat > archive_job <<END
#!/bin/bash -l
#PBS -l walltime=12:00:00
#PBS -q transfer
#PBS -A Project_ID
#PBS -l mppwidth=0
#PBS -j oe
#PBS -S /bin/bash
cd $WORKDIR/$JOBID
rsh $ARCHIVE_HOST mkdir $ARCHIVE_HOME/$JOBID
rcp -r $JOBID $ARCHIVE_HOST:$ARCHIVE_HOME/
rsh $ARCHIVE_HOST ls -l $ARCHIVE_HOME/$JOBID
# Remove scratch directory from the file system.
cd $WORKDIR
rm -rf $JOBID
END

# Submit the archive job script.
qsub archive_job

8.2. MPI Script (accessing more memory per process)

By default, an MPI job runs one process per core, with all processes sharing the available memory on the node. If you need more memory per process, then your job needs to run fewer MPI processes per node.

The following script requests 128 cores, but uses only one core per node for an MPI job running for 20 hours in the standard queue.

Note the use of the $BC_NODE_ALLOC environment variable (nodes allocated for this job). This would run 8 processes with access to 32 GBytes of memory.

Also note the use of the "-N" option of aprun, which specifies the number of MPI processes to start on each node.

#!/bin/bash -l
## Required Directives ------------------------------------
#PBS -l mppwidth=128
#PBS -l walltime=20:00:00
#PBS -q standard
#PBS -A Project_ID
#
## Optional Directives ------------------------------------
#PBS -N testjob
#PBS -j oe
#PBS -m be
#PBS -V
#PBS -S /bin/bash
#
## Execution Block ----------------------------------------
# cd to your scratch directory in /work
cd $WORKDIR
#
# create a job-specific subdirectory based on JOBID and cd to it
JOBID=`echo $PBS_JOBID | cut -d '.' -f 1`
if [ ! -d $JOBID ]; then
  mkdir -p $JOBID
fi
cd $JOBID
#
# stage input data from archive
rcp $ARCHIVE_HOST:$ARCHIVE_HOME/my_data_dir/"*.dat" .
#
# copy the executable from $HOME
cp $HOME/my_prog.exe .
#
## Launching ----------------------------------------------
aprun -n $BC_NODE_ALLOC -N 1 ./my_prog.exe > my_prog.out
#
## Cleanup ------------------------------------------------
# archive your results
# Using the "here document" syntax, create a job script
# for archiving your data.
cd $WORKDIR
rm -f archive_job
cat > archive_job <<END
#!/bin/bash -l
#PBS -l walltime=12:00:00
#PBS -q transfer
#PBS -A Project_ID
#PBS -j oe
#PBS -S /bin/bash
cd $WORKDIR/$JOBID
rsh $ARCHIVE_HOST mkdir $ARCHIVE_HOME/$JOBID
rcp -r $JOBID $ARCHIVE_HOST:$ARCHIVE_HOME/
rsh $ARCHIVE_HOST ls -l $ARCHIVE_HOME/$JOBID
# Remove scratch directory from the file system.
cd $WORKDIR
rm -rf $JOBID
END

# Submit the archive job script.
qsub archive_job

8.3. OpenMP Script

The following script is for an OpenMP job using one thread per core on a single node and running for 20 hours in the standard queue. Note the use of the $BC_CORES_PER_NODE environment variable. Also remember that the number of cores will be rounded up to the nearest complete node.

#!/bin/bash -l
## Required Directives ------------------------------------
#PBS -l mppwidth=1 
#PBS -l walltime=20:00:00
#PBS -q standard
#PBS -A Project_ID
#
## Optional Directives ------------------------------------
#PBS -N testjob
#PBS -j oe
#PBS -m be
#PBS -V
#PBS -S /bin/bash
#
## Execution Block ----------------------------------------
# cd to your scratch directory in /work
cd $WORKDIR
#
# create a job-specific subdirectory based on JOBID and cd to it
JOBID=`echo $PBS_JOBID | cut -d '.' -f 1`
if [ ! -d $JOBID ]; then
  mkdir -p $JOBID
fi
cd $JOBID
#
# stage input data from archive
rcp $ARCHIVE_HOST:$ARCHIVE_HOME/my_data_dir/"*.dat" .
#
# copy the executable from $HOME
cp $HOME/my_prog.exe .
#
## Launching ----------------------------------------------
export OMP_NUM_THREADS=$BC_CORES_PER_NODE
aprun -n 1 -d $BC_CORES_PER_NODE ./my_prog.exe > my_prog.out
# If you want to start fewer threads, replace $BC_CORES_PER_NODE in
# this section with a lower value.
#
## Cleanup ------------------------------------------------
# archive your results
# Using the "here document" syntax, create a job script
# for archiving your data.
cd $WORKDIR
rm -f archive_job
cat > archive_job <<END
#!/bin/bash -l
#PBS -l walltime=12:00:00
#PBS -q transfer
#PBS -A Project_ID
#PBS -l mppwidth=0
#PBS -j oe
#PBS -S /bin/bash
cd $WORKDIR/$JOBID
rsh $ARCHIVE_HOST mkdir $ARCHIVE_HOME/$JOBID
rcp -r $JOBID $ARCHIVE_HOST:$ARCHIVE_HOME/
rsh $ARCHIVE_HOST ls -l $ARCHIVE_HOME/$JOBID
# Remove scratch directory from the file system.
cd $WORKDIR
rm -rf $JOBID
END

# Submit the archive job script.
qsub archive_job

8.4. SHMEM Script

The following script is for a 128 core SHMEM job running for 20 hours in the standard queue.

#!/bin/bash -l
## Required Directives ------------------------------------
#PBS -l mppwidth=128
#PBS -l walltime=20:00:00
#PBS -q standard
#PBS -A Project_ID
#
## Optional Directives ------------------------------------
#PBS -N testjob
#PBS -j oe
#PBS -m be
#PBS -V
#PBS -S /bin/bash
#
## Execution Block ----------------------------------------
# cd to your scratch directory in /work
cd $WORKDIR
#
# create a job-specific subdirectory based on JOBID and cd to it
JOBID=`echo $PBS_JOBID | cut -d '.' -f 1`
if [ ! -d $JOBID ]; then
  mkdir -p $JOBID
fi
cd $JOBID
#
# stage input data from archive
rcp $ARCHIVE_HOST:$ARCHIVE_HOME/my_data_dir/"*.dat" .
#
# copy the executable from $HOME
cp $HOME/my_prog.exe .
#
## Launching ----------------------------------------------
aprun -n 128 ./my_prog.exe > my_prog.out
#
## Cleanup ------------------------------------------------
# archive your results
# Using the "here document" syntax, create a job script
# for archiving your data.
cd $WORKDIR
rm -f archive_job
cat > archive_job <<END
#!/bin/bash -l
#PBS -l walltime=12:00:00
#PBS -q transfer
#PBS -A Project_ID
#PBS -j oe
#PBS -S /bin/bash
cd $WORKDIR/$JOBID
rsh $ARCHIVE_HOST mkdir $ARCHIVE_HOME/$JOBID
rcp -r $JOBID $ARCHIVE_HOST:$ARCHIVE_HOME/
rsh $ARCHIVE_HOST ls -l $ARCHIVE_HOME/$JOBID
# Remove scratch directory from the file system.
cd $WORKDIR
rm -rf $JOBID
END
# Submit the archive job script.
qsub archive_job

8.5. Hybrid MPI/OpenMP Script

The following script uses 128 cores with one MPI task per node and one thread per core. Note the use of the $BC_CORES_PER_NODE and $BC_NODE_ALLOC environment variables.

#!/bin/bash -l
## Required Directives ------------------------------------
#PBS -l mppwidth=128
#PBS -l walltime=20:00:00
#PBS -q standard
#PBS -A Project_ID
#
## Optional Directives ------------------------------------
#PBS -N testjob
#PBS -j oe
#PBS -m be
#PBS -V
#PBS -S /bin/bash
#
## Execution Block ----------------------------------------
# cd to your scratch directory in /work
cd $WORKDIR
#
# create a job-specific subdirectory based on JOBID and cd to it
JOBID=`echo $PBS_JOBID | cut -d '.' -f 1`
if [ ! -d $JOBID ]; then
  mkdir -p $JOBID
fi
cd $JOBID
#
# stage input data from archive
rcp $ARCHIVE_HOST:$ARCHIVE_HOME/my_data_dir/"*.dat" .
#
# copy the executable from $HOME
cp $HOME/my_prog.exe .
#
## Launching ----------------------------------------------
export OMP_NUM_THREADS=$BC_CORES_PER_NODE
aprun -n $BC_NODE_ALLOC -N 1 -d $BC_CORES_PER_NODE ./my_prog.exe > my_prog.out
#
## Cleanup ------------------------------------------------
# archive your results
# Using the "here document" syntax, create a job script
# for archiving your data.
cd $WORKDIR
rm -f archive_job
cat > archive_job <<END
#!/bin/bash -l
#PBS -l walltime=12:00:00
#PBS -q transfer
#PBS -A Project_ID
#PBS -j oe
#PBS -S /bin/bash
cd $WORKDIR/$JOBID
rsh $ARCHIVE_HOST mkdir $ARCHIVE_HOME/$JOBID
rcp -r $JOBID $ARCHIVE_HOST:$ARCHIVE_HOME/
rsh $ARCHIVE_HOST ls -l $ARCHIVE_HOME/$JOBID
# Remove scratch directory from the file system.
cd $WORKDIR
rm -rf $JOBID
END
# Submit the archive job script.
qsub archive_job