• 1. Introduction
• 1.1. Document Scope and Assumptions
• 1.2. Policies to Review
• 1.2.1. Login Node Abuse Policy
• 1.2.2. Workspace Purge Policy
• 1.3. Obtaining an Account
• 1.4. Requesting Assistance
• 2. System Configuration
• 2.1. System Summary
• 2.2. Processors
• 2.3. Memory
• 2.4. Operating System
• 2.5. File Systems
• 2.5.1. /u/home/
• 2.5.2. /scr/
• 2.5.3. /p/cwfs/
• 2.6. Peak Performance
• 3. Accessing the System
• 3.1. Kerberos
• 3.2. Logging In
• 3.3. File Transfers
• 4. User Environment
• 4.1. User Directories
• 4.1.1. Home Directory
• 4.1.2. Work Directory
• 4.1.3. Center Wide File System Directory
• 4.2. Shells
• 4.3. Environment Variables
• 4.3.1. Login Environment Variables
• 4.3.2. Batch-Only Environment Variables
• 4.4. Modules
• 4.5. Archive Usage
• 4.5.1. Archival Command Synopsis
• 5. Program Development
• 5.1. Programming Models
• 5.1.1. Message Passing Interface (MPI)
• 5.1.2. Open Multi-Processing (OpenMP)
• 5.1.3. Hybrid Processing (MPI/OpenMP)
• 5.2. Available Compilers
• 5.2.1. Intel C, C++ and Fortran Compiler
• 5.2.2. GNU Programming Environment
• 5.3. Relevant Modules
• 5.4. Libraries
• 5.4.1. Intel Math Kernel Library (MKL)
• 5.4.2. Additional PETTT Libraries
• 5.5. Debuggers
• 5.5.1. TotalView
• 5.5.2. DDT
• 5.5.3. gdb
• 5.6. Code Profiling and Optimization
• 5.6.1. gprof
• 5.6.2. codecov
• 5.6.3. Additional Profiling Tools
• 5.6.4. Program Development Reminders
• 5.6.5. Compiler Optimization
• 5.6.6. Performance Optimization Methods
• 6. Batch Scheduling
• 6.1. Scheduler
• 6.2. Queue Information
• 6.3. Interactive Logins
• 6.4. Interactive Batch Sessions
• 6.5. Batch Request Submission
• 6.6. Batch Resource Directives
• 6.7. Launch Commands
• 6.8. Sample Scripts
• 6.9. PBS Commands
• 6.10. Advance Reservations
• 7. Software Resources
• 7.1. Application Software
• 7.2. Useful Utilities
• 7.3. Sample Code Repository
• 8. Links to Vendor Documentation
• 8.1. IBM Links
• 8.2. Red Hat Links
• 8.3. GNU Links
• 8.4. Intel Links

1. Introductionto top

1.1. Document Scope and Assumptions

This document provides an overview and introduction to the use of the IBM iDataPlex (Kilrain) located at the Navy DSRC, along with a description of the specific computing environment on Kilrain. The intent of this guide is to provide information that will enable the average user to perform computational tasks on the system. To receive the maximum benefit from the information provided here, you should be proficient in the following areas:

1.2. Policies to Review

Users are expected to be aware of the following policies for working on Kilrain.

1.2.1. Login Node Abuse Policy

The login nodes provide login access for Kilrain and support such activities as compiling, editing and general interactive use by all users. Consequently, memory or CPU intensive programs running on the login nodes can significantly affect all users of the system. Therefore, only small serial applications requiring less than 15 minutes of compute time and less than 8 GBytes of memory are allowed on the login nodes. Any jobs running on the login nodes that exceed these limits will be terminated.

1.2.2. Workspace Purge Policy

Close management of space in the /scr file system is a high priority. Files in the /scr file system that have not been accessed in 30 days are subject to the purge cycle. If available space becomes critically low, a manual purge may be run, and all files in /scr are eligible for deletion. Using the touch command (or similar commands) to prevent files from being purged is prohibited. Users are expected to keep up with file archival and removal within the normal purge cycles.

1.3. Obtaining an Account

The process of getting an account on the HPC systems at any of the DSRCs begins with getting an account on the HPCMP Portal to the Information Environment, commonly called a "pIE User Account". If you do not yet have a pIE User Account, please visit the Consolidated Customer Assistance Center (CCAC) Accounts page and follow the instructions there. Once you have an active pIE User Account, visit the Navy DSRC Accounts page for instructions on how to request accounts on the Navy DSRC HPC systems. If you need assistance with any part of this process, please contact CCAC at accounts@ccac.hpc.mil.

1.4. Requesting Assistance

The Consolidated Customer Assistance Center (CCAC) is available to help users with unclassified problems, issues, or questions. Analysts are on duty 8:00 a.m. - 11:00 p.m. Eastern, Monday - Friday (excluding Federal holidays).

You can contact the Navy DSRC for after-hours support and for support services not provided by CCAC. You can contact us in any of the following ways:

For more detailed contact information, please see our Contact Page.

2. System Configurationto top

2.1. System Summary

Kilrain is an IBM iDataPlex. The login and compute nodes are populated with 2.6-GHz Intel Xeon Sandy Bridge E5-2670 16-core processors. Kilrain uses the FDR-10 InfiniBand interconnect in a Fat Tree configuration as its high-speed network for MPI messages and IO traffic. Kilrain uses IBM's General Parallel File System (GPFS) to manage its parallel file system that targets IBM's IS4600 (Infinite Storage) RAID arrays. Kilrain has 1,176 compute nodes that share memory only on the node; memory is not shared across the nodes. Each login node has two 8-core processors (16 cores) with its own Red Hat Enterprise Linux operating system, sharing 64 GBytes of memory, with no user-accessible swap space. Each compute node has two 8-core processors (16 cores) with its own Red Hat Enterprise Linux operating system, sharing 32 GBytes of memory, with no user-accessible swap space. Kilrain is rated at 391 peak TFLOPS and has 2.4 PBytes (formatted) of disk storage.

Kilrain is intended to be used as a batch-scheduled HPC system. Its login nodes are not to be used for large computational (e.g., memory, IO, long executions) work. All executions that require large amounts of system resources must be sent to the compute nodes by batch job submission.

2.2. Processors

Kilrain uses 2.6-GHz Intel Sandy Bridge processors on its login and compute nodes. There are two processors per node, each with 8 cores, for a total of 16 cores per node. In addition, these processors have 8x256 KBytes of L2 cache and 20 MBytes of L3 cache.

2.3. Memory

Kilrain uses both shared- and distributed-memory models. Memory is shared among all the cores on a node, but is not shared among the nodes across the cluster.

Each login node contains 64 GBytes of main memory. All memory and cores on each node are shared among all users who are logged in to that node. Therefore, users should not use more than 8 GBytes of memory at any one time. Memory is not distributed across the login nodes.

Each compute node contains 27 GBytes of user-accessible shared memory.

2.4. Operating System

The operating system on Kilrain is Red Hat Linux. The operating system supports 64-bit software.

2.5. File Systems

Kilrain has the following file systems available for user storage:

2.5.1. /u/home/

This file system is locally mounted from Kilrain's GPFS file system. It has a formatted capacity of 15 TBytes. All users have a home directory located on this file system that can be referenced by the environment variable $HOME. This file system is not backed up. Users are responsible for making backups of their files to the archive server, Newton, or to some other local system.

2.5.2. /scr/

This file system is locally mounted from Kilrain's GPFS file system. It has a formatted capacity of 2.4 PBytes. All users have a work directory located on this file system which can be referenced by the environment variable $WORKDIR. This file system is not backed up. Users are responsible for making backups of their files to the archive server, Newton, or to some other local system.

2.5.3. /p/cwfs/

This path is directed to the Center Wide File System (CWFS) which is meant for short term storage (no longer than 30 days). All users have a directory defined in this file system which can be referenced by the environment variable $CENTER. This directory is accessible from both the login and compute nodes on the utility server and from the login nodes on Kilrain. The CWFS has a formatted capacity of 728 TBytes.

2.6. Peak Performance

Kilrain is rated at 391 peak TFLOPS.

3. Accessing the Systemto top

3.1. Kerberos

A Kerberos client kit must be installed on your desktop to enable you to get a Kerberos ticket. Kerberos is a network authentication tool that provides secure communication by using secret cryptographic keys. Only users with a valid HPCMP Kerberos authentication can gain access to Kilrain. More information about installing Kerberos clients on your desktop can be found at the CCAC Support page.

3.2. Logging In

3.3. File Transfers

File transfers to DSRC systems (except for those to the local archive server) must be performed using the following kerberized tools: krcp, kftp, scp, sftp, and mpscp. Before using any of these tools, you must use a Kerberos client to obtain a Kerberos ticket. Information about installing and using a Kerberos client can be found at the CCAC Support page.

The command below uses secure copy (scp) to copy a single local file into a destination directory on a Kilrain login node. The mpscp command is similar to the scp and krcp commands, but has a different underlying means of data transfer, and may enable greater transfer rate. The mpscp and krcp commands have the same syntax as scp.

The three commands, scp, krcp, and mpscp, can be used to send multiple files. The following command transfers all files with the .txt extension to the same destination directory.

The example below uses the secure file transfer protocol (sftp) to connect to Kilrain, then uses the sftp cd and put commands to change to the destination directory and copy a local file there. The sftp quit command ends the sftp session. Use the sftp help command to see a list of all sftp commands.

The Kerberized file transfer protocol (kftp) command differs from sftp in that you are prompted for your username.

Windows users may use a graphical file transfer protocol (ftp) client such as Filezilla.

4. User Environmentto top

4.1. User Directories

The following user directories are provided for all users on Kilrain.

4.1.1. Home Directory

When you log on to Kilrain, you will be placed in your home directory, /u/home/username. The environment variable $HOME is automatically set for you and refers to this directory. $HOME is visible to both the login and compute nodes, and may be used to store small user files. It has an initial quota of 10 GBytes and is not backed up and therefore should not be used for long-term storage.

4.1.2. Work Directory

Kilrain has one large file system (/scr) for the temporary storage of data files needed for executing programs. You may access your personal working directory under /scr by using the $WORKDIR environment variable, which is set for you upon login. Your $WORKDIR directory has no disk quotas, and files stored there do not affect your permanent file quota usage. Because of high usage, the /scr file system tends to fill up frequently. Please review the Purge Policy and be mindful of your disk usage.

REMEMBER: /scr is a "scratch" file system and is not backed up. You are responsible for managing files in your $WORKDIR by backing up files to the archive system and deleting unneeded files when your jobs end. See the section below on Archive Usage for details.

All of your jobs should execute from your $WORKDIR directory, not $HOME. While not technically forbidden, jobs that are run from $HOME are subject to disk space quotas and have a much greater chance of failing if problems occur with that resource.

To avoid unusual errors that can arise from two jobs using the same scratch directory, a common technique is to create a unique subdirectory for each batch job by including the following lines in your batch script:

4.1.3. Center Wide File System Directory

The path for your working directory on the Center-Wide File System is /p/cwfs/username. The environment variable $CENTER is automatically set to point to this directory. The main purpose of this area is as a staging area for production system output files that require post-processing using the Utility Server.

Because the compute nodes are unable to see /p/cwfs on Kilrain, you will need to transfer output files from $WORKDIR to $CENTER from a login node. This may be done manually or through the transfer queue, which executes on the login nodes.

4.2. Shells

The following shells are available on Kilrain: csh, bash, ksh, tcsh, and sh. To request a change of your default shell, contact the Consolidated Customer Assistance Center (CCAC).

4.3. Environment Variables

A number of environment variables are provided by default on all HPCMP HPC systems. We encourage you to use these variables in your scripts where possible. Doing so will help to simplify your scripts and reduce portability issues if you ever need to run those scripts on other systems. The following environment variables are automatically set in your login environment:

4.3.1. Login Environment Variables

4.3.2. Batch-Only Environment Variables

In addition to the variables listed above, the following variables are automatically set only in your batch environment. That is, your batch scripts will be able to see them when they run. These variables are supplied for your convenience and are intended for use inside your batch scripts.

4.4. Modules

Software modules are a convenient way to set needed environment variables and include necessary directories in your path so that commands for particular applications can be found. All accounts have a set of modules loaded on login. These modules add compilers to the PATH, etc. For additional information on using modules, please refer to the Modules User Guide.

4.5. Archive Usage

All of our HPC systems have access to an online archival mass storage system that provides long-term storage for users' files on a Petascale archival storage system that resides on a robotic tape library system. A 60-TByte disk cache frontends the tape file system and temporarily holds files while they are being transferred to or from tape.

Tape file systems have very slow access times. The tapes must be robotically pulled from the tape library, mounted in one of the limited number of tape drives, and wound into position for file archival or retrieval. For this reason, users should always tar up their small files in a large tarball when archiving a significant number of files. A good maximum target size for tarballs is about 200 GBytes or less. At that size, the time required for file transfer and tape I/O is reasonable. Files larger than 1 TByte may span more than one tape, which will greatly increase the time required for both archival and retrieval.

The environment variables $ARCHIVE_HOST and $ARCHIVE_HOME are automatically set for you. $ARCHIVE_HOST can be used to reference the archive server, and $ARCHIVE_HOME can be used to reference your archive directory on the server. These can be used when transferring files to/from archive.

4.5.1. Archival Command Synopsis

A synopsis of the main archival utilities is listed below. For information on additional capabilities, see the Archive User Guide or read the online man pages that are available on each system. These commands are non-Kerberized and can be used in batch submission scripts if desired.

• Copy one or more files from the archive system
  rcp ${ARCHIVE_HOST}:${ARCHIVE_HOME}/file_name ${WORKDIR}/proj1

• List files and directory contents on the archive system
  rsh ${ARCHIVE_HOST} ls [lsopts] [file/dir ...]

• Create directories on the archive system
  rsh ${ARCHIVE_HOST} mkdir][-p] [-s] dir1 [dir2 ...]

• Copy one or more files to the archive system
  rcp ${WORKDIR}/proj1/file_name ${ARCHIVE_HOST}:${ARCHIVE_HOME}/proj1

5. Program Developmentto top

5.1. Programming Models

Kilrain supports two base programming models: Message Passing Interface (MPI) and Open Multi-Processing (OpenMP). A Hybrid MPI/OpenMP programming model is also supported. MPI is an example of the message- or data-passing models, while OpenMP only uses shared memory on a node by spawning threads, and the hybrid model combines both shared and distributed memory usage.

5.1.1. Message Passing Interface (MPI)

Kilrain has three MPI-2.0 standard library suites: IntelMPI, OpenMPI and IBM PE. The modules for these MPI libraries are: mpi/intel/impi/#.#.#, mpi/intel/ibmpe/#.#.#, and mpi/intel/openmpi/#.#.#, where #.#.# is the library version number.

When building an MPI program on Kilrain, ensure the following:

• That the MPI module you wish to use is loaded. By default, the mpi/intel/impi module is loaded.
• That the compiler you wish to use is loaded. By default, the Intel compiler module is loaded.
• That the source code includes one of the following lines:

  #include <mpi.h>        ## for C, or
  INCLUDE "mpif.h"        ## for Fortran

Using the Intel MPI Library

When building your program with the default Intel MPI library and the default Intel compiler, use the following:

mpiicc -o mpiprog.exe mpi_prog.c        ## for C
mpiicpc -o mpiprog.exr mpi_prog.c       ## for C++
mpiifort -o mpiprog.exe mpi_prog.f      ## for Fortran

When building your program with the default Intel MPI library and the GNU compiler, swap the default Intel compiler module with a GNU compiler module and swap the Intel MPI module with a GNU Intel MPI module, as follows:

module swap compiler/intel compiler/gcc
module swap mpi mpi/gnu/impi

And, compile the program with the following:

mpicc -o mpiprog.exe mpi_prog.c     ## for C
mpicxx -o mpiprog.exr mpi_prog.c    ## for C++
mpif77 -o mpiprog.exe mpi_prog.f    ## for Fortran77
mpif90 -o mpigpro.exe mpi_prog.f    ## for Fortran 90

To run an MPI program built with the Intel MPI library using the Intel or GNU compilers within a batch script, use the following command:

mpirun -np N $WORKDIR/mympidirectory/mpiprog.exe

The mpirun utility executes across a specified number of compute nodes. The "-np N" option specifies the number of cores to start. Users should ensure that all files needed by the MPI job are located in $WORKDIR. Additional information on the mpirun utility can be found in the online man pages.

Using the OpenMPI MPI Library

When building your program with the OpenMPI MPI library on Kilrain and the default Intel compiler, swap the default Intel MPI module for an OpenMPI module as follows:

module swap mpi mpi/intel/openmpi

And, compile the program with the following:

mpicc -o mpiprog.exe mpi_prog.c     ##for C
mpicxx -o mpiprog.exr mpi_prog.c    ##for C++
mpif77 -o mpiprog.exe mpi_prog.f    ##for Fortran 77
mpif90 -o mpiprog.exe mpi_prog.f    ## for Fortran 90

When building your program with the OpenMPI MPI library on Kilrain and the GNU compiler, swap the default Intel compiler module for a GNU compiler module and swap the default Intel MPI module for a GNU OpenMPI module as follows:

module swap compiler/intel compiler/gcc
module swap mpi mpi/gnu/openmpi

And, compile the program with the following:

mpicc -o mpiprog.exe mpi_prog.c     ##for C
mpicxx -o mpiprog.exr mpi_prog.c    ##for C++
mpif77 -o mpiprog.exe mpi_prog.f    ##for Fortran77
mpif90 -o mpiprog.exe mpi_prog.f    ## for Fortran 90

To run an MPI program built with the OpenMPI MPI library using the Intel or GNU compilers within a batch script, use the following command:

mpirun -np N $WORKDIR/mympidirectory/mpiprog.exe

The mpirun utility executes across a specified number of compute nodes. The "-np N" option specifies the number of cores to start. Users should ensure that all files needed by the MPI job are located in $WORKDIR. Additional information on the mpirun utility can be found in the online man pages.

Using the IBM PE Library

When building your program with the IBM PE library on Kilrain and the default Intel compiler, swap the default Intel MPI module for an IBM PE module, as follows:

module swap mpi mpi/intel/ibmpe

And, compile the program with the following:

mpiicc -o mpiprog.exe mpi_prog.c     ## for C
mpiicpc -o mpiprog.exr mpi_prog.c    ## for C++
mpiifort -o mpiprog.exe mpi_prog.f   ## for Fortran

When building your program with the IBM PE library on Kilrain and the GNU compiler, swap the default Intel compiler module for a GNU compiler module and swap the default Intel IBM PE module for a GNU IBM PE module as follows:

module swap compiler/intel compiler/gcc
module swap mpi mpi/gnu/ibmpe

And, compile the program with the following:

mpicc -o mpiprog.exe mpi_prog.c     ## for C
mpicxx -o mpiprog.exr mpi_prog.c    ## for C++
mpif77 -o mpiprog.exe mpi_prog.f    ## for Fortran77
mpif90 -o mpiprog.exe mpi_prog.f    ## for Fortran 90

To run an MPI program built with the IBM PE library on Kilrain using the Intel or GNU compilers within a batch script, use the following command:

poe -procs N $WORKDIR/mympidirectory/mpiprog.exe

The poe utility executes across a specified number of cores. The "-procs N" option specified the number of cores to start. Additional information on the poe utility can be found in the online man pages.

5.1.2. Open Multi-Processing (OpenMP)

OpenMP is a portable, scalable model that gives programmers a simple and flexible interface for developing parallel applications. It supports shared-memory multiprocessing programming in C, C++, and Fortran, and consists of a set of compiler directives, library routines, and environment variables that influence compilation and run-time behavior.

When creating an OpenMP program on Kilrain, ensure the following:

• That the source code includes one of the following:

  #include <omp.h>    ## for C, or
  INCLUDE 'omp.h'     ## for Fortran

• That the compile command includes the "-openmp" option.

To compile an OpenMP program, use the following examples.

Intel Programming Environment

icc -openmp -o openmpprog.exe openmp_prog.c     ## for C, or
ifort -openmp -o openmpprog.exe openmp_prog.f   ## for Fortran

GNU Programming Environment

cc -o openmpprog.exe -fopenmp openmp_prog.c         ## for C, or
gfortran -o openmpprog.exe -fopenmp openmp_prog.f   ## for Fortran

To run an OpenMP program within a batch script, just call the executable:

$WORKDIR/job1/openmpprog.exe

5.1.3. Hybrid Processing (MPI/OpenMP)

In hybrid processing, all intranode parallelization is accomplished using OpenMP, while all internode parallelization is accomplished using MPI. Typically, there is one MPI task assigned per node, with the number of OpenMP threads assigned to each node set as the number of cores available on the node.

5.2. Available Compilers

There are two compiler suites available on Kilrain:

• Intel
• GNU

All versions of MPI share a common base set of compilers that are available on both the login and compute nodes. Additional compiler wrapper scripts are available for Intel MPI, OpenMPI, and IBM PE.

To use these compilers, you will first need to load the appropriate module. The compiler can then be invoked using the appropriate environment variable or wrapper script listed in the tables above. The Intel compiling environment is loaded for you by default. To use a different suite, you will need to swap modules. See Relevant Modules (below) to learn how.

5.2.1. Intel C, C++ and Fortran Compiler

The Intel Programming Environment provides a large number of options that are the same for all compilers in the suite. The following table lists some of the more common options that you may use.

Detailed information about these and other compiler options is available in the Intel compiler (ifort and icc) man pages on Kilrain.

5.2.2. GNU Programming Environment

The GNU Programming Environment provides a large number of options that are the same for all compilers in the suite. The following table lists some of the more common options that you may use.

Detailed information about these and other compiler options is available in the GNU compiler (gfortran, gcc, and g++) man pages on Kilrain.

5.3. Relevant Modules

By default, Kilrain loads the Intel programming environment for you. The GNU environment is also available. To use the GNU environment, the Intel module must be unloaded and replaced with the one you wish to use. To do this, use the "module swap" command as follows:

module swap compiler/intel/#.# compiler/gcc/#.#.#

When you compile your own code, you will need to select which compiler and MPI version you want to use. If you are using the default programming environment compiler, then you need only to load the desired MPI module. The same module commands should be executed in your batch script before executing your program.

The table below lists the naming convention used for various modules:

For more information on using modules, see the Modules User Guide.

5.4. Libraries

Intel's Math Kernel Library (MKL) is available on Kilrain. In addition, an extensive suite of math and science libraries are available in the $PET_HOME directory.

5.4.1. Intel Math Kernel Library (MKL)

The Intel Math Kernel Library (MKL) is a library of numerical processing functions that have been optimized for math, scientific and engineering applications. The MKL includes:

The MKL can be loaded into your path using the following command:

module load mkl/#.#

5.4.2. Additional PETTT Libraries

There is also an extensive set of Math libraries available in the $PET_HOME/MATH directory on Kilrain. A list of PETTT and other commonly supplied tools, such as HDF5, CMake, and NetCDF can be found at http://www.pettt-ace.com.

5.5. Debuggers

Kilrain provides the TotalView, DDT and the GNU Project Debugger (gdb) debuggers to assist users in debugging their code.

5.5.1. TotalView

TotalView is a debugger that supports threads, MPI, OpenMP, C/C++, and Fortran, mixed-language codes, advanced features like on-demand memory leak detection, other heap allocation debugging features, and the Standard Template Library Viewer (STLView). Unique features like dive, a wide variety of breakpoints, the Message Queue Graph/Visualizer, powerful data analysis, and control at the thread level are also available.

Currently on Kilrain, to display the source code, you must limit your debug job to 1 node (16 cores). Debug jobs using multiple nodes will display only assembler instructions.

Follow these steps to use TotalView on Kilrain via a UNIX X-Windows interface:

1. Open a console window to Kilrain.

   ssh kilrain#.navo.hpc.mil (# = 1 - 2)

2. Start a 16-core, interactive session with X-Windows forwarding enabled:

   qsub -l walltime=07:00:00 -l select=1:ncpus=16:mpiprocs=16 -A Project_ID -q debug -I -X

   After a short while a message similar to the following will appear:

   qsub:  waiting for job ######.##### to start
   qsub:  job ######.##### ready

   You are now logged into an interactive batch session.

3. Load the modules that you used to compile your code and load the TotalView module.

   module load compiler/intel/#.#.# mpi/intelmpi/#.#.# totalview

4. Now start TotalView: type "totalview" and wait a minute or so for the TotalView windows to pop up.
5. Under the TotalView Window named "New Program" select the "Browse" button and select your program executable.
6. Click the "Parallel" tab and select the appropriate MPI suite from the pull down list.
7. In the same tab, click the "up" arrow on the "Tasks" to 16. This will allow a 16-MPI-task job. Also, increase the node button to "1" from "0".
8. Click OK. Your source code should pop up, allowing you to enter stop points, watchpoints, etc.

For more information on using TotalView, see the TotalView Documentation page.

5.5.2. DDT

DDT is an intuitive, scalable, graphical debugger capable of debugging a wide variety of scenarios found in today's development environments. DDT supports threads, MPI, OpenMP, C/C++, Fortran, Coarray Fortran, UPC and CUDA. Memory debugging and data visualization are supported for large-scale parallel applications. The Parallel Stack viewer is a unique way to see the program state of all processes and threads at a glance.

Follow these steps to use DDT on Kilrain via a UNIX X-Windows interface:

1. Open a console window to Kilrain.

   ssh kilrain#.navo.hpc.mil (# = 1 - 2)

2. Start a 16-core, interactive session with X-Windows forwarding enabled:

   qsub -l walltime=07:00:00 -l select=1:ncpus=16:mpiprocs=16 -A Project_ID -q debug -I -X

   After a short while a message similar to the following will appear:

   qsub:  waiting for job ######.##### to start
   qsub:  job ######.##### ready

   You are now logged into an interactive batch session.

3. Load the modules that you used to compile your code and load the DDT module.

   module load compiler/intel/#.#.# mpi/intelmpi/#.#.# ddt

4. Now start DDT: type "ddt" and wait a minute or so for the DDT windows to pop up.

   IMPORTANT ! Follow this section only if running DDT for the first time!

   The first time you run DDT, the DDT Configuration Wizard will pop up.

   1. Select "Create a New Configuration File." Click "Next."

   2. Select the appropriate MPI suite from the pull-down list. If your code is compiled with Intel-MPI, select "intel-mpi". Click "Next."

      You'll get an error about a Mismatched Host Name, followed by the question, "Do these host names refer to the same machine?"
      Click "Yes."

   3. The next window asks a Job Scheduling question. Select "Skip this step." Click "Next."
   4. The next window asks a Site-wide configuration question. Select "Skip this step." Click "Next."
   5. Click "Finish."
5. Under the "DDT - Welcome" window, select "Run and Debug a Program," and then select your program. Your source code should pop up.
6. In the same window, click the "up" arrow on "Number of processes" until it reaches 16.
7. Click "Run."

5.5.3. gdb

The GNU Project Debugger (gdb) is a source level debugger that can be invoked either with a program for execution or a running process id. To launch your program under gdb for debugging, use the following command:

gdb a.out corefile

To attach gdb to a program that is already executing on this node, use the following command:

gdb a.out pid

For more information, the GDB manual can be found at http://www.gnu.org/software/gdb/.

5.6. Code Profiling and Optimization

Profiling is the process of analyzing the execution flow and characteristics of your program to identify sections of code that are likely candidates for optimization, which increases the performance of a program by modifying certain aspects for increased efficiency.

We provide two utilities, gprof and codecov, to assist you in the profiling process. In addition, a basic overview of optimization methods with information about how they may improve the performance of your code can be found in Performance Optimization Methods (below).

5.6.1. gprof

The gprof utility may be used to analyze a profile created by compiling and linking with the "-pg" option. After being run, a gmon.out file will be created for each task. To create a report, run your code and run gprof on the resulting files creating a summary file:

mpiicc -g -pg -o myprog.exe myprog.c
mpirun ./myprog.exe      ## from within a PBS batch script or interactive session
gprof myprog.exe gmon.out > gprof.output

For more information, the gprof manual can be found at: http://sourceware.org/binutils/docs/gprof/index.html.

5.6.2. codecov

The Intel Code Coverage Tool (codecov) can be used in numerous ways to improve code efficiency and increase application performance. The tool leverages Profile-Guided optimization technology as discussed in Section 5.6.6. Coverage can be specified in the tool as file-level, function-level or block-level. Another benefit to this tool is the ability to compare the profiles of two application runs to find where the optimizations are making a difference. More detailed information on this tool can be found at http://www.intel.com/software/products/compilers.

5.6.3. Additional Profiling Tools

There is also a set of profiling tools available in the $PET_HOME/pkgs directory on Kilrain. Information about these tools may be found on the Baseline Configuration Web site at BC policy FY07-02. A list of commonly provided profiling tools can also be found at http://www.pettt-ace.com.

5.6.4. Program Development Reminders

If an application is not programmed for distributed memory, then only the cores on a single node can be used. This is limited to 16 cores on Kilrain.

Keep the system architecture in mind during code development. For instance, if your program requires more memory than is available on a single node, you will need to parallelize your code so that it can function across multiple nodes.

5.6.5. Compiler Optimization

The "-Olevel" option enables code optimization when compiling. The level that you choose (0-3) will determine how aggressive the optimization will be. Increasing levels of optimization may increase performance significantly, but you should note that a loss of precision may also occur. There are also additional options that may enable further optimizations. The following table contains the most commonly used options.

5.6.6. Performance Optimization Methods

Optimization generally increases compilation time and executable size, and may make debugging difficult. However, it usually produces code that runs significantly faster. The optimizations that you can use will vary depending on your code and the system on which you are running.

Note: Before considering optimization, you should always ensure that your code runs correctly and produces valid output.

In general, there are four main categories of optimization:

Global Optimization

A technique that looks at the program as a whole and may perform any of the following actions:

Loop Optimization

A technique that focuses on loops (for, while, etc.) in your code and looks for ways to reduce loop iterations or parallelize the loop operations. The following types of actions may be performed:

Interprocedural Analysis and Optimization (IPA)

A technique that allows the use of information across function call boundaries to perform optimizations that would otherwise be unavailable.

Function Inlining

A technique that seeks to reduce function call and return overhead.

6. Batch Schedulingto top

6.1. Scheduler

The Portable Batch System (PBS) is currently running on Kilrain. It schedules jobs and manages resources and job queues, and can be accessed through the interactive batch environment or by submitting a batch request. PBS is able to manage both single-processor and multiprocessor jobs.

6.2. Queue Information

The following table describes the PBS queues available on Kilrain:

Queue Descriptions and Limits

Priority	Queue Name	Job Class	Max Wall Clock Time	Max Cores Per Job	Comments
Highest	urgent	Urgent	24 Hours	4096	Designated urgent projects by DoD HPCMP
	high	High	168 Hours	6144	Designated high-priority projects by service/agency
	challenge	Challenge	168 Hours	6144	Challenge projects only
	special	N/A	24 Hours	4096	Access available by request
	debug	Debug	30 Minutes	1024	User diagnostic jobs
	standard	Standard	168 Hours	4096	Non-challenge user jobs
	transfer	N/A	12 Hours	1	Data transfer jobs
Lowest	background	Background	4 Hours	512	User jobs that will not be charged against the project allocation

6.3. Interactive Logins

When you log in to Kilrain, you will be running in an interactive shell on a login node. The login nodes provide login access for Kilrain and support such activities as compiling, editing, and general interactive use by all users. Please note the Login Node Abuse policy. The preferred method to run resource intensive executions is to use an interactive batch session.

6.4. Interactive Batch Sessions

An interactive session on a compute node is possible using the PBS qsub command with the "-I" option from a login node. Once PBS has scheduled your request to the specified queue, you will be directly logged into a compute node, and this session can last as long as your requested wall time. For example:

Your batch shell request will be placed in the specified queue and scheduled for execution. Depending on system load, this may take a few minutes. Once your shell starts, you can run or debug interactive applications, execute job scripts, or start an execution on the compute nodes via the appropriate MPI launch command. See Launch Commands below for more information.

6.5. Batch Request Submission

PBS batch jobs are submitted via the qsub command. The format of this command is:

qsub [ options ] batch_script_file

qsub options may be specified on the command line or embedded in the batch script file by lines beginning with "#PBS".

For a more thorough discussion of PBS Batch Submission, see the Kilrain PBS Guide.

6.6. Batch Resource Directives

Batch resource directives allow you to specify to PBS how your batch jobs should be run and what resources your job requires. Although PBS has many directives, you only need to know a few to run most jobs.

The basic syntax of PBS directives is as follows:

#PBS option[[=]value]

where some options may require values to be included. For example, to set the number of cores for your job, you might specify the following:

#PBS -l select=1:ncpus=16

The following directives are required for all jobs:

A more complete listing of batch Resource Directives is available in the Kilrain PBS Guide.

6.7. Launch Commands

On Kilrain, PBS batch scripts and interactive batch sessions run on service nodes instead of compute nodes. There are different commands for launching MPI executables from within a batch job depending on whether your script uses Intel MPI, OpenMPI, or IBM PE.

The mpirun command is used to send your Intel MPI or OpenMPI executables to the compute nodes. The following example command line can be used in a batch script or in an interactive batch session, sending the executable $WORKDIR/a.out to 16 compute cores:

mpirun -np 16 $WORKDIR/a.out

The poe command is used to send your IBM PE executable to the compute nodes.

poe $WORKDIR/a.out

No command is needed to launch OpenMP executables.

For more information about mpirun or poe, see the corresponding man page.

6.8. Sample Scripts

The following example is a good starting template for a batch script to run a serial job for one hour:

#!/bin/bash -l  ## Specify your shell
# Specify name of the job
#PBS -N serialjob
#
# Append std output to file serialjob.out 
#PBS -o serialjob.out
#
# Append std error to file serialjob.err
#PBS -e serialjob.err
#
# Specify Project ID to be charged (Required)
#PBS -A Project_ID
#
# Request wall clock time of 1 hour (Required)
#PBS -l walltime="01:00:00"
#
# Specify queue name (Required)
#PBS -q standard
#
# Specify the number cores (Required)
#PBS -l select=1:ncpus=1
#
#PBS -S /bin/bash
# Change to the specified directory
cd $WORKDIR
#
# Execute the serial executable on 1 core
./serial_fort.exe
# End of batch job

The first few lines tell PBS to save the standard output and error output to the given files, and to give the job a name. Skipping ahead, we estimate the run-time to be about one hour and know that this is acceptable for the standard batch queue. We need one core in total, so we request one core.

The following example is a good starting template for a batch script to run a parallel (MPI) job that was built with IntelMPI for two hours:

#!/bin/bash -l
## The first line (above) specifies the shell to use for parsing 
## the remaining lines of the batch script.
#
## Required PBS Directives --------------------------------------
#PBS -A Project_ID
#PBS -q standard
#PBS -l select=2:ncpus=16:mpiprocs=16
#PBS -l walltime=12:00:00
#
## Optional PBS Directives --------------------------------------
#PBS -N Test_Run_1
#PBS -j oe
#PBS -V
#PBS -S /bin/bash
#
## Execution Block ----------------------------------------------
# Environment Setup
module swap mpi mpi/intel/impi/4.0.3
# cd to your personal directory in the scratch file system
cd $WORKDIR
#
# create a job-specific subdirectory based on JOBID and cd to it
JOBID=`echo $PBS_JOBID | cut -d '.' -f 1`
if [ ! -d $JOBID ]; then
  mkdir -p $JOBID
fi
cd $JOBID
#
# Launching
# copy executable from $HOME and submit it
cp $HOME/myimpiprog.exe .
mpirun ./myimpiprog.exe > myimpiprog.out
#
# Clean up
# archive your results
# Using the "here document" syntax, create a job script
# for archiving your data.
cd $WORKDIR
rm -f archive_job
cat > archive_job << END
#!/bin/bash
#PBS -l walltime=6:00:00
#PBS -q transfer
#PBS -A Project_ID
#PBS -l select=1:ncpus=1
#PBS -j oe
#PBS -S /bin/bash
cd $WORKDIR
rsh $ARCHIVE_HOST mkdir $ARCHIVE_HOME/$JOBID
rcp -r $JOBID $ARCHIVE_HOST:$ARCHIVE_HOME/
rsh $ARCHIVE_HOST ls -l $ARCHIVE_HOME/$JOBID
# Remove scratch directory from the file system.
rm -rf $JOBID
END
#
# Submit the archive job script.
qsub archive_job
# End of batch job

The first few lines tell PBS to save the standard output and error output to the given files, and to give the job a name. Skipping ahead, we estimate the run-time to be about 12 hours and know that this is acceptable for the standard batch queue. The next couple of lines set the total number of cores and the number of cores per node for the job. This job is requesting 32 total cores and 16 cores per node allowing the job to run on 2 nodes. The default value for number of cores per node is 16.

Additional examples are available in the Sample Code Repository ($SAMPLES_HOME) on Kilrain.

6.9. PBS Commands

The following commands provide the basic functionality for using the PBS batch system:

A more complete list of PBS commands is available in the Kilrain PBS Guide.

6.10. Advance Reservations

The Advance Reservation Service (ARS) is available on Kilrain for reserving up to 4,096 cores for use, starting at a specific date/time, and lasting for a specific number of hours. The ARS is accessible via most modern web browsers at https://reservation.hpc.mil. Authenticated access is required. An ARS User Guide is available online once you have logged in.

7. Software Resourcesto top

7.1. Application Software

All Commercial Off The Shelf (COTS) software packages can be found in the /site/applic/COTS directory. A complete listing of installed versions can be found on our software page. The general rule for all COTS software packages is that the two latest versions will be maintained on our systems. For convenience, modules are also available for most COTS software packages.

7.2. Useful Utilities

The following utilities are available on Kilrain.

7.3. Sample Code Repository

The Sample Code Repository is a directory that contains examples for COTS batch scripts, building and using serial and parallel programs, data management, and accessing and using serial and parallel math libraries. The $SAMPLES_HOME environment variable contains the path to this area, and is automatically defined in your login environment. Users should look in the $SAMPLES_HOME directory for simple examples. Examples and useful tips are periodically added to this directory.

8. Links to Vendor Documentationto top

8.1. IBM Links

IBM Home: http://www.ibm.com
IBM IDataPlex: http://www-03.ibm.com/systems/x/hardware/rack/dx360m4/index.html

8.2. Red Hat Links

Red Hat Home: http://www.redhat.com

8.3. GNU Links

GNU Home: http://www.gnu.org
GNU Compiler: http://gcc.gnu.org

8.4. Intel Links

Intel Home: http://www.intel.com
Intel Sandy-Bridge Processor: http://www.intel.com/technology/architecture-silicon/next-gen/
Intel Software Documentation Library: http://software.intel.com/en-us/articles/intel-software-technical-documentation/