TESTRUN v3–07 User Guide

by Bob Yantosca
Last Updated: October 8, 2009

Introduction | Directory Structure | Shell Scripts | Job Files | Log Files
Installation | Customization | Execution | Documentation

Click HERE to download a gzipped TAR file containing the latest TESTRUN release!
(Netscape users: try shift + left mouse button)

1. Introduction

TESTRUN is Perl-based software which you can use to submit GEOS–Chem model runs. A simple Perl script named trun creates scripts which compile GEOS–Chem, move the executable file to the proper run directory, and start the GEOS–Chem simulation. Output files will be given a unique timestamp, in order to prevent them from being overwritten by successive runs. This greatly facilitates testing and debugging of GEOS–Chem.

TESTRUN is meant to be used for running several GEOS–Chem jobs in the same run directory (i.e. for sensitivity studies or code development). The run directory must be set up manually. You may also keep several different run directories for various different versions of GEOS–Chem.

TESTRUN v3–07 (08 Oct 2009) is designed to be used with GEOS–Chem v8–01–01 and higher. It will compile GEOS–Chem source code using the proper makefile for the type of compiler that you are using. It is now possible to compile and run the GEOS–Chem code as separate processes. Also, users may customize TESTRUN based on their particular platform and local batch queue system.

TESTRUN v3–07 contains some minor improvements that will enable it to run more easily on Linux machines (under the "bash" shell) or other machines (under the "sh" shell). Some hardwiring of path names in the defaults files have also been removed. It also contains minor updates for compatibility with the new "Ceres" cluster at the Harvard University Atmospheric Chemistry Modeling Group.

TESTRUN v3–07 includes:

  • a total rewrite of the main trun script (first released in v3–01) in order to make it compatible with a newer version of the PBS batch queue system. Also, the format of the defaults files are now much more flexible and easy to use.

  • compatibility with the Intel Fortran Compiler (ifort) versions 9 and 10

  • compatibility with the SunStudio 11 & SunStudio 12 compilers

  • minor changes to make it more flexible to run with the Portable Batch System (PBS) queuing system

  • compatibility with the Sun Grid Engine (SGE) queuing system

  • ability to create reference documentation files in *.ps and *.pdf formats

  • compatibility with the new makefile structure in GEOS–Chem v8–02–03 and higher

2. TESTRUN Directory Structure

./testrun          -- directory containing the "trun" shell script
./testrun/defaults -- directory for defaults files for various queue systems
./testrun/doc/     -- directory where reference documentation will be built
./testrun/jobs/    -- directory where job scripts will be stored
./testrun/logs/    -- directory where GEOS–Chem log files will be stored
./testrun/runs/    -- directory containing GEOS–Chem run directories

3. TESTRUN Scripts

trun — Perl script which does the following:

  • Creates a compilation script (in Perl) which
    • Compiles the GEOS–Chem code using the proper Makefile for your operating system

    • Moves the GEOS–Chem executable to a subdirectory of ./testrun/runs/

    • Pipes information from the GEOS–Chem run to a log file in ./testrun/logs

    • Sends compilation error output to the error log ./testrun/logs/log.compile

  • Creates a job script (in Perl) which
    • Runs the GEOS–Chem code and issues the job a unique timestamp

    • Pipes information from the GEOS–Chem run to a log file in ./testrun/logs

    • Sends run error output to the error log ./testrun/logs/log.error

    • Appends a unique timestamp GEOS–Chem to output files to avoid overwriting of files

  • Submits compilation and job scripts to your batch queue system (or runs them directly if you do not have a batch queue system installed on your platform)

4. TESTRUN Job Files (in the testrun/jobs directory)

cmp.vX-YY-ZZ.DDDHHMM

trun creates this Perl script, which contains the commands to compile the GEOS–Chem model. The script will also move the executable from the code directory to a subdirectory of testrun/runs, in which the code will run.

For platforms which allow auto-submit, this script will also submit the GEOS–Chem run script (job.vX-YY-ZZ.DDDHHMM, discussed below), to the queue system.

The timestamp DDDHHMM contains the day of the year (DDD, from 0-365 or 0-366 in leap years), the hour (HH), and the minutes (MM) when the job was submitted. Each job submitted by trun will have a unique timestamp, therefore insuring that results from one run will not be overwritten by successive runs.

The version number is represented as vX-YY-ZZ, where X is the major version series number, YY is the minor version number, and ZZ is the release number.

   
run.vX-YY-ZZ.DDDHHMM

trun also creates this Perl script, which contains the commands to start a GEOS–Chem model simulation job. This script will also stamp the binary punch file (ctm.bpch) with the same DDDHHMM timestamp so that it may easily be identified.

5. TESTRUN Log Files (in the testrun/logs directory)

log.vX-YY-ZZ.DDDHHMM

This log file corresponds to the GEOS–Chem run denoted by timestamp DDDHHMM. It will contain output from both the compilation and execution phases of the job.

   
log.compile

This log file contains the standard error (stderr) output from the GEOS–Chem compilation job. If there were any compile-time errors, then these will be included in log.compile.

   
log.error

This log file contains the standard error (stderr) output from the GEOS–Chem run job. If the GEOS–Chem job terminates abnormally, then log.error should contain important information as to why the run stopped.

6. Installation

Let us assume that we want to use TESTRUN to start a job for GEOS–Chem v8–02–02. We will make the following assumptions in the examples below.

  • Our username is smith (a common last name!)
  • Our home directory is /home/smith
  • We have a GEOS–Chem v8–02–03 source code directory installed in ~/Code.v8-02-03
    /as/home/smith/Code.v8-02-03

Installation Instructions:

1

Change to your home directory:

cd /home/smith
   
2 Unzip and extract the scripts from the testrun.v3-07.tar file as follows:

tar xvzf testrun.v3-07.tar

This will create the following directory:

/home/smith/testrun

which has the following contents:

trun defaults/ logs/ jobs/ runs/

Where

trun is a Perl script and
defaults/
, jobs/, logs/, and runs/ are subdirectories of the testrun directory.

   
3

Create the following GEOS–Chem run directory:

cd /home/testrun/runs/
mkdir run.v8-02-03

and copy to this directory all of the relevant input and restart files for the particular GEOS–Chem simulation that you wish to run.

You may now customize TESTRUN to work with your particular batch queue system. TESTRUN currently supports the following queuing systems:

  • SGE (Sun Grid Engine) -- this is the default
  • PBS (Portable Batch System)
  • LSF (Load Sharing Facility)

TESTRUN may be also used even if you do not have a batch queue system installed on your computer.

7. Customizing TESTRUN for your platform and batch queue system

The testrun/defaults directory contains the following files:

defaults File contaning default settings for TESTRUN (read by trun script)
defaults.lsf File containing TESTRUN defaults for the LSF batch queuing system
defaults.pbs File containing TESTRUN defaults for the PBS batch queuing system
defaults.sge File containing TESTRUN defaults for the Sun Grid Engine (SGE) batch queuing system
defaults.sge.harvard File containing TESTRUN defaults for the Sun Grid Engine (SGE) batch queuing system -- for the configuration used at Harvard University. (Members of the Jacob group should use this file.)
defaults.none File containing TESTRUN defaults for no batch queuing system

The defaults* files allow you to specify the default GEOS–Chem code directory path, the default run directory path, the default queue, and the default queue submission commands. You may also specify the default testrun root directory.

NOTE: The defaults* files as packaged in the testrun.v3-07.tar.gz distribution file are customized to the Harvard computer systems. If you are located at a different institution you will have to edit these files to include the settings particular to your system.

If your machine uses the SGE (Sun Grid Engine) queuing system, then

  1. Type cd ~/testrun/defaults
  2. Add your changes (if any) to the file defaults.sge
  3. Type cp -f defaults.sge defaults

If your machine uses the PBS (Portable Batch System) queuing system, then

  1. Type cd ~/testrun/defaults
  2. Add your changes (if any) to the file defaults.pbs
  3. Type cp -f defaults.pbs defaults

If your machine uses the LSF (Load Sharing Facility) queuing system, then

  1. Type cd ~/testrun/defaults
  2. Add your changes (if any) to the file defaults.lsf
  3. Type cp -f defaults.lsf defaults

If your machine uses no batch queue system, then

  1. Type cd ~/testrun/defaults
  2. Add your changes to the file defaults.none
  3. Type cp -f defaults.none defaults

A sample file (defaults.sge) is provided below.

# $Id: defaults.sge,v 1.6 2009/08/12 20:47:31 bmy Exp $
#------------------------------------------------------------------------------
#          Harvard University Atmospheric Chemistry Modeling Group            !
#------------------------------------------------------------------------------
#BOP
#
# !INCLUDE: defaults.sge
#
# !DESCRIPTION: Defaults for TESTRUN and Sun Grid Engine queue system
# This file may be modified as is necessary for your system.
#\\
#\\
# TESTRUN compiles and executes GEOS-Chem as two separate processes.  A 
# separate compilation queue and run queue must be specified.  For some
# environments the code can be executed as soon as the compilation process
# is finished.
#\\
#\\
# !REMARKS:
# Replaceable tokens in default settings:
# =============================================================================
# {HOME}     is replaced by the user's home path dir (e.g. /home/bmy)
# {USER}     is replaced by the user's login id (e.g. bmy, phs)
# {VSTR}     is replaced by the GEOS-Chem version string (e.g. v8-01-01)
# {DSTR}     is replaced by the date string (e.g. DDDHHMM)
# {CQUEUE}   is replaced by the compilation queue name (e.g. sp)
# {CPE}      is replaced by the compilation parallel env string (e.g. sp 1)
# {CLOGERR}  is replaced by the compilation error logfile (e.g. log.compile)
# {CSCRIPT}  is replaced by the compilation script name 
# {RQUEUE}   is replaced by the run queue name (e.g. smp4)
# {RPE}      is replaced by the run parallel env string (e.g. smp4 4)
# {RLOGERR}  is replaced by the run error logfile (e.g. log.error)
# {RSCRIPT}  is replaced by the run script name
# {SUBMIT}   is replaced by the PBS command (e.g. qsub)
#
# Options for the SGE "qsub" command:
# =============================================================================
# -cwd          Executes the job from the current working directory
# -r y          Specifies that the job is re-runable
# -b y          Tells SGE not to parse {CSCRIPT}, {RSCRIPT} thru a Unix shell
# -j y          Merges the stderr stream with the stdout stream
# -o {CLOGERR}  Specifies stdout+stderr path from compiliation phase
# -o {RLOGERR}  Specifies stdout+stderr path from run phase
# -pe {CPE}     Specifies the compile queue name and # of processors
# -pe {RPE}     Specifies the run queue name and # of processors
#

==> Queue System
SGE

==> Queue Commands
qsub    qsub 
getPE   /as/bin/getPE

==> Code Directory
{HOME}/Code.{VSTR}

==> Default TESTRUN Root Directory
{HOME}/testrun

==> Default Run Directory
{HOME}/testrun/runs/run.{VSTR}

==> Default Compilation Queue 
smp

==> Default Compilation Command
{SUBMIT} -cwd -r y -b y -j y -o {CLOGERR} -pe {CPE} {CSCRIPT}

==> Default Run Queue
smp4

==> Default Run Command
{SUBMIT} -cwd -r y -b y -j y -o {RLOGERR} -pe {RPE} {RSCRIPT}

==> Platforms Supporting Auto-Submit

==> Executable File Name
geos

END OF FILE
#
# !REVISION HISTORY: 
#  09 Sep 2008 - R. Yantosca - Initial version
#  12 Aug 2009 - R. Yantosca - Removed support for Altix platform
#  12 Aug 2009 - R. Yantosca - Added ProTeX headers
#EOP
#------------------------------------------------------------------------------

Note that the default settings for code directory, and run directory will be used unless you explicitly specify these when calling the trun script. Also, if you wish to modify the

  • default compilation queue
  • default run queue
  • compilation script submit command
  • job script submit command

then please consult with your local computer guru.

NOTES:

  1. On some types of systems it possible for the trun compile script (i.e. cmp.vX–YY–ZZ.DDHHMM) to auto-submit the run script (i.e. run.vX–YY–ZZ.DDHHMM) to a different queue. You may list the system types which allow auto-submit below the tag which reads "==> Platforms Supporting Auto-Submit". This feature may or may not always work, depending on the configuration of your system.

8. Running GEOS–Chem with TESTRUN

Now that we have installed TESTRUN, we can use it to start a GEOS–Chem job. Change to your testrun directory, for example:

cd /home/smith/testrun

You may call the trun script with the following options:

-v  VERSION     specifies the GEOS-Chem version number (e.g. 8.01.01)
                VERSION may also be a character string (e.g. current)
-cq CQUEUE      specifies the queue in which the model will be compiled
-q  QUEUE       specifies the queue in which the model will run
-f  OPT1:OPT2:  specifies options for the "make" command
-r  RUNDIR      specifies the GEOS-Chem run directory
-c  CODEDIR     specifies the GEOS-Chem code directory
-x  DDDHHMM     will manually submit job script with id DDDHHMM
-w              will remove ("wipe") prior job scripts and log files

Here are some examples:

1

We will compile and run a GEOS–Chem job on a Linux compute node. We'll assume that this platform does not support auto-submit.

trun -v 8.02.03 -cq smp

We specify the compilation queue with the -cq smp option. If you would like to compile GEOS–Chem in the default compilation queue (which is specified in the defaults file), then you may omit this option.

  • The default run directory is assumed to be: ~/testrun/runs/run.v8-02-03.
  • The default source code directory is assumed to be: ~/Code.v8-02-03.
  • Output from the compilation phase will be sent to the log file: logs/log.v8-02-03.DDDHHMM
  • The stdout and stderr streams from the compilation phase will be sent to the log file named: logs/log.compile

After the model has been compiled successfully, you will have to call trun again to run GEOS–Chem:

trun -v 8.02.03 -q smp4 -x DDDHHMM

We specify the run queue with the -q smp4 option. If you would like to run GEOS–Chem in the default run queue (which is specified in the defaults file), then you may omit this option.

We also use the -x DDDHHMM option to specify that this is a continuation of the same GEOS–Chem simulation, rather than the start of a new simulation. This will execute GEOS–Chem in the run queue and append output from the model to the same log file logs/log.v8-02-03.DDDHHMM.

   

2

We will now compile and run a GEOS–Chem job again on a Linux compute node. Let us run in a queue with a longer time limit and with more available processors than in the previous example.

trun -v 8.02.03 -cq smp -w

We specify the compile queue with the -cq smp option.

The -w argument removes all previous files in the jobs/ and logs/ directories while creating the following new files:

jobs/cmp.v8-02-03.DDDHHMM   -- Compilation script
jobs/run.v8-02-03.DDDHHMM   -- Run script 
logs/log.v8-02-03.DDDHHMM   -- Log file

You can use -w to clear out the jobs and logs directories periodically, or if you are testing or debugging the model and don't want to save the scripts or log file output.

As in the previous example, you will have to call trun again to start the GEOS–Chem simulation. Type:

trun -v 8.02.03 -q smp8 -x DDDHHMM

Here we have used the option -q smp8 to specify the run queue. As before, the output from GEOS–Chem will be appended to the log file logs/log.v8-02-03.DDDHHMM.

   
3

We will compile and run a GEOS–Chem job on a Linux compute node. Here we will run the code in a different run directory than the default, but keep all of the other options from Example 2 above.

trun -v 8.02.03 -cq sp -r ~/smith/tmpruns -w

We specify the run directory with -r ~/smith/tmpruns.

The -w argument removes all previous files in the jobs and logs directories (see #2 above).

As in the previous example, you will have to call trun again to start the GEOS–Chem simulation. Type:

trun -v 8.02.03 -q smp8 -x DDDHHMM

Here we have used the option -q smp8 to specify the run queue. As before, the output from GEOS–Chem will be appended to the log file logs/log.v8-02-03.DDDHHMM.

   
4

We will compile and run a GEOS–Chem job on a Linux compute node. Here we will compile the code from a different code directory than the default. Otherwise we will keep everything the same as in Example 2 above.

trun -v 8.02.03 -cq sp -c ~/smith/Code.temporary -w

We specify the code directory with -c ~/smith/Code.temporary .

The -w argument removes all previous files in the jobs/ and logs/ directories (see #2 above).

After the model is compiled successfully, the GEOS–Chem run script should begin executing and the output will be appended to the same log file: logs/log.v8-02-03.DDDHHMM

As in the previous example, you will have to call trun again to start the GEOS–Chem simulation. Type:

trun -v 8.02.03 -q smp8 -x DDDHHMM

Here we have used the option -q smp8 to specify the run queue. As before, the output from GEOS–Chem will be appended to the log file logs/log.v8-02-03.DDDHHMM.

5

If you wish to pass options to the make command, then you can use the -f keyword to trun, as shown here:

trun -v 8.02.03 -f NTRAC=54:KPPSOLVER=lsodes:TRACEBACK=yes ...

You may specify as many options as you wish; however each option must be be separated with a semicolon, colon, or comma. You must not leave spaces between each option, or else trun will not be able to interpret the string correctly.

9. Building the TESTRUN reference documentation

The trun script and default input files use the ProTeX automatic documentation system. This enables you to create reference documents in *.pdf and *.ps formats from the comments in the subroutine headers.

To build the reference documents, make sure you are in the doc/ subdirectory, then type:

make doc

This will create the following documents in the doc/ subdirectory:

trun.pdf   : Reference document in *.pdf format
trun.ps    : Reference document in *.ps format
trun.tex   : Reference document in LaTeX script

The reference documents contain a description of each subroutine within the trun Perl script, the variables that are passed to it as input & output arguments, and the revision history. The full text of each input file in the defaults/ directory is also listed.

If you wish to remove the trun reference documentation files, then make sure you are in the doc/ subdirectory and type:

make clean

Bob Yantosca
yantoscaseas.harvard.edu
08 Oct 2009