Graham Tips

The following are a list of scripts, functions and aliases that will make your life on the Sharcnet Graham cluster much easier. Section 1, Interactive Jobs, provides information on running interactive jobs on compute nodes. Section 2, Submitting Jobs, provides submission scripts to your jobs. Section 3, ~/.bashrc, lists other useful commands for checking job information (such as memory usage, expected start time, etc), changing directories to a running job. Section 4, Visualization provides some information on running visualization software on Graham.

Interactive Jobs

Graham does not have any nodes set aside specifically for interactive work. Instead, users can request time on the regular compute nodes for interactive jobs. This can be done using salloc (Slurm allocation). Some things to keep in mind are:

Once granted, you have sole access to the requested processors (unlike the Orca development nodes, which were shared)
You won't necessarily get immediate access to the nodes. Depending on how many processors / for how long / how much memory you request, you may have to wait several minutes or more before being granted the allocation.
After calling salloc, your terminal session will be automatically redirected to the assigned compute node. Once you're finished, exit relinquishes the compute nodes.
The environment variable SLURM_NTASKS stores the number of requested processors, in case you forget.

To request interactive processors, use:

salloc --time=DD-HH:MM --mem_per_cpu=<number>G --ntasks=<number> --account=<your_account>

where --time=DD-HH:MM specifies the days (DD), hours (HH), and minutes (MM), for which you would like the compute nodes, --mem_per_cpu=<number>G specifies the number of GB per processor that you require, --ntasks=<number> specifies the number of processors, and --account-<your_account> specifies to which account the cpu-hours should be charged. In general, your default account should be fine.

In general, X-forwarding is not enabled by default. To enable X-forwarding add the flag --x11 in the salloc command.

Notes

The allocation, and any processes running therein, will be terminated, without warning, after the requested amount of run time expires, so be sure to request a sufficient amount of time.
If you exceed the amount of requested memory, the salloc session will terminate.

See Changing Prompt Colour when on Compute Nodes to set up your terminal prompt to automatically change colour when using a compute node.

For more information, see the official Graham documention.

Interactive Jupyter (iPython)

This sharcnet page provides information on running interactive Jupyter sessions on Graham. It allows you to host the session on Graham (using multiple processors, a large amount of memory, as well as GPUs), while interacting with the session on your local machine via your favourite web browser. There are a few steps required to set it up, but once done it is very straight-forward.

Jupyter notebooks (formerly iPython notebooks) allow an interactive environment for python (it can also be used for other things, such as Matlab, R, etc., but in this instance is only set up for python), which, when combined with access to the Graham servers, can provide a useful way to interact with your data.

Jupyter sessions are killed without warning when the requested time limit has been exceeded, so remember to save often!

Visualization

See Visualization for more information about using ParaView or VisIt on Graham.

Submitting Jobs

The following are recommendations for the use of our contributed resources on Graham. As the UW fluids group, we have been allocated 832 processor years (for 2017-2018). Until these processor years have been expended, we have priority to run jobs on any processor ahead of regular users. Much of what you need to know for submitting and monitoring jobs is presented here. Further questions may be answered on the Compute Canada Graham page Running jobs.

Remember to be courteous about your memory usage and use only what you need! However, if you are using entire nodes then this does not apply since no one else will have access to that memory.

Submit script

An MPI job may be submitted to the Graham scheduler with either of the following bash scripts (A suggested name for them is submit.sh). The first script is for use of entire nodes, while the second allows for the processors to be spread out over many nodes. The first will take longer in queue, but should in theory run quicker since all processors have fewer connections. The second will likely start quicker since there is no requirement to wait for entire nodes to become available and can start whenever enough processors become available. Each script is broken into two parts: 1) the run dependent parameters and 2) the permanent parameters.

To submit the job execute sbatch submit.sh in a login window. Since this file won't go anywhere it is a handy way to look up what submission parameters you used (number of processors, memory, etc).

One thing to be aware of is that the memory requirement MUST be an integer. Decimals will not be accepted. The unit can be changed to Gigs (G) or other.

The requested time will play are large role in the time the job spends in queue. This is because the nodes on Graham have been specified to only accept jobs with a run time less than specific values. That is, there are far fewer nodes that accept jobs with a run time of 28 days than there are which accept a 3 hour run time. This is because the 3 hour job can run on any node (through back-filling into gaps on the other nodes), while the larger duration job will only run on nodes which accept it. The partitions are split into the following ways:

3 hours or less
12 hours or less
24 hours (1 day) or less
72 hours (3 days) or less
7 days or less
28 days or less

In summary, pick a value on this list and not something just larger than that (for example, pick 24 hours, not 25).

Complete Nodes

The run dependent parameters are the first three items: the number of nodes (and therefore the total number of processors since each node has 32), the duration of the job, and an identifiable name for the job. The remaining permanent parameters need not be changed from run to run. You will need to replace the mail option to use your email address. Additional options exist for when to be emailed (i.e. start/end/failure of job), and many other possibilities such as waiting for another job to complete (which is useful as a post-processing job). See the sbatch manual page for more information.

The following script will use the UW Fluids contributed account which has high priority. If for any reason you decide to submit the job to the regular queue, change the account info to --account=def-supervisor where supervisor is the username of your supervisor (this could be def-mmstastn, or def-kglamb, ...).

Here is the script to put into submit.sh:

#!/bin/bash
# bash script for submitting a job to the sharcnet Graham queue

#SBATCH --nodes=2               # number of nodes to use
#SBATCH --time=03-00:00         # time (DD-HH:MM)
#SBATCH --job-name="A name"     # job name

#SBATCH --ntasks-per-node=32                # tasks per node
#SBATCH --mem=128000M                       # memory per node
#SBATCH --output=sim-%j.log                 # log file
#SBATCH --error=sim-%j.err                  # error file
#SBATCH --mail-user=username@uwaterloo.ca   # who to email
#SBATCH --mail-type=FAIL                    # when to email
#SBATCH --account=ctb-mmstastn              # UW Fluids designated resource allocation
srun ./case1.x

Partial Nodes

The run dependent parameters are the first four items: the number of processors ("tasks"), the memory per processor, the duration of the job, and an identifiable name for the job. As in the complete node section the remaining permanent parameters need not be changed from run to run.

Here is the script to put into submit.sh:

#!/bin/bash
# bash script for submitting a job to the sharcnet Graham queue

#SBATCH --ntasks=64             # number of MPI processes
#SBATCH --mem-per-cpu=3G        # memory per processor (default in Mb)
#SBATCH --time=03-00:00         # time (DD-HH:MM)
#SBATCH --job-name="A name"     # job name

#SBATCH --output=sim-%j.log                 # log file
#SBATCH --error=sim-%j.err                  # error file
#SBATCH --mail-user=username@uwaterloo.ca   # who to email
#SBATCH --mail-type=FAIL                    # when to email
#SBATCH --account=ctb-mmstastn              # UW Fluids designated resource allocation
srun ./case1.x

Automatically Specifying SPINS Runtime

Newer versions of SPINS include the spins.conf-specified variable compute_time. This variable tells SPINS how much time has been allocated, so that SPINS can save the state when nearing the end of the run-time, even if it's not near a regular output time (such saves have the suffix .dump). Previously, the user had to specify this variable each time. However, thanks to the SLURM scheduler used on Graham, it is possible to have the submit script automatically detect the requested run-time and update the spins.conf accordingly.

There are two steps to using this feature:

bash function

First, include the following function in your ~/.bashrc. Note: it is very important that the string lines DO NOT begin with whitespace. Ugly though it may look, it is necessary, so please be careful to copy the function as-is.

# Determine, in seconds, the amount of time remaining for a job
#   Input to the function is the jobid
function remainingJobTime() {
    SLURM_JOB_ID=$1

    # A bit of bash magic, courtesy of Tyson Whitehead:
    #    https://www.sharcnet.ca/my/problems/ticket/33549
    local secondsleft=$( sacct -n -j ${SLURM_JOB_ID} --format=elapsed,timelimit \
        | awk -e '$2 != ""'\
'{ '\
'    patsplit($1,now,"[0-9]+"); '\
'    patsplit($2,end,"[0-9]+"); '\
'} END '\
'{ '\
'    endraw = length(end)==4 ? ((end[1]*24+end[2])*60+end[3])*60+end[4] : (end[1]*60+end[2])*60+end[3]; '\
'    nowraw = length(now)==4 ? ((now[1]*24+now[2])*60+now[3])*60+now[4] : (now[1]*60+now[2])*60+now[3]; '\
'    print(endraw-nowraw); '\
'}' )
    echo $secondsleft
}
export -f remainingJobTime

# Clean function wrapper to handle updating the spins.conf with accurate run-time variables
function updateSPINSruntime() {
    sed -i '/compute_time/c\'"compute_time = `remainingJobTime ${SLURM_JOB_ID}`" spins.conf
}
export -f updateSPINSruntime

submit script command

Next, include the following line in your submit.sh script after the #SBATCH flags but before calling mpiexec (or mpirun, srun, etc.).

updateSPINSruntime

Recording submitted jobs

It is sometimes useful to be able to review your submitted jobs that have already finished (trying to recall what you ran last week, etc.). While the sq_hist function provides that, it may not be the most convenient. Another option is to store a log of your submitted jobs. The following will allow you to automatically log information about jobs that start running into the file ~/my_jobs.log.

`~/.bashrc function`

# Update a logfile with job information
#   first (and only) argument is a string indicating the
#   log-file. This is useful for keeping separate log files
#   for separate simulation sets.
function updateJobSummary() {

    # If a log file is not provided, use the default
    DEFAULT_LOG_FILE="${HOME}/my_jobs.log"
    LOGFILE=${1:-${DEFAULT_LOG_FILE}};

    # Check if something has previously been logged for this jobid
    #    if yes: we're now recording the end
    #    if no : we're recording the beginning
    oldSum=`grep ${SLURM_JOB_ID} ${LOGFILE}`
    if [ "${oldSum}" == "" ]; then
        FLAG="BEG";
    else
        FLAG="END";
    fi

    date_beg=`squeue --noheader -o "%S" -j ${SLURM_JOB_ID}`

    if [ "$FLAG" == "BEG" ]; then
        date_end=`squeue --noheader -o "%e" -j ${SLURM_JOB_ID}`
        date_end="${date_end} (est.)";
    else
        date_end=$(perl -e 'use POSIX;print strftime "%Y-%m-%dT%H:%M:%S",localtime time;');
    fi

    if [ -z "${SLURM_NTASKS_PER_NODE+x}" ] || [ "${SLURM_NTASKS_PER_NODE}" == "" ] ; then
        procs="${SLURM_NPROCS} (total procs)"
    else
        procs="${SLURM_NSTASKS_PER_NODE} (procs per node)";
    fi

    if [ -z "${SLURM_MEM_PER_CPU+x}" ]; then
        mem="${SLURM_MEM_PER_NODE} (mem per node)"
    else
        mem="${SLURM_MEM_PER_CPU} (mem per cpu)";
    fi

    jobsum="${date_beg} : ${date_end} : ${SLURM_JOB_ID} : ${SLURM_JOB_NAME} : ${SLURM_NNODES} (nodes) : ${procs} : ${mem} : ${SLURM_SUBMIT_DIR} "

    if [ "$FLAG" == "BEG" ]; then
        echo ${jobsum} >> ${LOGFILE};
    else
        sed -i "s@${oldSum}@${jobsum}@" $LOGFILE
    fi
}
export -f updateJobSummary

`submit.sh`

After defining the updateJobSummary function in ~/.bashrc, all that you need to do is call updateJobSummary both before and after calling you code in your submit.sh script. An example is given below.

#SBATCH flags...

# Update job summary with initial information
updateJobSummary

# Update spins.conf with accruate run-time information
updateSPINSruntime

# run spins
mpiexec casefile.x

# Update the job summary with the final time
updateJobSummary

~/.bashrc

Copy the following aliases and scripts into ~/.bashrc to make them available to you at the command line.

Changing Prompt Colour when on Compute Nodes

The following snippet will set the prompt to a golden yellow colour when on a Graham login node, and a red-purple, when on a compute (via salloc) node.

if [ -z "${SLURM_NTASKS+x}" ]; then
    export PS1='\[\e[1;33m\][\u@\h \W]\$\[\e[0m\] '
else
    export PS1='\[\e[38;5;125m\][\u@\h \W]\$\[\e[0m\] '
fi

Check Job Status and Nodes Usage

sqm gives a summary of all of the running jobs by ${USER}
sqa gives a summary of all jobs running with the UW Fluids group contributed resources
ssm gives the the fair share values and recent cpu-seconds used by ${USER}
ssa gives the the fair share values and recent cpu-seconds used of all users of the UW fluids contributed resources

${USER} will be automatically replaced by your userid when called.

alias sqm='squeue -u ${USER}'
alias sqa='squeue --account=ctb-mmstastn_cpu,rrg-mmstastn_cpu'
alias ssm='sshare -U ${USER}'
alias ssa='sshare -a -A ctb-mmstastn_cpu -A rrg-mmstastn_cpu'

See Scheduling Policies to find out more about fair share, but the basic idea is that values closer to 1 have highest priority and values close to 0 have lowest. 0.5 will result in a job wait time being roughly the "average" wait time for all jobs on the cluster. Depending on recent usage of your default account or the ctb-mmstastn account, either account could have higher priority. The ssm command will easily show the fair share value of each of the accounts your group is allocated. A job submitted with the account that has the higher fair share value will start faster than any other account.

Currently, we have 832 processor-years allocated to our group for the year (2017-Apr 2018). We would like to have, on average, about 832 processors running at any given moment. There is no harm in going over or under this number, so long as we roughly complete 832 processor-years by the time our allocation expires. To check the current amount of cpus that are running or pending run

function nodeUsage() {
    jobsR=`squeue --account=ctb-mmstastn_cpu,rrg-mmstastn_cpu -o %C -t R`
    jobsP=`squeue --account=ctb-mmstastn_cpu,rrg-mmstastn_cpu -o %C -t PD`

    cpuSUM_R=0
    cpuSUM_P=0
    count=0

    IFS='
    '
    for x in $jobsR;
    do
        if [ "$count" -gt "0" ]; then
            cpuSUM_R=$((x + cpuSUM_R));
        else
            count=1;
        fi
    done

    count=0;

    IFS='
    '
    for x in $jobsP;
    do
        if [ "$count" -gt "0" ]; then
            cpuSUM_P=$((x + cpuSUM_P));
        else
            count=1;
        fi
    done

    perc_R=$((100*$cpuSUM_R/832 + 200*$cpuSUM_R/832 % 2)) # 2nd term is for rounding
    perc_P=$((100*$cpuSUM_P/832 + 200*$cpuSUM_P/832 % 2)) # 2nd term is for rounding
    perc_all=$((100*($cpuSUM_R + $cpuSUM_P)/832 + 200*($cpuSUM_R + $cpuSUM_P)/832 % 2))

    echo "Processors running:   $cpuSUM_R ($perc_R%)";
    echo "Processors pending:   $cpuSUM_P ($perc_P%)";
    echo "Processors total:     $((cpuSUM_P+cpuSUM_R)) ($perc_all%)";
    echo "Processors available: $((832 - cpuSUM_R-cpuSUM_P))";
}

Resource Allocation Usage

The total amount of cpu-years charged to the our contributed resources is not accessible within a login node and must be accessed online. Follow these steps to see the allocated resource usage:

sign into the Compute Canada Database
Select View Group Usage under My Account
Select By Resource Allocation Project
Select a given year and then choose a project (Currently it is pim-260-ab)

Memory storage can be presented in a login window with the command diskusage_report

Job details

The following three commands give information about a particular job.

scj (slurm control job): processors and nodes used by a job
saj (slurm account job): memory and time used by a job (use on completed jobs)
ssj (slurm status job): job memory (use on running jobs)
job_summary: print all of the above

The memory used is the AveRSS (in KB), which is the average memory per task (cpu).

function scj() {
    scontrol show jobid -dd $1
}   
function saj() {
    sacct --format=jobid,JobName,ncpus,ntasks,state,reserved,elapsed,End,MaxVMSize,AveVMSize,MaxRSS,ReqMem -j $1
}
function ssj() {
    sstat --format=jobid,ntasks,AveVMSize,MaxVMSize,AveRSS,MaxRSS -j $1
}
function job_summary() {
    scj $1;
    saj $1;
    ssj $1;
}

Usage is:

$ scj <jobID>

Move to a Simulation/Job Directory

You may find that your directory tree becomes rather involved after a while, and so changing into a simulation directory (or just remembering the path) can start to be cumbersome. A useful function is cdJob, which takes you into the working directory for a submitted job, provided that you know the jobID (which can be given by sqm).

This command only works for running jobs.

function cdJob() {
    pth=$(squeue -o %Z -j $1 | sed '1d')
    echo "cd-ing to ${pth}"
    cd ${pth}
}

Usage is:

$ cdJob <jobID>

List Submitted Jobs

sq_hist defaults to showing the jobs submitted in the last week. Option argument is of the form YYYY-MM-DD.

Note that this does not display any jobs from salloc / interactive jobs that are not assigned a name.

function sq_hist() {
    # Date for one week ago
    dt=$(perl -e 'use POSIX;print strftime "%Y-%m-%d",localtime time-604800;')

    # If argument given (YYYY-MM-DD), use that. Else default to last week.
    TIME=${1:-${dt}};
    echo $TIME;
    sacct --starttime ${TIME} -X --format=jobid,jobname,reserved,alloccpus,state,exitcode  | grep -v " sh ";
}

Graham Tips

Contents

Interactive Jobs

Notes

Interactive Jupyter (iPython)

Visualization

Submitting Jobs

Submit script

Complete Nodes

Partial Nodes

Automatically Specifying SPINS Runtime

bash function

submit script command

Recording submitted jobs

`~/.bashrc function`

`submit.sh`

~/.bashrc

Changing Prompt Colour when on Compute Nodes

Check Job Status and Nodes Usage

Resource Allocation Usage

Job details

Move to a Simulation/Job Directory

List Submitted Jobs

Navigation menu

Graham Tips

Interactive Jobs

Notes

Interactive Jupyter (iPython)

Visualization

Submitting Jobs

Submit script

Complete Nodes

Partial Nodes

Automatically Specifying SPINS Runtime

bash function

submit script command

Recording submitted jobs

~/.bashrc function

submit.sh

~/.bashrc

Changing Prompt Colour when on Compute Nodes

Check Job Status and Nodes Usage

Resource Allocation Usage

Job details

Move to a Simulation/Job Directory

List Submitted Jobs

Navigation menu

Search

`~/.bashrc function`

`submit.sh`