View source for info specific to sutton, rondeau, pelee



A CFI proposal by Profs. Stastna, Poulin, and Lamb resulted in acquisition of new servers in late 2023.

== Hostnames and Hardware ==

* sutton.math.private.uwaterloo.ca  (Dell R7625)
** 2 AMD EPYC 9454 2.75 GHz 48-core CPUs, 768 GB RAM, ~90 TB storage
* pelee.math.private.uwaterloo.ca (Dell XE8545)
** 2x AMD EPYC 7713 2.0 GHz 64-core CPUs, 1 TB RAM, four NVIDIA A100 40 GB GPUs
* rondeau.math.private.uwaterloo.ca (Dell XE8545)
** same as pelee

== Operating System Software and vendor add-ons ==

* Ubuntu 22.04 LTS

== System Administration ==

* MFCF administers these machines. Users do not have root access.
* System management is done by SaltStack software, unlike hood which is managed using MFCF's XHier
** this means things will be different
** applications are not under the /software directory anymore
** default PATH does not have everything in it
** details below

== File Systems ==

* home directories are NFS-mounted from sutton
** that's where you are when you log in
** do not run models from your home directory: it's too small
* the old /fsys2, /fsys3, /fsys4, /fsys5, /fsys6, /fsys7 file systems are NFS-mounted from hood.math
** these are under /mnt/autofs/hood.math/
** just 'cd' there and they will appear
* /fsys1 and /fsys2 are large file systems on sutton exported by NFS to the others
** these are under /mnt/autofs/sutton.math/
** just 'cd' there and they will appear
* /scratch is a large file system on bow exported by NFS to the others
** this is under /mnt/autofs/bow.math/
** just 'cd' there and it appears
* /local_scratch is a 3.4 TB local file system on each of pelee and rondeau
** this will be fastest for saving output of model runs
** when done, copy important results elsewhere (i.e. fsysN) for safekeeping and delete from /local_scratch so there's room for others to do their runs
* if you don't have a personal directory in these various places, ask MFCF

== Application Software, Development Environment ==

Wherever satisfactory versions are provided by the OS distribution, things like library and include files are installed at default locations such as /usr/lib64 and /usr/include.

Third party applications are installed as modules under /opt. Run the command <code>module avail</code> to see what's available.



=== Login shell environment ===

* the recommended .cshrc and .profile files set up your environment automatically with recommended compilers, libraries, etc. mentioned below to work with the configuration files included with models such as SPINS, MITgcm, IGW
** see the [[Login script]] wiki page for details
* some optional software comes in "module" format (see <code>man module</code>)
** to see a list of available modules, run <code>module avail</code>
*** this also shows which ones you have loaded already
** load one using its exact name as shown in the list e.g. MATLAB, <code>module load matlab/2022b-research</code>
** use <code>module unload ''modulename''</code> to unload a module, e.g. if you want to switch to an alternative one


=== Compilers ===

* gcc 11.4 is in standard search rules, no need to load it with a module command

=== MPI environments ===

* MPICH is not installed yet. Let MFCF know if it is needed.

* OpenMPI 4.1.2

=== MATLAB ===

* version 2202b
* <code>module load matlab/2022b-research</code>
* check <code>module avail</code> to see available versions

=== Python ===

* default python3 is 3.10.x
* NumPY, SciPY, etc. are installed
* you can install other packages, such as matplotlib, via: <syntaxhighlight lang="bash" inline> pip3 install --user matplotlib </syntaxhighlight>

=== Models ===

* MIT GCM - in progress
** eventually, see config file called sutton_gcc_openmpi in the MITgcm section of the main Models page in this wiki
** ensure you call genmake2 with the -mpi flag

* NCL NCARG - not installed yet
** set environment variable NCARG_ROOT to /opt/ncl-6.4.0 and add $NCARG_ROOT/bin to your $PATH
** optionally, make a .hluresfile in your home directory if you want to customize your NCL graphical environment

* SPINS
** use the sutton.gcc.openmpi.blas.sh configuration file from the SPINS systems folder
** spins2netcdf SPINS to NetCDF converter not installed yet

=== Visualization ===

* ParaView - 5.10.1
** <code>module load paraview/5.10.1</code>

* VisIT - not installed yet

* FFMPEG 4.4.x
** in standard search rules

* the ImageMagick suite
** commands such as 'display' and 'convert'; see "man imagemagick"

=== GPU computing ===

* pelee and rondeau have GPUs and the CUDA development environment
** <code>module load cuda/12.3.2</code>
** use <code>module avail</code> to check for other versions

<!--

== Scheduler - NOT UPDATED FOR THESE MACHINES YET ==

These machines use SLURM to schedule jobs. 
This means that instead of running jobs live, they need to be submitted to a queue.
See the Submitting Jobs section of the [[Graham Tips]] page for more about SLURM

=== Expected use of SLURM ===

* SLURM head node is fluids-pr1-01.math.private.uwaterloo.ca, alias fluidssubmit.math.private.uwaterloo.ca
** runs a different OS from compute nodes
** can't compile there, only submit jobs there
* need SLURM module in your environment
** recommended .cshrc and .profile do that for you (see the [[Login script]] page)
* must include <syntaxhighlight lang="bash" inline>#SBATCH --partition=<partition_name></syntaxhighlight> flag in your submit script
** use partition name <syntaxhighlight lang="bash" inline>fluids_short</syntaxhighlight> for jobs under 8 hours
** use partition name <syntaxhighlight lang="bash" inline>fluids_long</syntaxhighlight> for jobs up to 40 days

=== MPI jobs that span the machines: The easy way (but not recommended) ===

When you want to span an MPI job across multiple machines, performance should be better if you force the use of the private 40 Gb network that the
three machines share.  This is a bit awkward with SLURM (see next section).
As long as there is no contention for resources, you may run such jobs outside of SLURM.
With the mpirun command, use "-a mountainlakes" to specify the machine-spanning array, and comma-separated special hostnames with -np for number of processes.
The special hostnames have "-pn" (for "private network") appended to identify the dedicated high-speed network interfaces.
For example (assuming you are using HPE/SGI MPT, not OpenMPI or MPICH):

<pre>
% mpirun -a mountainlakes waterton-pn -np 2, minnewanka-pn -np 2 ./a.out
Hello, world! I am 2 of 4 on host minnewanka.
Hello, world! I am 0 of 4 on host waterton.
Hello, world! I am 3 of 4 on host minnewanka.
Hello, world! I am 1 of 4 on host waterton.
</pre>

See "man array" for ways you can inspect activities across the array of machines.

=== MPI jobs that span the machines: The awkward way (recommended) ===

If there is any contention for resources, use SLURM. Special steps are needed to force it to use the private high-speed network because the SLURM head node cannot see that network itself. By using the --nodelist option, the SLURM_JOB_NODELIST environment variable, and some list manipulation, we can pass the special private network hostnames (suffix "-pn") through to the mpirun command.  Here is an example sbatch script:

<pre>
 #!/bin/bash
#SBATCH --mail-user=<userid>@uwaterloo.ca
#SBATCH --mail-type=BEGIN,END,FAIL
#SBATCH --job-name="mpi_job"
#SBATCH --account=igw_acct       #<== your slurm account
#SBATCH --partition=fluids_long  #<== partition: fluids_long or fluids_short
#SBATCH --time=00:01:00
#SBATCH --nodelist=waterton,minnewanka  #<== specify machines
#SBATCH --ntasks=16
#SBATCH --ntasks-per-node=8
#SBATCH --mem-per-cpu=1GB
#SBATCH --output=%j.out
#SBATCH --error=%j.err

# Your code below this line
# First set the environment for using  mpt
module load mpt

# compile your code
make clean
make

# create an array of nodes list from  SLURM_JOB_NODELIST env. variable
nodes=($(echo $SLURM_JOB_NODELIST | tr "," " "))

# debugging  messages
echo "=========================================="
echo "slurm ntasks      = $SLURM_NTASKS         "
echo "slurm ntasks/Node = $SLURM_NTASKS_PER_NODE"
echo "slurm Nodes = $SLURM_JOB_NODELIST"
echo "slurm Node[0] = ${nodes[0]}"
echo "slurm Nodes[1] = ${nodes[1]}"
echo "=========================================="
echo
echo
set -x

mpirun -a mountainlakes ${nodes[0]}-pn -np $SLURM_NTASKS_PER_NODE, ${nodes[1]}-pn -np  $SLURM_NTASKS_PER_NODE ./my_mpi_program


</pre>

-->

<!-- 

=== Checking Node Usage ===

The following function displays the extent of current processor usage, including pending and running jobs.
It has been adapted from [[Graham Tips#Check Job Status and Nodes Usage|the Graham function]].

<syntaxhighlight lang="bash">
function nodeUsage() {
    jobsR=`squeue --account=fluids_acct -o %C -t R`
    jobsP=`squeue --account=fluids_acct -o %C -t PD`

    cpuSUM_R=0
    cpuSUM_P=0
    count=0

    IFS='
    '
    for x in $jobsR;
    do
        if [ "$count" -gt "0" ]; then
            cpuSUM_R=$((x + cpuSUM_R));
        else
            count=1;
        fi
    done

    count=0;

    IFS='
    '
    for x in $jobsP;
    do
        if [ "$count" -gt "0" ]; then
            cpuSUM_P=$((x + cpuSUM_P));
        else
            count=1;
        fi
    done

    perc_R=$((100*$cpuSUM_R/84 + 200*$cpuSUM_R/84 % 2)) # 2nd term is for rounding
    perc_P=$((100*$cpuSUM_P/84 + 200*$cpuSUM_P/84 % 2)) # 2nd term is for rounding
    perc_all=$((100*($cpuSUM_R + $cpuSUM_P)/84+ 200*($cpuSUM_R + $cpuSUM_P)/84 % 2))

    echo "Processors running:   $cpuSUM_R ($perc_R%)";
    echo "Processors pending:   $cpuSUM_P ($perc_P%)";
    echo "Processors total:     $((cpuSUM_P+cpuSUM_R)) ($perc_all%)";
    echo "Processors available: $((84 - cpuSUM_R-cpuSUM_P))";
}
</syntaxhighlight>

-->