info specific to sutton, rondeau, pelee: Difference between revisions

From Fluids Wiki
Jump to navigation Jump to search
(describe the Stastna/Poulin/Lamb CFI-funded servers)
 
(Add GPU section, update file system info and assorted application info, comment out all Slurm stuff)
(4 intermediate revisions by the same user not shown)
Line 32: Line 32:
* the old /fsys2, /fsys3, /fsys4, /fsys5, /fsys6, /fsys7 file systems are NFS-mounted from hood.math
* the old /fsys2, /fsys3, /fsys4, /fsys5, /fsys6, /fsys7 file systems are NFS-mounted from hood.math
** these are under /mnt/autofs/hood.math/
** these are under /mnt/autofs/hood.math/
** just 'cd' there and they will appear
* /fsys1 and /fsys2 are large file systems on sutton exported by NFS to the others
* /fsys1 and /fsys2 are large file systems on sutton exported by NFS to the others
** if you don't have a personal drectory here, ask MFCF
** these are under /mnt/autofs/sutton.math/
 
** just 'cd' there and they will appear
* /scratch is a large file system on bow exported by NFS to the others
** this is under /mnt/autofs/bow.math/
** just 'cd' there and it appears
* /local_scratch is a 3.4 TB local file system on each of pelee and rondeau
** this will be fastest for saving output of model runs
** when done, copy important results elsewhere (i.e. fsysN) for safekeeping and delete from /local_scratch so there's room for others to do their runs
* if you don't have a personal directory in these various places, ask MFCF


== Application Software, Development Environment ==
== Application Software, Development Environment ==
Line 51: Line 59:
** to see a list of available modules, run <code>module avail</code>
** to see a list of available modules, run <code>module avail</code>
*** this also shows which ones you have loaded already
*** this also shows which ones you have loaded already
** load one using its exact name as shown in the list e.g. MATLAB, <code>module load matlab/2017a-research</code>
** load one using its exact name as shown in the list e.g. MATLAB, <code>module load matlab/2022b-research</code>
** use <code>module unload ''modulename''</code> to unload a module, e.g. if you want to switch to an alternative one
** use <code>module unload ''modulename''</code> to unload a module, e.g. if you want to switch to an alternative one


Line 61: Line 69:
=== MPI environments ===
=== MPI environments ===


Choose from MPICH, OpenMPI,
* MPICH is not installed yet. Let MFCF know if it is needed.
* MPICH ???


* OpenMPI 4.1.2
* OpenMPI 4.1.2


=== MATLAB ===
=== MATLAB ===


* not installed yet
* version 2202b
* <code>module load matlab/2022b-research</code>
* check <code>module avail</code> to see available versions
* check <code>module avail</code> to see available versions
* <code>module load matlab/2017a-research</code>, or
* <code>module load matlab/2020a-research</code>


=== Python ===
=== Python ===
Line 82: Line 87:
=== Models ===
=== Models ===


* MIT GCM - NOT PREPARED YET - OLD NOTES
* MIT GCM - not prepared yet
** see config file called bow_mpt in the MITgcm section of the main Models page in this wiki
** eventually, see config file called sutton_mpi in the MITgcm section of the main Models page in this wiki
** first need to load an MPI module, such as mpt/2.16 or hmpt/2.25
** ensure you call genmake2 with the -mpi flag
** ensure you call genmake2 with the -mpi flag
** GNU Fortran compiler seems pickier about some things than Intel compiler so you may see new complaints about previously-working code
** here is a [[sample SLURM MITgcm script]].  See more SLURM information below.


* NCL NCARG version 6.4.0, without OPeNDAP - NOT PREPARED YET - OLD NOTES
* NCL NCARG - not installed yet
** set environment variable NCARG_ROOT to /opt/ncl-6.4.0 and add $NCARG_ROOT/bin to your $PATH
** set environment variable NCARG_ROOT to /opt/ncl-6.4.0 and add $NCARG_ROOT/bin to your $PATH
** optionally, make a .hluresfile in your home directory if you want to customize your NCL graphical environment
** optionally, make a .hluresfile in your home directory if you want to customize your NCL graphical environment


* SPINS - NOT PREPARED YET - OLD NOTES
* SPINS - coming soon
** so far we have one system configuration file
** spins2netcdf SPINS to NetCDF converter not installed yet
*** bow.gcc.mpt.blas.sh
*** with symbolic link to that called bow.sh, since it's the only one we have so far
** as the name suggests, it is set up to expect GCC, SGI/HPE MPT, and default BLAS and LAPACK
*** so you need to load the MPT module first
** we expect to develop alternative configuration files for other compilers, MPI implementations, and numerical libraries for comparison to find an optimum set-up
** the spins2netcdf SPINS to NetCDF converter is available in standard search rules at /usr/local/bin/spins2netcdf  (Version 2.0.0.0 from April 2018)


=== Visualization ===
=== Visualization ===


* ParaView 5.4.1 - NOT DONE YET
* ParaView - not installed yet
** installed at /opt/paraview/


* VisIT 2.13.0 - NOT DONE YET
* VisIT - not installed yet
** installed at /opt/visit/


* FFMPEG 4.4.x
* FFMPEG 4.4.x
Line 115: Line 109:
* the ImageMagick suite
* the ImageMagick suite
** commands such as 'display' and 'convert'; see "man imagemagick"
** commands such as 'display' and 'convert'; see "man imagemagick"
=== GPU computing ===
* pelee and rondeau have GPUs and the CUDA development environment
** <code>module load cuda/12.3.2</code>
** use <code>module avail</code> to check for other versions
<!--


== Scheduler - NOT UPDATED FOR THESE MACHINES YET ==
== Scheduler - NOT UPDATED FOR THESE MACHINES YET ==
Line 198: Line 200:


</pre>
</pre>
-->


<!--  
<!--  

Revision as of 12:26, 16 April 2024


A CFI proposal by Profs. Stastna, Poulin, and Lamb resulted in acquisition of new servers in late 2023.

Hostnames and Hardware

  • sutton.math.private.uwaterloo.ca (Dell R7625)
    • 2 AMD EPYC 9454 2.75 GHz 48-core CPUs, 768 GB RAM, ~90 TB storage
  • pelee.math.private.uwaterloo.ca (Dell XE8545)
    • 2x AMD EPYC 7713 2.0 GHz 64-core CPUs, 1 TB RAM, four NVIDIA A100 40 GB GPUs
  • rondeau.math.private.uwaterloo.ca (Dell XE8545)
    • same as pelee

Operating System Software and vendor add-ons

  • Ubuntu 22.04 LTS

System Administration

  • MFCF administers these machines. Users do not have root access.
  • System management is done by SaltStack software, unlike hood which is managed using MFCF's XHier
    • this means things will be different
    • applications are not under the /software directory anymore
    • default PATH does not have everything in it
    • details below

File Systems

  • home directories are NFS-mounted from sutton
    • that's where you are when you log in
    • do not run models from your home directory: it's too small
  • the old /fsys2, /fsys3, /fsys4, /fsys5, /fsys6, /fsys7 file systems are NFS-mounted from hood.math
    • these are under /mnt/autofs/hood.math/
    • just 'cd' there and they will appear
  • /fsys1 and /fsys2 are large file systems on sutton exported by NFS to the others
    • these are under /mnt/autofs/sutton.math/
    • just 'cd' there and they will appear
  • /scratch is a large file system on bow exported by NFS to the others
    • this is under /mnt/autofs/bow.math/
    • just 'cd' there and it appears
  • /local_scratch is a 3.4 TB local file system on each of pelee and rondeau
    • this will be fastest for saving output of model runs
    • when done, copy important results elsewhere (i.e. fsysN) for safekeeping and delete from /local_scratch so there's room for others to do their runs
  • if you don't have a personal directory in these various places, ask MFCF

Application Software, Development Environment

Wherever satisfactory versions are provided by the OS distribution, things like library and include files are installed at default locations such as /usr/lib64 and /usr/include.

Third party applications are installed as modules under /opt. Run the command module avail to see what's available.


Login shell environment

  • the recommended .cshrc and .profile files set up your environment automatically with recommended compilers, libraries, etc. mentioned below to work with the configuration files included with models such as SPINS, MITgcm, IGW
  • some optional software comes in "module" format (see man module)
    • to see a list of available modules, run module avail
      • this also shows which ones you have loaded already
    • load one using its exact name as shown in the list e.g. MATLAB, module load matlab/2022b-research
    • use module unload modulename to unload a module, e.g. if you want to switch to an alternative one


Compilers

  • gcc 11.4 is in standard search rules, no need to load it with a module command

MPI environments

  • MPICH is not installed yet. Let MFCF know if it is needed.
  • OpenMPI 4.1.2

MATLAB

  • version 2202b
  • module load matlab/2022b-research
  • check module avail to see available versions

Python

  • default python3 is 3.10.x
  • NumPY, SciPY, etc. are installed
  • you can install other packages, such as matplotlib, via: pip3 install --user matplotlib

Models

  • MIT GCM - not prepared yet
    • eventually, see config file called sutton_mpi in the MITgcm section of the main Models page in this wiki
    • ensure you call genmake2 with the -mpi flag
  • NCL NCARG - not installed yet
    • set environment variable NCARG_ROOT to /opt/ncl-6.4.0 and add $NCARG_ROOT/bin to your $PATH
    • optionally, make a .hluresfile in your home directory if you want to customize your NCL graphical environment
  • SPINS - coming soon
    • spins2netcdf SPINS to NetCDF converter not installed yet

Visualization

  • ParaView - not installed yet
  • VisIT - not installed yet
  • FFMPEG 4.4.x
    • in standard search rules
  • the ImageMagick suite
    • commands such as 'display' and 'convert'; see "man imagemagick"

GPU computing

  • pelee and rondeau have GPUs and the CUDA development environment
    • module load cuda/12.3.2
    • use module avail to check for other versions


Retrieved from "http://wiki.math.uwaterloo.ca/fluidswiki/index.php?title=info_specific_to_sutton,_rondeau,_pelee&oldid=1845"