SPINS User Guide

Welcome to the SPINS user guide.

Other information can be found on this SPINS Documentation page, though not everything listed there is fully incorporated into the master SPINS branch.

The basics

The SPINS model is a Navier-Stokes solver that gets parameters and initial/boundary conditions from calls to user-provided routines. The user-provided routines are encapsulated in class derived from BaseCase (see BaseCase.hpp).

Creating your own custom configuration involves supplying the user-provided routines in a derived class based on BaseCase. The case file cases/doc_minimal.cpp shows the structure of a case file. It usually makes sense to start with a similar case file and customise it.

SPINS components

SPINS consists of a bunch of C++ source files and a bunch of case files, and it requires four libraries. UMFPack, AMD, and Blitz++ are supplied with SPINS, and it uses the system-installed FFTW. As a parallel program, it also requires MPI, so you must ensure that your environment refers to a system-installed MPI suite.

Directory structure:

• spins/src - SPINS source files
• spins/src/cases - A few dozen example case files
• spins/src/Science - functions for real-time analysis
• spins/matlab - Some helper functions for MATLAB analysis

How to get SPINS running

SPINS is hosted in a git repository on the UW git server. A guide to using git can be found here: http://git-scm.com/book. If your system does not have a working copy of git that can access http-based repositories (winisk and kazan are currently notable examples), you may get a full copy of the current repository from Christopher Subich (this replaces step #1 below).

You will need to get the code, build the dependencies, build the model and then run it.

Extracting the code from the git repository

• Create a directory in which to store the code.
• In that directory type "git clone https://git.uwaterloo.ca/SPINS/SPINS_main.git".
• This will create a directory called SPINS_main.

Building SPINS

One-time setup

• Go to the systems directory.
• Type ./makemake.sh [system]. This will construct the system-specific settings for the makefile.
  • This script reads and processes an appropriate script from the systems/ subdirectory; these files contain variable definitions for compiler names, include/library options, and other attributes that are necessary at build time.
  • There are several scripts in the systems/ subdirectory, and in general one needs to be written for each unique system based on its idiosyncrasies.
  • makemake.sh takes the system name as an optional argument. If not specified, it will try to guess the appropriate host file based on the current hostname. E.g., building on winisk will try to read systems/winisk.sh. If this is inappropriate (such as on clusters, where login nodes are numbered), then the system name can be specified at the command-line, e.g. ./makemake.sh graham to read systems/graham.sh.
• Execute ./make_deps.sh in the main repository by typing ./make_deps.sh [system] -j. This file may need to be edited to select which libraries aren't present in the current build environment; when in doubt build everything. FFTW and Boost are the libraries most likely to be already installed system-wide.

To build your case-file

• Enter the src directory.
• Type make cases/case_directory/your_case.x
• This requires a file called your_case.cpp in the cases/case_directory directory. There are several cases included with the code so you may want to start with one of those.
• After successful compilation, an executable called your_case.x is created.

Running SPINS

• Please be careful not to run in your home directory on machines like boogaloo or graham. Typical cases output a great deal of data, and can quickly fill up your /home quota (or the full partition, on non-quota systems).
• The code can be executed using mpirun e.g "mpirun -np 4 ./your_case.x".
  • ensure your environment is set up for MPI first
  • e.g. module load openmpi-1.10-x86_64; see machine-specific info on the LINUX/UNIX systems page
• Most cases require a configure file called spins.conf. Options included in the command-line or spins.conf are configured in the main() function of the respective case.

Updating SPINS

Development of spins is on-going and it is advisable to have the most up-to-date version.

To update spins, make sure the SPINS directory is first clean in the git sense (no uncommitted and changed files). Check this by running git status. Once the directory is clean, run git pull.

SPINS was recently (Spring 2018) moved from Belize (which has been decommissioned) to the UW git server. If you installed SPINS prior to this transfer, a git pull will not work as the remote branch is still looking to Belize. Update the remote with

git remote set-url origin https://git.uwaterloo.ca/SPINS/SPINS_main.git

before running git pull.

Examples of common operations

You can find examples of how to do various operations by digging through the case files. Some of the common operations are reproduced here.

Using wave_reader.cpp

The case file wave_reader.cpp is a special case used for initializing the model with specified velocity and density fields. A configuration file, spins.conf, is used to hand parameters to SPINS. An example spins.conf is provided below (# are comments) for a periodic box initialized from 3D files:

  Nx = 256
  Ny = 64
  Nz = 128

  type_x = FOURIER
  type_y = FOURIER
  type_z = FREE_SLIP

  Lx = 20
  Ly = 5
  Lz = 1

  min_x = 0
  min_y = -1
  min_z = 0

  mapped_grid = false
  # xgrid = xgrid.bin
  # zgrid = zgrid.bin

  file_type = FULL

  u_file = input_u
  v_file = input_v
  w_file = input_w
  rho_file = input_rho

  enable_tracer = false
  # tracer_file = tracer.bin
  # tracer_kappa = kappa

  g = 9.81
  rot_f = 0.0
  rho_0 = 1000.0
  visco = 0.0
  kappa_rho = 0.0

  perturb = 0.0

  # init_time = 0.0
  final_time = 100
  plot_interval = 1

  restart = false
  # restart_time = 0
  # restart_sequence = 0

For a short explanation of each option, the wave_reader help is reproduced here:

  mpirun -np 1 ./wave_reader_x --help

  SPINS: baseline options:
    --config arg (=spins.conf) Configuration file
    --help                     Print this set of options and exit

  Grid Options:
    --Nx arg                Number of points in X
    --Ny arg (=1)           Number of points in Y
    --Nz arg                Number of points in Z
    --type_x arg            Grid type in X.  Valid values are:
                               FOURIER: Periodic
                               FREE_SLIP: Cosine expansion
                               NO_SLIP: Chebyhsev expansion
    --type_y arg (=FOURIER) Grid type in Y
    --type_z arg            Grid type in Z
    --Lx arg                X-length
    --Ly arg (=1)           Y-length
    --Lz arg                Z-length
    --min_x arg (=0)        Unmapped grids: Minimum X-value
    --min_y arg (=0)        Minimum Y-value
    --min_z arg (=0)        Minimum Z-value

  Grid mapping options:
    --mapped_grid arg (=0) Use a mapped (2D) grid -  Note: this must be a 2D grid even if you are restarting a 3D simulation.
    --xgrid arg            x-grid filename
    --zgrid arg            z-grid filename

  Input data:
    --file_type arg       Format of input data files, including that for the 
                          mapped grid.Valid options are:
                             MATLAB: Row-major 2D arrays of size Nx x Nz
                             CTYPE:  Column-major 2D arrays (including that 
                                     output by 2D SPINS runs)
                             FULL:   Column-major 3D arrays; implies CTYPE for 
                                     grid mapping if enabled
    --u_file arg          U-velocity filename
    --v_file arg          V-velocity filename
    --w_file arg          W-velocity filename
    --rho_file arg        Rho (density) filename

  Passive tracer:
    --enable_tracer       Enable evolution of a passive tracer
    --tracer_file arg     Tracer filename
    --tracer_kappa arg    Diffusivity of tracer

  Physical parameters:
    --g arg (=9.8100000000000005) Gravitational acceleration
    --rot_f arg (=0)              Coriolis force term
    --rho_0 arg (=1000)           Reference density
    --visco arg (=0)               Kinematic viscosity
    --kappa_rho arg (=0)              Thermal diffusivity
    --perturb arg (=0)       Velocity perturbation (additive or multiplicative white noise, depending on your version of wave_reader) applied to read-in data.

  Running options:
    --init_time arg (=0)  Initial time, no longer needed for newer versions of SPINS
    --final_time arg      Final time
    --plot_interval arg   Interval between output times

  Restart options:
    --restart               Restart from prior output time.  OVERRIDES many other
                            values.  NOT TO BE USED TO EXTEND FROM 2D DATA. For extending set restart to false
    --restart_time arg (=0) Time to restart from
    --restart_sequence arg  Sequence number to restart from (if plot_interval has changed)

Also see Matlab-to-SPINS grid ordering.

Generating the grid files: regular grid

For an unmapped grid, include the following call in your case file's constructor to generate the grid files and grid file readers:

  automatic_grid(MinX,MinY,MinZ);

where MinX, MinY and MinZ are the coordinates of the starting corner of your grid.

Sending SPINS Output to a Log-File

For single-processor jobs, it is quite simple to route standard output (stdout) from the terminal to a log-file using myrun_x > logfile.log. With mpirun, this won't do the trick and the process is a bit more convoluted. The command would be:

mpirun -np <numProcs> myrun_x > logfile.log 2>&1 < /dev/null &

The > logfile.log 2>&1 part routes both stdout (1) and stderr (2) to logfile.log. The < /dev/null part tells mpirun to accept 'null' as input instead of the command-line (this allows you to close the terminal while the job is running). Finally, the & at the end makes the whole process run in the background and returns you to the shell prompt.

SPINS problem resolution

SPINS problems and their solutions are listed here.

Restarting 3D simulations with mapped grids (wave_reader.x)

Problem: I've had some problems restarting 3D simulations with mapped grids using wave_reader.x. It seems that wave_reader.x requires 2D (x,z) grid files when restarting a simulation with mapped grids, even if the original simulation was 3D.

Resolution: I suggest building a 2D grid and referring to this grid in the spins.conf file. I'm not sure if the same issue occurs with non-mapped grids since I haven't been working with non-mapped grids.

Initializing 3D simulations with 2D data (wave_reader.x)

Make sure that restart=false in spins.conf. The restart flag represents a true restart simulation and cannot be used to extend a 2D simulation to 3D.

Additionally, ensure that the 2D intialization files have a different name than the 3D output files.

Restarting from dump after changing plot_interval Suppose you ran a simulation, then restarted it at a particular time but with higher temporal resolution (smaller plot_interval). When restarting from dump, spins is then unable to correctly predict the correct time for the next output write since the output numbers do not multiply consistently throughout the entire simulation. You'll need to restart from the last output manually.