SPINS User Guide

Welcome to the SPINS user guide.

The basics

The SPINS model is a Navier-Stokes solver that gets parameters and initial/boundary conditions from calls to user-provided routines. The user-provided routines are encapsulated in class derived from BaseCase (see BaseCase.hpp).

Creating your own custom configuration involves supplying the user-provided routines in a derived class based on BaseCase. The case file cases/doc_minimal.cpp shows the structure of a case file. It usually makes sense to start with a similar case file and customise it.

SPINS components

SPINS consists of a bunch of C++ source files and a bunch of case files, and it requires four libraries. UMFPack, AMD and Blitz++ are supplied with SPINS, and it uses the system-installed FFTW.

Directory structure:

• spins/src - SPINS source files
• spins/src/cases - A few dozen example case files
• spins/matlab - Some helper functions for MATLAB analysis

How to get SPINS running

SPINS is hosted in a git repository on belize. A guide to using git can be found here: http://git-scm.com/book. If your system does not have a working copy of git that can access http-based repositories (winisk and kazan are currently notable examples), you may get a full copy of the current repository at: http://belize.math.uwaterloo.ca/~csubich/spins_daily.tgz (this replaces step #1 below).

You will need to get the code, build the dependencies, build the model and then run it.

Extracting the code from the git repository

• Create a directory in which to store the code.
• In that directory type "git clone http://belize.math.uwaterloo.ca/~csubich/spins.git".
• This will create a directory called spins.

Building SPINS

One-time setup

• Go to the systems directory.
• Type ./makemake.sh [system]. This will construct the system-specific settings for the makefile.
  • This script reads and processes an appropriate script from the systems/ subdirectory; these files contain variable definitions for compiler names, include/library options, and other attributes that are necessary at build time.
  • There are several scripts in the systems/ subdirectory, and in general one needs to be written for each unique system based on its idiosyncrasies.
  • makemake.sh takes the system name as an optional argument. If not specified, it will try to guess the appropriate host file based on the current hostname. E.g., building on winisk will try to read systems/winisk.sh. If this is inappropriate (such as on clusters, where login nodes are numbered), then the system name can be specified at the command-line, e.g. ./makemake.sh orca to read systems/orca.sh.
• Execute ./make_deps.sh in the main repository by typing ./make_deps.sh [system] -j. This file may need to be edited to select which libraries aren't present in the current build environment; when in doubt build everything. FFTW and Boost are the libraries most likely to be already installed system-wide.

To build your case-file

• Enter the src directory.
• Type make cases/your_case.x
• This requires a file called your_case.cpp in the cases directory. There are several cases included with the code so you may want to start with one of those.
• After successful compilation, an executable called your_case.x is created.

Running SPINS

• Please be careful not to run in your home directory on machines like belize or winisk. Typical cases output a great deal of data, and can quickly fill up your /home quota (or the full partition, on non-quota systems).
• The code can be executed using mpirun e.g "mpirun -np 4 ./your_case.x".
• Please note that some cases may require command line options or a configure file called spins.conf. Options included in the command-line or spins.conf are configured in the main() function of the respective case.

Examples of common operations

You can find examples of how to do various operations by digging through the case files. Some of the common operations are reproduced here.

Using wave_reader.cpp

The case file wave_reader.cpp is a special case used for initialising the model with specified velocity and density fields. A configuration file, spins.conf, is used to hand parameters to SPINS. An example spins.conf is provided below (# are comments) for a periodic box initialised from 3D files:

  Nx = 256
  Ny = 64
  Nz = 128

  type_x = FOURIER
  type_y = FOURIER
  type_z = FREE_SLIP

  Lx = 20
  Ly = 5
  Lz = 1

  min_x = 0
  min_y = -1
  min_z = 0

  mapped_grid = false
  # xgrid = xgrid.bin
  # zgrid = zgrid.bin

  file_type = FULL

  u_file = input_u
  v_file = input_v
  w_file = input_w
  rho_file = input_rho

  enable_tracer = false
  # tracer_file = tracer.bin
  # tracer_kappa = kappa

  g = 9.81
  rot_f = 0.0
  visc = 0.0
  kappa = 0.0

  perturbation = 0.0

  init_time = 0.0
  final_time = 100
  plot_interval = 1

  restart = false
  # restart_time = 0
  # restart_sequence = 0

For a short explanation of each option, the wave_reader help is reproduced here:

  mpirun -np 1 ./wave_reader_x --help

  SPINS: baseline options:
    --config arg (=spins.conf) Configuration file
    --help                     Print this set of options and exit

  Grid Options:
    --Nx arg                Number of points in X
    --Ny arg (=1)           Number of points in Y
    --Nz arg                Number of points in Z
    --type_x arg            Grid type in X.  Valid values are:
                               FOURIER: Periodic
                               FREE_SLIP: Cosine expansion
                               NO_SLIP: Chebyhsev expansion
    --type_y arg (=FOURIER) Grid type in Y
    --type_z arg            Grid type in Z
    --Lx arg                X-length
    --Ly arg (=1)           Y-length
    --Lz arg                Z-length
    --min_x arg (=0)        Unmapped grids: Minimum X-value
    --min_y arg (=0)        Minimum Y-value
    --min_z arg (=0)        Minimum Z-value

  Grid mapping options:
    --mapped_grid arg (=0) Use a mapped (2D) grid -  Note: this must be a 2D grid even if you are restarting a 3D simulation.
    --xgrid arg            x-grid filename
    --zgrid arg            z-grid filename

  Input data:
    --file_type arg       Format of input data files, including that for the 
                          mapped grid.Valid options are:
                             MATLAB: Row-major 2D arrays of size Nx x Nz
                             CTYPE:  Column-major 2D arrays (including that 
                                     output by 2D SPINS runs)
                             FULL:   Column-major 3D arrays; implies CTYPE for 
                                     grid mapping if enabled
    --u_file arg          U-velocity filename
    --v_file arg          V-velocity filename
    --w_file arg          W-velocity filename
    --rho_file arg        Rho (density) filename

  Passive tracer:
    --enable_tracer       Enable evolution of a passive tracer
    --tracer_file arg     Tracer filename
    --tracer_kappa arg    Diffusivity of tracer

  Physical parameters:
    --g arg (=9.8100000000000005) Gravitational acceleration
    --rot_f arg (=0)              Coriolis force term
    --visc arg (=0)               Kinematic viscosity
    --kappa arg (=0)              Thermal diffusivity
    --perturbation arg (=0)       Velocity perturbation (multiplicative white 
                                       noise) applied to read-in data.

  Running options:
    --init_time arg (=0)  Initial time
    --final_time arg      Final time
    --plot_interval arg   Interval between output times

  Restart options:
    --restart               Restart from prior output time.  OVERRIDES many other
                            values.  NOT TO BE USED TO EXTEND FROM 2D DATA. For extending set restart to false
    --restart_time arg (=0) Time to restart from
    --restart_sequence arg  Sequence number to restart from (if plot_interval has
                            changed)

Generating the grid files: regular grid

For an unmapped grid, include the following call in your case file's constructor to generate the grid files and grid file readers:

  automatic_grid(MinX,MinY,MinZ);

where MinX, MinY and MinZ are the coordinates of the starting corner of your grid.

Using a mapped grid

Can someone write this?

Analytic initialisation

Can someone write this?

Boundary conditions

Can someone write this?

Forcing / sponge regions

Can someone write this?

Online Analysis

Some analysis can be done online, some examples are shown below.

Energy Diagnostics

If you're using a periodic grid, use this for computing kinetic energy diagnostic

double dV = (Lx/Nx)*(Ly/Ny)*(Lz/Nz);
double ke = 0.5*rho_0*pssum(sum( u*u+v*v+w*w ))*dV; // KE

If you're on a Chebyshev grid, you can use this for the KE computation

double ke = pssum(sum((*get_quad_x())(ii)*(*get_quad_y())(jj)*(*get_quad_z())(kk)*
                  (pow(u(ii,jj,kk),2)+pow(v(ii,jj,kk),2)+pow(w(ii,jj,kk),2))));

and you will need to compute the quadrature weights in the constructor

// Compute the quadrature weights
compute_quadweights(size_x(),size_y(),size_z(),
                    length_x(),length_y(),length_z(),
                    type_x(),type_y(),type_z());

(Please note: cleaning up the quadrature code is somehwere buried on the to-do list, so at some point this rather long system should become nicer.)

Sending SPINS Output to a Log-File

For single-processor jobs, it is quite simple to route standard output (stdout) from the terminal to a log-file using myrun_x > logfile.log. With mpirun, this won't do the trick and the process is a bit more convoluted. The command would be:

mpirun -np <numProcs> myrun_x > logfile.log 2>&1 < /dev/null &

The > logfile.log 2>&1 part routes both stdout (1) and stderr (2) to logfile.log. The < /dev/null part tells mpirun to accept 'null' as input instead of the command-line (this allows you to close the terminal while the job is running). Finally, the & at the end makes the whole process run in the background and returns you to the shell prompt.

SPINS problem resolution

SPINS problems and their solutions are listed here.

Restarting 3D simulations with mapped grids (wave_reader.x)

Problem: I've had some problems restarting 3D simulations with mapped grids using wave_reader.x. It seems that wave_reader.x requires 2D (x,z) grid files when restarting a simulation with mapped grids, even if the original simulation was 3D.

Resolution: I suggest building a 2D grid and referring to this grid in the spins.conf file. I'm not sure if the same issue occurs with non-mapped grids since I haven't been working with non-mapped grids.

Initializing 3D simulations with 2D data (wave_reader.x)

Make sure that restart=false in spins.conf. The restart flag represents a true restart simulation and cannot be used to extend a 2D simulation to 3D.

Additionally, ensure that the 2D intialization files have a different name than the 3D output files.