SPINS User Guide
Welcome to the SPINS user guide.
The basics
The SPINS model is a Navier-Stokes solver that gets parameters and initial/boundary conditions from calls to user-provided routines. The user-provided routines are encapsulated in class derived from BaseCase (see BaseCase.hpp).
Creating your own custom configuration involves supplying the user-provided routines in a derived class based on BaseCase. The case file cases/doc_minimal.cpp shows the structure of a case file. It usually makes sense to start with a similar case file and customise it.
SPINS components
SPINS consists of a bunch of C++ source files and a bunch of case files, and it requires four libraries. UMFPack, AMD and Blitz++ are supplied with SPINS, and it uses the system-installed FFTW.
Directory structure:
- spins/src - SPINS source files
- spins/src/cases - A few dozen example case files
- spins/matlab - Some helper functions for MATLAB analysis
How to get SPINS running
SPINS is hosted in a git repository on belize. A guide to using git can be found here: http://git-scm.com/book. If your system does not have a working copy of git that can access http-based repositories (winisk and kazan are currently notable examples), you may get a full copy of the current repository at: http://belize.math.uwaterloo.ca/~csubich/spins_daily.tgz (this replaces step #1 below).
You will need to get the code, build the dependencies, build the model and then run it.
Extracting the code from the git repository
- Create a directory in which to store the code.
- In that directory type "git clone http://belize.math.uwaterloo.ca/~csubich/spins.git".
- This will create a directory called spins.
Building the dependencies
- Go to the spins directory.
- Type "./make_deps.sh".
Building SPINS
One-time setup
- Go to the systems directory.
- Type "./makemake.sh". This will build the code based on the system architecture.
- Note that there are several files in the systems/ subdirectory that include instructions for several systems e.g. gpc.sh.
- makemake.sh will try to guess which instructions to use based on the system hostname, preferring to select <hostname>.sh.
- If this file is not available (for example many clusters have confusingly-named login nodes), call ./makemake.sh <script name>, as in ./makemake.sh orca (for the sharcnet Orca cluster)
- Execute ./make_deps.sh in the main repository. This file will need to be edited to select which libraries aren't present in the current build environment; when in doubt build everything.
To build your case-file
- Enter the src directory.
- Type "make cases/your_case.x".
- This requires a file called your_case.cpp in the cases directory. There are several cases included with the code so you may want to start with one of those.
- After successful compilation, an executable called your_case.x is created.
Running SPINS
- Please be careful not to run in your home directory on machines like belize or winisk. Typical cases output a great deal of data, and can quickly fill up your /home quota (or the full partition, on non-quota systems).
- The code can be executed using mpirun e.g "mpirun -np 4 ./your_case.x".
- Please note that some cases may require command line options or a configure file called spins.conf. Options included in the command-line or spins.conf are configured in the main() function of the respective case.
Examples of common operations
You can find examples of how to do various operations by digging through the case files. Some of the common operations are reproduced here.
Using wave_reader.cpp
The case file wave_reader.cpp is a special case used for initialising the model with specified velocity and density fields. A configuration file, spins.conf, is used to hand parameters to SPINS. An example spins.conf is provided below (# are comments) for a periodic box initialised from 3D files:
Nx = 256
Ny = 64
Nz = 128
type_x = FOURIER
type_y = FOURIER
type_z = FREE_SLIP
Lx = 20
Ly = 5
Lz = 1
min_x = 0
min_y = -1
min_z = 0
mapped_grid = false
# xgrid = xgrid.bin
# zgrid = zgrid.bin
file_type = FULL
u_file = input_u
v_file = input_v
w_file = input_w
rho_file = input_rho
enable_tracer = false
# tracer_file = tracer.bin
# tracer_kappa = kappa
g = 9.81
rot_f = 0.0
visc = 0.0
kappa = 0.0
perturbation = 0.0
init_time = 0.0
final_time = 100
plot_interval = 1
restart = false
# restart_time = 0
# restart_sequence = 0
For a short explanation of each option, the wave_reader help is reproduced here:
mpirun -np 1 ./wave_reader_x --help
SPINS: baseline options:
--config arg (=spins.conf) Configuration file
--help Print this set of options and exit
Grid Options:
--Nx arg Number of points in X
--Ny arg (=1) Number of points in Y
--Nz arg Number of points in Z
--type_x arg Grid type in X. Valid values are:
FOURIER: Periodic
FREE_SLIP: Cosine expansion
NO_SLIP: Chebyhsev expansion
--type_y arg (=FOURIER) Grid type in Y
--type_z arg Grid type in Z
--Lx arg X-length
--Ly arg (=1) Y-length
--Lz arg Z-length
--min_x arg (=0) Unmapped grids: Minimum X-value
--min_y arg (=0) Minimum Y-value
--min_z arg (=0) Minimum Z-value
Grid mapping options:
--mapped_grid arg (=0) Use a mapped (2D) grid
--xgrid arg x-grid filename
--zgrid arg z-grid filename
Input data:
--file_type arg Format of input data files, including that for the
mapped grid.Valid options are:
MATLAB: Row-major 2D arrays of size Nx x Nz
CTYPE: Column-major 2D arrays (including that
output by 2D SPINS runs)
FULL: Column-major 3D arrays; implies CTYPE for
grid mapping if enabled
--u_file arg U-velocity filename
--v_file arg V-velocity filename
--w_file arg W-velocity filename
--rho_file arg Rho (density) filename
Passive tracer:
--enable_tracer Enable evolution of a passive tracer
--tracer_file arg Tracer filename
--tracer_kappa arg Diffusivity of tracer
Physical parameters:
--g arg (=9.8100000000000005) Gravitational acceleration
--rot_f arg (=0) Coriolis force term
--visc arg (=0) Kinematic viscosity
--kappa arg (=0) Thermal diffusivity
--perturbation arg (=0) Velocity perturbation (multiplicative white
noise) applied to read-in data.
Running options:
--init_time arg (=0) Initial time
--final_time arg Final time
--plot_interval arg Interval between output times
Restart options:
--restart Restart from prior output time. OVERRIDES many other
values. NOT TO BE USED TO EXTEND FROM 2D DATA. For extending set restart to false
--restart_time arg (=0) Time to restart from
--restart_sequence arg Sequence number to restart from (if plot_interval has
changed)
Generating the grid files: regular grid
For an unmapped grid, include the following call in your case file's constructor to generate the grid files and grid file readers:
automatic_grid(MinX,MinY,MinZ);
where MinX, MinY and MinZ are the coordinates of the starting corner of your grid.
Using a mapped grid
Can someone write this?
Analytic initialisation
Can someone write this?
Boundary conditions
Can someone write this?
Forcing / sponge regions
Can someone write this?
Online Analysis
Some analysis can be done online, some examples are shown below.
Energy Diagnostics
If you're using a periodic grid, use this for computing kinetic energy diagnostic
double dV = (Lx/Nx)*(Ly/Ny)*(Lz/Nz);
double ke = 0.5*rho_0*pssum(sum( u*u+v*v+w*w ))*dV; // KE
If you're on a Chebyshev grid, you can use this for the KE computation
double ke = pssum(sum((*get_quad_x())(ii)*(*get_quad_y())(jj)*(*get_quad_z())(kk)*
(pow(u(ii,jj,kk),2)+pow(v(ii,jj,kk),2)+pow(w(ii,jj,kk),2))));
and you will need to compute the quadrature weights in the constructor
// Compute the quadrature weights
compute_quadweights(size_x(),size_y(),size_z(),
length_x(),length_y(),length_z(),
type_x(),type_y(),type_z());
(Please note: cleaning up the quadrature code is somehwere buried on the to-do list, so at some point this rather long system should become nicer.)
Sending SPINS Output to a Log-File
For single-processor jobs, it is quite simple to route standard output (stdout) from the terminal to a log-file using myrun_x > logfile.log. With mpirun, this won't do the trick and the process is a bit more convoluted. The command would be:
mpirun -np <numProcs> myrun_x > logfile.log 2>&1 < /dev/null &
The > logfile.log 2>&1 part routes both stdout (1) and stderr (2) to logfile.log. The < /dev/null part tells mpirun to accept 'null' as input instead of the command-line (this allows you to close the terminal while the job is running). Finally, the & at the end makes the whole process run in the background and returns you to the shell prompt.
SPINS problem resolution
SPINS problems and their solutions are listed here.
Restarting 3D simulations with mapped grids (wave_reader.x)
Problem: I've had some problems restarting 3D simulations with mapped grids using wave_reader.x. It seems that wave_reader.x requires 2D (x,z) grid files when restarting a simulation with mapped grids, even if the original simulation was 3D.
Resolution: I suggest building a 2D grid and referring to this grid in the spins.conf file. I'm not sure if the same issue occurs with non-mapped grids since I haven't been working with non-mapped grids.
Initializing 3D simulations with 2D data (wave_reader.x)
Make sure that restart=false in spins.conf. The restart flag represents a true restart simulation and cannot be used to extend a 2D simulation to 3D.