Gravity wave tutorial

(WIP)

MIT GCM Tutorial -- Getting Started with the MITgcm

This is a short tutorial designed to walk someone through the main parts of the MITgcm and developing a sensible parameter set for your particular experiment. The MITgcm has a series of additional tutorials that may also be useful to you under the /verification/ folder. This tutorial is useful if you have never worked with the model before and want to very quickly get a sense of the main parts and offers a "hands on" alternative to sifting through the manual. The tutorial will also attempt to walk you through troubleshooting skills that will be useful as the manual becomes less so. For best results, do attempt the activities before looking at the walkthrough, and take note of references to the MITgcm user manual. Enjoy!

Troubleshooting Resources

The MITgcm Online Manual can be found here. A more useful, complete PDF file of the manual can be found here. The MITgcm help support mailing list can be found here, and it is often useful to search the help inquiries (google with the addition site: mitgcm.org) if you are having a specific issue. Finally, the MITgcm is supported by a code browser that allows you to track certain variables within the entire code base. Keep in mind that this search model may contain out-of-date code, but it is a useful resource for sourcing where problems may be found. The best option is always to track it down in your local version of the MITgcm!

Acquiring the Code

The MITgcm page provides a relatively easy-to-follow set of instructions to picking up the code. Of course, best practice is to set up CVS so that the model can be updated if you're working on it long term, but you may find it easier to just use the .tar (non-Linux users, this is the equivalent of a .zip.

Getting Started with the Tutorial

The tutorial materials and some additional helpful scripts can be found at the git repository developed by Emily Tyhurst https://git.uwaterloo.ca/etyhurst/MITgcmUtils. Change directories to where the MITgcm code was unpacked and run the following command:

  % git clone gitlab@git.uwaterloo.ca:etyhurst/MITgcmUtils.git

And the files you need will be managed by git.

For this tutorial you will need everything in the directory called tutorial wave, and a mpi file for whichever system you are running the MITgcm on. This tutorial does not cover how to write MPI files in order to find your local Fortran compiler, so hopefully you will be provided with one. There are a few mpi-files in the directory mpi_files but they may need revision if the systems have changed since this tutorial was written. The mpi file should be placed in the MITgcm directory tools/build_options

Case 1: Looking at the Example

For this tutorial you will be looking at a very simple 2D simulation. A density difference established via a temperature difference begins at a perfect vertical division at t=0s. The division becomes a simple gravity wave as the warmer water floats and the cooler water sinks below. Below is an image of the simulation in the IGW, next to an image of what your simulation will hopefully look like in MATLAB.

(!!!) add images h

A: Compile-Time Procedure

The first course of action is to compile the MITgcm into an executable file. Change directories into the 'build_folder' and run the following command:

 % ../../tools/genmake2 -mods=../code -of ../../tools/build_options/my_device_mpi

The first command tells the machine to use genmake2, an MITgcm script for creating all the appropriate make files. The -mods flag tells it where the compile-time options are located. You can cd into ../code if you want to take a look at these. We will address them later. The -of flag locates the mpi file for your particular machine, as mentioned in "getting started". Once genmake2 has concluded use the command:

 % make depend 

To compile all the Fortran dependencies. The directory should fill up with a large number of .F files. Then use the command:

 % make

To complete the creation of the executable. If you have errors they are most likely related to the mpi file at this stage. The error "Clock Skew detected" does not seem to affect the model in any major way. If it completes successfully, you should have an executable called mitgcmuv. Send a copy of this to the directory case_1_density (cp mitgcmuv ../case1_density if you're unfamiliar with Linux). This takes care of all the compile-time information. The main components set at compile time are the grid size (in computational grid cells, not m, see SIZE.h) and which packages are added (see packages.conf). It takes extra time to compile more packages, so don't compile them unless you need them!

B: Run-Time Procedure

Go to the data_files directory and take a look at the contents. One of the most important files there is data which sets the majority of the important parameters for the MITgcm. The first section of parameters contains a lot of the "fiddly" parameters that you'll want to decide whether or not you need them for your simulation. The MITgcm page "Customizing the Model" will tell you in detail most of the parameters that can be set in data. We will walk through a few of the useful parameters throughout the course of this tutorial. For now, familiarize yourself with the main types according to the PARM section labels. Take special note of the time stepping parameters (&PARM03)-- the startTime and the endTime tells you how many iterations the model will run, whereas deltaT is the time in seconds per iteration.

data.pkg specifies flags for turning on and off the compiled packages. This should probably match what is seem in the code/packages.conf file if you don't want to be compiling extraneous packages.

All the data files that end with an additional extension are specific to the package in question. For instance data.exf sets parameters related to the External Forcing Package. We will look in detail at data.obcsin Case 3.

Copy the contents of this directory to case_1_density (cp ./*../case1_density where . means "the directory I am in" and * is a wildcard. You can also use this wildcard to copy and list subsets of other things. For example ls data* should list everything that starts with the string "data". Alright you can get the rest of your regex lesson from the internet...).

Now move yourself to the directory input_files. There should be three python files inflow_outflow.py, initial_temp.py, bathymetry.py. You will only need to run initial_temp.py for now. Open it up in your favorite text editor and take a look before you run it. It specifies an initial temperature that should match your domain size. The temperature is split in left and right.

Run the file using the command:

   % python initial_temp.py

This will create the binary file initial_temp.bin. Copy it and the generating script into the case 1 directory. Change directories to the case 1 directory. You are now ready to run the model!

Once you are in case_1_density run the following command:

   % mpirun -np 3 ./mitgcmuv > output < /dev/null &

The component ./mitgcmuv runs the executable. The component mpirun -np 3 uses the parallelization MPI on 3 CPU cores. The last segment relays the console output. Note that when you are doing larger-scale, non-tutorial runs (especially when other people are using the machines), you may want to lock on to specific cores. You can look at this wiki page to find out how to do that.

For now, run the command:

   % ls T*data

You should start to see one more more files titled T.0000000___.data, each with a number of iterations. These, combined with the .meta files, are the output data of the MITgcm. T is for temperature.

C: Data Post-Processing

Case 2: Changing the Domain

The goals for this section are as follows. You are encouraged to try to figure them out before looking at the full instructions.

1. Change the grid size from 200x1x40 cells on 3 processors in x to 200x50x40 cells on 6 processors. (must happen at compile-time) 2. Change the grid resolution from

Your simulation will become a 3D simulation! This particular example does not have a lot of three dimensionality, so you will not see significant changes in the experiment behavior.

A: Compile-Time Procedure

Enter the code directory and open up SIZE.h (you can use vim with the command vi SIZE.h or perhaps gedit with edit SIZE.h. There are parameters for the number of sub-grid points, sub-grids, and processes in X and Y. Notice that the total number of grid points is the product. Make the following changes:

 % sNy = 1, => sNy=25,

 % nPy = 1, => nPy=2,

If you made different changes keep in mind that the number of processors in x and y is multiplicative as well (nPx=3 , nPy= 2 , total procs=2*3=6). Follow the procedure in the first example to recompile the model. It will overwrite your original mitgcmuv executable, so make sure you've made a copy in the case1 directory. Copy the newly compiled executable into the case2 directory.

B: Run-Time Procedure

Case 3: Using a Package (OBCS)

(!!!) Walk the person through how to use the OBCS package with minimal pain. Highlight as many parts of these that are general for using a package as possible.

Also walk them through how to edit data.diagnostics to output different data, at varying frequencies

Case 4: No Slip BCs and Troubleshooting

(!!!) Make a few changes to the first parameter set, notably adding the no-slip BC. Point out the comparison to IGW and how it is not matching.

Walk through how to use the code selector and how to print lines in Fortran to see what's going on.

Case 5: Bathymetry and Notes on Noise Control

(!!!) Have them add a step bathymetry and mention terrain following vs. the MITgcm coordinates. Give a few troubleshooting options if MITgcm bathymetry noise becomes an issue.