Gravity wave tutorial

(WIP)

MIT GCM Tutorial -- Getting Started with the MITgcm

This is a short tutorial designed to walk someone through the main parts of the MITgcm and developing a sensible parameter set for your particular experiment. The MITgcm has a series of additional tutorials that may also be useful to you under the /verification/ folder. This tutorial is useful if you have never worked with the model before and want to very quickly get a sense of the main parts and offers a "hands on" alternative to sifting through the manual. The tutorial will also attempt to walk you through troubleshooting skills that will be useful as the manual becomes less so. For best results, do attempt the activities before looking at the walkthrough, and take note of references to the MITgcm user manual. Enjoy!

Troubleshooting Resources

The MITgcm Online Manual can be found here. A more useful, complete PDF file of the manual can be found here. The MITgcm help support mailing list can be found here, and it is often useful to search the help inquiries (google with the addition site: mitgcm.org) if you are having a specific issue. Finally, the MITgcm is supported by a code browser that allows you to track certain variables within the entire code base. Keep in mind that this search model may contain out-of-date code, but it is a useful resource for sourcing where problems may be found. The best option is always to track it down in your local version of the MITgcm!

Acquiring the Code

The MITgcm page provides a relatively easy-to-follow set of instructions to picking up the code. Of course, best practice is to set up CVS so that the model can be updated if you're working on it long term, but you may find it easier to just use the .tar (non-Linux users, this is the equivalent of a .zip.

Getting Started with the Tutorial

The tutorial materials and some additional helpful scripts can be found at the git repository developed by Emily Tyhurst https://git.uwaterloo.ca/etyhurst/MITgcmUtils. Change directories to where the MITgcm code was unpacked and run the following command: (NOTE: Here all terminal commands will be preceded with a  %. All MATLAB commands will be preceded with a >>. All edits made to files will be referred to as old_text => new_text where new_text is the change to be made).

  % git clone gitlab@git.uwaterloo.ca:etyhurst/MITgcmUtils.git

And the files you need will be managed by git.

For this tutorial you will need everything in the directory called tutorial wave, and a mpi file for whichever system you are running the MITgcm on. This tutorial does not cover how to write MPI files in order to find your local Fortran compiler, so hopefully you will be provided with one. There are a few mpi-files in the directory mpi_files but they may need revision if the systems have changed since this tutorial was written. The mpi file should be placed in the MITgcm directory tools/build_options

Case 1: Looking at the Example

For this tutorial you will be looking at a very simple 2D simulation. A density difference established via a temperature difference begins at a perfect vertical division at t=0s. The division becomes a simple gravity wave as the warmer water floats and the cooler water sinks below. Below is an image of the simulation in the IGW, next to an image of what your simulation will hopefully look like in MATLAB.

A: Compile-Time Procedure

The first course of action is to compile the MITgcm into an executable file. Change directories into the 'build_folder' and run the following command:

 % ../../tools/genmake2 -mods=../code -of ../../tools/build_options/my_device_mpi

The first command tells the machine to use genmake2, an MITgcm script for creating all the appropriate make files. The -mods flag tells it where the compile-time options are located. You can cd into ../code if you want to take a look at these. We will address them later. The -of flag locates the mpi file for your particular machine, as mentioned in "getting started". Once genmake2 has concluded use the command:

 % make depend 

To compile all the Fortran dependencies. The directory should fill up with a large number of .F files. Then use the command:

 % make

To complete the creation of the executable. If you have errors they are most likely related to the mpi file at this stage. The error "Clock Skew detected" does not seem to affect the model in any major way. If it completes successfully, you should have an executable called mitgcmuv. Send a copy of this to the directory case_1_density (cp mitgcmuv ../case1_density if you're unfamiliar with Linux). This takes care of all the compile-time information. The main components set at compile time are the grid size (in computational grid cells, not m, see SIZE.h) and which packages are added (see packages.conf). It takes extra time to compile more packages, so don't compile them unless you need them!

B: Run-Time Procedure

Go to the data_files directory and take a look at the contents. One of the most important files there is data which sets the majority of the important parameters for the MITgcm. The first section of parameters contains a lot of the "fiddly" parameters that you'll want to decide whether or not you need them for your simulation. The MITgcm page "Customizing the Model" will tell you in detail most of the parameters that can be set in data. We will walk through a few of the useful parameters throughout the course of this tutorial. For now, familiarize yourself with the main types according to the PARM section labels. Take special note of the time stepping parameters (&PARM03)-- the startTime and the endTime tells you how many iterations the model will run, whereas deltaT is the time in seconds per iteration.

data.pkg specifies flags for turning on and off the compiled packages. This should probably match what is seem in the code/packages.conf file if you don't want to be compiling extraneous packages.

All the data files that end with an additional extension are specific to the package in question. For instance data.exf sets parameters related to the External Forcing Package. We will look in detail at data.obcsin Case 3.

Copy the contents of this directory to case_1_density (cp ./*../case1_density where . means "the directory I am in" and * is a wildcard. You can also use this wildcard to copy and list subsets of other things. For example ls data* should list everything that starts with the string "data". Alright you can get the rest of your regex lesson from the internet...).

Now move yourself to the directory input_files. There should be three MATLAB files inflow_outflow.m, initial_temperature.m, bathymetry.m. You will only need to run initial_temperature.m for now. Open up MATLAB (on hood) using the command matlab

Inside the MATLAB command, open the file using the prompt

   % edit initial_temperature.m

Examine the script and make sure you understand the comments. Then run it in the matlab terminal:

  % initial_temperature

This will create the binary file initial_temp.bin. Copy it and the generating script (all this does is keep a nice record into the case 1 directory. Change directories to the case 1 directory. You are now ready to run the model!

Once you are in case_1_density run the following command:

   % mpirun -np 3 ./mitgcmuv > output < /dev/null &

The component ./mitgcmuv runs the executable. The component mpirun -np 3 uses the parallelization MPI on 3 CPU cores. The last segment relays the console output. Note that when you are doing larger-scale, non-tutorial runs (especially when other people are using the machines), you may want to lock on to specific cores. You can look at this wiki page to find out how to do that.

For now, run the command:

   % ls T*data

You should start to see one more more files titled T.0000000___.data, each with a number of iterations. These, combined with the .meta files, are the output data of the MITgcm. T is for temperature. Wait until the iterates reach 6000 (this time is set in the data file). Then the model should conclude with the Fortran printout NORMAL END.

C: Data Post-Processing

Using the commands:

 % mv *.data data_processing/
 % mv *.meta data_processing/

Place all the outputted data files in the MATLAB data processing folder. Inside that folder is a number of MATLAB utilities, primarily aimed at creating movies for visualizing the MITgcm data. Open up MATLAB and peruse the file TwoDimensionalRho.m. This takes the density anomaly data from the MITgcm (which is generated by the specified diagnostics file) and plots it as a 2D color plot, from the two times specified in the time manager (it should start from 0 to 6000 in iterates of 100). The code is well commented, so be sure you understand it.

Run the MATLAB script and investigate the folders GIF_IMAGES, GIF_MOVIE, JPG_IMAGES, potentially using xanim to view the movies and display to view the images. The fluid should begin as a split in density on either side that quickly becomes a gravity wave sliding along the bottom surface.

Congratulations! You've completed your first run of the MIT gcm!

Case 2: Changing the Domain

The goals for this section are as follows. You are encouraged to try to figure them out before looking at the full instructions.

1. Change the grid size from 200x1x40 cells on 3 processors in x to 250x50x40 cells on 6 processors. (must happen at compile-time).

2. Change the grid resolution from 25.0m to 20.0 m. Change the time step from 1s/iteration to 5s/iteration, and for a length of 12000 seconds. Change the Rho diagnostics to output every 300s.

Your simulation will become a 3D simulation! This particular example does not have a lot of three dimensionality, so you will not see significant changes in the experiment behavior.

A: Compile-Time Procedure

Enter the code directory and open up SIZE.h (you can use vim with the command vi SIZE.h or perhaps gedit with edit SIZE.h. There are parameters for the number of sub-grid points, sub-grids, and processes in X and Y. Notice that the total number of grid points is the product. Make the following changes:

  sNx = 100 => 125
  sNy = 1, => sNy=25,

  nPy = 1, => nPy=2,

If you made different changes keep in mind that the number of processors in x and y is multiplicative as well (nPx=3 , nPy= 2 , total procs=2*3=6). Follow the procedure in the first example to recompile the model. It will overwrite your original mitgcmuv executable, so make sure you've made a copy in the case1 directory. Copy the newly compiled executable into the case2 directory.

Information in the MIT gcm manual on this procedure and the gridding.

B: Run-Time Procedure

Now copy the contents of data_files into case2_domainChange. Change to the case2 directory. Open up the file data and make the following changes:

  delX= 200*25.0 => delX= 250*20.0
  delY= 100.0 => delY= 50*20.0

This changes the grid resolution and accounts for your new grid shape.

  endTime= 6000 => endTime = 12000
  deltaTmom= 1.0 => deltaTmom=5.0
  deltatTtracer=1.0 => deltaTtracer=5.0

This edits the time parameters, the end time, and the amount of time per step. Information on the time-stepping in the MITgcm can be found here.

Save data and open up data.diagnostics. This contains all the information for outputting the binary .data and .meta files. Scroll down and change the line:

#-----------------
 frequency(4) = -100.0,
 timePhase(4) = 0.0,
 fields(1,4)  = 'Theta',
 filename(4)  = 'T',
&

=>

#-----------------
 frequency(4) = -300.0,
 timePhase(4) = 0.0,
 fields(1,4)  = 'Theta',
 filename(4)  = 'T',
&

This changes the output frequency of the temperature data specifically.

Now the input files must be updated to reflect the new domain. Enter the input_files folder and make the following changes to the

Copy a new version of mitgcmuv from the newly sized build that you compiled in step A. Run the MIT gcm using the same commands as above.

C: Data Post-Processing

Now that you have 3D data, you can begin looking at slices in different directions. Run the MATLAB script MITgcmGenMovie_Main. This script has a reasonably functional UX and can be used to create basic movies of any 3D MITgcm data (from binary files). Since the script itself should walk you through how to take slices, you are free to play around with different views here. Here are a few examples at certain time slices.

You are encouraged to read the help documentation for the various functions which make up the MITgcmGenMovie_Main script.

(!!!) IMAGES (!!!)

It may sometimes come into your interest to place several graphs of different views in the same file. Examine the MATLAB script GenCrossGraphs.m, taking note of how to use the subplot layout. Note that this file manually does the movie stitching, as opposed to using the MITgcmGenGifMovie function.

At this point you now should be familiar with how to use MATLAB scripts to interact with the MIT gcm (anything further would fall under the domain of a MATLAB tutorial). The rest of the tutorial will focus on more advanced parameters in the MIT gcm.

Case 3: Using a Package (OBCS)

The goal of this section of the tutorial will be to adjust the simulation from using a pre-set temperature difference to using the Open Boundary Conditions package. You are encouraged to try this out on your own first. Your goals are as follows:

1. Allow open boundary conditions on the east and west.

2. Provide arrays of appropriate length to fit those open boundaries. The eastern open boundary should have a temperature influx of 14C, with a velocity of 1.0m/s. The western open boundary will have the ambient temperature, 4.0C

3. Modify the initial temperature array to be a constant 4.0C.

You can read about the package on the MITgcm manual here.

A: Compile-Time Procedure

The first course of action is to add the OBCS package to the list of compiled packages. In the code directory open up packages.conf and add the line obcs (under diagnostics). When adding a new package this will always be necessary. Now open the file OBCS_OPTIONS.h. Make the following changes:

  #undef  ALLOW_OBCS_EAST => #define ALLOW_OBCS_EAST
  #undef ALLOW_OBCS_WEST => #define ALLOW_OBCS_WEST

When you enable a new package, almost all of them will require some form of PKG_OPTIONS.h file for compile-time parameters. The MIT gcm verification directory contains numerous examples which can help you for creating other .h files when compiling other packages.

Compile the code as you would normally.

B: Run-Time Procedure

Copy the data files from Case 2 into the Case 3 directory using the following series of commands (you'll be surprised how often you use these):

  % cp case2_domainChange data* case3_openBC
  % cp case2_domainChange eedata case3_openBC

Enter the case3 directory and first open data.pkg. Add the line:

  useOBCS=.TRUE.,

underneath the last package boolean. Like at compile-time, the packages need to be both enabled and specified at run-time. Now open up the file data.obcs, which is analogous to the file OBCS_OPTIONS.h (the same advice applies regarding looking at the verification examples if you need to add on a package that you have no idea how to to specify data files for). Make the following changes to the data.obcs file.

 OB_Ieast= 700*-1, => OB_Ieast=50*-1,
 OB_Iwest= 100*0, => OB_Iwest=50*1,

The first line specifies to create a vector of length 50 (the number of grid cells in y), and place it along the eastern boundary (the -1 wraps around to the last element of the x array). The second specifies a vector of length 50 and places it at the western boundary.

Notice how OBSCprescribe is turned on, and different binaries are specified. We'll now need to create those. Save the file and change directories to the input_files directory. If you're familiar with MATLAB or just want to get practice creating binaries using MATLAB, you can try to make your own inflow_outflow.m based on the initial_temperature.m script. It's simply a matter of creating the right arrays. If you're in a hurry, simply read through and run the inflow_outflow.m script. Copy the created binaries into your case3 directory.

Again, copy the mitgcmuv file you compiled and run the model. You can visualize the data to compare and contrast the differences. Here are a few example images:

(!!!) IMAGES (!!!)

Case 4: No Slip BCs and Troubleshooting

The goal of this section is to get familiar with the MIT gcm online code browser, the help documentation, and get into the nitty-gritty of what it means to debug the system. The goal is simple:

Activate proper no-slip boundary conditions in the MIT gcm

If you recall the initial comparison to the IGW, you'll notice that the gravity wave has a more smooth, continuous tail along the bottom boundary as opposed to the non-physical sharp edge of the MIT gcm wave.

When this case is complete, the two cases should match more accurately. The actual process of this case will be very artificial compared to the real procedure of troubleshooting. Nevertheless, hopefully this section may serve as a reference for what to do when unexpected behavior presents itself.

A: Initial run and Troubleshooting'

Walk through how to use the code selector and how to print lines in Fortran to see what's going on.

Case 5: Bathymetry and Notes on Noise Control

(!!!) Have them add a step bathymetry and mention terrain following vs. the MITgcm coordinates. Give a few troubleshooting options if MITgcm bathymetry noise becomes an issue.