FEniCS

Installing FEniCS

The following is a set of instructions for installing FEniCS 2017.2.0 and related libraries on Graham. Note that the installation should be done on a Graham login node, as it seems compute nodes on Graham are unable to clone git repos.

Step 0 (Optional): Clean out your environment

Before installing FEniCS, you might want to remove previous installations of petsc4py/FEniCS, in order to prevent conflicts when installing. Consider doing the following:

• Remove directories that contain previous installations of petsc4py/fenics. Directories can be removed by typing rm -rf directory_name.

• Previous installations of petsc4py/FEniCs might have data in hidden directories. Hidden directories can be seen by typing ls -a. Spend some time looking through hidden directories and remove anything associated with previous petsc4py/FEniCS installations.

• Check the environment variables (they can be listed with the env command) and remove anything that points to previous installations of petsc4py/FEniCS.

Step 1 (Optional): Install petsc4py

Installing petsc4py is not required if you just want to use standalone FEniCS. However, petsc4py is needed in order to directly interface with PETSc, if that is something you wish to do.

To install petsc4py on Graham, first go to your home directory by typing cd $HOME. Then, run the following script:

module purge
module load hdf5-mpi/1.8.18 boost eigen python/3.5 scipy-stack/2017b petsc/3.8.2 fftw-mpi/3.3.6 mpi4py/3.0.0

git clone https://bitbucket.org/petsc/petsc4py.git
cd petsc4py
git fetch && git checkout maint-3.8

python setup.py build
python setup.py install --user

export PYTHONPATH=$HOME/.local/lib/python3.5/site-packages:$PYTHONPATH

When the script completes, petsc4py should be installed. The contents of the script are based on the official petsc4py installation instructions (https://www.mcs.anl.gov/petsc/petsc4py-current/docs/usrman/install.html), so consider looking there if something goes wrong.

Step 2: Install FEniCS

To install FEniCS on Graham, first go to your home directory by typing cd $HOME. Then, run the following script:

export PYTHONPATH=$HOME/.local/lib/python3.5/site-packages:$PYTHONPATH

module purge
module load hdf5-mpi/1.8.18 boost eigen python/3.5 scipy-stack/2017b petsc/3.8.2 fftw-mpi/3.3.6 mpi4py/3.0.0 ipp/9.0.4

mkdir fenics && cd fenics
git clone https://bitbucket.org/fenics-project/fiat.git
git clone https://bitbucket.org/fenics-project/instant.git
git clone https://bitbucket.org/fenics-project/dijitso.git
git clone https://bitbucket.org/fenics-project/ufl.git
git clone https://bitbucket.org/fenics-project/ffc.git
wget https://bitbucket.org/fenics-project/dolfin/downloads/dolfin-2017.2.0.tar.gz
tar xvfz dolfin-2017.2.0.tar.gz
mv dolfin-2017.2.0 dolfin
chmod u+w ~/fenics/*/.git/objects/pack/*

pyvenv ~/fenics
source ~/fenics/bin/activate
cd fiat    && git fetch && git checkout 2017.2.0 && pip3 install . && cd -
cd instant && git fetch && git checkout 2017.2.0 && pip3 install . && cd -
cd dijitso && git fetch && git checkout 2017.2.0 && pip3 install . && cd -
cd ufl     && git fetch && git checkout 2017.2.0 && pip3 install . && cd -
cd ffc     && git fetch && git checkout 2017.2.0 && pip3 install . && cd -
pip3 install ply
cd dolfin
mkdir build && cd build

cmake .. -DDOLFIN_SKIP_BUILD_TESTS=true -DEIGEN3_INCLUDE_DIR=$EBROOTEIGEN/include  -DCMAKE_INSTALL_PREFIX=$HOME/software/dolfin -DCMAKE_SKIP_RPATH=ON -DRT_LIBRARY=$EBROOTNIXPKGS/lib64/librt.so -DHDF5_C_LIBRARY_dl=$EBROOTNIXPKGS/lib64/libdl.so -DHDF5_C_LIBRARY_m=$EBROOTNIXPKGS/lib64/libm.so -DHDF5_C_LIBRARY_pthread=$EBROOTNIXPKGS/lib64/libpthread.so -DHDF5_C_LIBRARY_z=$EBROOTNIXPKGS/lib/libz.so -DLIB_ifcore_pic=$EBROOTIFORT/lib/intel64/libifcore.so -DLIB_ipgo=$EBROOTIPP/lib/intel64/libipgo.a -DLIB_decimal=$EBROOTIPP/lib/intel64/libdecimal.a -DLIB_irc_s=$EBROOTIPP/lib/intel64/libirc_s.a

nice make -j 8 install && cd -
sed -i s'^export LD_LIBRARY_PATH=/lib^#export LD_LIBRARY_PATH=/lib^' ~/software/dolfin/share/dolfin/dolfin.conf

Once the script completes, FEniCS will be installed. To run FEniCS, use the following script:

module purge
module load hdf5-mpi/1.8.18 boost eigen python/3.5 scipy-stack/2017b petsc/3.8.2 fftw-mpi/3.3.6 mpi4py/3.0.0
source ~/software/dolfin/share/dolfin/dolfin.conf
source ~/fenics/bin/activate
export PYTHONPATH=$HOME/.local/lib/python3.5/site-packages:$PYTHONPATH

Upon running the above script, the FEniCS virtual environment will be activated. At this point you can simply run a python script containing FEniCS code in the usual way, i.e. by typing python3 script.py or mpirun -np 4 python3 script.py.

The above instructions are based on instructions from the Compute Canada wiki (https://docs.computecanada.ca/wiki/FEniCS), so consider looking there if something goes wrong.

Step 3 (Optional): Install mshr

mshr is an optional FEniCS library that provides meshing utilities. To install mshr, first make sure that you have installed FEniCS. Then, run the following script in your home directory:

module purge
module load hdf5-mpi/1.8.18 boost eigen python/3.5 scipy-stack/2017b petsc/3.8.2 fftw-mpi/3.3.6 mpi4py/3.0.0

export CMAKE_PREFIX_PATH=/home/$USER/software/dolfin/share/dolfin/cmake/:$CMAKE_PREFIX_PATH

git clone https://bitbucket.org/fenics-project/mshr.git
cd mshr
git fetch && git checkout 2017.2.0
mkdir build
cd build
cmake .. -DCMAKE_INSTALL_PREFIX=$HOME/software/mshr -DCMAKE_BUILD_TYPE=Release -DCMAKE_SKIP_RPATH=ON -DEIGEN3_INCLUDE_DIR=$EBROOTEIGEN/include
make
make install

Once the script completes, mshr will be installed. To run FEniCS with mshr, you will now have to use the following script:

module purge
module load hdf5-mpi/1.8.18 boost eigen python/3.5 scipy-stack/2017b petsc/3.8.2 fftw-mpi/3.3.6 mpi4py/3.0.0

source ~/software/dolfin/share/dolfin/dolfin.conf
source ~/fenics/bin/activate

export PYTHONPATH=$HOME/software/mshr/lib/python3.5/site-packages:$PYTHONPATH
export PYTHONPATH=$HOME/.local/lib/python3.5/site-packages:$PYTHONPATH
export LD_LIBRARY_PATH=$HOME/software/mshr/lib:$LD_LIBRARY_PATH

The above instructions are based on instructions from the Compute Canada wiki (https://docs.computecanada.ca/wiki/FEniCS), so consider looking there if something goes wrong.

FEniCS Tips

Sometimes there is weird behavior when using FEniCS on Graham. The following documents some of that behaviour and suggests workarounds.

Unable to perform just-in-time compilation of form

This error message can occur when running FEniCS in parallel on multiple nodes. It seems that FEniCS has difficulties with just-in-time compilation when ran on multiple nodes.

To fix this error, you must first do all the just-in-time compilation on one node. That is, run your script on one node only (you can change the size of your mesh to make this go faster), and stop the script once all of the just-in-time compilation is done. The just-in-time compiled code will be cached by FEniCs. You should now be able to run your script normally on as many nodes as you like.