Compiling

The PDC Cray system Lindgren currently has four compilers available:

 Intel, PGI, Gnu, Cray

To access the compilers use the ftn/cc/CC scripts e.g.

ftn foo.f
cc foo.c
CC foo.cpp

Always use ftn/cc/CC when compiling. You should never need to use anything else. These scripts automatically call the correct compiler (depending which PrgEnv module you have loaded). They also automatically link to the MPI libraries, and and other libraries depending which modules you have loaded  (e.g. to link to Lapack etc, just load the xt-libsci module and the scripts will handle the rest).

When you log on the PGI compiler will be active, but for example to make the gnu compiler active

module swap PrgEnv-pgi PrgEnv-gnu

to change this. Similarly if you want to use the intel compiler then

module swap PrgEnv-pgi PrgEnv-intel

There are many scientific libraries pre-installed by Cray. In general if you use the ftn/cc/CC scripts and have the module loaded then they will be included in the compile automatically, i.e. if you have the correct modules loaded you do not need to add any extra flags while linking the ftn/cc/CC-script takes care of all the needed flags.

e.g. invoking:

module load xt-libsci fftw

will make your code linked against lapack, blas, scalapack, blacs, fftw etc. without adding any flags to the ftn/cc/CC command line.

Be aware that not specifying flags in this way may confuse some automatic build scripts.

When using xt-libsci make sure you load (i.e. link) both FFTW and xt-libsci even if you have no need for ffts.

Other libraries that can be linked by just loading the module and by not adding any extra flags include

netcdf arpack hdf5

It is best if everything on the Cray is statically linked as using shared libraries on the compute nodes requires special treatment, and many shared libraries are not available. Some pre-compiled programs are for this reason a problem to run, which is why there is also "cluster compatibility mode" which invokes a more complete linux environment on the compute nodes before running programs.

Compiling Dynamically

By default on the Cray all programs are built using static libraries. This is recommended for most programs. Some programs however it is necessary to build using dynamic libraries. In this case the following procedure should be used

Set the following environment variable

export XTPE_LINK_TYPE=dynamic

then compile your code as normal. At runtime you also need to set

export CRAY_ROOTFS=DSL

to make sure you have access to the shared libraries.

Compiling Large Programs using Make

The login node on Lindgren is quite weak with only 6 cores and 12GB of ram, so large programs can also be compiled on the interactive nodes (which have 24 cores and 32 GB of ram)

To be able to access make on the interactive nodes you first need to set the environment variable CRAY_ROOTFS using (for bash)

export CRAY_ROOTFS=DSL

Then as long as your source code is in /cfs (remember your home directory is not accessible from the interactive nodes) then you can use

aprun -n 1 -d 24 make -j 24

to compiler your program. The "-d" option is important as it stops your make being locked to a single core.

You might also need

export TMPDIR=/cfs/klemming/scratch/u/username

as well.