AVL Fire at PDC

Fire Usage Information

Fire is commercial software and PDC does not have a site license for the software, so users must provide their own license to the software. The environment variable

LM_LICENSE_FILE

Should be set to point to the license server.

Running Fire at PDC

Lindgren

 On lindgren to use Fire then cluster compatability mode (CCM) must be used. Note your account must be specially enabled to use CCM on Lindgren. Contact support if you need your account to be able to use CCM.

Running interactively on Lindgren

It is not possible to use the interactive nodes to run CCM jobs. However it is possible to reserve some nodes in CCM mode and use them interactively. To do this use. If no nodes are available you will join the batch queue until they become available.

qsub -q ccm -I -l mppwidth=48 -l walltime=30:00

then fire can be started using

module load ccm avl
nodes=`hostconvert_fire.pl $PBS_NODEFILE`
ccmrun run_fire.sh $PBS_O_WORKDIR 48 $nodes CaseName

The default run_fire.sh is shown below. If you need different flags for  cfdwm_solver then a new run_fire.sh script can be made and used instead.

Also the command

ccmlogin

can be used to login to the head node of the ccm allocation, at which point it is possible to call cfdwm_solver directly. The enviroment variable $PBS_NODEFILE is not exported in this case though, so the nodes must be set carefully.

Running in batch mode on Lindgren

To run in batch mode it similar to running a normal job, except the special CCM queue must be used,

With CCM it is necessary to use two scripts. A PBS job script and the script that gets started by CCM on the reserved nodes.

Example PBS script

# The name of the script is myjob
#PBS -N myjob

#use ccm
#PBS -q ccm

# Only 1 hour wall-clock time will be given to this job
#PBS -l walltime=1:00:00

# Number of cores to be allocated is 288.
# always request full nodes, i.e. mppwidth should normally be a multiple of 24
#PBS -l mppwidth=288

# Change to the work directory
cd $PBS_O_WORKDIR

#enable modules in the queue system
. /opt/modules/default/etc/modules.sh

module load ccm avl

#build the string so that fire knows which nodes are available
nodes=`hostconvert_fire.pl $PBS_NODEFILE`

# start the avi_fire job
# run_fire.sh takes 4 values. The work directory, the number of nodes
# the nodes string built above and the casename. 
ccmrun run_fire.sh $PBS_O_WORKDIR 288 $nodes CaseName

The default run_fire.sh script looks like

#!/bin/bash
. /opt/modules/default/etc/modules.sh
module unload heimdal
module load avl

cd $1
cfdwm_solver -name=fire -nomapping -mpi -cpu=$2 -hostlist=$3 -project_dir=$PWD -case=$4 -solver_vers=v2010

Note that the "-ssh" option should not be used in cfdwm_solver if you make your own script. The default rsh should be used by mpirun to start tasks on remote nodes.

Running on Lindgren using only 12 cores per node

It has been reported that better performance can be seen on Lindgren using just 12 cores per node to run fire processes rather than the default 24.

This can be acheived using a submit file such as the following.

# The name of the script is myjob
#PBS -N myjob

#use ccm
#PBS -q ccm

# Only 1 hour wall-clock time will be given to this job
#PBS -l walltime=1:00:00

# Number of cores to be allocated is 288.
# always request full nodes, i.e. mppwidth should normally be a multiple of 24
#PBS -l mppwidth=288

# Change to the work directory
cd $PBS_O_WORKDIR

#enable modules in the queue system
. /opt/modules/default/etc/modules.sh

module load ccm avl

#build the string so that fire knows which nodes are available
nodes=`hostconvert_fire_half.pl $PBS_NODEFILE`

# start the avi_fire job
# run_fire.sh takes 4 values. The work directory, the number of nodes
# the nodes string built above and the casename. 
ccmrun run_fire.sh $PBS_O_WORKDIR 144 $nodes CaseName

Note that on the

#PBS -l mppwidth=288

line you give the total number of cores on the nodes you will be using. Then later on the

ccmrun run_fire.sh $PBS_O_WORKDIR 144 $nodes CaseName

you give half this number.