StarCCM+

STAR-CCM+ is a commercial computational fluid dynamics package from CD-adapco.

In order to use StarCCM+ at PDC you must provide your own license for the software. The environment variable

CDLMD_LICENSE_FILE

should be set to point to the correct license server

Running Star-CCM+ on Lindgren in batch mode

The Star-CCM files can be accessed by loading the appropriate modules. To see which versions of Star-CCM+ are available use the command

module avail starccm+

The version of VASP required can then be accessed using for example

module add starccm+/6.06.008

Scania users should use the default Scania module which will load the latest version of StarCCM+ and set up the environment so that the Scania licenses are used. This is done using

module add starccm+/scania

Example job script for scania users

# The name of the script is myjob
#PBS -N myjob

# Only 1 hour wall-clock time will be given to this job
#PBS -l walltime=24:00:00

# Number of cores to be allocated is 288.
# always ask for complete nodes (i.e. mppwidth should normally
# be a multiple of 24)
#PBS -l mppwidth=288

# Change to the work directory
cd $PBS_O_WORKDIR

# Run the executable named myexe 
# and write the output into my_output_file
#enable modules within the batch system
. /opt/modules/default/etc/modules.sh
module load starccm+/scania
sim_file="testcase.sim"
STARTMACRO="./starccmStart.java"
LETTER=`echo $USER | cut -c1`
LUSTRE_ROOT=/cfs/scania/nobackup/$LETTER/$USER/StarCCM_config

STARTMACRO=${STARTMACRO:+"-batch $STARTMACRO"}
 
starccm+ -power -collab -np 192 -pio -mppflags "-S 4" $STARTMACRO -batch -arch linux-x86_64-2.3.4 -mpidriver crayxt -nbuserdir $LUSTRE_ROOT ${sim_file}> starccmRun.${PBS_JOBID}.log 2>&1

Note you have to set the number of cores in two places at the top in the

#PBS -l mppwidth=288

line and at the bottom when you specify

-np 192

Note also that StarCCM+ requires 2 GB of RAM per core to work properly, so these numbers to not agree. As each Lindgren node only has 32 GB of RAM it is only possible to run using 16 cores per node instead of the full 24. This means that the number you give in mppwidth (which should be number of nodes*24) will not match the number given after -np (which should be number of nodes*16).

Note also that if you use a license server that is not inside Lindgren due to the poor connectivity between Lindgren and the outside world

Currently we have been having issues running starccm, it appears to work if you set it so that it only runs using 8 cores per node rather than 24. In this case you must change

-mppflags "-S 4"

to

-mppflags "-S 2"

in the submit script and make sure that the number given after the -np is number of nodes*8

e.g. if you use

#PBS -l mppwidth=288

and then should use

-np 96 -mppflags "-S 2"

when using an external license server