User information

Getting access to Ansys Fluent at PDC

In order to use ANSYS Fluent at PDC you need to:

1. Have an account at PDC.
2. Contact PDC to be added to the group of Fluent users. When contacting us, please state which person in the PDC ANSYS Fluent collaboration that you are working with.
3. Contact KTH (intranet) if you have problems/questions with Ansys/Fluent licenses.
4. Use command

module avail fluent

to list all available versions and make sure that the correct version is loaded in the job script. If there is no same (compatible) version as that creating the .cas file, please contact PDC.

 

Using Fluent on Lindgren

Fluent on lindgren works a little differently and is described on this page.

Submitting a Fluent job on Ferlin

Typically, you can read your mesh (generated in ICEM, for example) into Fluent on your local machine and setup all the necessary parameters interactively (directly using the GUI). You can then save this as a case file (something.cas.gz

), transfer it to PDC and then prepare the journal file using a text editor on PDC (vi or emacs for example). A small script must also be written to deal with the parallelization of Fluent on PDC.

Since the case file is compressed, it occupies less disk space on PDC (an important issue) and several parameters have already been setup in this file (the boundary conditions, basic

discretization schemes, fluid and turbulence models, etc.). This strategy greatly simplifies the writing of the journal file and avoids syntax errors.

Fluent Journal File

The journal file is the equivalent of the definition file in CFX. It is written in a dialect of Lisp called Scheme and contains all the instructions that are to be executed during the run. A basic form of this file, is as follows:

# -----------------------------------------------------------
# SAMPLE JOURNAL FILE
#
# read case file (*.cas.gz) that had previously been prepared
file/read-case something.cas.gz
# initialize flowfield
solve/initialize/initialize-flow
# run 100 iterations
solve/iterate 100
# write data
file/write-data something.dat.gz
# exit fluent
exit yes
#------------------------------------------------------------

Save the journal file as something.jou. In this simple example, we read in the something.cas.gz

file, which we had previously prepared (includes all the boundary conditions and models). We then initialized the flowfield in order to start

the iterations and we solved for 100 iterations. After 100 iterations had been performed, the data generated was written to the file something.dat.gz. We are assuming here that 100 iterations will suffice for a converged solution.

Convergence Issues

Up to here everything seems quite straightforward. However, imagine that during the run the solution diverged, i.e., it would arrive at iteration 100 and we would not have a converged solution or, the simulation would crash sometime in between. There are several ways to perform a more stable simulation. For example, start off with simpler turbulence models (or/and fluid models or/and more relaxed discretization schemes) and perform n number of iterations. Then change to more elaborate models a perform m iterations.
An example of this strategy would look something like this (we assume here that
something.cas.gz had been setup with the “simple” standard k-epsilon turbulence model):

# -------------------------------------------------------------------------
# SAMPLE JOURNAL FILE (attempt to avoid some convergence issues)
# read case file (*.cas.gz) that had previously been prepared with standard 
# k-epsilon
turbulence model
file/read-case something.cas.gz
solve/initialize/initialize-flow
solve/iterate 50
file/write-data something-50.dat.gz
# change to a more elaborate turbulence model
define/model/viscous/ke-realizable yes
# run another 50 iterations
solve/iterate 50
# write case and data files to current directory
file/write-case-data something-100.dat.gz
# exit fluent
exit yes
#--------------------------------------------------------------------------

Fluent Text Inputs for Journal Files (Scheme)

If you are not familiar with the nomenclature of the Fluent text inputs, do the following:

1. Open the GUI version of Fluent.
2. Once loaded, press Enter and you will see the text version of the “main menu”.

3. Type file and press Eenter.

4. Type read-case and press Enter.
5. Type q and press Enter to exit the submenus.

In a journal file, this would look something like this:

file/read-case/

or (in condensed notation)

 f/rc/

Furthermore, the scheme language is programmable. An experienced user can easily automate
several functionalities within the journal file.

Running Fluent on dedicated nodes

Connect to whatever cluster you will be using on PDC, remember to forward your forwardable Kerberos tickets. Once you have your journal file prepared, you will need to prepare a script to parallelize your Fluent run. Let us first look at the syntax involved in submitting a Fluent parallel simulation from the command line:

fluent <version> -g -p<comm> -mpi=openmpi -t<N> -cnf=<host list> -i <journal file>

The EASY scheduler system allocates nodes for your job. Once you type the esubmit command, several environment variables are generated, specific to the job you just submitted. One of these variables, the SP_HOSTLIST, contains a list of the hosts that have been allocated to you. Lets suppose that you have requested 3 nodes then the SP_HOSTLIST

would look something like this (one node per line):

node1.pdc.kth.se
node2.pdc.kth.se
node3.pdc.kth.se

Now you know which nodes will be allocated to you (the once specified by the environment variable SP_HOSTLIST). However, this file must be rewritten in the correct format for Fluent, depending on how many CPUs per node I want to use. How do you do this? This is where your shell scripting knowledge comes into scene. We know that if you request 3 nodes and you want to take advantage of the 2 CPU’s per node, then your Fluent host file will have to look something like this:

node1.pdc.kth.se
node2.pdc.kth.se
node3.pdc.kth.se
node1.pdc.kth.se
node2.pdc.kth.se
node3.pdc.kth.se

Fluent wants a host list with a name of a node for each CPU. If you want to use several CPUs on one node, that node name should be repeated one time for every processor. For the example above, the Fluent argument –t must have the number 6 in front of it (you will be running 3 x
2 = 6 processes (or tasks). All these adjustments can be done by writing a shell script, presented in the next section. Basically, the script just generates the Fluent syntax with the arguments in the correct format.

The fluent-pdc.sh Script

A script has been prepared called fluent-pdc.sh and presented on the following page. Note that this script does not include all the arguments that you can supply to Fluent, but you can/should add/replace whatever you want. As it is, it will work fine for your simulations if you follow the notation properly. You can copy this script to your home directory on PDC and save it as fluent-pdc.sh. Make sure its permissions are set to executable (chmod a+x fluent-pdc.sh). The fluent-pdc.sh script handles all the arguments for fluent and puts them in the correct format. So, in order to run a fluent parallel job on PDC, and once you have created your journal file, all you have to do is type the following:

module add easy
cd <where ever you have created the files>
esubmit -n 2 -t 15 ./fluent-pdc.sh 8 3ddp ./case.jou

In this example, we requested 2 nodes for 15 minutes using the esubmit command included in the easy module. When the queuing system has processed our request and allocated the node the fluent-pdc.sh script steps into action. We have chosen to run on 8 CPUs per node with the fluent 3d double precision solver. Note - we give the number of processes per node to the script, and when the script starts Fluent it will give 8*2=16 as an argument to the t-flag in Fluent. Instructions are given to the Fluent solver
via the journal file case.jou which is assumed to be located in the same directory as the fluent-pdc.sh script.

Note that the license for students only allows Fluent running one node with maximum four codes.

To confirm if Fluent is actually running on the requested node, log on to it and type top. Check the list to see if the Fluent executable appears and also look at the CPU
usage. Type q to exit top.
Once the job has finished, you will receive an email with the output. If you wish, you can
automatically generate output files in a certain location (if you are familiar with shell scripting this should be easy for you to incorporate into fluentpdc.esy).
A help menu has been incorporated into the script, so just type fluent-pdc.sh –h to access it if you need to be remembered of something.

#!/bin/bash
# fluent-pdc.sh script 
# This script was originally created by 
# Pedro Costa, pedro.costa@energy.kth.se
# ----------------------- HELP MENU -----------------------------------------------
mail="support@pdc.kth.se"
help()
{
cat <<HELP
fluent-pdc.sh -- run fluent on a PDC cluster. This script must 
always be preceded by the esubmit command and its respective
arguments.

USAGE:
fluent-pdc.sh [-h] -ppn -solver -journalfile
Note that the order in which these arguments appear must be the 
one specified here
-h help text (this is an option)
-ppn number of processes to run on each node (up to 8 for Ferlin)
-solver version of solver (2d, 2ddp, 3d, 3ddp)
-journalfile journal file (include path, if necessary)
EXAMPLE:
esubmit -n 2 -t 15 ./fluent-pdc.sh 8 3ddp ./case.jou
Here we requested 2 nodew for 15 minutes with esubmit. 
We chose to take advantage of 8 CPUs per node and run
the fluent 3d double precision solver. Instructions will 
be given to the solver via the file case.jou

BUGS/SUGGESTIONS:
$mail
HELP
exit 0
}
# -------------------------------------------------------------------------------------
error()
{
echo "$1"
exit 1
}
# --------------------------- READ USER SUPPLIED OPTIONS ------------------------------
while [ -n "$1" ]; do
case $1 in
-h) help;shift 1;;
-*) echo "error: no such option $1. -h for help";exit 1;;
*) break;;
esac
done
PPN=$1; # nr of processors per node
SOL=$2; # version of solver
JOU="$3"; # path to journalfile
LOGFILE=./$SP_JID.log
test $SP_PROCS || exit 255
module add heimdal fluent
# ------------------------- RUN FLUENT (PARALLELIZATION) ------------------------------
if [ $SP_PROCS -gt 1 ]; then
# use SP_HOSTFILE to create the correct format and rename it
i=0
while [ $i -lt $PPN ]; do
       cat $SP_HOSTFILE > $SP_JID.hosts
       i=$((i+1))
done
echo "CNF file is: ">>$LOGFILE
cat $SP_JID.hosts >>$LOGFILE
FLUENTARGS="-t`expr $PPN \* $SP_PROCS` -cnf=$SP_JID.hosts -pethernet -mpi=openmpi"
elif [ $PPN -gt 1 ]; then
# use shared memory MPI only
FLUENTARGS="-t$PPN"
fi
echo "Executing fluent in directory `pwd`">>$LOGFILE
echo "fluent $SOL -g $FLUENTARGS -i $JOU" >>$LOGFILE
fluent $SOL -g $FLUENTARGS -i "$JOU"
echo "Finished running Fluent" >>$LOGFILE

Summary

The steps required to submit a Fluent batch job on dedicated nodes to PDC are as follows:

1. Generate a mesh (in ICEM, for example). Save as something.msh.
2. Open Fluent. Read in mesh and setup parameters (boundary conditions, models, discretization schemes, etc.). Save as something.cas.gz and transfer it to your working directory at PDC.
3. Login to whatever cluster you will be using on PDC. Confirm the lifetime of your Kerberos tickets. Open an editor and write your journal file. Save it as something.jou. Load the easy module.
4. Create the fluent-pdc.sh script in your working directory. Make it executable and submit your job to the EASY scheduler using the esubmit command.
5. Once the nodes requested have been allocated to you, confirm that everything is running properly by connecting to one of the nodes and typing top.
6. Once your convergence criteria has been met (or if the job crashes!) you will receive an email with the respective output.

External links

Fluent information at the SNIC knowledge base