General information about python¶

Multiple versions of Python are installed on PDC machines.

To see which version of Python is available by default use the command:

python -V

Anaconda¶

Python has a very large number of optional packages for large-scale data processing and scientific computing which are not available in the default system Python. Many of these packages can be found in the enterprise-ready Anaconda Python distribution, which is installed as modules on Dardel. To list available Anaconda modules, type:

       $ module load PDC/22.06
$ module avail anaconda

For example, to load Anaconda version 2021.05 for Python 3.8, type:

$ module load Anaconda3/2021.05

After loading an Anaconda module, the Python version can be printed by:

       $ python --version
Python 3.8.17

The conda package manager can be used to list the packages installed in a given Anaconda module:

$ conda list

Available Anaconda environments can be listed by:

$ conda info --envs

If you need a package which is not available in any of the installed Anaconda modules or environments, either contact PDC support or follow one of the options described below.

Activating the base environment¶

Loading the Anaconda3 module does not automatically activate the “base” environment. To activate the base environment, you can create a script that contains the following lines

        # >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('//pdc/software/22.06/other/pdc-anaconda3/2021.05/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
    eval "$__conda_setup"
else
    if [ -f "/pdc/software/22.06/other/pdc-anaconda3/2021.05/etc/profile.d/conda.sh" ]; then
        . "/pdc/software/22.06/other/pdc-anaconda3/2021.05/etc/profile.d/conda.sh"
    else
        export PATH="/pdc/software/22.06/other/pdc-anaconda3/2021.05/bin:$PATH"
    fi
fi
unset __conda_setup
# <<< conda initialize <<<

       

If you save this script as conda.init.sh and then :

$ source conda.init.sh

The base environment will be activated, as indicated by (base) in front of your bash prompt.

Creating your own Python environment¶

If you create your own conda environment you would have full control and install everything you would want into that environment. For example you can create a conda environment under your Klemming folder:

        module load PDC/22.06
module load Anaconda3/2021.05
source conda.init.sh
conda create --name my-conda-env
conda activate my-conda-env
conda install ...

       

Please note that by default the conda environment will be stored in the .conda folder in your home directory. If you would like to store the conda environment elsewhere, you can use the --prefix flag to specify the path when creating the environment.

Installing your own Anaconda/Miniconda¶

If you need full control over your Python installation, we recommend that you install your own Anaconda or Miniconda distribution, which is relatively easy. The Anaconda distribution is rather large, while Miniconda is much more lightweight. Follow these links to find installation instructions for Anaconda and Miniconda, respectively.

Installing Python packages using pip¶

Another option for installing packages which are missing from the available Anaconda modules (and their conda environments) is to install them locally with pip:

        $ module load PDC/22.06
$ module load Anaconda3/2021.05
$ pip install --user <package-name>

       

This should install the package locally under ~/.local/lib/[.....].

If you would like to install your local package in a path other than ~/.local/lib/, you can use the PYTHONUSERBASE environment variable to specify the installation directory for Python packages that are installed with the --user flag.

Using Anaconda Python on interactive node¶

On Dardel, you instead run Python on the allocated interactive node using the srun command:

        $ salloc -A <your-project-ID> -t 1:0:0 -n 1 -p shared
$ module load PDC/22.06
$ module load Anaconda
$ srun -n 1 python some_script.py

       

Using Anaconda Python in batch job¶

        #!/bin/bash -l
# The -l above is required to get the full environment with modules

# Set the allocation to be charged for this job
# not required if you have set a default allocation
#SBATCH -A snic20XX-X-XX

# The name of the script is myjob
#SBATCH -J myjob

# Only 1 hour wall-clock time will be given to this job
#SBATCH -t 1:00:00

# Number of tasks
#SBATCH -n 1

# Job partition
#SBATCH -p shared

# load the anaconda module
module load PDC/22.06
module load Anaconda3/2021.05

# if you need the custom conda environment:
conda activate my-conda-env

# execute the program
srun -n 1 python some_script.py

# to deactivate the Anaconda environment
conda deactivate

       

Disclaimer¶

PDC takes no responsibility for the correctness of results produced with the binaries. Always evaluate the binaries against known results for the systems and properties you are investigating before using the binaries for production jobs.

Installed software¶

Cluster	How to use	How to build
Dardel	3.8.17	3.8.17

PDC Center for High Performance Computing