Clusters, nodes, processors, and cores¶
Like most high-performance computing facilities, PDC mainly features clusters. A computer cluster is in the broad sense terminology for a supercomputer, consisting of a set of connected computers working together so that they can be viewed as a single system. There are a number of different Clusters at PDC for various needs.
A node is the individual computer part of each cluster. Nodes are analogous to the computers we use everyday.
And each node in turn has processors made up of computing entities called cores.
For a more technical overview, please visit the Resources page.
Deciding whether you need PDC for your work¶
The key feature of PDC systems (or most HPC resources in general) is large-scale parallelization of computations. PDC resources are used in a wide-range of scientific disciplines. If you want to find out if you work could benefit from PDC resources, check if:
- your application require large computional resources
- your application require large memory resources
- your application require GPUs
- your application can be parallelized
If you decide to use PDC resources, welcome on board! PDC provides you with:
- supercomputer systems for large simulations and calculations
- systems for processing data before and after simulations or calculations
- software for simulation and modelling
- short term storage for large volumes of data
- assistance with using PDC’s computing and storage resources
- experts in different research fields to assist you with using and/or scaling software
Who may use PDC’s HPC services¶
PDC’s resources are available for both business and academic research. Companies wanting to use PDC’s systems should contact email@example.com to discuss their requirements. Academic researchers (from Sweden or Europe) may apply for free time/storage allocations at PDC. Swedish academic researchers and their collaborators will need to apply for an allocation of time/storage for their projects via SNIC https://www.pdc.kth.se/support/documents/getting_access/get_access.html, while researchers from other European countries may apply to use PDC’s resources through DECI http://www.prace-ri.eu/deci-projects/.
Account and Time Allocation¶
Before starting using PDC resources, you will need to get an account. Each user account must belong to one or more Time Allocations, since we allocate resources and manage job queueing based on the Time Allocation you belong to and not based on your individual user account. Each Time Allocation includes the following information:
- list of users belonging to that Time Allocation
- number of corehours allocated per 30 days period for all members of the Time Allocation
- an expiration date for the Time Allocation
- list of clusters available for running jobs
When you submit jobs in a cluster, you should belong to at least one Time Allocation, or the submission will fail. Using Time Allocations allows us to:
- Prioritize your submitted jobs compared to other user’s jobs.
- Keep track of compute-time used within the last 30 days by users and projects.
How much resources will be needed: corehours¶
At PDC, we allocate compute-time on our systems in corehours and you are granted an amount of corehours on a particular system. Corehours equals the number of cores used in how many hours. A time allocation gives you a certain amount of corehours per month.
Clusters at PDC¶
Today PDC have these clusters which are available to users
|Name||Types of allocation||Intended for…||Available to…|
|Dardel||Medium/Large||Jobs capable of using >128 cores (1 node)/job||
Scientific groups in the Stockholm area
|Beskow||Medium/Large||Jobs capable of using >128 cores (4 nodes)/job||SNIC|
|Tegner||Small/Medium, and pre/post allocation for Medium/Large Beskow allocations||jobs requiring Large memory, GPUs||Pre- and postprocessing for SNIC users of Beskow. Scientific groups in the Stockholm area|
You can read information about our systems at https://www.pdc.kth.se/hpc-services/computing-systems
Basic Linux for new HPC users¶
Working with PDC resources requires a basic understanding of Linux systems. Some of our users are new to using Linux-based systems and have asked for introductory materials. Here is a collection for the basic command-line operations needed to get started with PDC.
Using commands in the shell¶
The shell is the program from which the user controls everything in a text-interface. When you login to a PDC system remotely, you are already in the shell window of the system. If you login to your own system, you are probably on a graphical screen. From there, search for terminal or Ctrl+Alt+T to enter the shell terminal. In the shell, you can start typing commands to perform some action. The default shell on the PDC clusters is bash.
Useful Shell commands¶
Upon login we are greeted with the shell
ssh firstname.lastname@example.org Last login: Fri Aug 8 10:14:59 2017 from example.com user@tegner-login-1:~> _
Bash: Files and directories¶
Command pwd tells me where I am. After login I am in the “home”directory
I can change the directory with cd
cd tmp/talks /tmp/talks pwd /afs/pdc.kth.se/home/u/user/tmp/talks
I can go one level up with cd ..
I can return to my HOME folder with cd
List the contents with ls -l
user@machine:~/tmp/talks ls -l total 237 drwx------ 3 user csc-users 2048 Aug 17 15:21 img -rw------- 1 user csc-users 18084 Aug 17 15:21 pdc-env.html -rw------- 1 user csc-users 222051 Aug 17 15:22 remark-latest.min.js
Bash: creating directories and files¶
We create a new directory called results and change to it
mkdir results cd results
Creating and editing files
- Textfiles can be edited on your local computer and then transferred
- Textfiles can also be edited locally using text editor like nano/emacs/vim
Copying, moving renaming and deleting¶
# copy file cp draft.txt backup.txt # recursively copy directory cp -r results backup # move/rename file mv draft.txt draft_2.txt # move/rename directory mv results backup # move directory one level up mv results .. # remove file rm draft.txt # remove directory and all its contents rm -r results
File permissions with chmod¶
In Linux systems all files have a user, a group and a set of privileges which determines what resources a user can access. Every file has three different kind of access: read(r), write(w) and execute(x), as well as three different kind of permissions depending on if the person is the owner(u=user) of the file, in the same group(g) or someone else(o=other).
chmod g+w file
Adds(+) write(w) permissions for group(g) to the file.
There is another way to set the permissions by using numbers. Assume that each permission equals the number listed below:
chmod 753 file
Gives the user the read, write and execute permission(4+2+1), whereas users in the same group get read and execute permissions (4+1) while others get write and execute permissions (2+1).
The command getfacl is used to get file access control lists (ACLs)
getfacl dir # file: dir1 # owner: lama-tst # group: users user::rwx group::r-x other::r-x
the directory dir1 is owned by the user lama-tst with permissions read(r), write(w) and execute(x). The permissions of the group and others are set to r-x which means that the group and others can read and execute.
Use option -c to skip the comment header.
setfacl -m user:jon:rwx dir1 getfacl -c dir1 user::rwx user:jon:rwx group::r-x mask::rwx other::r-x
Now the user jon has been added and can access to dir1 with permission read(r), write(w) and execute(x).
The options -x is used to remove entries from the ACLs of file and -R to recurse into subdirectories.
Bash: history and tab completion¶
history preserve commands used
history 689 cd .. 691 cd Documents/ 692 cp -r introduction_PDC /afs/pdc.kth.se/home/h/hzazzi/Documents/Presentations 693 ssh beskow.pdc.kth.se 694 cd introduction_PDC/ 695 ls -l 696 pwd 697 history
If I want to repeat…
!696 pwd ~/Documents/introduction_PDC
Use also the TAB key for completion
CTRL/R to search for previous commands
Arrows up/down to scroll for earlier commands
Bash: finding things¶
Extract lines which contain an expression with grep
# extract all lines that contain searchme grep searchme draft.txt
If you do not know what a UNIX command does, examine it with man
Find files with find
find ~ | grep lostfile.txt
We can pipe commands and filter results with |
grep energy results.out | sort | uniq
Bash: Redirecting output¶
Print content of a file to screen
Redirect output to a file
cat test.out > myfile.txt
Append output to a file
cat test.out >> myfile.txt
Bash: Writing shell scripts¶
#!/bin/bash # here we loop over all files that end with *.out for file in *.out; do echo $file cat $file done
We make the script executable and then run it
# Make it executable chmod u+x my_script # run it ./my_script
Arguments to script can be passed by using $¶
#!/bin/bash echo "Hi" $1 $2
./myscript Nils Nilsson Hi Nils Nilsson
To starting executing such scripts, you would need to start with a text-editor. Choosing a text-editor is a matter of personal choice, the most popular ones being Vim and Emacs. But there are a lot more new and interesting ones. Open your favorite text-editor and copy-paste the file example above and save with file as <script>. Then run the script by typing
Information about shell commands¶
Information about a commands can be retrieved from the manual
Also you can get information about where the executable lies
Executing your software¶
Most commands are quite intuitive acronyms and are easy to remember once you start using them. The usual syntax is
command -option1 arg1 -option2 arg2
command is the name of the command,
-option2 specifies the particulars of the command (they are optional, there can be as many options as the specific command permits), and
arg2 are the value of the corresponding options. In general
Prints information about what options and arguments you can enter.
Editing your files¶
Editing your files on our cluster can be done with several text editors. Emacs and vi/vim are available on all PDC clusters and can be opened by
$ emacs [filename] $ vi [filename] $ vim [filename]
In addition, nano is available on Tegner and installed as a module on Beskow
$ module load nano # only needed on Beskow $ nano [filename] # both Beskow and Tegner
For Linux beginners, nano might be a good editor to start with. Emacs and vi have steeper learning curves, so a reference guide is provided here for the most common operations using these editors.
A complete tour of using emacs can be found at https://www.gnu.org/software/emacs/tour/
Below is a table summarizing the most frequently used operations.
C-x means pressing simultaneously the
C-x 2 means first pressing both
x and then pressing
C-x C-s means first pressing
x followed by
M-x means pressing the
meta-key (Alt-key) and
|$ emacs foo.dat||open foo.dat for editing|
|C-x C-s||save file|
|C-x C-c||quit emacs|
|C-x C-w foo.dat||save file as foo.dat|
|C-_ or C-/||undo last change (can be repeated)|
|C-g||cancel current command|
|C-f||Move forward one character|
|C-b||Move backward one character|
|M-f||Move forward one word|
|M-b||Move backward one word|
|C-a||Beginning of line|
|C-e||End of line|
|M-<||Beginning of buffer (file)|
|M->||End of buffer|
|C-k||Kill (cut) line|
|C-u 10 C-k||Kill 10 lines (C-u <N> can be used to repeat any command)|
|C-y||Yank (paste) line|
|C-SPC||Set mark at current location|
|C-w||Kill (cut) region between current location and last mark|
|M-w||Save (copy) region|
|C-x (||Start defining keyboard macro|
|C-x )||End defining macro|
|C-x e||Execute macro once|
You can find tips and tricks for using vi at https://www.cs.colostate.edu/helpdocs/vi.html
Below is a table summarizing the most frequently used operations of vi/vim.
|$ vim foo.dat||open foo.dat for editing|
|i||enter insert mode, insert before the cursor|
|a||enter insert mode, insert (append) after the cursor|
|o||enter insert mode, append a new line|
|Esc||exit insert mode|
|:wq||save and quit|
|:q!||quit without save|
|u||undo last change|
|Ctrl-r||redo last change|
|h||move cursor left|
|j||move cursor down|
|k||move cursor up|
|l||move cursor right|
|gg||move to the beginning of file|
|G||move to the end of file|
|10G||move to line 10|
|x||delete 1 character|
|dw||delete 1 word|
|dd||delete 1 line|
|d5d||delete 5 lines|
|yyp||duplicate 1 line|
|/pattern||search for pattern|
|?pattern||search backward for pattern|
|n||repeat search in same direction|
|N||repeat search in opposite direction|
|:s/old/new/||replace the first old with new in the current line|
|:s/old/new/g||replace all old with new in the current line|
|:%s/old/new/g||replace all old with new throughout file|
Learn more at http://www.fprintf.net/vimCheatSheet.html
The apperance of vim can be configured via the
~/.vimrc file. Below is an example
syntax on filetype plugin indent on set autoindent set hlsearch " highlight search set number " show line number set ruler " show ruler at bottom right " Return to last edit position when opening files autocmd BufReadPost * \ if line("'\"") > 0 && line("'\"") <= line("$") | \ exe "normal! g`\"" | \ endif
- The Linux Command Line by William E. Shotts, Jr.
- This book introduces the linux command line from the basics, and moves on to customizing the working environment and then finally to shell scripting. The entire book is available for free from the authors web page, and if you would like a paper copy you can order one from the publisher.
- UNIX / Linux Tutorial for Beginners
- The University of Surrey has an online tutorial that introduces the linux command line. The web page also has links to other recommended linux books.
Here you will find the basic terminologies and quick reference materials.
- Computer cluster is in the broad sense the terminology for a supercomputer, consisting of a set of connected computers working together so that they can be viewed as a single system. See resources to examine what clusters PDC currently has available.
- Nodes are components of a cluster and analogous to the computers we use every day. A supercomputer consists of a number of nodes that perform computations and run an own instance of the operating system.
- component of each node. Each node has a number of processors that are analogous to the CPU (Central processing unit) in a personal computer.
- component of each processor. The actual computing entity.
- Time Allocation
- Time Allocation is the technical term for “Project” that users need to be a part of to use the supercomputer. Time allocation contains information about the project and how much resources that have been allocated to the project. There are different types of time allocation depending on the size of the allocation, but the instructions for running jobs are the same.
- Principal Investigator (PI)
- Principal Investigator is the “Project manager” of a time allocation. This user can add/remove users from the project and is usually the one that applied for the time allocation. All project related utilities are managed through SUPR, https://supr.snic.se
- Corehours refers to the number of processor units (cores) used to run a simulation multiplied by the duration of the job (t) in hours. Corehours (c) can be calculated if the number of nodes (n), the duration in hours (t) and the number of cores per node (cpn) are known: c = n*cpn*t
- SNIC or Swedish National Infrastructure for Computing is a national research infrastructure for high-performance computing.
- SUPR or SNIC User and Project Repository is the SNIC database used to keep track of persons, projects, project proposals and more. https://supr.snic.se
- SNAC or Swedish National Allocations Committee handles all larger applications and allocations of SNIC.
- Swedish National Allocations Committee Working Group is a group comprising of SNAC members and experts from the various HPC centres in Sweden.
- SLURM or Simple Linux Utility for Resource Management is an open-source cluster management and job scheduling system extensively used by PDC.
- Kerberos is a computer network authentication protocol. It works on the basis of creating a ticket that is used for secure communication. You need to create a Kerberos ticket to log in to the clusters, run programs and get access to the home directory and transfer files.
- Andrew File System (AFS) is a distributed file system. This file system can be accessed without logging in to the clusters, and this is also the system where users home directory reside. To get access to AFS, one needs to use Kerberos.
- Lustre is a high-performance parallel file system which is mounted on the PDC clusters. Lustre provides fast access to large data files needed for large parallel applications, but is less suitable for dealing with many small operations on a large number of files.