Using SCP/RSYNC

Using SCP/RSYNC from Ubuntu (Linux)

SCP: (secure copy) copies files between hosts on a network. It uses SSH for data transfer, and uses the same authentication and provides the same security as ssh. Before using scp, make sure you have valid forwardable Kerberos tickets on your local machine and a working SSH setup.

Transferring from your local machine to PDC

Standing in a directory on your local computer containing the file localfile, you can copy it to the Private directory on your PDC home directory ~ using the command:

scp <localfile> <username>@t04n28.pdc.kth.se:~/Private/

where username is your username at PDC. For more information about what nodes to use see Nodes for file operations

Transferring from PDC to your local machine

Standing in a directory on your local computer where you want to put the file pdcfile located in /cfs/klemming/nobackup/<1st letter username>/<username>/ you can transfer it using the command:

scp <username>@t04n28.pdc.kth.se:/cfs/klemming/scratch/<1st letter username>/<username>/<pdcfile> .

where username is your username at PDC.

Note

Avoid overloading the login nodes

The login nodes of most clusters can be weak, so it is important not to over load these nodes. The currently recommended machines for transferring large amounts of data in and out of the Lustre filesystem are the two transfer nodes: t04n27.pdc.kth.se and t04n28.pdc.kth.se.

Note

If your .bashrc or other shell configuration files produce ANY output, then scp can fail. You can test this with the command below. If the command produces output, then you need to fix your shell configuration files so that they do not produce output.

ssh <username>@t04n28.pdc.kth.se /bin/true

For more information about what nodes to use see Nodes for file operations

Using SCP/PSFTP from Windows

If you’re using PuTTY to login to PDC clusters, you can use PSFTP or PSCP that follows the PuTTY installation. To use one of the transfer nodes (t04n27 and t04n28), you should set up a new session in PuTTY in the same way as for login but using the hostname of the transfer node, e.g. t04n27.pdc.kth.se. Save the session with for example the name filetransfer.

Using PSCP

To use PSCP, open up the command prompt, i.e run cmd. Use PSCP to transfer the files using the Saved Session you previously added in PuTTY. The syntax is similar to scp (i.e. -r for recursive etc).

To transfer a file from your local computer to Lustre (if you have saved the file transfer session in PuTTY as filetransfer)

"C:\Program Files\PuTTY\pscp.exe" -load filetransfer C:\<file to transfer> <username>@t04n28.pdc.kth.se:/cfs/klemming/nobackup/<1st letter username>/<username>

To transfer a file from AFS to your local computer

"C:\Program Files\PuTTY\pscp.exe" -load filetransfer <username>@t04n28.pdc.kth.se:/afs/pdc.kth.se/home/<1st letter username>/<username>/<file> C:\[folder to transfer to]

Using PSFTP

To start PSFTP navigate to the folder PuTTY is installed and double click on psftp.exe, or search for PSFTP on the Windows main menu. Note that just like PuTTY, you need a kerberos ticket to use PSFTP.

If you have multiple sessions in PuTTY clicking the PSFTP executable might load the wrong settings. If that happens you have to start PSFTP from the command prompt with an argument specifying the session you created for our cluster (e.g. filetransfer, see above).

"C:\Program Files\PuTTY\psftp.exe" -load filetransfer

When you have started PSFTP, a new terminal will open. Here, you first have to connect to the cluster. You can do this by typing

open <username>@t04n28.pdc.kth.se

This is a designated transfer node. If you have followed the step from Windows Login and saved a login session (example: filetransfer) you can also type `` open filetransfer`` instead. At the psftp> prompt you can then use the standard ftp commands (cd, lcd, get, put, …).

Now you have logged in to the cluster and you’re in your AFS Home Directory. You can change your remote directory location to your cfs directory with

cd /cfs/klemming/nobackup/<1st letter username>/<username>/<folder>

You can also change your local directory location with

lcd c:<1st letter username>/<username>/<file folder>

Keep in mind that the location should be specified in the same way you change directory on a Windows terminal, for lcd.

You can transfer files by using get or put. get will transfer files specified from remote location to current local directory, and put will transfer files from current local directory to the current remote directory.

get <filename>

For more information about PSFTP utility and commands, please look at http://the.earth.li/~sgtatham/putty/0.63/htmldoc/

Using SCP/RSYNC from Mac OS

scp and rsync work the same on Mac OS as they do on Linux, see information at Using SCP/RSYNC from Ubuntu (Linux)