Using SCP/RSYNC from Ubuntu (Linux)¶
SCP: (secure copy) copies files between hosts on a network. It uses SSH for data transfer, and uses the same authentication and provides the same security as
ssh. Before using
scp, make sure you have valid forwardable Kerberos tickets on your local machine and a working SSH setup.
Transferring from your local machine to PDC¶
Standing in a directory on your local computer containing the file
localfile, you can copy it to the
Private directory on your PDC home directory
~ using the command:
scp <localfile> <username>@t04n28.pdc.kth.se:~/Private/
username is your username at PDC. For more information about what nodes to use see Nodes for file operations
Transferring from PDC to your local machine¶
Standing in a directory on your local computer where you want to put the file pdcfile located in
/cfs/klemming/nobackup/<1st letter username>/<username>/ you can transfer it using the command:
scp <username>@t04n28.pdc.kth.se:/cfs/klemming/scratch/<1st letter username>/<username>/<pdcfile> .
username is your username at PDC.
Avoid overloading the login nodes
The login nodes of most clusters can be weak, so it is important not to over load these nodes. The currently recommended machines for transferring large amounts of data in and out of the Lustre filesystem are the two transfer nodes:
.bashrc or other shell configuration files produce ANY output, then
scp can fail. You can test this with the command below. If the command produces output, then you need to fix your shell configuration files so that they do not produce output.
ssh <username>@t04n28.pdc.kth.se /bin/true
For more information about what nodes to use see Nodes for file operations
Using SCP/PSFTP from Windows¶
If you’re using PuTTY to login to PDC clusters, you can use PSFTP or PSCP that follows the PuTTY installation. To use one of the transfer nodes (t04n27 and t04n28), you should set up a new session in PuTTY in the same way as for login but using the hostname of the transfer node, e.g. t04n27.pdc.kth.se. Save the session with for example the name filetransfer.
To use PSCP, open up the command prompt, i.e run cmd. Use PSCP to transfer the files using the Saved Session you previously added in PuTTY. The syntax is similar to scp (i.e. -r for recursive etc).
To transfer a file from your local computer to Lustre (if you have saved the file transfer session in PuTTY as filetransfer)
"C:\Program Files\PuTTY\pscp.exe" -load filetransfer C:\<file to transfer> <username>@t04n28.pdc.kth.se:/cfs/klemming/nobackup/<1st letter username>/<username>
To transfer a file from AFS to your local computer
"C:\Program Files\PuTTY\pscp.exe" -load filetransfer <username>@t04n28.pdc.kth.se:/afs/pdc.kth.se/home/<1st letter username>/<username>/<file> C:\[folder to transfer to]
To start PSFTP navigate to the folder PuTTY is installed and double click on psftp.exe, or search for PSFTP on the Windows main menu. Note that just like PuTTY, you need a kerberos ticket to use PSFTP.
If you have multiple sessions in PuTTY clicking the PSFTP executable might load the wrong settings. If that happens you have to start PSFTP from the command prompt with an argument specifying the session you created for our cluster (e.g. filetransfer, see above).
"C:\Program Files\PuTTY\psftp.exe" -load filetransfer
When you have started PSFTP, a new terminal will open. Here, you first have to connect to the cluster. You can do this by typing
This is a designated transfer node. If you have followed the step from Windows Login and saved a login session (example: filetransfer) you can also type `` open filetransfer`` instead. At the psftp> prompt you can then use the standard ftp commands (cd, lcd, get, put, …).
Now you have logged in to the cluster and you’re in your AFS Home Directory. You can change your remote directory location to your cfs directory with
cd /cfs/klemming/nobackup/<1st letter username>/<username>/<folder>
You can also change your local directory location with
lcd c:<1st letter username>/<username>/<file folder>
Keep in mind that the location should be specified in the same way you change directory on a Windows terminal, for
You can transfer files by using get or put. get will transfer files specified from remote location to current local directory, and put will transfer files from current local directory to the current remote directory.
For more information about PSFTP utility and commands, please look at http://the.earth.li/~sgtatham/putty/0.63/htmldoc/