Secure Copy Tools
How to transfer files using secure transfer mechanisms including scp, sftp and rsync
The SSH secure terminal tool also supports transferring files over a secure encrypted link. There are two mechanisms provided, Secure Copy (scp) and Secure File Transfer Protocol (sftp), the former is a scriptable one-off transfer of a file or folder, the latter an interactive session allowing you to up or download files and folders. You will typically need to use these tools to download MRI scans, transfer files to/from the WIN or BMRC clusters or collaborator's systems.
In addition to the command line tools (available on macOS, Linux and recent versions of Windows) there are various GUI applications that support the SFTP protocol. We recommend Cyberduck on macOS and Windows (OS App Store or https://cyberduck.io/) and FileZilla on Linux.
How to use the scp command to transfer files
The scp commands general syntax is:
scp [-r] source destination
The -r option is only necessary when the source is a folder, and will recursively copy the folder and it's contents.
Source or destination may be local or remote, depending on whether you are pulling (source is remote) or pushing (destination is remote) the file(s)/folder(s). The source supports the use of wildcards to allow the transfer of many files/folders with similar names, in this case destination must be a folder that they will be copied to.
When specifying a local target, use the absolute or relative path to the object(s) to transfer, ~/myfile would reference the file myfile in your home folder, or ../something would reference the file/folder something in the parent of the current folder.
When specifying a remote target use
username is optional if it matches on source and destination.
path supports absolute paths or with ~ as a prefix items relative to your home folder.
When not to use scp
scp is not resumable, so if you need to transfer a very large file (or a folder with many files) you should use some alternate tool, such as rsync. It also can't easily synchronise two folders - it only support replacing existing files and adding new ones, not removing files from the target that don't exist (anymore) on the source, again use rsync for this.
scp and multi-factor authentication can be irritating as you need to enter your password and second factor every time you transfer a file/folder - if the service supports SSH certificate authentication then this can be avoided, but otherwise use sftp or a GUI app.
How to transfer files using the SFTP command line tool
The sftp command provides an interactive file transfer client, although it can be used in a one-off transfer manner to get individual files. Whilst it is more complex to use than scp, it may be more efficient when you have many files to transfer that can't be specified (easily) with wildcards, especially when you are connecting to a system using multi-factor authentication (such as BMRC's cluster).
In addition, some services only support SFTP connections, such as the WIN MRI download system when used with SFTP Only accounts.
The basic sftp command syntax is
where destination is of the form user@host:path. As with scp, user@ is optional if the username is the same on your computer as the remote service and if path is not specified then it will use your remotely configured home folder.
When you have authenticated you will get a prompt that can be used to move around the remote file system; use 'cd' and 'ls' as you would on a UNIX system to change directory and list the contents. You can also interact with your local file system with the 'lcd' and 'lls' equivalents.
To upload files/folders use 'put' and to download use 'get' with the filename of the local or remote object respectively.
How to use the Rsync (remote synchronisation) tool
The rsync tool allows you to transfer files/folders between two locations in a resumable manner and optionally ensure that the two locations are identical. It uses file metadata to determine what needs to be transferred which means it is much slower to start transferring than a simple scp but for regular mirroring or large transfers, especially over intermittent network connections (such as WiFi) the efficiency gains over plain scp are significant.
Rsync is included with Linux and macOS, but currently the best way to use it with Windows is to install Windows Subsystem for Linux and use it from there.
The rsync command is very complex, so only the bare minimum will be covered here - see the 'man' entry or read online here https://download.samba.org/pub/rsync/rsync.1
Transferring a folder from your computer to a remote server can be achieved with
rsync -avz /path/to/folder/ email@example.com:/path/to/rfolder/
This will synchronise (but not delete on the remote folder if missing locally) /path/to/folder to /path/to/rfolder on remotehost.domain. username is optional if this matches at both ends.
Pulling from a remote host is possible by reversing the source and targets:
rsync -avz firstname.lastname@example.org:/path/to/rfolder/ /path/to/folder/
rsync -avz /path/to/file email@example.com:/path/to/folder/
Will copy/update /path/to/file to /path/to/folder (the remote folder does not need to exist as long as it has a trailing slash)
rsync -avz /path/to/file firstname.lastname@example.org:/path/to/rfile
Will copy/update /path/to/file to /path/to/rfile
The trailing slash on the folder specifications avoids some of the confusing rsync behaviour so it generally recommended.
The options above mean:
- a - archive mode. This turns on lots of useful options which make the target file/folder as closely match as possible but does not turn on the deletion options needed for a true mirror.
- v - verbose mode. This tells you what it is doing, don't specify this if scripting.
- z - compress during transfer. On slow links with compressible date this can speed up transfers considerably, but it may be counter productive for files such as JPEGs, PNGs, video files, NIFTI-GZ files and other pre-compressed files.
Options for scp/sftp with a GUI applications
There are many options available for transferring files, we recommend either CyberDuck (macOS and Windows) or FileZilla (Linux).
CyberDuck is free software, although it will periodically nag you for a donation - alternatively you can buy a copy from the Apple App Store or Microsoft Windows store. If your computer is Orchard managed this should already have been installed as part of the software bundle included.
Once installed and running you can open a connection using the Open Connection globe icon. From the drop-down choose SFTP (Secure File Transfer Protocol) and then enter the host name into the box provided. Unless told otherwise by instructions for the host you are connecting to leave the port set to 22.
You may enter your username and password or leave these blank to be prompted when connecting. Please do not store your password in the Keychain (on macOS) - we recommend using a Password Manager instead.
Click Connect to connect to the remote server. You can then navigate the remote folder and drag/drop files to/from as required.
If this is the first time you have connected to a particular server you will be given the 'fingerprint' of the server - if this is a WIN server you can check this on our SSH page. Tick the box to always accept this fingerprint - should it change on the server, you will be notified and connections will not be attempted.
Multi-Factor Authentication (MFA)
If the remote server requires MFA then the default settings for CyberDuck will mean that for each transfer you request you will need to enter your credentials afresh. To avoid this, change CyberDuck's preferences to use a persistent connection. Open the Preferences window then in Transfers > General, change Transfer Files to Use browser connection.
Linux users should consider using FileZilla (other clients are available). It should be a simple install from your OS's software centre.
When you launch FileZilla you can connect simply by filling out the Quickconnect section at the top, hostname, username and password and port 22 (for SFTP). The first icon in the tool bar allows you to save connections for multiple servers.
Never agree to store your password in FileZilla's store - consider using a Password Manager instead if you are unable to remember your password.
The first time you connect to a server you will be given a prompt advising you that the host key for this server is unknown - for WIN servers you can check this on our SSH page. Tick the 'Always trust this host...' box if it is correct and then click OK to continue. If the key were to change you will be warned and connections will not be made. Other services should publicise their server fingerprints.
Once connected you will have a view of your local files on the left and the remote folder on the right, drag/drop between these sections to transfer files.
To avoid the need to continually enter your password and second factor when transferring multiple files during a session, use the Site Manager (first icon on the tool bar) to create a new site. Give it a memorable name, e.g. BMRC, choose the SFTP - SSH File Transfer Protocol option, fill in the host field then in the Transfer Settings tab tick the Limit number of simultaneous connections and ensure that the maximum number of connections is set to 1.
Leave the Logon Type set to Ask for password - consider using a Password Manager instead if you are unable to remember your password.
Click OK to save.
To open a saved connection, visit the Site Manager, select the site in the tree on the left and click the Connect button.