Rsync

The rsync tool allows you to transfer files/folders between two locations in a resumable manner and optionally ensure that the two locations are identical. It uses file metadata to determine what needs to be transferred which means it is much slower to start transferring than a simple scp but for regular mirroring or large transfers, especially over intermittent network connections (such as WiFi) the efficiency gains over plain scp are significant.

Rsync is included with Linux and macOS, but currently the best way to use it with Windows is to install Windows Subsystem for Linux and use it from there.

The rsync command is very complex, so only the bare minimum will be covered here - see the 'man' entry or read online here https://download.samba.org/pub/rsync/rsync.1

Transferring a folder from your computer to a remote server can be achieved with

rsync -avz /path/to/folder/ username@remotehost.domain:/path/to/rfolder/

This will synchronise (but not delete on the remote folder if missing locally) /path/to/folder to /path/to/rfolder on remotehost.domain. username is optional if this matches at both ends.

Pulling from a remote host is possible by reversing the source and targets:

rsync -avz username@remotehost.domain:/path/to/rfolder/ /path/to/folder/

rsync -avz /path/to/file username@remotehost.domain:/path/to/folder/

Will copy/update /path/to/file to /path/to/folder (the remote folder does not need to exist as long as it has a trailing slash)

rsync -avz /path/to/file username@remotehost.domain:/path/to/rfile

Will copy/update /path/to/file to /path/to/rfile

The trailing slash on the folder specifications avoids some of the confusing rsync behaviour so it generally recommended.

The options above mean:

• a - archive mode. This turns on lots of useful options which make the target file/folder as closely match as possible but does not turn on the deletion options needed for a true mirror.
• v - verbose mode. This tells you what it is doing, don't specify this if scripting.
• z - compress during transfer. On slow links with compressible date this can speed up transfers considerably, but it may be counter productive for files such as JPEGs, PNGs, video files, NIFTI-GZ files and other pre-compressed files.