Cookies on this website

We use cookies to ensure that we give you the best experience on our website. If you click 'Accept all cookies' we'll assume that you are happy to receive all cookies and you won't see this message again. If you click 'Reject all non-essential cookies' only necessary cookies providing core functionality such as security, network management, and accessibility will be enabled. Click 'Find out more' for information on how to change your cookie settings.

Accept all cookies Reject all non-essential cookies Find out more

Submitting Jobs

How to run tasks on the cluster

Auto-submitting software

Some FSL commands and/or GUIs automatically queue themselves where appropriate, i.e., you do not need to use 'fsl_sub' to submit these programs. For a list of these FSL programs see: GE compatibility chart.
Please note that this list may not be exhaustive, so you may come across more commands which have been adapted to queue themselves.  If you do submit one of these tools to the queues then they will still run, but may not be able to make full use of the cluster resources (e.g. not be able to run multiple tasks in parallel).
Other commands run from the terminal command line will need to use the ''fsl_sub'' command, described below, to submit them to the queue.
Before submitting any tasks make sure you have loaded any shell modules you require - for example to use fsl_sub and the FSL tool set run the following:
module add fsl
module add fsl_sub
These lines can be added to your .bash_profile to ensure they take effect for every login session you have.

SUBMITTING JOBS WITH FSL_SUB

Typing fsl_sub before the rest of your command will send the job to the cluster. By default, this will submit your task to the short partition. fsl_sub can automatically choose a queue for you if you provide information about your job's requirements - we would strongly recommend that you provide at least an estimated maximum run time (--jobtime) to allow SLURM to efficiently schedule jobs

There are several ways to select a queue:
  1. Use the -R (--jobram) and -T (--jobtime) options to fsl_sub to specify the maximum memory and run-time requirements for your job (in GB and minutes of wall time*) respectively. fsl_sub will then select the most appropriate queue for you.
    GPU tasks can be requested using the --coprocessor options (see the Running GPU Tasks section).
  2. Specify a specific partition with the -q (--queue) option. For further information on the available queues and which to use when see the queues section.
Notes:
  • The command you want to run on the queues must be in your path - this does NOT include the current folder. If it isn't then you must specify the path to the command; commands/scripts in the current folder must be prefixed with './', e.g. ''./script''.
  • The BMRC cluster does not have a 'verylong' equivalent queue. See Long Running Tasks
  • Jobs submitted to the BMRC cluster do NOT inherit the 'environment' of your login shell, e.g. environment variables such as FSLDIR are not copied over to your job. Load software configuration (such as FSL) from shell modules or use the '--export' option to fsl_sub to copy the variables to your job (see Passing Envrionment Variables to Queued Jobs). 
  • Wall Time: Unlike the FMRIB cluster (which uses CPU time) the BMRC cluster measures job run-time in real time, often called wall time (as in the time on a clock on the wall).
    To assess the time necessary for your job to complete you can look at the run-times of similar previous jobs using the 'sacct' command (see Monitoring Tasks).

Example Usage

To queue a job which requires 10GiB of memory and runs for 2 hours use:
fsl_sub -T 120 -R 10 ./myjob
This will result in a job being put on the short partition.
If your software task automatically queues then you can also specify the memory you expect the task to require with the environment variable FSLSUB_MEMORY_REQUIRED, for example:
FSLSUB_MEMORY_REQUIRED=32G feat mydesign.feat
would submit a FEAT task informing fsl_sub that you expect to require 32GiB of memory. If units aren't specified then the integer is assumed to be in the units specified in the configuration file (default GiB).
The different partitions have different run-times and memory limits, when a task reaches these limits it will be terminated; also shorter queues take precedence over the longer ones. It is advantageous to provide the scheduler with as much information about your job's memory and time requirements.
The command you submit cannot run any graphical interface, as they will have no where to display the output. If you have a task that insists on producing some output contact the BMRC Team for advice.
If you want to run a non-interactive MATLAB task on the queues then see the MATLAB section. 

FSL_SUB OPTIONS

​To see a full list of the available options use:

​fsl_sub --help​

In addition to the list of options this will also display a list of partitions available for use with descriptions of allowed run times and memory availability. For details on how to use these options see the Advanced Usage section.

Long running tasks

Unlike the Jalapeno cluster, BMRC's cluster does not offer 'infinite' partitions (equivalent to verylong.q, bigmem.q and the cuda.q on the jalapeno cluster). You must break your task up into shorter components or regularly save state to allow restart and submit these parts (or resubmit the job continuing where it left off) using job holds to prevent tasks running before the previous one completes.​