Submitting Serial Jobs (Job Holds)
How to submit pipeline stages such that they wait for their predecessor to complete
If you have a multi-stage task to run, you can submit the jobs all at once, specifying that later stages must wait for the previous task to complete. This is achieved by providing the '-j' (or --jobhold) option with the job id of the task to wait for. For example:
jid=$(fsl_sub -q veryshort.q ./my_first_stage)
fsl_sub -q long.q -j $jid ./my_second_stage
Note the $() surrounding the first fsl_sub command, this captures the output of a command and stores the text in the variable 'jid'. This is then passed as the job id to wait for before running 'my_second_stage'.
Jobs set to wait for the completion of another task will appear in the queue with state 'hqw'.
It is also possible to submit array holds with the --array_hold command which takes the job id of the predecessor array task. This can only be used when both the first and subsequent job are both array tasks of the same size (same number of sub-tasks) and each sub-task in the second array depends only on the equivalent sub-task in the first array.