cwl¶

CWL v1.2 workflows from runs, executed with cwltool (or Toil on Slurm).

The same run as simrun.argo, with the same commands and environment, as one CWL Workflow. The steps work on the run store by absolute path, the workflow input store (a directory every step sees: a local directory, or the NFS mount on Slurm), and communicate through it: the ledger and the outputs live in the store. The only CWL output of a step is its standard output, which exists to order the steps.

A split task is three steps of command line tools: <task>-w0 (worker <task>-w0, SIMRUN_ROLE=base), the coordinator, which returns once the ledger is complete; <task>-extra, scattering over the worker indices 1..pods - 1 (workers <task>-w<index>, SIMRUN_ROLE=extra; left out for one pod), which exit when nothing is pending; and <task>-merge after <task>-w0 only. With --parallel the coordinator and the extra workers run side by side; without it the steps run one after the other and the run still ends, since either one worker computes everything or the next finds nothing pending. A task waits for the merge of the tasks it reads from. A failed extra worker does not fail the workflow (successCodes of the step), as continueOn in Argo; the coordinator releases its claims and computes them, and a deterministic unit failure fails the workflow through <task>-w0.

The export targets a shared file system without containers (cwltool --no-container, Toil on Slurm with the store on NFS): store is a path, not a CWL Directory, and the image is only a DockerRequirement hint. Running the steps in containers is deferred to the Slurm plan.

The scratch directory of a worker is <scratch>/<run>/<task>/<worker> below the workflow input scratch (<run>/<task>/merge for a merge, <run>/<task> for an unsplit task). The image is a DockerRequirement hint, so --no-container runs the commands of the environment.

to_cwl(
    record,
    plans,
    types,
    *,
    scratch=DEFAULT_SCRATCH,
    grace_s=default,
)

export_cwl(record, plans, types, path, **options)