argo¶

Argo workflows from runs: one DAG, a scatter of workers per split task.

A split task of pods pods is three DAG tasks of one worker template running simrun-worker: <task>-w0, the coordinator (worker <task>-w0, SIMRUN_ROLE=base, base priority), which returns once the ledger of the task is complete; <task>-extra over withSequence: {start: 1, count: pods - 1} (workers <task>-w<item>, SIMRUN_ROLE=extra, extra priority; left out for one pod), which exit when nothing is pending. The coordinator merges the units itself once the ledger is complete (SIMRUN_MERGE=1, TaskPlan.coordinator_merges; only the coordinator acts on it), so extra workers still waiting for admission never hold up the merge. A plan recorded before that has a node <task>-merge running simrun-merge, which depends on <task>-w0 only. An unsplit task is one pod running its command. A task waits for the tasks it reads from, a split one for their merge (its <task>-w0 when that merges).

The extra workers continue on failure (continueOn: {failed: true, error: true}), so an extra worker which fails does not fail the run; the coordinator releases the claims of workers whose heartbeat is older than SIMRUN_STALE_S (grace_s + 60) and computes them. The workers carry the input parameters worker, priority, task and role. Extra workers may still be pending or running after the last merge of a run: the controller stops the whole workflow then and records the state of the run itself.

Every pod carries the Kueue label of the LocalQueue, the WorkloadPriorityClass label and the Kubernetes priorityClassName (the coordinator, the merge and an unsplit task the base priority, the extra workers the extra priority), the annotation which requires one node per pod (topology aware scheduling), requests equal to limits, the run directory of the store mounted read-write and the snapshots read-only, a scratch directory on the local disk (<run>/<task>/<worker> for a worker, <run>/<task>/merge for a merge, <run>/<task> for an unsplit task), and the environment of the worker harness (simrun.worker). The label simrun/group is the group made a valid label value (label_value), the annotation simrun/group the group as given. The pods run as the service account ClusterConfig.service_account, which the wait container of the executor needs to write its results. The pods of a run with a reserved node select the node by its label simrun/reserved=<run id> and tolerate its taint of the same key and value whatever its effect (the backend taints with NoExecute).

The Argo features this relies on (Argo Workflows v4.1.6):

• retryStrategy.expression with lastRetry.exitCode, a string ("-1" when the pod has no exit code), and retryPolicy: Always, which must hold together to retry (docs/retries.md, "Conditional retries"; the type in docs/variables.md, "RetryStrategy"). The expression compares strings, since "3" != 3 holds in expr.
• {{retries}} and lastRetry.* in a container template with a retry strategy, substituted per attempt, lastRetry.exitCode "0" in the first attempt (docs/variables.md, "Container/Script Templates"; executeTemplate in workflow/controller/operator.go), also in {{=...}} expressions, which are left for the attempt until lastRetry is known (variablesToCheck in util/template/expression_template.go). So podSpecPatch sets the memory and SIMRUN_MEMORY_PER_WORKER_GB of an attempt after an exit 137 to the retry values, and every other attempt keeps the first values.
• podSpecPatch, a strategic merge patch of the pod spec (containers and env merged by name), also for terminationGracePeriodSeconds, which a template has no field for; the emissary executor reads it after the patch (createWorkflowPod in workflow/controller/workflowpod.go).
• {{item}} in the arguments of an expanded DAG task (processItem in workflow/controller/operator.go), and input parameters substituted anywhere in a template, its metadata and priorityClassName included (SubstituteParams in workflow/common/util.go).
• continueOn of a task which no task depends on: the DAG ignores its failure (assessDAGPhase in workflow/controller/dag.go).

workflow_name(run_id)

label_value(text)

to_argo(record, plans, types, config)