Skip to content

service.backend

The backend the service runs workflows on: Argo Workflows, or a fake in tests.

FailedAttempt

Bases: BaseModel

An attempt of a pod which failed.

Attributes:

Name Type Description
attempt int

The number of the attempt, from 0, as the pod has it in SIMRUN_ATTEMPT (Argo's {{retries}}) and in the name of its report.
exit_code int | None

The exit code of its main container, None without one (e.g. a pod deleted by an eviction of Kueue).
oom_killed bool

Whether the main container was killed by the out of memory killer (the reason OOMKilled); an exit 137 alone is no evidence of it.

PodLog

Bases: BaseModel

The log of the main container of one pod (one attempt) of a task.

Attributes:

Name Type Description
node str

The node of the pod in the workflow, e.g. squares-w0(1).
pod str

The name of the pod.
text str

The log.
done bool

Whether the pod has terminated, so its log is complete.

Placement

Bases: BaseModel

A pod of a run which has not terminated, with its node and requests.

Attributes:

Name Type Description
workflow str

The backend name of the run.
dag_task str

The DAG task of the pod: <task>-w0, <task>-extra, <task>-merge or <task>.
pod str

Name of the pod.
node str | None

The node the pod is bound to, None while it is not.
phase str

The phase of the pod.
cores float

The cores the pod requests.
memory_gb float

The GiB of memory the pod requests.
worker str | None

The worker id of the pod (SIMRUN_WORKER, the id of its ledger claims and report names), None for a pod without one, e.g. a merge.
attempt int | None

The attempt of the pod (SIMRUN_ATTEMPT, the number of its retries, the attempt of its report name), None when unknown.

BackendStatus

Bases: BaseModel

The state of a workflow in the backend.

The service, not the backend, decides the state of a run from it (see simrun.service.controller).

Attributes:

Name Type Description
phase str

Phase of the workflow, e.g. Pending, Running, Succeeded.
tasks dict[str, str]

Phase of every node of the DAG by its name: <task>-w0, <task>-extra(<i>:<item>) and <task>-merge of a split task, <task> of an unsplit one; a node not yet created is missing. A node whose pod was evicted and whose retry waits is Preempted (PREEMPTED_PHASE), one whose pod failed otherwise Retrying (RETRYING_PHASE).
failed dict[str, list[FailedAttempt]]

The attempts of pods which failed, in the order of their number: per worker id (<task>-w<i>), per task id of an unsplit task and per <task>-merge of a merge.

Backend

Bases: Protocol

What the service needs of a cluster.

runs_images instance-attribute 

runs_images

Whether the runs of the backend are containers: the service resolves the environment of a run and builds its image. False: it keeps the image of the record.

extra_workers instance-attribute 

extra_workers

Whether a split task may run as several pods, its coordinator and extra workers. False: the service plans one pod per split task, e.g. on one machine.

submit 

submit(record, plans, types, cluster)

Submit a run and return the name of its workflow.

Parameters:

Name Type Description Default
record RunRecord

The run, with its image set when the backend runs_images.

 required
plans dict[str, TaskPlan]

The plan of every task by id.

 required
types dict[str, TaskType]

The task types by name.

 required
cluster ClusterConfig

The configuration of the cluster.

 required

Returns:

Type Description
str

The name of the workflow of the run (argo.workflow_name).

Raises:

Type Description
ValueError

The run cannot be submitted (e.g. a task has no image); the message is the one of the failed run.

subscribe 

subscribe(callback)

Register a callback the backend calls when the status of a run changed.

The service registers the wake of its controller, so a backend which knows of a change makes it reconcile at once.

status 

status(name)

The status of a workflow.

Raises:

Type Description
KeyError

The workflow does not exist (any more).

cancel 

cancel(name)

Terminate a workflow and its pods.

stop 

stop(name)

Stop a workflow whose run is decided: pods still pending are removed.

logs 

logs(name, task_id, tail_lines=None)

The logs of the pods of a task of a workflow, oldest first.

Every attempt whose pod still exists; tail_lines limits each log to its last lines.

nodes 

nodes()

The nodes of the cluster.

placements 

placements()

The pods of the runs which have not terminated, with nodes and requests.

reserve 

reserve(node, run_id)

Reserve a node for a run.

The node gets the taint simrun/reserved=<run id>:NoExecute and the label simrun/reserved=<run id>.

unreserve 

unreserve(node, run_id)

Remove the taint and the label of the reservation of a run.

A node reserved by another run (or by none) is left as it is, so a late release never removes the reservation of a later run.

Raises:

Type Description
KeyError

The node does not exist.

build_image 

build_image(context, ref)

Build the image of a build context and push it as ref.

has_image 

has_image(ref)

Whether the registry has the image ref.