Skip to content

service.app

The API of the service: people and agents submit, follow and compare runs.

Every route of the API needs a bearer token of the tokens file (simrun.service.auth); only the files of the web app at /ui/ (setting web) answer without one. The identity of the token decides the priority of its runs. A submission is validated, its resources resolved and its tasks planned on the nodes of the backend, and the record (submitting) and the settings of every task written; POST /runs returns then, and the environment, the image and the workflow follow in the background (simrun.service.submission). The controller (simrun.service.controller) runs the submissions and follows the workflows in a background task.

Run it with uvicorn --factory simrun.service.app:serve --timeout-graceful-shutdown 5, which reads the ServiceSettings from the YAML file named by SIMRUN_CONFIG; the timeout ends the open event streams when the service stops.

ServiceSettings

Bases: BaseModel

The settings of the service.

Attributes:

Name Type Description
store Path

Root of the run store.
db Path

SQLite file of the resource observations and the index of the runs, on a local volume.
tokens Path

YAML file of the tokens, see simrun.service.auth.load_tokens.
owner str

The person who owns the cluster; their runs have the top priority.
cluster ClusterConfig

The cluster.
poll_s float

Seconds between two reconciliations of the controller when no change of the backend woke it before.
events_poll_s float

Seconds between two checks of an event stream.
events_max_s float

Seconds after which an event stream ends and the client reconnects.
default_image str | None

Image of a run whose model has no pyproject.toml.
simrun_wheel Path | None

The simrun wheel the images of the environments install.
python str

Python version of the images.
build_constraints Path | None

The pinned versions of the build backends of the environments, the file of simrun (simrun.environment.BUILD_CONSTRAINTS) by default.
backend str | None

module:callable returning the backend, for serve.
web Path | None

Directory of the built web app, served at /ui/ without a token; the app itself talks to the API with the token of the user. The one of the installation by default (simrun.local.web_dir), None serves none.
max_upload_bytes int

The largest body of an upload of a snapshot; a larger one is refused with 413.

SubmitRequest

Bases: BaseModel

A submission of a workflow.

Attributes:

Name Type Description
workflow Workflow

The workflow.
sources list[SourceProvenance]

The snapshots of the sources, the model first.
group str | None

Group of the run, e.g. a study.
split SplitMode | None

Split mode of every task, else the one of the task or the workflow.
reserve str | None

A node reserved for the run, else the one of the workflow.
local_commands Literal['direct', 'docker'] | None

How the local backend runs the command tasks: direct their args, docker in their images; None the choice of the backend. The Argo backend ignores it.

ValidationResult

Bases: BaseModel

The result of a validation: the errors of a workflow, none when valid.

TaskEstimate

Bases: BaseModel

The resources, the plan and the expected duration of a task.

Attributes:

Name Type Description
resources ResolvedResources

The resolved resources.
plan TaskPlan

The plan of the pods.
seconds float | None

The expected seconds of a split task: the median time of a unit in one process of past runs times the rounds of its units over the processes of its pods, None without past runs.

TaskProgress

Bases: BaseModel

The progress of a task: its ledger and the first and last lines of its progress file.

The first line is the first unit done, whose time is when the task began to deliver units.

RunItem

Bases: RunSummary

A run of the list of runs: its summary with its owner and workflow name.

Attributes:

Name Type Description
owner str

The owner of the run, of an agent the person it acts for.
workflow str

The name of the workflow.
progress float | None

The progress of an active run (submitting, queued, running) from 0 to 1, from the states of its tasks and their ledgers (run_progress); None for a run which ended.

PodUsage

Bases: BaseModel

A pod of a run on a server.

Attributes:

Name Type Description
run_id str | None

The run of the pod, None when the pod belongs to no known run.
task_id str

The task of the pod.
pod str

The name of the pod.
phase str

The phase of the pod.
cores float

The cores it requests.
memory_gb float

The GiB of memory it requests.

ServerUsage

Bases: BaseModel

A server of the cluster with the pods of runs on it.

Attributes:

Name Type Description
name str

The name of the server.
cores float

The cores the scheduler gives the pods of the runs: the allocatable ones less the requests of other pods, e.g. of DaemonSets.
memory_gb float

The GiB of memory the scheduler gives the pods of the runs.
gpus int

The GPUs of the server.
reserved_by str | None

The run which reserved the server, None if free.
pods list[PodUsage]

The pods of runs on the server, with their requests, which count against cores and memory_gb.

WorkerView

Bases: BaseModel

A worker of a task, from its reports, its pod and its claims.

Attributes:

Name Type Description
worker str

The worker id.
node str | None

The server of the attempt shown: of the live pod while it runs, None while that pod is not bound, else of the last report.
state Literal['running', 'done', 'failed']

running while a pod of an attempt without a report lives (a retry after an eviction, a preemption, a lost node or a kill for memory, as a worker reports also when it is stopped), else from the last report: failed when it was killed for memory or failed (a unit or its command failed), else done.
attempts int

The attempts of the worker: one more than the highest attempt of its reports and of its live pod.
processes int | None

The processes of its pool, None before a report.
units_done int

The distinct units its unit_done lines in the progress file of the task name, live and over all its attempts.
units_claimed int

The units it holds claimed now.
peak_memory_gb float | None

The peak memory of the last report.
memory_limit_gb float | None

The memory limit of the last report.
oom_killed bool

Whether the attempt of the last report was killed for memory.

OutputFile

Bases: BaseModel

A file of the output of a task, its path relative to the store.

HintView

Bases: BaseModel

The default resources of a task type.

TaskTypeView

Bases: BaseModel

A task type with the JSON Schema of its settings, for the editor.

of classmethod 

of(task_type)

The view of a task type.

CleanupRequest

Bases: BaseModel

Which ended runs a cleanup removes.

Attributes:

Name Type Description
older_than_days float

Runs which ended at least this many days ago.
group str | None

Only the runs of this group.
failed_only bool

Only failed runs.
dry_run bool

List what would be removed, remove nothing.

CleanupResult

Bases: BaseModel

What a cleanup or a deletion removed, paths relative to the store.

Usage

Bases: BaseModel

Bytes of the run store per run and per group ("" without a group).

CompareRequest

Bases: BaseModel

The runs to compare.

unknown_settings 

unknown_settings(model, settings)

The keys of settings which are no field of their model, sorted.

A field counts by its name and its aliases; a model which allows extra keys (extra="allow") has none unknown.

validate_workflow 

validate_workflow(workflow, types)

The errors of a workflow beyond its structure: types, settings and ports.

A setting which is no field of the settings model of its type is an error, so a misspelled one is not silently dropped. An invalid setting is one error per field, task '<id>': <loc>: <msg>, the location dotted.

Parameters:

Name Type Description Default
workflow Workflow

The workflow.

 required
types dict[str, TaskType]

The task types by name.

 required

Returns:

Type Description
list[str]

The errors, empty when the workflow is valid.

freed_bytes 

freed_bytes(paths)

The bytes removing paths frees: files of which no hard link stays.

disk_usage 

disk_usage(path)

The bytes of the files below path, every hard linked file once.

run_progress 

run_progress(tasks, counts)

The progress of a run from 0 to 1, as the web app counts it.

The mean over its tasks: a task which is over (done or skipped) counts whole, a task with a ledger the share of its units done, at most UNFINISHED_TASK_MAX (its merge and outputs are still to come), any other task nothing; so the progress is 1 only once every task is over.

Parameters:

Name Type Description Default
tasks dict[str, TaskState]

The states of the tasks.

 required
counts dict[str, LedgerCounts | None]

The ledger counts by task, None or missing without a ledger.

 required

Returns:

Type Description
float

The progress, 0 for a run without tasks.

create_app 

create_app(settings, backend, types=None)

Create the app of the service.

Parameters:

Name Type Description Default
settings ServiceSettings

The settings.

 required
backend Backend

The backend the workflows run on.

 required
types dict[str, TaskType] | None

The task types by name, the ones of the entry points by default.

 None

Returns:

Type Description
FastAPI

The app; its lifespan runs the controller every settings.poll_s seconds.

openapi_document 

openapi_document()

The OpenAPI document of the service, the input of the types of the frontend.

The app is created on a fake backend in a temporary directory; the routes do not depend on either, so the document is the same on every machine.

load_settings 

load_settings(path)

Read the settings of the service from a YAML file.

serve 

serve()

The app of the settings in the YAML file named by SIMRUN_CONFIG.

The backend is the backend of the settings, a module:callable called without arguments.

Raises:

Type Description
RuntimeError

SIMRUN_CONFIG or the backend is not set.