# simrun > Runner of simulation workflows on a k3s cluster with Argo Workflows and Kueue, for people and agents The complete documentation from https://matthiaskoenig.github.io/simrun, one section per page. --- # simrun: simulation workflows on a k3s cluster [![CI-CD](https://github.com/matthiaskoenig/simrun/actions/workflows/ci-cd.yml/badge.svg)](https://github.com/matthiaskoenig/simrun/actions/workflows/ci-cd.yml) [![Documentation](https://img.shields.io/badge/docs-simrun-3f51b5.svg)](https://matthiaskoenig.github.io/simrun) [![Version](https://img.shields.io/pypi/v/simrun.svg)](https://pypi.org/project/simrun/) [![Python Versions](https://img.shields.io/pypi/pyversions/simrun.svg)](https://pypi.org/project/simrun/) [![MIT License](https://img.shields.io/pypi/l/simrun.svg)](https://opensource.org/licenses/MIT) [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.24035618.svg)](https://doi.org/10.5281/zenodo.24035618) `simrun` runs simulation workflows on a small cluster of servers with [k3s](https://k3s.io), [Argo Workflows](https://argoproj.github.io/workflows/) and [Kueue](https://kueue.sigs.k8s.io). A workflow is a YAML file of tasks which read the outputs of other tasks. People and agents submit it together with the working trees of their model and their libraries, uncommitted changes included, and get a run id at once; the run is queued, placed, executed and recorded without the submitter staying connected, and its results and provenance stay in a shared run store. The source code is on [GitHub](https://github.com/matthiaskoenig/simrun). It is made for a research group which runs many analyses of models at once on its own servers, e.g. many parameter fits of individual models, each optimized by another agent in its own git worktree. A task which splits into units, such as the repeats of a fit or the points of a scan, spreads over the idle servers to finish as fast as possible and gives servers back when runs of a higher priority need them; the memory a task needs is learned from its earlier runs. ## A first run The package ships a demo task type, `demo.squares`, which squares the numbers `0..n-1`, one unit per number, and sums them. A workflow of one task of it, `examples/squares.yaml` of the repository: ```yaml title="squares.yaml" name: squares tasks: - id: squares type: demo.squares settings: n: 1000 sleep_s: 0.1 ``` The command line reads the address of the service, a token and the local mount of the run store from the environment, submits the workflow and waits for its end: ```bash export SIMRUN_URL=http://simrun.cluster.internal # the service export SIMRUN_TOKEN=... # a token of the tokens file export SIMRUN_STORE=/mnt/cluster # the run store as mounted here simrun estimate squares.yaml # the plan and the expected duration of every task simrun submit squares.yaml # prints the run id at once simrun wait # exit 0 succeeded, 1 failed or cancelled, 2 timeout simrun status # the tasks and their progress ``` The task runs as up to eight pods on as many servers, which claim the units from a ledger in the run store; its merge writes the sum into `runs//tasks/squares/out/summary.json` of the store, which `simrun compare` puts side by side for several runs. A model of your own is submitted with its sources, e.g. `simrun submit fit.yaml --source model=~/git/pkdb_models --source sbmlsim=~/git/sbmlsim`, see [Workflows](workflows.md). ## Features - [Concepts](concepts.md): runs, tasks and ports, the split of a task into units, priorities and preemption, learned resources, snapshots and environments. - [Workflows](workflows.md): the YAML format of a workflow, with a complete example. - [Command line](command-line.md): submit, follow, compare, cancel and clean up runs. - [Agents and MCP](mcp.md): the MCP server with which an agent submits and follows its runs. - [Plugins](plugins.md): task types as entry points of a python package, e.g. the scans and fits of [sbmlsim](https://github.com/matthiaskoenig/sbmlsim). - [Service](service.md): the HTTP API, its settings and the environment variables. - [Cluster deployment](deployment.md): k3s with Kueue, Argo Workflows, an NFS run store, the registry Zot and BuildKit. - [Testing](testing.md): the unit tests, the tests on a k3d cluster and a local test system on it. ## Questions and issues If you have any questions or issues please [open an issue](https://github.com/matthiaskoenig/simrun/issues). Contributions are welcome, see [Development](development.md). ## How to cite If you use `simrun` please cite the archived software on Zenodo: > König, M. *simrun: a runner of simulation workflows on a k3s cluster*. Zenodo. [https://doi.org/10.5281/zenodo.24035618](https://doi.org/10.5281/zenodo.24035618) This DOI stands for every version of `simrun`; the DOI of a single version is on its Zenodo page. The metadata of the citation are in [CITATION.cff](https://github.com/matthiaskoenig/simrun/blob/develop/CITATION.cff). ## License `simrun` is licensed under the [MIT License](https://github.com/matthiaskoenig/simrun/blob/develop/LICENSE). ## Funding Matthias König is supported by the Federal Ministry of Education and Research (BMBF, Germany) within the research network Systems Medicine of the Liver (**LiSyM**, grant number 031L0054) and by the German Research Foundation (DFG) within the Research Unit Programme FOR 5151 [QuaLiPerF](https://qualiperf.de) (Quantifying Liver Perfusion-Function Relationship in Complex Resection - A Systems Medicine Approach) by grant number 436883643 and by grant number 465194077 (Priority Programme SPP 2311, Subproject SimLivA). Matthias König has received funds from the EOSCsecretariat.eu which has received funding from the European Union's Horizon Programme call H2020-INFRAEOSC-05-2018-2019, grant Agreement number 831644. --- # Installation `simrun` needs python 3.14 or newer and runs on Linux. The same package is the client (the command line `simrun` and the MCP server `simrun-mcp`), the service and the harness of the pods (`simrun-worker`, `simrun-merge`, `simrun-exec`). Install it from [PyPI](https://pypi.org/project/simrun/) with pip: ```bash pip install simrun ``` ## With uv With [uv](https://docs.astral.sh/uv/), install the command line as a tool: ```bash uv tool install simrun ``` add `simrun` to a project, e.g. the one of a plugin, which locks it together with the rest of the environment: ```bash uv add simrun ``` or run it from a checkout of the repository, which is how the MCP server of an agent is usually registered (see [Agents and MCP](mcp.md)): ```bash git clone https://github.com/matthiaskoenig/simrun.git uv run --project simrun simrun --help ``` ## Check the installation ```bash simrun --help python -c "import simrun; print(simrun.__version__)" ``` The command line needs a running service, see [Command line](command-line.md), or runs everything on this machine with `--local`, see [Local mode](local.md); the service and the cluster are set up as described in [Cluster deployment](deployment.md). ## Optional packages | feature | install | | --- | --- | | the CWL export of a run (`simrun.cwl`), validated with cwl-utils and run with cwltool, locally or on Slurm | `pip install "simrun[cwl]"` | | the tools of development: tests, lint, types, documentation, release | `pip install "simrun[dev]"`, see [Development](development.md) | --- # Concepts This page explains how `simrun` turns a submitted workflow into pods on the cluster and what it records. The components are k3s on every server, Argo Workflows, which runs a run as a workflow of pods, Kueue, which admits the pods by cores, memory and GPUs and preempts lower priorities, a run store on NFS, which the service, the pods and the development server of the agents share, the registry Zot and BuildKit, which builds the images of the environments in the cluster. ```text people (CLI) agents (MCP) dev server (working trees, /mnt/cluster) \ | | +-------> simrun service (FastAPI) <---------------+ snapshots of the working trees | | learned resources | | submit, follow | v k3s: Argo Workflows (execution, retries) + Kueue (admission, priorities, preemption) | servers: pods of the tasks, image from the registry, snapshots and run store on NFS, scratch on the local disk ``` ## Runs A run is a submitted workflow. `POST /runs` validates the workflow, resolves the resources of its tasks, plans their pods on the nodes, writes the record of the run in the state `submitting` and returns its id at once; an invalid workflow, a task which fits on no node or a reserved node which is taken is refused right away. The environment of the run is resolved, its image built and its Argo workflow submitted in the background; the run is `queued` then, `running` once a pod runs, and ends `succeeded`, `failed` or `cancelled`. A run lives in `runs//` of the store: | path | content | | --- | --- | | `run.json` | the record: the workflow, the identity of the submitter, the plans, the provenance of the sources, the image, the states of the tasks and the out of memory kills | | `environment.lock` | the lock of the environment of the run | | `tasks//settings.json` | the settings of the task as submitted | | `tasks//provenance.json` | the provenance of the sources of the run, `SIMRUN_PROVENANCE` in the pods of the task | | `tasks//out/` | the outputs: a directory per output port and `summary.json`, the summary of the task, which `compare` reads | | `tasks//units/` | the ledger of a split task | | `tasks//workers/` | the reports of the pods: memory, units and their seconds | | `tasks//logs/` | the logs of the pods, kept once the task is final | | `tasks//progress.jsonl` | the progress of the task, one line per completed unit | The store is the source of truth. The service keeps an index of the runs in SQLite beside it (id, owner, agent, group, workflow, state, times, pending release and cancellation, snapshots and summary), rebuilt from the records at its start, so a restarted service continues where the one before stopped. The state of a run is the one of the service, not the one of Argo: a split task is done when its merge succeeded (its coordinator, which merges, succeeded and its output has the marker of the merge, which the service awaits up to two minutes for the attribute cache of NFS), an unsplit task when its pod succeeded; when every task is done the run succeeded, when a task failed the run failed, and in both cases the service stops the workflow. A run belongs to a group, e.g. a study or the subagents of one main agent, which lists and compares the runs of the group (`simrun list --group`, `simrun compare`). ## Tasks and ports A task has a type, settings and inputs. The type comes from a [plugin](plugins.md) and declares the settings as a pydantic model and its input and output ports. An output port `` of a task is the directory `tasks//out/`; a downstream task reads it as the input `.`. The tasks run in the order of their inputs, a cycle is refused. A setting which the settings model of its type does not have is an error of the validation and of the submission, so a misspelled setting never falls back to its default silently. ## Splitting into units: the ledger A task type with `count_units`, `compute_unit` and `merge` is split. The service counts the units of the task from its settings (e.g. the repeats of a fit or the points of a scan) when it plans the run. The units are claimed from a ledger, a directory `units/` of the task with `pending/`, `claimed/@` and `done/`: every transition is one `os.rename`, atomic on a local file system and on NFS v4.1 or newer, so a unit is held by at most one worker. A worker claims a unit, computes it in a process of its local pool and marks it done; it touches `heartbeats/`, so the claims of a worker which died without releasing them are found by the age of its heartbeat. A unit may be computed more than once: by a worker which is evicted before it completed the unit, or by a restarted pod whose earlier incarnation still runs. So `compute_unit` writes outputs which depend on the settings and the index only, each written atomically. The merge combines the units once all of them are done; the result does not depend on the number of pods the task ran on. ## Coordinator and extra workers A split task of `pods` pods is three nodes of its Argo DAG: - `-w0`, the coordinator, a worker which claims units like the others and returns only once the ledger is complete and then merges the units. It releases the claims of workers whose heartbeat is older than the grace period plus a minute and computes them itself. - `-extra`, the extra workers `-w1` to `-w`, which exit when nothing is pending. An extra worker which fails does not fail the run. The coordinator merges the units itself once the ledger is complete (`SIMRUN_MERGE=1`), so extra workers still waiting for admission never hold up the merge. A run recorded before that has a node `-merge` running `simrun-merge` after the coordinator only. An unsplit task is one pod which runs the command of its type through `simrun-exec`. ## Split modes `fastest` (the default) splits a task into as many pods as there are slots on the nodes, at most eight (`MAX_PIECES`) and no pod without a unit for each of its processes; each pod takes the processes its node allows by cores and memory. Once the seconds of a unit are learned, a task gets a further pod only while every pod still has at least `pod_start_s` (10) of work: a pod which takes longer to start than its share of the work would find the units taken by the others and only hold cores. `single-server` places one pod on one server. The mode is set for the workflow, for a task or for the submission (`--single-server`). Extra pods are not limited by the idle capacity: Kueue queues them until a node is free. ## Priorities and preemption The identity of the token of a submission decides the priority of its run: | priority | who | base | extra | | --- | --- | --- | --- | | `owner` | the owner of the cluster and the owner's agents | 1000 | 900 | | `standard` | the other people | 500 | 400 | | `agent` | the agents of the other people | 200 | 100 | Every priority is a Kueue `WorkloadPriorityClass` and a Kubernetes `PriorityClass` of the same name and value (`simrun--base`, `simrun--extra`). The coordinator (with its merge) and an unsplit task have the base priority of the run, the extra workers the extra priority. Kueue admits the pods and preempts workloads of a lower priority; the scheduler itself never preempts. ## Take-back by preemption Since the extra workers have a lower priority than the base pods of their own level, a new run takes back the extra workers of the running runs of the same level: the pods of the new run preempt them, and every running run keeps its coordinator. A preempted worker stops on SIGTERM within the grace period, releases its claims and writes its report; Argo retries the pod, which waits for admission again. The units it gave back are computed by the coordinator or by another worker, so splitting is elastic and the results do not change. Workloads of equal priority never preempt each other, so the extra workers of two runs of one priority wait for each other's end. ## Learned resources A pod requests `base + per_worker x processes` of memory, with the limit equal to the request. Every worker (and every unsplit plugin task) writes a report with the largest sample of the anonymous memory of its cgroup, sampled every second, and the seconds of the units it completed. A request is resolved from the last 20 observations of its key (the task type, the model and the value of the `cost_setting` of the type): the 95th percentile of the measured memory per process times 1.1, and at least 1.5 times the memory per process of a pod killed for memory, until a later run of the key succeeded. Without observations the resource hint of the type is used. `simrun estimate` gives a split task the median seconds of a unit times the rounds of its units over the processes of its pods. A pod killed for memory (the evidence of the out of memory killer: the reason `OOMKilled`, or a report counting kills in its cgroup) is retried once with 1.5 times the memory per process; a second kill fails the run with a message which names the retry memory. Exit codes of a command: 0 done, 137 (or killed with SIGKILL) retried with the retry memory, 143 stopped on SIGTERM and retried, any other a failure, not retried. ## Snapshots and environments A source is a git working tree, the model first. The client copies the files git knows of it (tracked and untracked, not ignored, uncommitted changes included) into `snapshots//` of the store and sends its provenance (branch, commit, changed files); a client without the store mounted sends the list of the files with the hashes of their contents to the service, then a tar of the files whose content the store lacks, and the service writes the same snapshot. Files whose content the last snapshot of the same name has are hard links to it (`.files.json` beside it maps its paths to the hashes), so a snapshot of an unchanged tree costs nearly nothing and its upload sends no file. A snapshot is named by its content and its files are read-only. The service never reads the disk of the client. The image of a run whose model has a `pyproject.toml` holds the third party dependencies of the model and of its snapshotted libraries, locked with `uv lock` against the snapshots, and their build backends at the versions of the constraints file of the service, so a new release of a backend changes no lock and rebuilds no image. The code comes from the snapshots, which a pod mounts read-only and installs editable at its start without network access. A task imports from the root of the model first, as `python -m` in its checkout does, so a project at the root of a checkout which no package installs (the `examples` of sbmlsim) is found; the client plans its tasks with the same path. The image installs the Debian packages of `system_packages` on the base image first, the shared libraries which wheels link but do not bundle (`libexpat1`, which python-libsbml needs), and `git` when a dependency is locked from a git repository (e.g. a branch of pkpdsim). Its packages are installed with their bytecode compiled, so a pod does not compile what it imports (about 15 s for the scientific stack), and the downloads of uv stay in a cache of BuildKit outside of the image; simrun is its last layer, so a new simrun rebuilds only that layer (seconds). It is named by the hash of the lock, the python version, the simrun version and its Dockerfile (the base image and the system packages) and built with BuildKit only when the registry does not have it, so a change of the code never rebuilds an image and a change of a dependency does. A run whose model has no `pyproject.toml` runs in the `default_image` of the service. ## Reservations A node can be reserved for one run, e.g. for a benchmark (`reserve:` of the workflow or `simrun submit --reserve `). The service taints the node with `simrun/reserved=:NoExecute`, which evicts the pods of other runs, and labels it `simrun/reserved=`, which the pods of the run select and tolerate. The planner leaves a reserved node out for every other run, and the reservation is released when the run ends. --- # Workflows A workflow is a YAML file of tasks, usually kept in the repository of a project, such as `pkdb_models/caffeine_pbpk/workflows/fit.yaml`. People and agents submit the same files; a single fit is a workflow of one task. The model is `simrun.workflow.Workflow`. ## A complete example ```yaml title="fit.yaml" name: hctz-fit split: fastest # or single-server: the pieces of every task on one server reserve: null # a node reserved for the run tasks: - id: scan # [a-z][a-z0-9-]* type: sbmlsim.scan # a task type of a plugin settings: # validated by the settings model of the type n: 1000 resources: # optional, replaces the learned resources memory_per_worker_gb: 4.0 scratch_gb: 10 gpus: 0 - id: report type: sbmlsim.report inputs: results: scan.results # the output port `results` of the task `scan` stage: [results] # copied to the local scratch before the task starts split: single-server # the split mode of this task - id: plot type: command # the built-in type: a command in an image of its own settings: image: registry.cluster.internal:5000/tools:1 args: [plot, --all] cores: 2 memory_gb: 4.0 ``` Submit it with the sources of the model and of the libraries it uses, the model first: ```bash simrun submit fit.yaml --source model=~/git/pkdb_models --source sbmlsim=~/git/sbmlsim --group hctz ``` ## The workflow | field | meaning | | --- | --- | | `name` | the name of the workflow | | `tasks` | the tasks, see below | | `split` | the split mode of every task, `fastest` (default) or `single-server`, see [Split modes](concepts.md#split-modes) | | `reserve` | a node reserved for the run, see [Reservations](concepts.md#reservations) | The tasks run in the order of their inputs; a cycle, a duplicate id and an input which names an unknown task are refused. ## A task | field | meaning | | --- | --- | | `id` | the identifier of the task within the workflow, `[a-z][a-z0-9-]*` | | `type` | the name of a task type of a [plugin](plugins.md), or `command` | | `settings` | the settings handed to the task type, validated by its settings model; an unknown setting is an error | | `inputs` | input port to the output it reads, `.` | | `split` | the split mode of this task, the one of the workflow by default | | `resources` | resources which replace the learned or default ones: `workers`, `memory_per_worker_gb`, `scratch_gb`, `gpus` | | `stage` | input ports copied to the local scratch of the node before the task starts | The service writes the settings as submitted to `tasks//settings.json`, and the pods validate them with the plugin of the snapshot. ## The type `command` The built-in type `command` runs a command in an image of its own, without the harness of simrun: it does not split, it cannot stage its inputs, and its resources are its settings: | setting | meaning | | --- | --- | | `image` | the image, from the registry of the cluster | | `args` | the argv of the command; as in the command of a [plugin](plugins.md), `{out}` is the output directory of the task, `{settings}` the path of its settings and a part with `{input}` is repeated once per input port, sorted by port, as `=`, e.g. `--input={input}` | | `cores` | the cores of the pod | | `memory_gb` | the memory of the pod | | `scratch_gb` | the local scratch of the pod | | `gpus` | the GPUs of the pod | ## Validation and estimate `POST /validate` checks a workflow against the task types the service knows without submitting it, and `simrun estimate` shows the plan of every task (pods, processes and memory per pod) and the expected duration from the learned resources: ```bash simrun estimate fit.yaml --source model=~/git/pkdb_models ``` --- # Command line `simrun` is the command line of people and scripts: it submits, follows, compares, cancels and cleans up runs, and `simrun edit` opens a workflow file in the [Editor](web-app.md#the-editor) of the web app. On a cluster it runs on the machine of the working trees and reads three environment variables, which the [MCP server](mcp.md) reads alike: | variable | meaning | | --- | --- | | `SIMRUN_URL` | the address of the service, e.g. `http://simrun.cluster.internal` | | `SIMRUN_TOKEN` | a token of the tokens file of the service, which names the person or the agent | | `SIMRUN_STORE` | optional: the run store as it is mounted on this machine, e.g. `/mnt/cluster`, where the client writes the snapshots itself; without it the client uploads them to the service | ```bash export SIMRUN_URL=http://simrun.cluster.internal export SIMRUN_TOKEN=... export SIMRUN_STORE=/mnt/cluster ``` Every command takes `--json` (except `logs`, which prints the logs as text, and `edit`, which runs until it is stopped) and prints machine-readable JSON instead of tables. A refusal of the service is printed to stderr with exit code 1. With `--local` on a command, no variable is needed: it talks to the local service of the current directory, see [Local mode](local.md). ## Commands | command | what it does | | --- | --- | | `simrun submit WORKFLOW` | snapshot the sources and submit a workflow; prints the run in the state `submitting`, or with `--wait` follows it to its end | | `simrun wait RUN` | wait for a run to end; exit 0 if it succeeded, 1 if it failed or was cancelled, 2 on timeout | | `simrun status RUN` | the state of a run and its tasks with their progress | | `simrun list` | the runs, filtered by group, agent, state and owner | | `simrun logs RUN TASK` | the logs of the pods of a task | | `simrun outputs RUN TASK` | list the output files of a task, or download them with `--to DIR` | | `simrun edit WORKFLOW` | serve the Editor of the web app for a workflow file: saves go into the file, runs use the working trees | | `simrun estimate WORKFLOW` | the plan and the expected duration of every task of a submission | | `simrun compare RUN RUN...` | the provenance and the summaries of the tasks of runs side by side | | `simrun cancel RUN` | cancel a run | | `simrun cleanup` | remove ended runs and the snapshots nothing refers to | | `simrun usage` | the bytes of the store per group and run | | `simrun serve` | run the service in the foreground: `--local` the local service of the current directory, else the service of `SIMRUN_CONFIG` | ### submit ```bash simrun submit fit.yaml --source model=~/git/pkdb_models --source sbmlsim=~/git/sbmlsim --group hctz ``` | option | meaning | | --- | --- | | `--source NAME=PATH` | a source, the path of a git working tree; repeated, the model first | | `--group GROUP` | the group of the run, e.g. a study | | `--single-server` | place the pieces of every task on one server | | `--reserve NODE` | reserve a node for the run | | `--local` | use the local service of the current directory, see [Local mode](local.md) | | `--docker` | with `--local`: run a `command` task in its image with docker instead of directly | | `--snapshot` | with `--local`: copy and install the sources as on the cluster instead of recording their provenance only | | `--wait` / `--no-wait` | follow the run to its end (a live line per task on a terminal, plain lines otherwise) and exit like `wait`; the default with `--local`, else `--no-wait` | | `--timeout SECONDS` | with `--wait`: give up after this long, exit 2; refused without waiting | The client copies every working tree with its uncommitted changes into `snapshots//` of the store, or without `SIMRUN_STORE` uploads the files the store lacks to the service which writes the snapshot, and sends its provenance, see [Snapshots and environments](concepts.md#snapshots-and-environments). ### edit ```bash simrun edit fit.yaml --source model=~/git/pkdb_models --source sbmlsim=~/git/sbmlsim ``` | option | meaning | | --- | --- | | `--source NAME=PATH` | a source, as for `submit`; snapshotted when the session opens and at every run | | `--group GROUP` | the group of the runs started from the Editor which name none | | `--local` | use the local service of the current directory | | `--snapshot` | with `--local`: copy and install the sources at every run | The command opens an edit session in the service, prints the link of the [Editor](web-app.md#the-editor) and runs until Ctrl-C, which closes the session and exits 0 (a second Ctrl-C exits without waiting for the close): it writes the saves of the browser into the file, runs the workflow of the browser as `submit` does, and sends a change of the file or of the working trees to the browser. A file which does not exist is a new workflow. ### wait ```bash simrun wait --timeout 3600 ``` `--timeout` is in seconds, without it the command waits for the end of the run. The service answers a long poll which the client repeats, so no connection is held for hours. ### outputs ```bash simrun outputs # the files and their sizes simrun outputs --to results # download them into results/ ``` The files come through the API, so no mount of the store is needed. With `--json` the list is `[{"path", "bytes"}]`, or with `--to` the paths written. ### status, list and logs ```bash simrun status simrun list --owner mk --state running simrun logs --tail 0 ``` `list` takes `--group`, `--agent`, `--state` (`submitting`, `queued`, `running`, `succeeded`, `failed`, `cancelled`) and `--owner`; the runs of an owner include the ones of their agents. `logs` prints the last `--tail` lines (300 by default, 0 for all) of every pod of the task: the logs kept in the store once the task is final, else the ones of its live pods. ### estimate and compare ```bash simrun estimate fit.yaml --source model=~/git/pkdb_models simrun compare ``` `estimate` takes the options of `submit` but `--group`. `compare` reads the provenance of the runs and the `summary.json` of their tasks. ### cancel, cleanup and usage ```bash simrun cancel simrun cleanup --older-than 30 --dry-run simrun usage ``` `cleanup` removes the runs which ended more than `--older-than` days ago (30 by default), only the ones of `--group` or only `--failed` runs, and the snapshots no remaining run refers to; `--dry-run` lists what it would remove. ### serve ```bash simrun serve --local # the local service of the current directory simrun serve --local --idle-exit 1800 # exits after 30 min without a request and without a live run simrun serve --local --stop # stops the local service of the current directory, cancelling its runs simrun serve --port 8000 # the service of SIMRUN_CONFIG, as uvicorn --factory simrun.service.app:serve ``` `serve --local` runs the service of the current directory with the local backend, which runs the tasks as processes on this machine: its store is `.simrun/store/`, its scratch `.simrun/scratch/`, its index `.simrun/simrun.sqlite` and its tokens `.simrun/tokens.yaml` (with a `.gitignore` of everything, so none of it lands in git), it listens on `127.0.0.1` only, on a free port unless `--port` names one, and serves the [web app](web-app.md) at `/ui/`. It prints `simrun at http://127.0.0.1:/ui/ (token in .simrun/service.json)`; `.simrun/service.json` (mode 0600) holds the pid, the port, the token, the URL, a random instance of this start and the environment of the service (its python, its simrun version, a hash of the task types it found, whether it runs in the background) while it runs; a client sends the token only to a server which answers that instance at `/local/instance`. Ctrl-C, SIGTERM or SIGHUP (the terminal closed) cancels the runs and waits for their processes, printing what it waits for, so no process of a run outlives a graceful stop; a service which starts after one which was killed (SIGKILL, out of memory) kills the processes that one left running. `--stop` stops the service of the current directory the same way and waits for its end. One service runs per directory; a second one is refused with exit code 3. A command which needs the local service starts it in the background when none runs (`python -m simrun serve --local --idle-exit 1800`, its output in `.simrun/service.log`); a service in the background which runs with another environment than the command is restarted when it has no live run and no open request, else the command warns and uses it. --- # Local mode The local mode runs a workflow on this machine, without a cluster, a registry or a mounted store. It is the same service, planner, harness, records and [web app](web-app.md) as on the cluster, with a backend which runs the tasks as processes. The time of a run is the time of its processes plus a fraction of a second per task. ```bash pip install simrun simrun submit fit.yaml --source model=. --local simrun outputs --to results ``` ## The switch `--local` on a command makes it talk to the local service of the current directory instead of `SIMRUN_URL`; no environment variable is needed. It is accepted by `submit`, `estimate`, `wait`, `status`, `list`, `logs`, `outputs`, `cancel`, `compare`, `cleanup`, `usage` and `edit`. Without `--local` the same command runs on the cluster, see [Command line](command-line.md). `submit --local` waits for the run by default and prints a live line per task on a terminal and plain lines otherwise; it exits like `wait` (0 if the run succeeded, 1 if it failed or was cancelled, 2 on `--timeout`). `--no-wait` returns at once with the run, as on the cluster. ## The service of a project The local service keeps everything in `.simrun/` of the current directory, like `.venv`: `store/` (`runs/`, `snapshots/`), `scratch/`, `envs/` (the environments of the runs with `--snapshot`), the index `simrun.sqlite`, the generated tokens `tokens.yaml`, `service.json`, `service.log` and a `.gitignore` of everything, so none of it lands in git. It listens on `127.0.0.1` on a free port only. A `--local` command starts the service in the background when none runs, waits for its `service.json` and uses it. The service exits after 30 minutes without a request and without a live run, and a stopped service is started again by the next command. The service runs the python, the simrun and the task types of the command which started it; `service.json` records them, and a command whose environment differs (another venv, a new or upgraded plugin, another simrun) restarts a service in the background which has no live run and no open request, else it warns and uses the running one. `simrun serve --local --stop` stops the service of the directory (its runs are cancelled) and waits for its end. `simrun serve --local` runs the service in the foreground instead and prints its address; Ctrl-C, SIGTERM or SIGHUP cancels the runs and waits for their processes (up to 60 s, printing what it waits for), so no process of a run outlives a graceful stop. A service killed without its cleanup (SIGKILL, the out of memory killer) leaves the processes of its runs running; the next service of the directory kills them when it starts. One service runs per directory. See [serve](command-line.md#serve) for the file `service.json` and the check of the instance before the token is sent. ## The web app The service serves the web app at `/ui/`: the Executions with live updates, and the Editor (`simrun edit --local`). The wheel ships the built app (`simrun/web`); a checkout serves `frontend/dist` once it is built, see [Development](development.md). The app asks for a token: the one in `.simrun/service.json`. The server strip shows this machine as the one server. ![The web app of the local service: a run of three tasks on this machine](images/executions-running.png) ## How tasks run The `LocalBackend` of the service runs the pods of a run as processes of this machine: - a split task is one coordinator process (`simrun-worker`) with as many processes as the machine has cores, which computes the units and merges them itself; there are no extra workers, and the plan (also of `estimate --local`) has one pod - a task of a plugin type runs through `simrun-exec`, without an image: in the environment of the service, activated (its python first on `PATH`), and with the working tree of the model first on `PYTHONPATH`, as `python -m` in the checkout - a `command` task runs its args directly by default, or in its image with `docker run` when the run is submitted with `--docker`; the run records which. The store and the scratch directory are mounted at the same paths - a task starts when the tasks it reads from succeeded and its cores are free; the cores are shared by all runs of the service, in the order of submission, and a waiting task is `queued` - a process stopped from outside (no exit code, 137 or 143) is retried as on the cluster, after a kill (137) with the retry memory per process, so the harness runs fewer processes; there is no memory limit per task, a process which takes too much is killed by the out of memory killer of the machine The backend sets `SIMRUN_CORES` (the cores of the worker), `SIMRUN_MEMORY_LIMIT_GB` (its memory, by which the harness sizes its pool of processes; nothing kills a process above it) and `SIMRUN_MEMORY_SOURCE=process` in the environment of a process: the processes share the cgroup of the machine or of the session, so the harness measures the memory of its own process tree in `/proc` instead of the one of the cgroup, and learns the resources of a task from it as on the cluster. The scratch directories are in `.simrun/scratch/`, the logs of every process are in `.simrun/store/runs//tasks//logs/`, and `simrun logs` reads them as it does on the cluster. ## Sources Locally the sources are recorded, not copied: the provenance (branch, commit, uncommitted files, the hash of the diff) is stored as on the cluster, and the processes run with the python environment of the command, so the code is the working tree as installed in it, e.g. editable. Nothing is snapshotted or installed, which keeps a run as fast as running the code by hand. `--snapshot` on `submit` and `edit` copies the working trees into `snapshots/`, when a run must not see later changes of the tree. The run gets an environment of its own, a venv in `.simrun/envs//` which sees the packages of the command's python, and the snapshots are installed into it as editable packages without their dependencies, never into the environment of the command; the harness runs with its python, and a `command` task finds it first on `PATH`. The installer is `uv` (else `pip`), which builds the snapshots with their build backends from the index or its cache. The venv is removed at the end of the run. ## What differs from the cluster | | cluster | local | | --- | --- | --- | | the machines | the servers of the cluster | this machine | | priorities and preemption | Kueue, six priority classes | none, the tasks run in the order of submission | | reservations of nodes | yes | refused | | images | built with BuildKit per environment | none; a `command` may use its image with `--docker` | | sources | snapshots, installed in the image | provenance only, unless `--snapshot`: installed in a venv of the run | | store | shared drive, or uploads through the API | `.simrun/store/` | | extra workers of a split task | spread over idle servers | none, one worker with all cores | The records, logs and outputs have the layout of a cluster run, see [Concepts](concepts.md). --- # Agents and MCP `simrun-mcp` is an [MCP](https://modelcontextprotocol.io) server on stdio, a local process of an agent: it runs on the machine of the working trees, takes the snapshots of the sources the agent names and calls the service with the token of the agent. It reads `SIMRUN_URL`, `SIMRUN_TOKEN` and the optional `SIMRUN_STORE` like the [command line](command-line.md). ## Setup Register the server with the environment of the client, for Claude Code e.g. ```bash claude mcp add simrun --env SIMRUN_URL=http://simrun.cluster.internal \ --env SIMRUN_TOKEN= --env SIMRUN_STORE=/mnt/cluster \ -- uv run --project ~/git/simrun simrun-mcp ``` or with an installed package, `-- simrun-mcp`. The token of an agent names its owner in the tokens file of the service; the runs of the agents of the owner of the cluster get the priority `owner`, the runs of other agents `agent`, see [Priorities and preemption](concepts.md#priorities-and-preemption). ## The tools | tool | arguments | answer | | --- | --- | --- | | `submit_workflow` | `workflow_path`, `sources` (name to the path of a working tree, the model first), `group`, `single_server` | the summary of the run at once, in the state `submitting` | | `wait_for_run` | `run_id`, `timeout_s` (600) | the summary of the run once it ended or after the timeout | | `run_status` | `run_id` | the record: state, tasks, plans and the provenance of the sources | | `run_progress` | `run_id` | the progress of every task: the counts of its ledger and its first and last progress lines | | `list_runs` | `group`, `agent`, `state`, `owner` | the runs; the runs of an owner include the ones of their agents | | `compare_runs` | `run_ids` | the provenance and the summaries of the tasks side by side | | `cancel_run` | `run_id` | the summary of the cancelled run | | `run_logs` | `run_id`, `task_id`, `tail_lines` (300, 0 for all) | the logs of the pods of the task | | `run_outputs` | `run_id`, `task_id` | the output files of the task | | `estimate_workflow` | `workflow_path`, `sources`, `single_server` | the plan and the expected duration of every task | Paths in the answers are relative to the store, and every answer carries `store`, the local mount of the store, so an agent opens a result with `store / path`. A refusal of the service is returned as `{"error": detail, "status": code}` instead of raised, so the agent can read it; a service which cannot be reached answers the status 503, an error on the side of the client 400. ## Groups of agents A main agent gives its subagents a group id; every subagent works on one variant of a model in its own git worktree and submits its runs with that group (`submit_workflow(..., group=...)`). The main agent lists the group, also while runs are going (`list_runs(group=...)`), compares the runs (`compare_runs`) and takes the winner from its provenance. --- # Web app The service serves a web app at `/ui/`, e.g. `http://localhost:8000/ui/` on the [local test system](testing.md#a-local-test-system), or on the [local service](local.md) of a project, which needs no cluster. It has two modes, switched in its top bar: the **Executions**, the runs of people and agents live, and the **Editor**, in which a workflow is built or changed, estimated and run while its file stays next to the code. The app talks only to the API of the service, with the token of the user; the service serves it from the directory of its setting `web` (`/opt/simrun/web` in the [service image](deployment.md#the-service-image)), and without that setting there is no `/ui/`. ## Login The app asks for a token of the tokens file of the service (see [Authentication](service.md#authentication)), checks it with `GET /me` and keeps it in the local storage of the browser. Every request and every event stream sends it as a bearer token, never in a URL. A token the service no longer accepts (removed from the tokens file) returns the app to the login and removes the token from the browser. ## Executions The view has the runs on the left, the canvas of the selected run in the center with the run bar above it and the servers below it, and the inspector on the right. ![The Executions: a running run with its task selected](images/executions-running.png) Everything is live: the app reads the event streams `GET /events` (the runs and the servers) and `GET /runs/{run}/events` (the record, the progress and the workers of the selected run). When a stream drops, e.g. while the service restarts, the top bar shows **reconnecting** and the app connects again, waiting 1 s, doubling up to 10 s, and keeps the selected run. - **Top bar**: where you are (the workflow of the selected run), the capacity of the cluster (the cores the pods of all servers request out of all cores, and the number of queued runs) and the identity of the token. - **Runs**: a card per run, the 200 most recent, with the workflow, the run, who submitted it (an agent with its owner), its state, its progress and, for a failed run, its message. The chips **All**, **Mine** (the runs of the person of the token and of the person's agents), **Agents** and **Active** (submitting, queued or running) filter them, a menu selects a group. A click selects the run; the address `/ui/#/runs/` keeps it on a reload. - **Run bar**: the run, who submitted it and when, its state and its progress (tasks done, units done of all units), and **Stop**, which cancels a run that is not over after a confirmation in the bar. - **Canvas**: a card per task, in columns by the depth of the task in the graph of its inputs, with the edges of the inputs. A card shows the type, the task, its state, its settings, the units of its ledger with a progress bar and the workers of a split task on their servers; a task split over several workers is drawn as a stack. A task without a ledger shows its state without a progress bar. - **Servers**: a tile per server of the cluster with the cores and memory the pods of runs on it request, out of what the scheduler gives them (the allocatable cores and memory less the requests of other pods, e.g. of DaemonSets), a segment per pod, those of the selected run in blue; a reserved server says by which run. - **Inspector**: without a selected task, the run: its message, who submitted it with which priority and when, its kills for memory and its sources (the snapshots with branch, commit and changed files). With a task selected on the canvas, the tabs **Overview** (why a failed task failed, the exit of its last pod; the units, the elapsed and the remaining time, or the phase, `starting` or `merging`; the numeric fields of the last progress line, the workers with their servers, processes and memory, and the resources its plan requested), **Outputs** (the output files of the task) and **Logs** (the last lines of every pod, read again every 5 s while the task runs; tracebacks and errors in red, the lines of the Argo executor hidden unless asked for, long lines scrolled sideways or, with **wrap lines**, wrapped at the width of the inspector). A server is named by the last two parts of its host name everywhere, `agent-0` for `k3d-simrun-test-agent-0`, the full name in a tooltip. ![A failed task: the message names the pod and its exit](images/executions-failed.png) ![The outputs of a task](images/executions-outputs.png) ## The Editor { #the-editor } The Editor edits a workflow file which stays on the machine with the code: the browser never reads or writes that machine, a command running there does. The browser can be on any machine which reaches the service. ### The command On the machine with the working trees, with `SIMRUN_URL`, `SIMRUN_TOKEN` and `SIMRUN_STORE` set as for every [command](command-line.md) and the store mounted: ```bash simrun edit fit.yaml --source model=~/git/pkdb_models --source sbmlsim=~/git/sbmlsim ``` It reads the file (a file which does not exist yet is a new workflow named by the file stem, without tasks), snapshots the sources, opens an edit session in the service and prints the link of the Editor, e.g. `http://localhost:8000/ui/#/edit/`. It keeps running: it answers the saves and runs of the browser, and every 2 s it sends the file and the state of the working trees (branch, commit, uncommitted files) to the session when they changed. While a run is being submitted (plans, snapshots) it keeps telling the service that it is alive, and when the service is not reachable or a proxy answers 502, 503 or 504 (e.g. during a redeploy), it retries with a backoff of 1 s doubling to 10 s; a request is done once, also when its answer had to be sent again. Ctrl-C (or SIGTERM) closes the session and ends the command with exit code 0; when the service does not answer the close within 5 s, the command ends anyway and the session shows not connected after 30 s. A second Ctrl-C ends the command at once. `--group` is the group of the runs started from the Editor which name none. ### The session A session lives in the store under `edits//`, so a restart of the service keeps it. The page `#/edit` (the **Editor** switch of the top bar) lists the sessions of your token by file name, with the directory, the machine, the workflow and whether the session is connected; the sessions which are no longer connected are listed under **Closed**, the ones not seen for a day behind **Show older**. A click opens one; the breadcrumb of the top bar then names the file. Saves and runs of a session are allowed only to the person whose token opened it. A session is **connected** while its command runs and asked the service for requests within the last 30 s. ### Editing The Editor has the palette of the task types of the service on the left, the canvas in the middle with the bar floating above it, and the inspector on the right. ![The Editor: the palette, the canvas with the estimate on the cards, and the settings and resources of the selected task](images/editor.png) - **Palette**: the task types installed in the service, grouped by the part of their name before the first `.` (`demo.squares` under "Demo", `command` under "General"), with the `description` of the type (see [Plugins](plugins.md#the-contract)) and a search; its footer says which file a save writes on which machine. A click adds a task right of the right-most one and pans the canvas to show it, a drag drops it where it is let go. A new task is named by the last part of its type, with `-2`, `-3` for further ones. - **Canvas**: a card per task with a handle per output port on the right and per input port of its type on the left; a `command` has a handle per input it reads and a `+`, where a connection adds an input named after the task it reads. Dragging from an output to an input connects them; a connection which closes a cycle or reaches a port which is already connected is refused with a note under the bar. Delete or Backspace removes the selected task (and the inputs which read it) or edge. The tasks are laid out when the session opens and when the workflow comes from disk; afterwards they stay where they are dragged. Positions are not saved in the file. - **Inspector**: with a task selected, its type and id, its settings as a form of the JSON Schema of the settings of its type, without the fields the schema marks `readOnly`, which the plan of the type sets (numbers, choices and switches in two columns; an empty field is not written, so the default of the type applies; what the form cannot edit is edited as JSON), its inputs and the ports staged to local scratch, its resources (workers and memory per worker overriding the learned ones, its split; a `command` has the resources of its settings and stages nothing) and the estimate: memory per worker of the plan, duration and placement (the pods, the worker processes of a pod and the memory of a pod). Without a selected task, the workflow: its name, its split, the reservation of a server, the group of the run and the sources. - **Validation and estimate**: every change is validated by the service 400 ms after the last one; the errors of a task show on its card and in the inspector, an error of a setting under its field, the others in the inspector of the workflow. A valid workflow is estimated with the last snapshots of the sources, and the cards show the chips of the estimate (workers, memory per worker, duration, pods). The estimate is of the workflow as edited: a task type which plans its settings does so only when the workflow runs. ![A `command` task in the Editor: its args one per line, the inputs it reads and a `+` handle for another](images/editor-command.png) ### Save and Run The bar shows the branch, the commit and the uncommitted files of the model source, the machine of the session and the state of the draft: **valid, saved**, **unsaved changes**, the number of errors, or **changed on disk**. - **Save** sends the draft to the command, which writes it into the file atomically, in the form `simrun` writes a workflow (the fields which differ from the defaults, a list indented below its key): comments and the formatting of a hand-written file are not kept. Save is enabled only with unsaved changes (or a file changed on disk), so opening a hand-written file and looking at it never rewrites it. A save or a run waits up to 60 s for the command; it fails at once when the command stops meanwhile. - **Run workflow** sends the draft, saved or not, to the command, which snapshots the working trees again, plans the tasks with the task types installed there and submits the run, as `simrun submit` does with the same sources. The file is not written by a run. The Editor then shows the run in the Executions. ### Conflicts A change of the file on disk (by another editor, or by `git checkout`) reaches the Editor within a few seconds. Without unsaved changes the Editor takes the new workflow and lays it out again. With unsaved changes the bar says **changed on disk** and offers **Reload** (drop the draft for the file on disk) and **Save anyway** (overwrite the file on disk with the draft, whatever changed there). A plain Save is never written over a file which changed since the draft started from it: the command refuses it and the bar offers the same two choices. A file which appears on disk after the session opened on a missing file is such a change as well. ### Without a connected machine When the command stopped, or its machine has not asked for 30 s (asleep, off the network), the bar says that the machine is **not connected**. **Save** becomes **Download YAML**, which downloads the draft as `.yaml` in the form a save writes, and the state says **unsaved changes, download to keep them** once the draft was changed. **Run workflow** submits the draft from the browser with the last snapshots of the session, as edited, without the plans of the task types. Starting the command again opens a new session. **Rerun with changes** in the run bar of the Executions opens the workflow and the sources of a run in the Editor (`#/runs//edit`). There is no machine behind it, the bar says **from run #**: Download YAML keeps the changes, and Run workflow submits the changed workflow with the snapshots, the group and the reservation of the run (the reservation shows in the inspector of the workflow and can be changed there). ## Development The app is a Vite project in `frontend/` (Vue 3, TypeScript, Vue Flow, Tailwind CSS) with node 24 and the versions of `frontend/package-lock.json`: ```bash cd frontend npm ci # the dependencies of the lock npm run dev # the dev server at http://localhost:5173/ui/, with reload on every change npm test # the tests (Vitest, jsdom) npm run typecheck # vue-tsc npm run build # into frontend/dist, which the service serves with `web: frontend/dist` ``` The dev server proxies the API to the service of `SIMRUN_URL`, else `http://localhost:8000`, so it runs against the service of the [local test system](testing.md#a-local-test-system): start the service as described there, submit a run or start `simrun edit`, and open `http://localhost:5173/ui/` with the token `t-mk`. The service of the local test system also serves the built app at `http://localhost:8000/ui/` once `npm run build` made `frontend/dist`. The types of the API in `frontend/src/api/schema.d.ts` are generated from the OpenAPI document of the service. After a change of the API: ```bash uv run python scripts/openapi.py # frontend/src/api/openapi.json, which a test keeps equal to the API cd frontend && npm run types # frontend/src/api/schema.d.ts ``` `frontend/mockup/` is the approved mockup the app is built from, a single page with data in memory. --- # Plugins A task type comes from a plugin: a python package with entry points of the group `simrun.task_types`, each a `simrun.plugins.TaskType`. simrun itself registers the built-in type `command` and the demo `demo.squares` (`simrun.demo`), which is the smallest complete example of a split type. ```toml title="pyproject.toml of the plugin" [project.entry-points."simrun.task_types"] "sbmlsim.scan" = "sbmlsim.simrun:SCAN" ``` ```python from simrun.plugins import ResourceHint, TaskType SCAN = TaskType( name="sbmlsim.scan", settings=ScanSettings, # a pydantic model inputs=("model",), # input ports outputs=("results",), # output ports hint=ResourceHint(memory_base_gb=2.0, memory_per_worker_gb=3.0), count_units="sbmlsim.simrun:count_units", # settings -> number of units setup="sbmlsim.simrun:setup", # settings, paths; once per process compute_unit="sbmlsim.simrun:compute_unit", # settings, index, paths merge="sbmlsim.simrun:merge", # settings, paths; after all units plan="sbmlsim.simrun:plan", # settings -> settings; in the client cost_setting="subset", # resources learned per subset description="Parameter scans of a model", # the subtitle in the Editor ) ``` The functions are `"module:function"` strings, so a worker process resolves them by import. `paths` is a `simrun.plugins.TaskPaths`: the store, the run and task directories, `out`, the inputs by port and the local scratch. ## The contract - **Split or unsplit.** A type with `count_units`, `compute_unit` and `merge` is split: its units are claimed from the ledger by the processes of its pods (`simrun-worker`), and the coordinator pod merges once every unit is done (`simrun-merge` remains for runs recorded before). Other types run their `command`, an argv template with `{settings}` (the path of `settings.json`), `{out}` (the output directory) and `{input}`, through `simrun-exec`; the command validates the settings itself with the settings model. A part of the template with `{input}` is repeated once per input port of the task, sorted by port, with `{input}` replaced by `=`, and dropped when the task has no inputs: the path is the absolute path of the output the port reads as the pod sees it, or `/` for a port the task stages, the local copy `simrun-exec` makes before the command starts. `--input={input}` becomes `--input=a=/store/runs//tasks//out/` and, for a staged port `b`, `--input=b=/scratch/b`. - **Deterministic, atomic unit outputs.** A unit may be computed more than once (a worker evicted before it completed the unit, a restarted pod whose earlier incarnation still runs), so `compute_unit` writes outputs which depend on the settings and the index only, each written atomically: a temporary file in the same directory, then renamed. - **Ports are directories.** An output port `` of a task is the directory `out/` (`paths.out / port`); a downstream task reads it as the input `.` (`paths.inputs[]`). `out/summary.json` is the summary of the task, no port, which `compare` reads. - **`uses_cores`.** An unsplit plugin task is sized as one process (one core, `base + per_worker`), unless its type sets `uses_cores`: then it takes the processes a node allows, like a worker, and its command runs a pool of its own. - **`plan`** completes the settings of a task before it is submitted, e.g. with values derived from its model: `plan(settings) -> settings` gets the settings validated with the settings model and returns the planned ones. The client (`simrun submit`, `simrun estimate`, the MCP server) calls it for every task whose type is installed where the client runs, before it takes the snapshots, and submits the planned settings, the fields which were given or which the plan set (`model_dump(mode="json", exclude_unset=True)`), so the defaults stay with the plugin of the snapshot which validates them in the pods; `simrun validate` plans the same way, so it validates what `simrun submit` submits. A task of a type which is not installed locally is submitted as written. - **Provenance.** Every task gets `tasks//provenance.json`, the JSON list of the sources of its run (name, branch, commit, dirty, changed files, diff hash, snapshot), which the service writes next to `settings.json`; its path is `SIMRUN_PROVENANCE` in every pod of the task, so a command or a merge can record where its results come from. - **`cost_setting`** names the setting which changes the cost of a task, e.g. the subset of the data of a fit: the resources are learned per value of it. - **`description`** says in a few words what a task of the type does; the palette of the [Editor](web-app.md#the-editor) shows it under the name of the type ("Split over servers" or "One process" without one). - **`hint`** is the `ResourceHint` of a type without observations: the base memory, the memory per process, the scratch and the GPUs. - **Images need simrun.** A type with an `image` of its own runs its pods in that image, which must have simrun installed (`simrun-worker`, `simrun-merge` and `simrun-exec`, which every plugin command runs through); the image of an environment has it. ## Exit codes `simrun-exec` installs the snapshots of the run, stages the inputs, sets `OMP_NUM_THREADS`, `OPENBLAS_NUM_THREADS` and `MKL_NUM_THREADS` to 1, passes a SIGTERM on to the command and writes a report of its memory. Its exit codes: | code | meaning | | --- | --- | | 0 | done | | 3 | the command failed, not retried | | 4 | a configuration error | | 137 | the command was killed, e.g. by the out of memory killer; retried once with 1.5 times the memory per process | | 143 | stopped on SIGTERM, e.g. preempted; retried | ## Plugins of the service and of the snapshots The service imports the plugins it knows: it validates the settings with their models and counts the units of a split type when it plans a run. A task type which only exists in a snapshot (a plugin in a working tree) is unknown to the service: the service image must carry every plugin whose task types are submitted, and a new task type, a split one in particular, needs a rebuilt service image before its first run, see [The service image](deployment.md#the-service-image). The pods run the plugin of the snapshot. --- # Service The service is the API (FastAPI) which the command line, the MCP server and scripts call, and the controller behind it, which runs the submissions in the background and follows the workflows of the runs. It runs in the cluster as the Deployment `simrun` of `deploy/cluster/simrun.yaml`: ```bash SIMRUN_CONFIG=/etc/simrun/settings.yaml uvicorn --factory simrun.service.app:serve --host 0.0.0.0 --port 8000 --timeout-graceful-shutdown 5 ``` ## Authentication Every route needs a bearer token of the tokens file, a YAML mapping of token to identity: ```yaml title="tokens.yaml" : {name: mk, kind: person} : {name: fit-agent-1, kind: agent, owner: mk} ``` The identity of the token is recorded with the run and decides its priority: the owner of the cluster and the owner's agents get `owner`, other people `standard`, their agents `agent`. The service has no documentation routes, which would answer without a token. ## Endpoints | method and path | what it does | | --- | --- | | `POST /validate` | validate a workflow against the known task types; `{ok, errors}` | | `POST /estimate` | the resources, the plan and the expected seconds of every task of a submission | | `POST /runs` | submit a workflow with the provenance of its sources; returns the run in the state `submitting` (201) | | `GET /runs` | the runs, filtered by `group`, `owner`, `agent` and `state` | | `GET /runs/{run}` | the record of a run | | `GET /me` | the identity of the token | | `GET /servers` | the servers of the cluster with their reservation, the cores and memory the scheduler gives the pods of runs (allocatable less the requests of other pods) and the pods of runs on them with their requests | | `GET /events` | server-sent events: `runs` (the runs, filtered as `GET /runs`, with owner and workflow) and `servers`, each when it changed | | `GET /runs/{run}/events` | server-sent events: `run` (the record), `progress` and `workers` of a run, each when it changed; it ends after a run ended | | `GET /runs/{run}/tasks/{task}/workers` | the workers of a task: server, state, attempts, processes, memory and units | | `GET /runs/{run}/wait` | a long poll: the summary once the run ended or after `timeout` seconds (60, at most 300) | | `GET /runs/{run}/progress` | the progress of every task: the counts of its ledger and its first and last progress lines | | `GET /runs/{run}/logs/{task}` | the logs of the pods of a task as text, the last `tail_lines` of each (300, 0 for all) | | `GET /runs/{run}/outputs/{task}` | the output files of a task, paths relative to the store | | `POST /snapshots/{name}/missing` | the paths of a file list (path, mode, `sha256` or `link`) whose content the latest snapshot of a source lacks; 413 above 64 MiB | | `POST /snapshots/{name}` | write a snapshot from a multipart body: `scan` (the git state and the file list as JSON) and `files` (a tar of the missing files); new files are renamed into the snapshot, files and directories keep their modes; 422 for a path outside the tree, a link out of it, a file mode beyond read and execute, a member which is not a listed regular file or a hash which does not match, 409 when the store lost a content meanwhile, 413 above `max_upload_bytes` | | `POST /runs/{run}/cancel` | cancel a run | | `DELETE /runs/{run}` | remove an ended run and the snapshots nothing refers to | | `POST /cleanup` | remove the runs which ended more than `older_than_days` ago, of a `group` or only failed ones (`failed_only`), `dry_run` lists them | | `GET /usage` | the bytes of the store per group and run | | `POST /compare` | the provenance and the summaries of the tasks of runs side by side | `simrun.client.Client` is the python client of these endpoints, which the command line and the MCP server use. ## Settings The service reads a YAML file named by `SIMRUN_CONFIG`, a `simrun.service.app.ServiceSettings`: | setting | meaning | | --- | --- | | `store` | the root of the run store, `/store` in the pod | | `db` | the SQLite file of the learned resources and the index of the runs, on a local volume | | `tokens` | the YAML file of the tokens, see above | | `owner` | the person who owns the cluster: their runs and their agents' get the top priority | | `cluster` | the cluster, a `ClusterConfig`, see below | | `poll_s` | seconds between two reconciliations of the controller when no change of a workflow woke it before (5) | | `events_poll_s` | seconds between two checks of an event stream (0.5) | | `events_max_s` | seconds after which an event stream ends and the client connects again (300) | | `default_image` | the image of a run whose model has no `pyproject.toml` | | `simrun_wheel` | the simrun wheel the images of the environments install | | `python` | the python of the images of the environments (3.14) | | `build_constraints` | the pinned versions of the build backends, `src/simrun/build_constraints.txt` by default | | `backend` | `module:callable` of the backend, `simrun.service.argo_backend:backend` | | `max_upload_bytes` | the largest body of an upload of a snapshot (2 GiB); the upload is spooled to the temporary directory of the service (`TMPDIR`, else `/tmp`) before it is checked, so that directory must hold this much for every concurrent upload | | `web` | the directory of the built [web app](web-app.md), served at `/ui/` without a token; by default the one of the installation (`simrun/web` of the package, else `frontend/dist` of a checkout), `null` for no `/ui/` | `cluster`, a `simrun.config.ClusterConfig`: | setting | meaning | | --- | --- | | `registry` | the registry inside the network, as the nodes and BuildKit name it in image references | | `registry_url` | the API of the registry as the service reaches it, `http://` by default | | `base_image` | the base image of the images of the environments, uv and the python of the service; a repository of the registry by default (`astral-sh/uv:0.13.7-python3.14-trixie-slim@sha256:...`), which the registry mirrors with its digest | | `system_packages` | the Debian packages the images of the environments install on the base image, the shared libraries which wheels link but do not bundle (`[libexpat1]`, which python-libsbml needs) | | `namespace` | the namespace of the workflows and their pods (`simrun`) | | `local_queue` | the Kueue LocalQueue of the pods (`simrun`) | | `store_claim` | the PersistentVolumeClaim of the run store (`simrun-store`) | | `store_mount` | the path of the run store in a pod (`/store`) | | `scratch_path` | the local scratch, a hostPath on the local disk of a node (`/scratch`) | | `service_account` | the service account of the pods of the workflows (`simrun-workflow`) | | `buildkit` | the address of the BuildKit daemon (`tcp://buildkitd.simrun:1234`) | | `buildctl_image` | the image of the job which sends a build to BuildKit | | `grace_s` | the seconds a worker has to stop after SIGTERM (60) | | `ttl_after_finished_s` | the seconds a finished workflow is kept (86400) | | `pod_start_s` | about the seconds from the creation of a pod to its first unit; a split task gets a further pod only while every pod has at least this much work (10), see [Split modes](concepts.md#split-modes) | To raise a pin of the build backends, edit `build_constraints.txt` (or point `build_constraints` at a file of your own) and redeploy the service; the images of the models which use the backend are built again. ## Environment variables | variable | read by | meaning | | --- | --- | --- | | `SIMRUN_URL` | command line, MCP server | the address of the service | | `SIMRUN_TOKEN` | command line, MCP server | the token of the person or the agent | | `SIMRUN_STORE` | command line, MCP server | optional: the run store as mounted on the machine of the client, which then writes the snapshots itself instead of uploading them | | `SIMRUN_CONFIG` | service | the YAML file of the `ServiceSettings` | | `KUBECONFIG` | service | the cluster outside of a pod; in the pod the service account | | `SIMRUN_TASK_DIR`, `SIMRUN_TASK_TYPE`, `SIMRUN_INPUTS`, `SIMRUN_STAGE`, `SIMRUN_EDITABLE`, `SIMRUN_SCRATCH`, `SIMRUN_PROVENANCE`, `SIMRUN_MEMORY_BASE_GB`, `SIMRUN_MEMORY_PER_WORKER_GB`, `SIMRUN_WORKER`, `SIMRUN_ROLE`, `SIMRUN_ATTEMPT`, `SIMRUN_STALE_S`, `SIMRUN_MERGE`, `SIMRUN_NODE`, `SIMRUN_CORES`, `SIMRUN_MEMORY_LIMIT_GB` | pods | the environment of the harness of a pod, set by the workflow, see `simrun.podspec` and `simrun.worker`; `SIMRUN_CORES` caps the cores of a worker and `SIMRUN_MEMORY_LIMIT_GB` is its memory limit without one of its cgroup (the local backend sets both, a pod has the quota and the limit of its cgroup) | | `SIMRUN_K3D`, `SIMRUN_K3D_KEEP` | tests | run the tests on the k3d cluster and keep it, see [Testing](testing.md) | ## Submissions in the background `POST /runs` validates and plans a run, writes its record in the state `submitting` and returns its id at once. The environment is resolved, the image built and the workflow submitted in the background, in a pool of the controller; a failure lands in the record (the run fails with the message). Two submissions of the same new environment build its image once. The service does not wait for the builds when it stops: a run submitting when the service stopped is resumed at its next start from its record (an image which was pushed is found, a reserved node kept, a workflow which was created taken as submitted). ## The controller The controller reconciles whenever a workflow of a run changed (the backend watches the workflows; the local backend reports the start and end of every process) and every `poll_s` seconds otherwise: it reads the records of the live runs from the index, the states of their workflows from the backend, records the states of the tasks and the kills for memory, records the reports of the workers as resource observations once a task is final, keeps the logs of the pods in the store, and ends a run when every task is done or one failed. It keeps no state in memory, so a restarted service continues where the one before stopped. A release of a reserved node or a cancellation of a workflow which fails stays in the record and is retried at the next polls. --- # Cluster deployment `simrun` runs on a k3s cluster of dedicated Linux servers. `deploy/cluster` is the whole cluster as a kustomization: | component | file | role | | --- | --- | --- | | namespace | `namespace.yaml` | the namespace `simrun` of the service, the workflows and their pods | | priorities | `priority-classes.yaml` | the six priority classes of simrun, each a Kueue `WorkloadPriorityClass` and a Kubernetes `PriorityClass` of the same value, see [Priorities and preemption](concepts.md#priorities-and-preemption) | | Kueue v0.20.5 | `kueue/` | the release manifests, committed unchanged, and the queue: a `Topology` by host, a `ResourceFlavor` over the nodes labelled `simrun/worker=true`, the `ClusterQueue` `simrun`, which preempts lower priorities, and the `LocalQueue` `simrun` | | Argo Workflows v4.1.6 | `argo/` | the release manifests of a namespace install, committed unchanged, the configuration of the controller, its requeue time of 1 s instead of 10 s (`requeue.yaml`, so a run shows running soon after its pods start and the next task of the DAG starts about a second after the one before it) and the service account `simrun-workflow` of the pods | | run store | `store-settings.yaml`, `store.yaml` | the run store on NFS (csi-driver-nfs), mounted by the service and every pod with `nfsvers=4.2` and `lookupcache=positive` | | registry | `zot.yaml` | [Zot](https://zotregistry.dev), the image registry inside the network | | builds | `buildkit.yaml` | the BuildKit daemon, which builds the images of the environments, and its state (`buildkitd-state`, 50 GiB of `local-path`): the layers and the downloads of uv of the builds, kept when its pod restarts | | service | `simrun.yaml` | the service: its settings, its tokens, its volumes and its rights | Since the release manifests are committed, a setup needs no download. Apply it with ```bash kubectl apply --server-side -k deploy/cluster ``` The resources of Kueue are accepted once its CRDs and its webhook are up, so the first apply may have to be repeated. The store is mounted with `lookupcache=positive`: the NFS client caches no negative lookups, so a file a pod wrote, e.g. the marker `out/.simrun-merged` of a merge, is seen by the service at once even when it looked for the file before. The mount options are the ones of the StorageClass when the volume of the claim was provisioned: an existing PersistentVolume keeps its options, so add the option to `spec.mountOptions` of that PersistentVolume (or provision it again) and restart the pods which mount it. Without the option the service waits up to two minutes for the marker of a merge before it fails the task. ## Nodes The nodes of the runs carry the label `simrun/worker=true`, which the `ResourceFlavor` of Kueue needs (a flavor with a topology needs node labels) and by which the backend lists the nodes: ```bash kubectl label node simrun/worker=true ``` The infrastructure (the controllers of Argo and Kueue, BuildKit and its build Jobs, Zot, the service) has a node affinity off these nodes (`off-workers.yaml`), so Kueue and the planner count their resources as free for the runs. The backend subtracts from the allocatable resources of a node the wait container of the Argo executor and the requests of the pods which are no pods of runs (the DaemonSets of the cluster, e.g. of the NFS driver and of the monitoring), rounded down to whole cores and GiB, so a pod the planner sizes for a node fits it. The local scratch of a pod is a hostPath on the local disk of its node, `/scratch` by default. ## The values a setup fills in The manifests carry the values of the cluster of the design. A setup replaces them: | value | where | meaning | | --- | --- | --- | | `server: storage.cluster.internal`, `share: /tank/simrun` | `store-settings.yaml` | the NFS export of the run store, substituted into the StorageClass by the replacements of the kustomization; NFS v4.1 or newer, the ledger relies on its atomic renames | | `registry.cluster.internal:5000` | `zot.yaml`, `buildkit.yaml`, `simrun.yaml`, `deploy/service/Dockerfile` | the address of the registry as the nodes and BuildKit name it | | `owner: mk` | `simrun.yaml`, ConfigMap `simrun-config` | the person who owns the cluster | | `{}` | `simrun.yaml`, Secret `simrun-tokens` | the tokens of the people and agents, see [Authentication](service.md#authentication) | | `simrun-service:0.3.2`, `simrun-0.3.2-py3-none-any.whl` | `simrun.yaml` | the image of the service and the wheel of simrun it carries, see below | | `nominalQuota` of cpu, memory, GPUs and ephemeral storage | `kueue/queues.yaml` | the allocatable resources of the worker nodes | The Service `simrun` (port 80) is reached by the clients, e.g. through an ingress or a DNS name of the network; the examples of this documentation use `http://simrun.cluster.internal`. The development server of the agents mounts the run store, e.g. at `/mnt/cluster`, which is `SIMRUN_STORE` of the clients. ## The service image `deploy/service/Dockerfile` is the image `simrun-service` of `deploy/cluster/simrun.yaml`: uv (the service resolves the environments of the runs), simrun with the dependencies of its lock, its wheel at `/opt/simrun/` (the `simrun_wheel` of the settings), the plugins and the [web app](web-app.md) at `/opt/simrun/web` (the `web` of the settings, served at `/ui/`). Build it from a context of the requirements of the lock, the wheel, the wheels of the plugins and the built web app (node 24): ```bash mkdir -p build/service/plugins uv build --wheel --out-dir build/service uv export --frozen --no-dev --no-emit-project --no-header --no-annotate \ > build/service/requirements.txt uv export --frozen --no-dev --no-emit-project --no-header --no-annotate --no-hashes \ > build/service/constraints.txt cp ~/git/sbmlsim/dist/sbmlsim-*.whl build/service/plugins/ # the plugins (cd frontend && npm ci && npm run build) # the web app rm -rf build/service/web && cp -r frontend/dist build/service/web cp deploy/service/Dockerfile build/service/ docker build -t registry.cluster.internal:5000/simrun-service:0.3.2 \ --build-arg WHEEL=simrun-0.3.2-py3-none-any.whl build/service docker push registry.cluster.internal:5000/simrun-service:0.3.2 ``` The plugins are installed with their dependencies at the versions of the lock of simrun (`constraints.txt`): a plugin which needs another version of a dependency of simrun fails the build instead of moving it. The base (`--build-arg BASE`) is the base image of the environments in the registry, which the registry mirrors with its digest (`ClusterConfig.base_image`). At runtime the service keeps the cache of uv in `/var/cache/uv`, a volume of the deployment, so a submission does not download the metadata and the build backends again. A client without the store mounted (`SIMRUN_STORE`) uploads its snapshots through the API, and the service spools an upload to its temporary directory (`TMPDIR`, else `/tmp`) before it moves the files into the store: the container's `/tmp` must hold up to `max_upload_bytes` (2 GiB by default) per concurrent upload, so give the pod the ephemeral storage for it or mount a volume at `/tmp`. ## Builds of the images of the environments An image is built by a Job of `buildctl`, which sends the build context (a ConfigMap of its files, owned by the Job) to the BuildKit daemon of the cluster and pushes the image to the registry with the credentials of the optional Secret `simrun-registry-auth`; whether the registry has an image is asked its API. ## The k3d overlay `deploy/k3d` is the overlay of the [integration tests](testing.md#tests-on-a-k3d-cluster): the store is a hostPath which k3d mounts from the host into every node, the registry is the one of k3d (`k3d-simrun-registry:5000`, no Zot), the service runs in the process of the tests (the Deployment has no replica), the infrastructure runs on the server node, which has no label `simrun/worker`, and the quotas are the allocatable resources of the three agents. ## Findings on the cluster The integration tests verified the design on k3s v1.35.5 (k3d v5.9.0) with Kueue v0.20.5 and Argo Workflows v4.1.6; what differed from the assumptions is in the manifests and the code: - **Plain pods of Argo are Kueue workloads.** The queue label of a template reaches the pod; Kueue gates it, creates a workload per pod and admits it. An Argo node whose pod is gated is `Pending` (its `Retry` node too), so the backend reports it queued. - **Preemption and the retries of Argo.** Kueue preempts a workload of a lower priority by deleting its pod within the grace period: the worker stops on SIGTERM, releases its claims and writes its report. Argo marks the attempt `Error` with the message `pod deleted` and no exit code, so the retry strategy retries it; the new pod is gated again and waits. The backend reports such a node `Preempted` until its next pod runs (a node whose pod failed otherwise, a crash or a kill after its grace period, is `Retrying`). A pod preempted before Argo saw it running is created again by Argo as the same attempt, without a retry, so such a node is reported queued and not `Preempted`. - **Topology aware scheduling.** A `ResourceFlavor` with a topology needs node labels, hence the label `simrun/worker=true`. On admission Kueue adds the host and the flavor labels to the node selector of the pod; it honours the node selector and the toleration of a reserved run, and the taint `NoExecute` of a reservation keeps the pods of other runs off the node. Kueue preempts for the capacity of a node as well. - **The executor counts.** Kueue counts every container, so the init and wait containers of Argo request what they limit (`executor.resources` in the controller config). The init container copies the argoexec binary (about 150 MB) into the pod and has 256Mi; with 128Mi it was killed for memory now and then. - **Out of memory.** The kernel kills the whole container (cgroup v2, `memory.oom.group`), so the worker writes no report; Argo names the container in the message, `main: OOMKilled (exit code 137)`. A kill for memory needs this evidence or a report which counts the kills of the out of memory killer in its cgroup; an exit 137 without either is a pod killed after its grace period, an eviction. Only a pod which was stopped is retried: without an exit code (a lost node, a pod deleted before it ran), killed (137) or stopped on SIGTERM (143); any other exit fails its task at once, and the message of the run names the pod and its exit. The retry limit of a pod is 100: every eviction by Kueue counts against it. - **The retry memory after a preemption.** Argo's expressions of a retry know only the last attempt (`lastRetry`), so an attempt after one which was preempted following a kill for memory gets the first memory again, not the retry memory. The controller does not count a kill of such an attempt toward the limit; the attempt after it has the retry memory again. - **Stopping a workflow.** After `shutdown: Stop` Argo fails the nodes still pending; the backend deletes their gated pods. - **Logs.** The logs of an attempt whose pod Kueue deleted are gone with the pod; the controller keeps the logs of the pods of a task in the store once the task is final. --- # Testing The tests are written with pytest and live in `tests/`. Most of them are unit tests, which run anywhere; the tests in `tests/cluster` run on a k3d cluster and are deselected by default (`-m 'not cluster'` of the `addopts` of `pyproject.toml`). The [local mode](local.md) needs no cluster: `simrun submit examples/squares.yaml --local` runs a workflow on this machine, and its tests (`tests/test_local.py`, `tests/test_cli_local.py`) are part of the unit tests. ## Unit tests ```bash pytest # the unit tests, in parallel with pytest-xdist pytest -n 0 # in one process, e.g. for --pdb pytest tests/test_ledger.py # a single module pytest tests/test_ledger.py::test_claim_order # a single test tox r -e py3.15 # on one python in a fresh environment tox run-parallel # on every supported python ``` The tests of the service run the API with an in-memory backend (`simrun.testing.FakeBackend`) which the test advances, so the controller, the submissions and the index are tested without a cluster; the tests of the environments and the snapshots call `uv` and `git`, the tests of the CWL export run `cwltool`. Continuous integration runs them on python 3.14 and 3.15 on Linux. ## Tests on a k3d cluster { #tests-on-a-k3d-cluster } ```bash SIMRUN_K3D=1 uv run pytest -m cluster tests/cluster SIMRUN_K3D=1 SIMRUN_K3D_KEEP=1 uv run pytest -m cluster tests/cluster # keep the cluster ``` They need docker, node 24 (the web app of the image of the service), about 20 GiB of free disk and the network (the images of k3s, Kueue, Argo, BuildKit, `python:3.14-slim-trixie` and the uv base image). The session downloads k3d and kubectl into `.tools/bin`, creates the k3d cluster `simrun-test` (one server with the infrastructure; three agents of 6 GiB labelled `simrun/worker=true`) with the registry `k3d-simrun-registry:5000` (`localhost:5000` from the host), writes its kubeconfig to `.tools/kubeconfig` (the contexts of the user are not touched), applies `deploy/k3d`, builds the image of the runs without sources from the working tree (`tests/cluster/Dockerfile`) and pushes it, and copies the base image of the images of environments (`ClusterConfig.base_image`, from `ghcr.io/astral-sh/uv`) into the registry with its digest. Each test runs the service in its process with an `ArgoBackend` and the store `.tools/k3d/store`, which k3d mounts into every node: | module | what it checks | | --- | --- | | `test_cluster.py` | the central assumptions of the design: a run end to end, the take-back of extra workers by a run of the same priority, the owner preempting an agent, the retry after a kill for memory, a reservation, a lost worker and the build of an image | | `test_sources.py` | the path of a model with sources end to end: a model repository (hatchling, `uv.lock`) with task types of its own and a library (setuptools, a dependency, an uncommitted change) are snapshotted, the service resolves their environment and builds its image with BuildKit, the pods install the read-only snapshots, a split and an unsplit task run, and the merged result, the report and the learned resources are checked | | `test_capacity.py` | a DaemonSet with requests on the agents, and the nodes planned without them | | `test_service_image.py` | the image of the service, built from `deploy/service/Dockerfile` with a plugin and the web app built in `frontend/`, serving the API and the app at `/ui/` | At the end the cluster, the registry, the store and the images are deleted (the files the pods of an environment wrote as root are given back to the user first), unless `SIMRUN_K3D_KEEP=1`; a kept cluster is used by the next session. The tests run in one worker of pytest-xdist, which shares the nodes, and take about nine minutes on a created cluster. ## A local test system { #a-local-test-system } The k3d cluster of the cluster tests also serves as a local test system: the service runs on the host against it, and runs are submitted with the command line or the MCP server as on the real cluster. It needs docker, about 20 GiB of free disk and the network, like the tests. ```bash uv sync --extra dev uv run python tests/cluster/local.py up # about two minutes, reuses a cluster which exists ``` `up` creates the cluster `simrun-test` as the session of the cluster tests does (or reuses it), builds the image of the runs from the working tree and pushes it, and writes the settings and the tokens of the service into `.tools/k3d/local`. It ends with the commands of the next two steps. Start the service in one shell: ```bash export KUBECONFIG=$PWD/.tools/kubeconfig export SIMRUN_CONFIG=$PWD/.tools/k3d/local/settings.yaml uv run uvicorn --factory simrun.service.app:serve --port 8000 --timeout-graceful-shutdown 5 ``` and submit from another, e.g. the workflow `examples/squares.yaml` of [A first run](index.md#a-first-run): ```bash export SIMRUN_URL=http://localhost:8000 export SIMRUN_TOKEN=t-mk # the owner; t-mk-bot, t-alice and t-alice-bot test the priorities export SIMRUN_STORE=$PWD/.tools/k3d/store # the run store, mounted into every node uv run simrun submit examples/squares.yaml uv run simrun wait uv run simrun status ``` To edit a workflow in the [Editor](web-app.md#the-editor) of the web app, start the command which serves it in the same environment (on a copy, as a save writes the file): ```bash cp examples/squares.yaml /tmp/squares.yaml uv run simrun edit /tmp/squares.yaml # prints the link of the Editor, Ctrl-C ends the session ``` The service also serves the [web app](web-app.md) at `http://localhost:8000/ui/` (log in with `t-mk`): `up` points the setting `web` at `frontend/dist` when the app is built (`npm ci && npm run build` in `frontend/`, then `up` again). For the development of the app, run its dev server against this service, which reloads on every change of the code: ```bash cd frontend && npm run dev # http://localhost:5173/ui/, the API proxied to http://localhost:8000 ``` The service knows the task types installed in `.venv`; the runs without sources use the image built by `up`, a model submitted with `--source` gets an image of its environment built by BuildKit in the cluster. The workflows and pods are seen with `.tools/bin/kubectl --kubeconfig .tools/kubeconfig -n simrun get workflows,pods`. After a change of the code of the pods, run `up` again, which builds a new image, and restart the service. The cluster is the one `SIMRUN_K3D_KEEP=1` keeps, so the cluster tests and the local system share it. `down` deletes the cluster, the registry, the store and the settings: ```bash uv run python tests/cluster/local.py down ``` ## Smoke workflows { #smoke-workflows } The smoke workflows run the analyses of sbmlsim and pkdb_models through simrun, to find what breaks a real model before the migration of the models does: `tests/smoke/sbmlsim.yaml` every kind of analysis of the HCTZ example of sbmlsim (simulate, fit, identifiability, fisher, report, petab, scan, sensitivity, uncertainty) with the checkout of sbmlsim as its source, and `tests/smoke/pkdb_models.yaml` the simulations, a fit with the analyses which read it and the build of the models of the HCTZ model of pkdb_models, with the checkouts of pkdb_models and sbmlsim as its sources. They need checkouts of both (by default `~/git/sbmlsim` and `~/git/pkdb_models`): ```bash uv run python tests/smoke/run.py # --local, on this machine (about 5 minutes) uv run python tests/smoke/run.py --k3d # on the local test system, after `tests/cluster/local.py up` uv run python tests/smoke/run.py --only sbmlsim --sbmlsim ../sbmlsim --pkdb-models ../pkdb_models ``` `run.py` installs this checkout of simrun and the two checkouts editable into an environment of its own, `.tools/smoke/venv`, so the client plans with the plugin of sbmlsim and the service knows its task types. Locally it uses the local service of `.tools/smoke/local` and stops it at the end; with `--k3d` it runs a service of that environment against the local test system on a free port, and the first run of each workflow builds the image of its environment. `--local --k3d` runs both, one after the other. After every run it prints the timeline of the run (`tests/smoke/timeline.py`, also on its own for any run of a store) and the load of the machine, which other work on it changes: ``` smoke-pkdb-models 20261108-235754-cc7e82 succeeded in 263 s environment and image 1 s, workflow 0 s task start waited dag admit start prep work merge end cpu procs thrott factory +6 6 1 1 3 1 8 0 +15 9 1 0 fit +6 6 1 1 3 2 102 1 +110 193 2 0 simulate +6 6 1 1 3 1 46 0 +53 19 11 0 ... identifiability +116 6 2 1 2 2 24 20 +163 26 1 0 report +171 9 4 1 2 1 87 0 +259 88 1 0 ``` `start` and `end` are seconds from the creation of the run; `waited` the seconds from the end of the tasks a task reads (or the creation of the run) to the start of its harness, the submission and the start of its pod on the cluster, the free cores locally; `prep` the install of the sources, the restart and the imports; `work` its command or its units; `merge` the merge of a split task by its coordinator; `cpu` the cpu seconds of its processes; `procs` its processes. On k3d the pods split `waited` into `dag` (to the creation of the pod), `admit` (Kueue) and `start` (the image and Argo's init container), `thrott` is the time the CPU quota of its pods held it back, and `environment and image` the seconds the service took to resolve the environment and to build the image when it was missing. The smoke workflows read the checkouts as they are: for numbers to compare, point `--sbmlsim` and `--pkdb-models` at clones of fixed commits. The reports of the workers carry these times (`started`, `prepare_seconds`, `cpu_seconds`, `throttled_seconds`). The exit code is 1 when a run did not succeed. ## Continuous integration In continuous integration the job `cluster` runs the tests on the k3d cluster on a push to `develop`, on a release, when the workflow `CI-CD` is started by hand and on a pull request with the label `cluster`, see [Development](development.md#pull-requests). --- # API reference The API reference is generated from the docstrings of the package. ## simrun The workflows, the plugins, the clients, the planning and the harness of the pods. | module | description | | --- | --- | | [workflow](workflow.md) | the workflow model: tasks, the references between them and their order | | [plugins](plugins.md) | the plugin interface: task types, their resource hints and their discovery | | [plugins_builtin](plugins_builtin.md) | the built-in task types | | [demo](demo.md) | the demo plugin: sums of squares, split into one unit per number | | [client](client.md) | the client of the service: snapshots working trees, then calls the API | | [cli](cli.md) | the command line of simrun: submit, follow and compare runs | | [local](local.md) | the local service of a project in `.simrun/`, started on demand | | [mcp](mcp.md) | the MCP server of simrun, a local stdio process of an agent | | [planner](planner.md) | the planner: the shape of a worker pod and the number of pieces of a task | | [resources](resources.md) | resources learned from the reports of workers | | [ledger](ledger.md) | the unit ledger: workers of a split task claim units through atomic renames | | [worker](worker.md) | the worker harness: a pod of a split task computes units of the ledger in a pool | | [cgroup](cgroup.md) | the limits and the usage of the cgroup (v2) a worker runs in | | [snapshot](snapshot.md) | snapshots of working trees with provenance | | [environment](environment.md) | the environment of a run: a lock from the model and its snapshotted libraries | | [store](store.md) | the run store: run records and the layout of a run on the shared drive | | [config](config.md) | the configuration of the cluster the runs are submitted to | | [podspec](podspec.md) | the parts of a pod which every backend shares: image, shapes, command, environment | | [argo](argo.md) | Argo workflows from runs: one DAG, a scatter of workers per split task | | [cwl](cwl.md) | CWL v1.2 workflows from runs, executed with cwltool (or Toil on Slurm) | | [testing](testing.md) | a backend in memory for tests of the service, advanced by the test | ## simrun.service The service: the API, the controller and the backends, see [Service](../service.md). | module | description | | --- | --- | | [service.app](service.app.md) | the API of the service: people and agents submit, follow and compare runs | | [service.auth](service.auth.md) | identities of the callers of the service and the priorities of their runs | | [service.backend](service.backend.md) | the backend the service runs workflows on: Argo Workflows, or a fake in tests | | [service.argo_backend](service.argo_backend.md) | the backend of the service on a cluster: Argo Workflows, Kueue and BuildKit | | [service.local_backend](service.local_backend.md) | the backend of the local mode: the tasks of a run as processes on this machine | | [service.local_env](service.local_env.md) | the venv of a local run with snapshots, never the environment of the service | | [service.controller](service.controller.md) | the controller: follows the workflows of the runs and records their end | | [service.submission](service.submission.md) | the submission of a run in the background: its environment, image and workflow | | [service.index](service.index.md) | the index of the runs in SQLite, beside the run store | | [service.logs](service.logs.md) | the logs of the pods of a task, kept in the run store | | [service.compare](service.compare.md) | the comparison of runs: provenance and the summaries of their tasks side by side | --- # simrun.workflow The workflow model: tasks, the references between them and their order. ## class `Priority(*values)` The priority of a run in the queue. ## class `ResourceOverride(*, workers: int | None = None, memory_per_worker_gb: float | None = None, scratch_gb: float | None = None, gpus: int | None = None) -> None` Resources of a task which replace the learned or default ones. ## class `SplitMode(*values)` How the pieces of a task are placed on the nodes. ## class `TaskSpec(*, id: Annotated[str, _PydanticGeneralMetadata(pattern='^[a-z][a-z0-9-]*$')], type: str, settings: dict[str, typing.Any] = , inputs: dict[str, str] = , split: simrun.workflow.SplitMode | None = None, resources: simrun.workflow.ResourceOverride | None = None, stage: list[str] = ) -> None` A task of a workflow. Attributes: id: Identifier of the task within the workflow. type: Name of the task plugin. settings: Settings handed to the plugin. inputs: Input port to the output of another task, `.`. split: Split mode of the task, the one of the workflow by default. resources: Resources replacing the learned or default ones. stage: Input ports copied to local scratch. ### `TaskSpec.dependencies(self) -> list[str]` Ids of the tasks this task reads from, in input order, without repeats. ## class `Workflow(*, name: str, tasks: list[simrun.workflow.TaskSpec], split: simrun.workflow.SplitMode = , reserve: str | None = None) -> None` A workflow: tasks which read the outputs of each other. ### `Workflow.order(self) -> list[simrun.workflow.TaskSpec]` Return the tasks in topological order, stable in file order. Raises: WorkflowError: If the references form a cycle. ### `Workflow.task(self, task_id: str) -> simrun.workflow.TaskSpec` Return the task with the given id. Raises: WorkflowError: If there is no such task. ## class `WorkflowError` A workflow which is not valid. ## function `dump_workflow(workflow: simrun.workflow.Workflow, path: pathlib.Path) -> None` Write a workflow as a YAML file. ## function `load_workflow(path: pathlib.Path) -> simrun.workflow.Workflow` Read a workflow from a YAML file. Raises: WorkflowError: If the file is not a valid workflow. ## function `workflow_yaml(workflow: simrun.workflow.Workflow) -> str` The compact YAML of a workflow: the fields which differ from the defaults. `name` and `tasks` are always written, also when they are the defaults. A sequence is indented below its key, as the workflows of the docs are written. Args: workflow: The workflow. Returns: The YAML text. --- # simrun.plugins The plugin interface: task types, their resource hints and their discovery. ## class `ResourceHint(memory_base_gb: float = 2.0, memory_per_worker_gb: float = 6.0, scratch_gb: float = 0.0, gpus: int = 0) -> None` Default resources of a task type. Attributes: memory_base_gb: Base memory of a task in GiB. memory_per_worker_gb: Memory per worker in GiB. scratch_gb: Local scratch in GiB. gpus: Number of GPUs. ## class `TaskPaths(store: pathlib.Path, run_dir: pathlib.Path, task_dir: pathlib.Path, out: pathlib.Path, inputs: dict[str, pathlib.Path], scratch: pathlib.Path) -> None` The locations a task works in. Attributes: store: Root of the run store. run_dir: Directory of the run. task_dir: Directory of the task. out: Output directory of the task, `task_dir / "out"`. inputs: Input port to the directory of the output it reads. scratch: Local scratch directory. ## class `TaskType(name: str, settings: type[pydantic.main.BaseModel], inputs: tuple[str, ...] = (), outputs: tuple[str, ...] = ('out',), hint: simrun.plugins.ResourceHint = ResourceHint(memory_base_gb=2.0, memory_per_worker_gb=6.0, scratch_gb=0.0, gpus=0), checkpoints: bool = False, command: str | None = None, uses_cores: bool = False, count_units: str | None = None, setup: str | None = None, compute_unit: str | None = None, merge: str | None = None, plan: str | None = None, image: str | None = None, cost_setting: str | None = None, description: str = '') -> None` A type of task, registered as an entry point of `ENTRY_POINT_GROUP`. The callables are `"module:function"` strings, so a worker process resolves them by import, see `resolve`. Signatures: `count_units(settings) -> int`, `setup(settings, paths) -> None` (once per process), `compute_unit(settings, index, paths) -> None`, `merge(settings, paths) -> None` and `plan(settings) -> settings`, which the client calls before a submission when the type is installed locally (`simrun.client.plan_tasks`). A unit may be computed more than once: by a worker which is evicted before it completes the unit, or by a restarted incarnation of a worker whose earlier one still runs. So `compute_unit` writes outputs which depend on the settings and the index only (deterministic), each written atomically (a temporary file in the same directory, then renamed), so a reader never sees a partial unit. Attributes: name: Name of the type, as used in a workflow. settings: Pydantic model of the settings. inputs: Names of the input ports. outputs: Names of the output ports. hint: Default resources. checkpoints: Whether the task resumes from checkpoints. command: Argv template of unsplit tasks with `{settings}`, `{out}` and `{input}`: a part with `{input}` is repeated once per input port of the task, sorted by port, with `=` (`/` for a staged port), and dropped without inputs. It runs through `simrun-exec` (see `simrun.worker.exec_main`). uses_cores: Whether the command of an unsplit task uses the cores of its pod (a pool of processes); else the pod is sized for one process. count_units: Reference to the function counting the units of work. setup: Reference to the function preparing a process. compute_unit: Reference to the function computing one unit. merge: Reference to the function merging the units. plan: Reference to the function which completes the settings of a task before it is submitted, e.g. with values derived from its model. image: Image of the task, None for the environment image of the run. cost_setting: Name of the setting which changes the cost of a task, e.g. the subset of the data of a fit; the resources are learned per value of it (`simrun.resources.resource_key`). description: What a task of the type does in a few words, the subtitle of the type in the palette of the Editor. ## function `load_task_types() -> dict[str, simrun.plugins.TaskType]` Load the task types of all entry points. Returns: The task types by name. Raises: ValueError: A name is registered twice. ## function `resolve(ref: str) -> collections.abc.Callable[..., typing.Any]` Import the function a `"module:function"` string names. Args: ref: Reference of the form `module:function`. Returns: The function. --- # simrun.plugins_builtin The built-in task types. ## class `CommandSettings(*, image: str, args: list[str], cores: int, memory_gb: float, gpus: int = 0, scratch_gb: float = 0) -> None` Settings of a plain command task. Attributes: image: Image the command runs in. args: Argument vector of the command. cores: Number of cores. memory_gb: Memory in GiB. gpus: Number of GPUs. scratch_gb: Local scratch in GiB. --- # simrun.demo The demo plugin: sums of squares, split into one unit per number. A unit writes its square into the output port `squares` (`out/squares/.json`), the merge the sum into the summary of the task (`out/summary.json`, which `compare` reads; no port). ## class `SquaresSettings(*, n: int, sleep_s: float = 0.0, fail_at: int | None = None, memory_mb: int = 0) -> None` Settings of the squares demo. Attributes: n: Number of units. sleep_s: Seconds each unit sleeps. fail_at: Index of a unit which fails. memory_mb: Megabytes each unit allocates. ## function `compute_unit(settings: simrun.demo.SquaresSettings, index: int, paths: simrun.plugins.TaskPaths) -> None` Write the square of the index, atomically (temporary file, rename). Args: settings: Settings of the task. index: Index of the unit. paths: Locations of the task. Raises: RuntimeError: The unit is the one set to fail. ## function `count_units(settings: simrun.demo.SquaresSettings) -> int` Count the units of work. Args: settings: Settings of the task. Returns: The number of units. ## function `merge(settings: simrun.demo.SquaresSettings, paths: simrun.plugins.TaskPaths) -> None` Write the sum of the squares. Args: settings: Settings of the task. paths: Locations of the task. ## function `setup(settings: simrun.demo.SquaresSettings, paths: simrun.plugins.TaskPaths) -> None` Create the directory of the output port, once per process. Args: settings: Settings of the task. paths: Locations of the task. --- # simrun.client The client of the service: snapshots working trees, then calls the API. A snapshot is written into the store when it is mounted here (`SIMRUN_STORE`), else uploaded: the client asks the service which contents the store lacks and sends a tar of only those files. The CLI (`simrun.cli`) and the MCP server (`simrun.mcp`) both go through `Client`. ## class `Client(url: str, token: str, store: pathlib.Path | None, sources: Literal['store', 'upload', 'provenance'] = 'store', transport: httpx.BaseTransport | None = None) -> None` A client of the service of one user or agent. Args: url: Base URL of the service. token: Bearer token. store: The run store as mounted locally, where snapshots are written; None for a client which sends no snapshots. sources: How a source is sent: `store` copies a snapshot into `store`, `upload` sends the files the store of the service lacks, which writes the snapshot, `provenance` sends its provenance only (the local service runs the tasks with the environment of the command). transport: An HTTP transport, e.g. for tests. ### `Client.cancel(self, run_id: str) -> simrun.store.RunSummary` Cancel a run. ### `Client.cleanup(self, older_than_days: float, group: str | None = None, failed_only: bool = False, dry_run: bool = False) -> simrun.service.app.CleanupResult` Remove ended runs and the snapshots nothing refers to any more. ### `Client.close(self) -> None` Close the connections. ### `Client.close_edit(self, session_id: str, timeout: float = 60.0) -> None` Close an edit session, the command stopped, waiting up to `timeout` s. ### `Client.compare(self, run_ids: list[str]) -> simrun.service.compare.CompareTable` Provenance and summaries of runs side by side. ### `Client.delete(self, run_id: str) -> simrun.service.app.CleanupResult` Delete an ended run. ### `Client.edit_requests(self, session_id: str, timeout: float) -> list[simrun.service.edits.EditRequest]` Long poll the open requests of the browser, for up to `timeout` s. ### `Client.edit_result(self, session_id: str, request_id: str, result: simrun.service.edits.EditResult) -> None` Answer a request of the browser. ### `Client.edit_state(self, session_id: str, state: simrun.service.edits.EditState) -> simrun.service.edits.EditSession` Send what was read from disk to an edit session. ### `Client.estimate(self, workflow: pathlib.Path | simrun.workflow.Workflow, sources: dict[str, pathlib.Path], group: str | None = None, split: simrun.workflow.SplitMode | None = None, reserve: str | None = None) -> dict[str, simrun.service.app.TaskEstimate]` The resources, plans and durations of the tasks of a submission. ### `Client.list_runs(self, group: str | None = None, agent: str | None = None, state: simrun.store.RunState | None = None, owner: str | None = None) -> list[simrun.store.RunSummary]` The runs, filtered by group, agent, state and owner. ### `Client.logs(self, run_id: str, task_id: str, tail_lines: int = 300) -> str` The logs of the pods of a task, the last `tail_lines` of each (0: all). ### `Client.open_edit(self, body: simrun.service.edits.OpenEdit) -> simrun.service.edits.EditSession` Open an edit session for the workflow file of this machine. ### `Client.output_archive(self, run_id: str, task_id: str, to: pathlib.Path) -> list[pathlib.Path]` Extract the output of a task into the directory `to`. Returns: The extracted files, sorted. ### `Client.output_file(self, run_id: str, task_id: str, path: str) -> bytes` The bytes of a file of the output of a task, `path` relative to it. ### `Client.outputs(self, run_id: str, task_id: str) -> list[simrun.service.app.OutputFile]` The output files of a task, paths relative to the store. ### `Client.progress(self, run_id: str) -> dict[str, simrun.service.app.TaskProgress]` The progress of the tasks of a run. ### `Client.source(self, repo: pathlib.Path, name: str) -> simrun.snapshot.SourceProvenance` The provenance of a source, its snapshot written to the store first. Raises: RuntimeError: When the client has `store` sources without a store, or git fails. ClientError: When the service refuses an upload. ### `Client.status(self, run_id: str) -> simrun.store.RunRecord` The record of a run. ### `Client.submit(self, workflow: pathlib.Path | simrun.workflow.Workflow, sources: dict[str, pathlib.Path], group: str | None = None, split: simrun.workflow.SplitMode | None = None, reserve: str | None = None, local_commands: str | None = None) -> simrun.store.RunSummary` Plan the tasks, snapshot the sources and submit a workflow. The settings of a task whose type is installed locally and has a `plan` are replaced by the planned ones, see `plan_tasks`. Args: workflow: The workflow or its YAML file. sources: Working trees by source name, the model first. group: Group of the run. split: Split mode of every task. reserve: A node to reserve. local_commands: With the local service, `docker` runs a `command` task in its image, `direct` (the default) as a process. Returns: The summary of the run, `submitting` while the service resolves its environment, builds its image and submits its workflow. Raises: ClientError: When the service refuses the submission. ### `Client.usage(self) -> simrun.service.app.Usage` Bytes of the store per run and group. ### `Client.validate(self, workflow: pathlib.Path | simrun.workflow.Workflow) -> simrun.service.app.ValidationResult` Validate a workflow with its tasks planned, as `submit` submits it. Raises: ValueError: The settings of a task are invalid for its local type. ### `Client.wait(self, run_id: str, timeout_s: float | None = None) -> simrun.store.RunSummary` Wait for a run to end, or for `timeout_s` seconds (None: forever). A dropped connection or a restarting service is retried with a backoff of 1 s doubling to 10 s, until the timeout. Returns: The summary of the run, not terminal when the timeout passed; when the timeout passes during an outage, the last summary obtained. Raises: ClientError: For an unknown run or another refusal. httpx.TransportError: When the service was never reached before the timeout. ### `Client.wait_live(self, run_id: str, on_update: collections.abc.Callable[[dict[str, typing.Any]], None], timeout_s: float | None = None) -> dict[str, typing.Any]` Follow a run through its event stream until it is final. The stream ends when the run is final and also every `events_max_s` of the service; it is opened again then. A dropped connection or a restarting service is retried with a backoff of 1 s doubling to 10 s. Args: run_id: The run. on_update: Called with the latest `{"run": RunRecord, "progress": dict[str, TaskProgress]}` whenever one of them changed. timeout_s: Seconds to follow (None: until the run is final). Returns: The last update; its run is not terminal when the timeout passed. Raises: ClientError: For an unknown run or another refusal. httpx.TransportError: When the service was never reached before the timeout. ## class `ClientError(status: int, detail: str) -> None` The service refused a request. Attributes: status: The HTTP status. detail: The message of the service. ## function `plan_tasks(workflow: simrun.workflow.Workflow, types: dict[str, simrun.plugins.TaskType], root: pathlib.Path | None = None) -> simrun.workflow.Workflow` The workflow with the settings of its tasks planned by their types. A task whose type is in `types` and has a `plan` gets the settings the plan returns for its validated settings, the fields which were given or which the plan set, dumped as JSON: the defaults stay with the plugin of the snapshot, which validates the settings in the pods. Every other task is kept as written, e.g. a task of a type which is not installed locally. Args: workflow: The workflow, which is not changed. types: The task types installed locally, by name. root: The working tree of the model, first on the path of the imports while planning, as a task imports from it (`podspec.import_root`). Returns: A copy of the workflow with the planned settings. Raises: ValueError: The settings of a task are invalid for its type. --- # simrun.cli The command line of simrun: submit, follow and compare runs. Every command prints the detail of a refusal of the service to stderr and exits 1; `--json` prints machine-readable JSON instead of tables. ## function `cancel(run: Annotated[str, ], local: Annotated[bool, ] = False, as_json: Annotated[bool, ] = False) -> None` Cancel a run. ## function `cleanup(older_than: Annotated[float, ] = 30, group: Annotated[str | None, ] = None, failed: Annotated[bool, ] = False, dry_run: Annotated[bool, ] = False, local: Annotated[bool, ] = False, as_json: Annotated[bool, ] = False) -> None` Remove ended runs and the snapshots nothing refers to. ## function `compare(runs: Annotated[list[str], ], local: Annotated[bool, ] = False, as_json: Annotated[bool, ] = False) -> None` Compare the provenance and the summaries of runs. ## function `edit(workflow: Annotated[pathlib.Path, ], source: Annotated[list[str] | None, ] = None, group: Annotated[str | None, ] = None, local: Annotated[bool, ] = False, snapshot: Annotated[bool, ] = False) -> None` Serve the Editor of the web app for a workflow file until stopped. Saves of the browser are written into the file, a run snapshots the sources again. A missing file is a new workflow. Ctrl-C closes the session and exits, a second Ctrl-C exits without waiting for the close. With --local the session is on the local service of the current directory. ## function `estimate(workflow: Annotated[pathlib.Path, ], source: Annotated[list[str] | None, ] = None, single_server: Annotated[bool, ] = False, reserve: Annotated[str | None, ] = None, local: Annotated[bool, ] = False, as_json: Annotated[bool, ] = False) -> None` Show the plan and the expected duration of every task of a submission. ## function `list_(group: Annotated[str | None, ] = None, agent: Annotated[str | None, ] = None, state: Annotated[simrun.store.RunState | None, ] = None, owner: Annotated[str | None, ] = None, local: Annotated[bool, ] = False, as_json: Annotated[bool, ] = False) -> None` List runs. ## function `logs(run: Annotated[str, ], task: Annotated[str, ], tail: Annotated[int, ] = 300, local: Annotated[bool, ] = False) -> None` Print the logs of the pods of a task. ## function `make_client() -> simrun.client.Client` The client of the environment (`SIMRUN_URL`, `SIMRUN_TOKEN`, `SIMRUN_STORE`). ## function `outputs(run: Annotated[str, ], task: Annotated[str, ], to: Annotated[pathlib.Path | None, ] = None, local: Annotated[bool, ] = False, as_json: Annotated[bool, ] = False) -> None` List the output files of a task, or extract them with --to. ## function `serve(local: Annotated[bool, ] = False, port: Annotated[int | None, ] = None, host: Annotated[str, ] = '127.0.0.1', idle_exit: Annotated[float | None, ] = None, stop: Annotated[bool, ] = False) -> None` Run the service until stopped. With --local the service of the current directory with the local backend: the tasks run as processes on this machine, Ctrl-C cancels the runs; exit 3 when the service of the directory runs already. --stop stops it instead. Without, the service of the settings named by SIMRUN_CONFIG, as `uvicorn --factory simrun.service.app:serve` runs it. ## function `status(run: Annotated[str, ], local: Annotated[bool, ] = False, as_json: Annotated[bool, ] = False) -> None` Show the state of a run and its tasks. ## function `submit(workflow: Annotated[pathlib.Path, ], source: Annotated[list[str] | None, ] = None, group: Annotated[str | None, ] = None, single_server: Annotated[bool, ] = False, reserve: Annotated[str | None, ] = None, local: Annotated[bool, ] = False, docker: Annotated[bool, ] = False, snapshot: Annotated[bool, ] = False, wait_: Annotated[bool | None, ] = None, timeout: Annotated[float | None, ] = None, as_json: Annotated[bool, ] = False) -> None` Snapshot the sources and submit a workflow. ## function `usage(local: Annotated[bool, ] = False, as_json: Annotated[bool, ] = False) -> None` Show the bytes of the store per group and run. ## function `wait(run: Annotated[str, ], timeout: Annotated[float | None, ] = None, local: Annotated[bool, ] = False, as_json: Annotated[bool, ] = False) -> None` Wait for a run to end; exit 0 if it succeeded, 1 if not, 2 on timeout. --- # simrun.local The local service of a project: the service with the local backend in `.simrun/`. `.simrun/` of a project directory holds the store, the index, the token of the user and, while the service runs, `service.json` (pid, port, token, url and the environment it runs with). `simrun serve --local` runs the service in the foreground; `ensure_local` finds the running one or starts it in the background, which exits after `IDLE_EXIT_S` without a request and without a live run. A service in the background which runs with another environment than the command (python, simrun, task types) is restarted when it is idle. The service listens on `127.0.0.1` only, and its end cancels the runs of the local backend, so no process of a run outlives a graceful stop; a service which starts kills the processes one which died left behind (`LocalBackend.kill_orphans`). ## class `LocalService(instance: str, pid: int, port: int, token: str, url: str, python: str = '', version: str = '', types: str = '', background: bool = False) -> None` A running local service, as its `service.json` names it. Attributes: instance: A random value of this start of the service, which it answers at `INSTANCE_PATH` without a token: a client checks it before it sends the token. pid: The process of the service. port: The port on `127.0.0.1`. token: The bearer token of the user. url: The URL of the API, `http://127.0.0.1:`. python: The interpreter of the service. version: The version of simrun of the service. types: The hash of the task types the service found (`environment`). background: Whether it was started in the background (`ensure_local`), which exits when idle and is restarted for another environment. ## class `ServiceBusy` The local service has a live run or an open request, so it was not stopped. ## class `ServiceRunning` The local service of the directory runs already. ## function `ensure_local(root: pathlib.Path, timeout_s: float = 10.0, warn: collections.abc.Callable[[str], None] | None = None) -> simrun.local.LocalService` The local service of a project directory, started in the background if none runs. A service runs when the pid of `service.json` is alive, the server answers the instance of the file and `GET /me` answers. The start runs under an exclusive lock of `.simrun/start.lock`, so concurrent calls start one service; a start refused while an old service still ends is repeated once it has ended. A service whose environment (python, simrun, task types) differs from the one of this process runs code this command does not: one in the background without a live run and without an open request is stopped and started again; else it is used with a warning. Args: root: The project directory. timeout_s: Seconds to wait for a started service. warn: Called with a warning for the user, by default logged. Returns: The running service. Raises: RuntimeError: The service did not start in time or exited; the message has the end of `.simrun/service.log`. ## function `environment() -> dict[str, str]` The environment of this process as a local service records it. Returns: `python` (the interpreter), `version` (of simrun) and `types`, a hash of the entry points of the task types with the names and versions of their distributions, so a new, removed or upgraded plugin changes it. ## function `local_dir(root: pathlib.Path) -> pathlib.Path` `.simrun/` of a project directory, created with a `.gitignore` of everything. ## function `local_settings(root: pathlib.Path) -> simrun.service.app.ServiceSettings` The settings of the local service of a project directory. The token file is generated once with a random token of the user, a person who owns the service. Args: root: The project directory. Returns: The settings: the store, the index and the tokens in `.simrun/`. ## function `read_service(root: pathlib.Path) -> simrun.local.LocalService | None` The service `.simrun/service.json` names. Returns: The service, None without a readable file or with a URL which is not the one of its port on `127.0.0.1`. ## function `serve_local(root: pathlib.Path, port: int = 0, idle_exit_s: float | None = None, ready: collections.abc.Callable[[simrun.local.LocalService], None] | None = None, stopping: collections.abc.Callable[[str], None] | None = None) -> None` Run the local service of a project directory until stopped. Writes `.simrun/service.json` once the service listens and removes it at the end; the end cancels every run of the local backend and waits for its processes. At the start it kills the processes a service before left running when it died. Args: root: The project directory. port: The port on `127.0.0.1`, 0 for a free one. idle_exit_s: Exit after this many seconds without a request and without a live run; None runs until stopped. Set for a service in the background. ready: Called with the service once it listens. stopping: Called with what the end of the service waits for, the open requests and the processes of the runs, when it waits for any. Raises: ServiceRunning: The local service of the directory runs already. RuntimeError: The tokens file has no token of the user. OSError: The port cannot be bound. ## function `stop_local(root: pathlib.Path, idle: bool = False, stopping: collections.abc.Callable[[simrun.local.LocalService], None] | None = None, timeout_s: float = 90.0) -> simrun.local.LocalService | None` Stop the local service of a project directory and wait for its end. The end of the service cancels its runs, whose processes get `worker.GRACE_S`. Args: root: The project directory. idle: Stop it only when no run is live and no request is open. stopping: Called with the service once it stops, before the wait. timeout_s: Seconds to wait for its end. Returns: The service which was stopped, None when none runs. Raises: ServiceBusy: With `idle`, the service has a live run or an open request. RuntimeError: The service refused to stop or did not end in time. ## function `web_dir() -> pathlib.Path | None` The built web app: the one in the package, else the one of a checkout. Returns: `simrun/web` of the installed package if it has an `index.html`, else `frontend/dist` of the checkout of simrun it is imported from (a `pyproject.toml` of the project simrun beside `frontend/`), else None. --- # simrun.mcp The MCP server of simrun, a local stdio process of an agent. The tools go through the `Client`; they return the JSON of the API and `store`, the absolute local prefix of the paths in it, which are relative to the store, so an agent opens a result with `store / path`; it is null without a store mounted here. A refusal is returned as `{"error": detail, "status": code}` instead of raised, so an agent can read it. ## function `build_server(client: simrun.client.Client) -> fastmcp.server.server.FastMCP` The MCP server whose tools call the service through `client`. Every answer carries `store`, the path of the store as mounted locally; it is null for a client without a mounted store: a client of the cluster without `SIMRUN_STORE`, which uploads its snapshots, and a client of the local service without snapshots. ## function `main() -> None` Run the server on stdio with the client of the environment. --- # simrun.planner The planner: the shape of a worker pod and the number of pieces of a task. ## class `Node(name: str, cores: int, memory_gb: float, gpus: int = 0, reserved_by: str | None = None, allocatable_cores: float | None = None, allocatable_memory_gb: float | None = None) -> None` A node of the cluster. Attributes: name: Name of the node. cores: The whole cores a worker pod of a run can have on the node, what the planner plans with. memory_gb: The GiB of memory a worker pod of a run can have on the node, what the planner plans with. gpus: Number of GPUs. reserved_by: Name of the reservation holding the node, None if free. allocatable_cores: The cores the scheduler gives the pods of the runs (sidecars included), as it counts their requests. allocatable_memory_gb: The GiB of memory the scheduler gives the pods of the runs, as it counts their requests. ## class `PlanError` A task which cannot be planned on the nodes. ## class `PodShape(cores: int, memory_gb: int, processes: int, per_worker_gb: float, retry_memory_gb: int, retry_per_worker_gb: float, scratch_gb: float, gpus: int) -> None` The resources of a worker pod. Attributes: cores: Number of cores. memory_gb: Memory in GiB, rounded up. processes: Number of worker processes. per_worker_gb: Memory per worker process in GiB. retry_memory_gb: Memory of a retry in GiB. retry_per_worker_gb: Memory per worker process of a retry in GiB. scratch_gb: Local scratch in GiB. gpus: Number of GPUs. ## class `TaskPlan(task_id: str, pods: int, shape: simrun.planner.PodShape, base_priority: str, extra_priority: str, units: int | None, coordinator_merges: bool = False) -> None` The plan of a task. Attributes: task_id: Id of the task. pods: Number of worker pods. shape: Shape of a pod. base_priority: Priority class of the first pods. extra_priority: Priority class of the further pods. units: Number of units, None for an unsplit task. coordinator_merges: Whether the coordinator of a split task merges once its ledger is complete, so the task has no merge pod. False for plans recorded before the merge moved into the coordinator. ## function `plan_task(task: simrun.workflow.TaskSpec, task_type: simrun.plugins.TaskType, resources: simrun.resources.ResolvedResources, nodes: list[simrun.planner.Node], mode: simrun.workflow.SplitMode, priority: simrun.workflow.Priority, units: int | None, pod_start_s: float = 10.0) -> simrun.planner.TaskPlan` Plans the pods of a task. A pod of a split task has no more processes than the task has units. A split task of the mode `fastest` gets at most `MAX_PIECES` pods, as many as fit on the nodes, and no pod without a unit for each of its processes. With the learned seconds of a unit it gets a further pod only while every pod still has at least `pod_start_s` of work: a pod which takes longer to start than its share of the work would find the units taken by the others and only hold cores. Args: task: The task. task_type: Its type. resources: The resolved resources. nodes: The nodes of the cluster. mode: The split mode. priority: The priority of the run. units: The number of units, None for an unsplit task. pod_start_s: About the seconds a pod takes to start on the cluster. Returns: The plan; extra pods are not limited by idle capacity, Kueue queues them. Raises: PlanError: The task fits on no node. ## function `shape(resources: simrun.resources.ResolvedResources, nodes: list[simrun.planner.Node], max_processes: int | None = None) -> simrun.planner.PodShape` Sizes a pod for the most common node which is not reserved. When one worker does not fit that node, the largest usable node is taken. Args: resources: The resources of the task. nodes: The nodes of the cluster. max_processes: At most this many processes, as many as fit by default. Returns: The shape of a pod. Raises: PlanError: One worker fits on no node. --- # simrun.resources Resources learned from the reports of workers. An observation is the memory per worker process and the seconds of a unit in one process of one report of a worker, keyed by the task type, the model and a variant. A request is resolved from the last `WINDOW` observations of its key: the 95th percentile of the measured memory per worker times `MARGIN`, and at least `RETRY_FACTOR` times the memory per worker of a pod killed for memory, until a later run of the key succeeded (its measurement then says what the key needs). The memory measured is anonymous memory (see `simrun.worker.MemorySampler`), and a worker counts as out of memory only when the backend reports its pod killed for memory. ## class `ResolvedResources(memory_base_gb: float, memory_per_worker_gb: float, scratch_gb: float, gpus: int, seconds_per_unit: float | None, samples: int, source: str) -> None` The resources requested for a task. Attributes: memory_base_gb: Base memory in GiB. memory_per_worker_gb: Memory per worker in GiB. scratch_gb: Local scratch in GiB. gpus: Number of GPUs. seconds_per_unit: Median seconds of a unit in one process, None without records. samples: Number of memory observations the memory was learned from. source: `learned from N runs`, `plugin hint`, `default` or `override`. ## class `ResourceKey(task_type: str, model: str, variant: str = '') -> None` What observations of a task are grouped by. Attributes: task_type: Name of the task type. model: Name of the first source, empty without. variant: A setting the plugin names as cost relevant, empty by default. ### `ResourceKey.text(self) -> str` Returns the key as stored. ## class `ResourceStore(db: pathlib.Path) -> None` Observations of the workers of past tasks in a SQLite file. The file must be on a local volume. A connection is opened per call, so the store is safe to use from several threads of one process. ### `ResourceStore.record(self, key: simrun.resources.ResourceKey, report: simrun.worker.WorkerReport, base_gb: float, oom: bool = False, run: str = '') -> None` Records the report of a worker. Args: key: The key of the task. report: The report. base_gb: The base memory of the task in GiB. oom: Whether the worker was killed for memory (see `simrun.service.controller.is_oom`); the limit is used then. run: The id of the run of the report. ### `ResourceStore.resolve(self, key: simrun.resources.ResourceKey, hint: simrun.plugins.ResourceHint, override: simrun.workflow.ResourceOverride | None) -> simrun.resources.ResolvedResources` Returns the resources to request. Args: key: The key of the task. hint: The hint of the plugin. override: The override of the workflow. Returns: The resolved resources; an override wins over learned values. ## function `resource_key(task_type: simrun.plugins.TaskType, settings: collections.abc.Mapping[str, typing.Any], model: str) -> simrun.resources.ResourceKey` The key of the observations of a task, the one place it is derived. Args: task_type: The type of the task. settings: The settings of the task as submitted. model: The name of the model, the first source of the run, empty without. Returns: The key; its variant is the value of the `cost_setting` of the type (the default of the settings when the task does not set it), text as is and other values as JSON, empty for a type without one. --- # simrun.ledger The unit ledger: workers of a split task claim units through atomic renames. ## class `Ledger(root: pathlib.Path, n_units: int) -> None` A directory with `pending/`, `claimed/@` and `done/`. Workers touch `heartbeats/`, so that the claims of a worker which died without releasing them are found by the age of its heartbeat (`release_stale`). Every transition is one `os.rename`, atomic on a local file system and on NFS v4, so a unit is held by at most one worker. On NFS the mount must be NFS >= 4.1 (sessions give renames exactly-once semantics; `claim` also tolerates a resent rename that answers ENOENT although it happened). ### `Ledger.claim(self, worker: str) -> int | None` Claim the lowest pending unit, or return `None` when none is pending. ### `Ledger.claims(self) -> dict[str, list[int]]` The claimed units by worker id, the indices sorted. ### `Ledger.complete(self, index: int, worker: str) -> None` Mark a unit claimed by `worker` as done. ### `Ledger.complete_ledger(self) -> bool` Whether every unit is done. ### `Ledger.counts(self) -> simrun.ledger.LedgerCounts` The number of units per state. ### `Ledger.heartbeat(self, worker: str) -> None` Touch the heartbeat file of `worker`, created with its directory lazily. Its modification time is set by the file server, so the heartbeats of workers on different servers compare without their clocks. Raises: OSError: The time of the file was not set. ### `Ledger.missing(self) -> list[int]` The units not done, sorted. ### `Ledger.release(self, index: int, worker: str) -> None` Return a unit claimed by `worker` to pending. ### `Ledger.release_all(self, worker: str) -> list[int]` Free every unit claimed by `worker`, as a restarted worker does. Returns: The indices released, sorted. ### `Ledger.release_stale(self, stale_s: float, reference: str) -> list[int]` Return the claims of workers without a recent heartbeat to pending. The heartbeat of `reference` is touched and its time is "now"; a claim of another worker which has no heartbeat or one older than `now - stale_s` is released. The claims of `reference` are never released. Args: stale_s: Seconds after which a heartbeat is stale. reference: The worker which releases, e.g. the coordinator. Returns: The indices released, sorted. Raises: OSError: The heartbeat of `reference` was not touched. ## class `LedgerCounts(pending: int, claimed: int, done: int, total: int) -> None` The number of units per state. --- # simrun.worker The worker harness: a pod of a split task computes units of the ledger in a pool. `simrun-worker` (`main`) claims units of the ledger of its task, computes them in a local process pool with the functions of the task type and writes a report; `simrun-merge` (`merge_main`) merges the units once the ledger is complete; `simrun-exec` (`exec_main`) runs the command of an unsplit plugin task and writes a report of it. A pod is evicted with SIGTERM and killed after its grace period, and is restarted with the same worker id, which releases the claims of the incarnation before. ## class `ConfigurationError` The environment or the settings of a worker are wrong. ## class `MemorySampler(source: MemorySource = 'cgroup') -> None` The maximum of the anonymous memory of the worker over its samples. The anonymous memory is what the processes allocated: of the cgroup (`memory.stat` `anon`), or of the process tree of the harness (`process`, see `process_tree_anon`). `memory.peak` also counts the page cache of staging, installing and writing outputs, which the kernel reclaims at the limit instead of killing. Attributes: source: Where the memory is read. peak: The largest sample in bytes, None before one was taken. ### `MemorySampler.sample(self) -> None` Read the anonymous memory and keep it when it is the largest. ## class `Terminated` The worker stopped on SIGTERM before its units were done. ## class `UnitError(index: int) -> None` A unit failed in the function of the task type; it was released. ## class `WorkerReport(worker: str, node: str, processes: int, units_done: list[int] = , seconds: float = 0.0, peak_memory_bytes: int | None = None, memory_limit_bytes: int | None = None, per_worker_gb: float = 0.0, oom_kills: int = 0, unit_seconds: float = 0.0, failed: bool = False, started: float | None = None, prepare_seconds: float = 0.0, cpu_seconds: float = 0.0, throttled_seconds: float | None = None) -> None` What a worker did, written as `task_dir/workers/.json`. Attributes: worker: The worker id. node: The server the worker ran on. processes: The number of processes of the pool. units_done: The units the worker completed. seconds: The duration of the worker. peak_memory_bytes: The largest sample of the anonymous memory of the cgroup (`MemorySampler`), without the page cache. memory_limit_bytes: The memory limit of the cgroup. per_worker_gb: The memory per process in GiB the worker was sized with. oom_kills: The processes of its cgroup the out of memory killer killed while it ran (`memory.events` `oom_kill`), the evidence that an exit 137 was a kill for memory. unit_seconds: The seconds of the units it completed, each measured in the process which computed it (the setup of a process in its first unit), summed: `unit_seconds / len(units_done)` is the time of a unit in one process, without the waits of the worker (for the units of other workers, a coordinator) and the processes idle at its end. failed: Whether the worker failed: a unit failed (exit 3), or the command of `simrun-exec` exited with an error; a stop or a kill is no failure. started: When the harness started, in seconds since the epoch, before it installed the sources (`HARNESS_STARTED`); None in a report of simrun 0.3.0 and before. prepare_seconds: The seconds from the start of the harness to the start of its work: the install of the sources, the restart, the imports and the staging of the inputs. cpu_seconds: The user and system seconds of the harness, of the children it reaped (the command of `simrun-exec`) and of the units it completed, each measured in the process of the pool which computed it. throttled_seconds: The seconds the processes of the cgroup of the worker waited for its CPU quota during the work, None without a cgroup of its own (the local backend). ## function `cores(env: collections.abc.Mapping[str, str]) -> int` The cores of a worker: the quota of its cgroup, else the CPUs of the process. `SIMRUN_CORES` caps them, e.g. the cores of the plan of a task on the local backend, which has no cgroup per task. Raises: ConfigurationError: `SIMRUN_CORES` is not a positive integer. ## function `cpu_seconds() -> float` The user and system seconds of this process and of its ended children. ## function `exec_main() -> None` Run `simrun-exec `, the command of an unsplit plugin task. The harness installs the snapshots (`SIMRUN_EDITABLE`), stages the inputs, sets the thread variables of BLAS and OpenMP to 1 and runs the command as its child, while its heartbeat thread samples the memory. A SIGTERM is passed on to the command, which is killed after `GRACE_S`. The report `task_dir/workers/-.json` (the worker is the task id, or `SIMRUN_WORKER`) is written in every case after the command started, so the controller learns the memory of the task. Exit codes: 0 done, 3 the command failed, 4 a configuration error (also a command which cannot be started), 137 the command was killed (by the out of memory killer, retried with more memory) or exited 137, 143 stopped on SIGTERM (retried). ## function `harness_started() -> float` When the harness started, set in `HARNESS_STARTED` by its first call. The variable survives the restart after the install of the sources, so the start is the one before the install. Returns: Seconds since the epoch. ## function `install_editable(snapshots: list[str], python: str = '/home/runner/work/simrun/simrun/.venv/bin/python', root: pathlib.Path | None = None) -> None` Install snapshots of sources as editable packages, without dependencies. The image has the dependencies and the build backends of the sources (see `simrun.environment`), so the snapshots are built without isolation and without an index. A snapshot is mounted read-only: each is installed from a writable shadow tree (`shadow_tree`) below `root`, whose files are links to the snapshot. Args: snapshots: The paths of the snapshots; nothing is installed without one. python: The interpreter of the environment to install into. root: The directory of the shadow trees, `SOURCES_DIR` below the temporary directory by default. Raises: subprocess.CalledProcessError: The installation failed. OSError: A shadow tree cannot be created. ## function `install_sources(env: collections.abc.Mapping[str, str], stop: threading.Event) -> None` Install the snapshots of `SIMRUN_EDITABLE`, then restart in a fresh interpreter. A source can be a package this process imported already, simrun itself when a run has the simrun repository as a source: its editable install removes the package directory the imported modules came from, so a later import in this interpreter fails. The command therefore starts again (`restart`) with `SOURCES_INSTALLED` set, and the new interpreter imports the installed sources. A SIGTERM during the install (`stop`) skips the restart, so the caller exits. Args: env: The environment of the pod. stop: Set on SIGTERM. Raises: subprocess.CalledProcessError: The installation failed. OSError: A shadow tree cannot be created. ## function `main() -> None` Run `simrun-worker` in a pod of a split task; exits with its code. SIGTERM is handled from the start, so a pod evicted while it prepares (as PID 1 without a handler it would ignore SIGTERM) stops before its first claim. The worker of `SIMRUN_ROLE=base` is the coordinator, which returns once the ledger is complete (stale claims released after `SIMRUN_STALE_S`); `extra` (the default) returns when nothing is pending. Exit codes: 0 done, 3 a unit failed (logged and released), 4 a configuration error, 137 a process of the pool died (retried with more memory), 143 stopped on SIGTERM (retried). ## function `memory_limit(env: collections.abc.Mapping[str, str]) -> int | None` The memory limit of a worker in bytes: the one of its cgroup. Without a limit of the cgroup, or with the memory of the process tree (`SIMRUN_MEMORY_SOURCE=process`, whose cgroup is not the worker's), it is `SIMRUN_MEMORY_LIMIT_GB`, the memory of the plan of the task, by which the harness sizes its pool; None without either. Raises: ConfigurationError: `SIMRUN_MEMORY_LIMIT_GB` or `SIMRUN_MEMORY_SOURCE` is invalid. ## function `memory_source(env: collections.abc.Mapping[str, str]) -> MemorySource` Where the harness reads its memory, `SIMRUN_MEMORY_SOURCE`, `cgroup` by default. Raises: ConfigurationError: The value is neither `cgroup` nor `process`. ## function `merge_main() -> None` Run `simrun-merge` after the workers of a split task; exits with its code. SIGTERM is handled from the start: a merge stopped before it began exits 143. A successful merge writes `out/.simrun-merged` (`MERGED_MARKER`). Exit codes: 0 merged, 3 the ledger is incomplete or the merge failed, 4 a configuration error, 143 stopped on SIGTERM before the merge. ## function `paths_from_env(env: collections.abc.Mapping[str, str]) -> simrun.plugins.TaskPaths` The locations of a task from the environment of its pod. The task directory is `/runs//tasks/` (`SIMRUN_TASK_DIR`), the inputs are `port=path` pairs (`SIMRUN_INPUTS`, a relative path is relative to the store) and the scratch directory is `SIMRUN_SCRATCH` (default `/scratch`). Args: env: The environment. Returns: The locations. Raises: ConfigurationError: `SIMRUN_TASK_DIR` is not set or a pair is malformed. ## function `process_tree_anon(pid: int, proc: pathlib.Path = PosixPath('/proc')) -> int | None` The anonymous memory of a process and all of its descendants in bytes. The sum over the processes of `RssAnon` (`VmRSS` without it): pages shared after a fork count once per process, so it is an upper bound. Args: pid: The root of the tree. proc: The mount of procfs. Returns: The bytes, None when the process is gone. ## function `processes(cores: int, memory_bytes: int | None, base_gb: float, per_worker_gb: float) -> int` The number of processes the cores and the memory allow. Args: cores: The number of cores. memory_bytes: The memory limit in bytes, `None` without a limit. base_gb: The memory of the task besides its processes in GiB. per_worker_gb: The memory per process in GiB. Returns: `min(cores, floor((memory - base) / per_worker))`, at least 1. Raises: ValueError: Not even one process fits into the memory. ## function `restart() -> None` Replace the process with a new interpreter of the same command line. ## function `run_merge(task_type: simrun.plugins.TaskType, settings: pydantic.main.BaseModel, paths: simrun.plugins.TaskPaths) -> None` Merge the units of a task once its ledger is complete. Args: task_type: The task type, splittable. settings: The settings of the task. paths: The locations of the task. Raises: ValueError: The task type is not splittable. RuntimeError: Units are not done; the message names them. ## function `run_worker(task_type: simrun.plugins.TaskType, settings: pydantic.main.BaseModel, paths: simrun.plugins.TaskPaths, worker: str, *, cores: int, memory_bytes: int | None, base_gb: float, per_worker_gb: float, grace_s: float = 60.0, attempt: int = 0, stop: threading.Event | None = None, heartbeat_s: float = 30.0, coordinator: bool = False, stale_s: float | None = None, poll_s: float = 0.5, memory_source: MemorySource = 'cgroup') -> simrun.worker.WorkerReport` Compute units of the ledger of a task until none is pending. The worker touches its heartbeat in the ledger every `heartbeat_s` from its start to its end. A coordinator does not return when nothing is pending: until the ledger is complete it waits, releases the claims of workers whose heartbeat is older than `stale_s` every `poll_s` and computes them. Releases the claims of an earlier incarnation of `worker`, creates the ledger `task_dir/units`, keeps `processes(...)` units in flight in a pool of processes which run the setup of the task type once, completes a unit after its function returned and appends a `unit_done` line to `task_dir/progress.jsonl`. On SIGTERM (or when `stop` is set, also before the first claim) it stops claiming, waits for the running units up to `grace_s` and releases the rest. The report `task_dir/workers/-.json` is written in every case. Args: task_type: The task type, splittable. settings: The settings of the task. paths: The locations of the task. worker: The worker id, the same for every incarnation of a pod. cores: The number of cores. memory_bytes: The memory limit in bytes, `None` without a limit. base_gb: The memory of the task besides its processes in GiB. per_worker_gb: The memory per process in GiB. grace_s: Seconds the running units get after SIGTERM. attempt: The attempt of the pod, which names its report. stop: Set on SIGTERM by a caller which handles it already, e.g. `main`. heartbeat_s: Seconds between two heartbeats. coordinator: Whether the worker continues until the ledger is complete. stale_s: Seconds after which the heartbeat of another worker is stale, `grace_s + 60` by default. poll_s: Seconds between two releases of stale claims by a coordinator. memory_source: Where the memory of the report is read, see `MemorySampler`. Returns: The report. Raises: ValueError: The task type is not splittable, the memory is too small for one process, the worker id is invalid or the ledger has another size. UnitError: A unit failed; it was released. BrokenProcessPool: A process of the pool died, e.g. killed for its memory. Terminated: The worker stopped on SIGTERM. ## function `shadow_tree(source: pathlib.Path, target: pathlib.Path) -> pathlib.Path` A writable tree of a read-only one: its directories, links to its files. A build backend may write into the tree it builds (a version file, metadata, a build directory); it writes into the writable directories of the shadow, and never into the source. A link (to a file or a directory) of the source is a link to it in the shadow. Args: source: The read-only tree. target: The shadow to create, replaced when it exists. Returns: `target`. ## function `stage(pairs: list[tuple[str, str]], store: pathlib.Path, scratch: pathlib.Path) -> None` Copy inputs to the scratch directory. Args: pairs: `(source, destination)`; a relative source is relative to the store, a relative destination to the scratch directory. store: The root of the run store. scratch: The scratch directory. ## function `write_report(report: simrun.worker.WorkerReport, task_dir: pathlib.Path, attempt: int = 0) -> pathlib.Path` Write the report as `task_dir/workers/-.json`. The file is written as a temporary file and renamed. Args: report: The report. task_dir: The task directory. attempt: The attempt of the pod. Returns: The path of the report. --- # simrun.cgroup The limits and the usage of the cgroup (v2) a worker runs in. ## function `cpu_quota(root: pathlib.Path = PosixPath('/sys/fs/cgroup')) -> int | None` The number of cores the CPU quota of `cpu.max` allows. Args: root: The directory of the cgroup. Returns: The quota over the period, rounded down and at least 1, or `None` without a quota or without the file. ## function `cpu_throttled_seconds(root: pathlib.Path = PosixPath('/sys/fs/cgroup')) -> float | None` The seconds the processes of the cgroup waited for its CPU quota. `throttled_usec` of `cpu.stat`: time a process could have run but the quota of `cpu.max` held it back, the evidence that a task needs more cores. Returns: Seconds, `None` without the file or the field. ## function `memory_anon(root: pathlib.Path = PosixPath('/sys/fs/cgroup')) -> int | None` The anonymous memory of `memory.stat` (`anon`) in bytes. The memory the processes allocated, without the page cache which `memory.peak` counts and the kernel reclaims at the limit instead of killing. Returns: The bytes, `None` without the file or the field. ## function `memory_limit(root: pathlib.Path = PosixPath('/sys/fs/cgroup')) -> int | None` The memory limit of `memory.max` in bytes, `None` without a limit. ## function `memory_oom_kills(root: pathlib.Path = PosixPath('/sys/fs/cgroup')) -> int | None` The processes of the cgroup the out of memory killer killed (`oom_kill`). The counter of `memory.events`, the evidence of a kill for memory: a process which exits 137 may have been killed by anyone with SIGKILL. Returns: The count, `None` without the file or the field. --- # simrun.snapshot Snapshots of working trees with provenance. A snapshot is a copy of the files git knows of a working tree (tracked and untracked, not ignored, uncommitted changes included) in the run store. A snapshot is made in two steps: `scan` lists the files of a working tree with their modes and content hashes, `assemble` writes them into the store; `snapshot` does both on the machine of the store, the service does the second for an upload of a client (`check_scan`, `missing`, `upload`). Next to every snapshot `.files.json` maps its paths to the hashes of their contents, so files whose content the latest snapshot of the same name has are hard links to it (copies when their modes differ) and an upload sends only the others. The files of a snapshot are read-only (`a-w`, part of the hash of its content): a snapshot is named by its content and never changes, and a build backend which writes through a link of a shadow tree into one fails loudly instead. Submodules are out of scope: their content is not copied. ## class `FileEntry(*, path: str, mode: int, sha256: str | None, link: str | None) -> None` A file of a working tree as a snapshot holds it. Attributes: path: Path below the tree, with `/`. mode: Mode bits without the write bits (a snapshot is read-only), special bits kept as release 0.1.0 kept them; the ones of the link for a symbolic link. sha256: Hash of the content of a regular file, None for a link. link: Target of a symbolic link, None for a regular file. ## class `MissingContent` A file of an upload whose content neither the archive nor the store has. The latest snapshot changed since the client asked what is missing; the client asks again. ## class `Scan(*, branch: str | None, commit: str, dirty: bool, changed_files: list[str], patch: bytes, files: list[simrun.snapshot.FileEntry], dirs: dict[str, int] = {}) -> None` What a snapshot of a working tree is made of, before it is copied. Attributes: branch: Branch of the working tree, `None` for a detached HEAD. commit: Commit of HEAD. dirty: Whether the working tree differs from HEAD. changed_files: Porcelain status lines, e.g. `M src/a.py`. patch: `git diff HEAD --binary`, base64 in JSON. files: The files of the snapshot, in the order of git. dirs: The mode bits of the directories which hold the files, by path; the snapshot keeps them as release 0.1.0 did, with the owner bits added. ## class `SnapshotError` An upload which cannot become a snapshot. A path outside the tree, a member of the archive which is not a listed file, or a hash which does not match. ## class `SourceProvenance(*, name: str, branch: str | None, commit: str, dirty: bool, changed_files: list[str], diff_sha256: str, snapshot: str, patch: str | None, path: str | None = None) -> None` Where the files of a snapshot come from. Attributes: name: Name of the source. branch: Branch of the working tree, `None` for a detached HEAD. commit: Commit of HEAD. dirty: Whether the working tree differs from HEAD. changed_files: Porcelain status lines, e.g. `M src/a.py`. diff_sha256: Hash of the snapshot tree (paths, types, modes, contents); the patch stays as provenance. snapshot: Directory of the snapshot, relative to the store root. patch: Stored `git diff HEAD --binary`, relative to the store root. path: The working tree of a source recorded without a snapshot (the local mode), None for a snapshot. ## function `assemble(store: pathlib.Path, name: str, scan: simrun.snapshot.Scan, files: collections.abc.Callable[[str], pathlib.Path | bytes], move: bool = False) -> simrun.snapshot.SourceProvenance` Write a scanned tree into `store/snapshots/`. The files whose content the latest snapshot of the name has are hard links to it (copies when their modes differ), the others come from `files`: the file to copy or the content. A snapshot is assembled in a temporary directory, its directories get the modes of the scan, it is named by its content and renamed into place; one which exists already is kept. Args: store: Root of the run store. name: Name of the source. scan: The files and the git state of the tree. files: The content of a path of `scan`, a file or bytes; a file which is gone (`FileNotFoundError`) is left out of the snapshot. move: The files of `files` are given up: renamed into the snapshot instead of copied (on the file system of the store), their content trusted to have the hash of the scan. Returns: The provenance of the snapshot. ## function `check_files(files: list[simrun.snapshot.FileEntry]) -> None` Refuse a file list which would write outside the tree of a snapshot. Raises: SnapshotError: For a path which is absolute, not normalized, has a NUL or a component `..` or `.git`, which is listed twice or below another entry; a regular file without a hash or with write or special mode bits; a link whose target is absolute or leaves the tree. ## function `check_name(name: str) -> None` Refuse a source name which is not a directory name of `snapshots/`. Raises: SnapshotError: For a name of other characters than letters, digits, `-`, `_` and `.`, or one which does not start with a letter or digit. ## function `check_scan(scan: simrun.snapshot.Scan) -> None` Refuse a scan which cannot become a snapshot, see `check_files`. Raises: SnapshotError: For a file list `check_files` refuses, a commit which is not a hexadecimal hash, or a directory which is not one of the tree, is also a file, or has other mode bits than permissions and setgid. ## function `manifest_path(snapshot_dir: pathlib.Path) -> pathlib.Path` The manifest of a snapshot: its paths and the hashes of their contents. ## function `missing(store: pathlib.Path, name: str, files: list[simrun.snapshot.FileEntry]) -> list[str]` The paths of regular files whose content the latest snapshot lacks. Args: store: Root of the run store. name: Name of the source. files: The files of a scan, checked with `check_files`. ## function `provenance(repo: pathlib.Path, name: str) -> simrun.snapshot.SourceProvenance` The provenance of a working tree without copying it. The same git calls as `snapshot`; `diff_sha256` is the hash a snapshot of the tree would have, `snapshot` is empty and there is no patch. Args: repo: A git working tree. name: Name of the source. Returns: The provenance, with `snapshot=""`, `patch=None` and the absolute `path` of the working tree. ## function `scan(repo: pathlib.Path) -> simrun.snapshot.Scan` List the files of a working tree with their modes and content hashes. A file which is gone between the listing of git and its hash is left out. Args: repo: A git working tree (a worktree with a `.git` file works too). Returns: The scan of the tree. Raises: RuntimeError: When git fails. ## function `snapshot(repo: pathlib.Path, store: pathlib.Path, name: str) -> simrun.snapshot.SourceProvenance` Copy a working tree into `store/snapshots/`. Args: repo: A git working tree (a worktree with a `.git` file works too). store: Root of the run store. name: Name of the source. Returns: The provenance of the snapshot. Raises: RuntimeError: When git fails. ## function `upload(store: pathlib.Path, name: str, scan: simrun.snapshot.Scan, archive: IO[bytes]) -> simrun.snapshot.SourceProvenance` Write a snapshot from a scan and a tar of the files the store lacks. Every member of the tar is a regular file of the scan whose content has the hash of the scan; it is extracted, under a name of its own, into a temporary directory beside the snapshots of the name, and the snapshot assembled from it (its files renamed into place) and the latest snapshot. Args: store: Root of the run store. name: Name of the source, checked with `check_name`. scan: The files and the git state of the tree, checked with `check_scan`. archive: The tar, read once from its start. Returns: The provenance of the snapshot. Raises: SnapshotError: For a member which is not a regular file, not a file of the scan or sent twice, or whose content does not match its hash; or a tar which cannot be read. MissingContent: For a file whose content neither the tar nor the latest snapshot has. --- # simrun.environment The environment of a run: a lock from the model and its snapshotted libraries. The image of a run holds only the third party dependencies. The code of the model repository and of the libraries comes from snapshots which the pod mounts read-only and installs editable at start (`SIMRUN_EDITABLE`, see `simrun.worker`), so a change of the code never rebuilds an image and a change of a dependency does. A pod installs the snapshots without build isolation and without an index, so the image holds their build backends: the `[build-system].requires` of the model and of every library (the legacy backend of setuptools without a `[build-system]`, PEP 517) and what the backend asks for an editable build (`get_requires_for_build_editable`, PEP 660, e.g. `editables` of hatchling), which a hook asks at resolve time in an environment of `uv run` with the static requirements, in a shadow tree of the snapshot. They are dependencies of the temporary project, so the lock pins them with the third party dependencies and the image installs them. Their versions are the pins of a constraints file of the service (`build_constraints.txt`, `BUILD_CONSTRAINTS`), for the packages the `uv.lock` of the model does not have, so a release of a backend changes no lock. The lock is resolved in a temporary project, a writable shadow tree of the model (`simrun.worker.shadow_tree`: its directories, links to its files) with its `pyproject.toml` changed and a copy of its `uv.lock`, and a directory `.simrun-store` with a shadow tree of every library at `.simrun-store/`. `uv lock` builds the metadata of a source with dynamic metadata, which a backend writes next to the sources (e.g. the `*.egg-info` of setuptools): into the shadows, never into the snapshots, which are named by the hash of their content. The sources of the libraries are the relative paths `.simrun-store/`, so the lock carries these paths and not the place the store is mounted at (`/mnt/cluster` on the development server, `/store` in the service): the same snapshots give the same lock and hash anywhere. ## class `Environment(lock: str, lock_sha256: str, python: str, simrun_version: str, editable: list[str], requirements: str = '', build_requirements: tuple[str, ...] = ()) -> None` The resolved environment of a run. Attributes: lock: Text of the `uv.lock`. lock_sha256: Hash of the lock and the python version, the identity of the image. python: Python version of the image. simrun_version: Version of the simrun wheel the image installs, with the hash of the wheel when the service has one. editable: Snapshot directories of the model and the libraries, relative to the store in the order of the sources, installed at pod start. requirements: The third party dependencies of the lock as pinned requirements, without the model and the snapshotted libraries. build_requirements: The requirements of the build backends of the sources, which the lock pins with the dependencies. ## function `build_requirements(directory: pathlib.Path, name: str, python: str, pins: dict[str, str] | None = None) -> list[str]` What building a source editable needs: its build system and its backend's hook. Args: directory: The snapshot of the source, with its `pyproject.toml`. name: The name of the source, for messages. python: The python version the hook runs with. pins: The pinned versions of build backends (`load_constraints`), applied to the environment the hook runs in. Returns: `[build-system].requires` and the result of the backend's `get_requires_for_build_editable`. Raises: RuntimeError: The backend cannot be loaded or its hook fails. ## function `dockerfile(env: simrun.environment.Environment, base_image: str, wheel: str, system_packages: collections.abc.Sequence[str] = ()) -> str` Dockerfile of the image: the dependencies, build backends and `simrun`, no source. The packages are installed with their bytecode compiled, so a pod does not compile the modules it imports, and with the downloads of uv in a cache of BuildKit (`UV_CACHE`), so a build after a change of a dependency downloads only what changed. `simrun` is the last layer: a new harness rebuilds only it. The pinned requirements are written with a heredoc, so no source is copied from the build context. The model and the libraries are installed at pod start. Args: env: The environment. base_image: The base image, with uv and the python of the environment, in the registry of the cluster (`ClusterConfig.base_image_ref`). wheel: File name of the simrun wheel in the build context, the only file copied; the harness of the service, never installed by name. system_packages: Debian packages installed before the requirements, the shared libraries their wheels link (`ClusterConfig.system_packages`); `git` as well when a requirement is locked from a git repository. ## function `image_ref(env: simrun.environment.Environment, registry: str, dockerfile_text: str) -> str` Reference of the image of an environment in the registry. Args: env: The environment. registry: The registry of the cluster. dockerfile_text: The Dockerfile of the image (`dockerfile`); with the lock it is the identity of the image, so another base image or other system packages give another image instead of the one built before. ## function `load_constraints(path: pathlib.Path) -> dict[str, str]` The pins of a constraints file by normalised name, e.g. `hatchling==1.33.0`. Raises: ValueError: A line is no exact pin `==`. ## function `normalise(name: str) -> str` Normalised name of a package (PEP 503), as `uv.lock` writes it. ## function `resolve_environment(store: pathlib.Path, model: simrun.snapshot.SourceProvenance, libraries: list[simrun.snapshot.SourceProvenance], python: str = '3.14', *, simrun_version: str, build_constraints: pathlib.Path = PosixPath('/home/runner/work/simrun/simrun/src/simrun/build_constraints.txt')) -> simrun.environment.Environment` Lock the dependencies of the model with its libraries as snapshot paths. Args: store: Root of the run store. model: Snapshot of the model repository, which has the `pyproject.toml`. libraries: Snapshots of the libraries; each replaces the entry of `[tool.uv.sources]` of its package name, the other entries are kept. python: Python version of the image. simrun_version: Version of the simrun wheel of the image, part of the hash (the service adds the hash of the wheel, `Submitter._harness`). build_constraints: The pinned versions of the build backends; a pin of a package the `uv.lock` of the model does not have constrains the lock. Returns: The environment. Raises: ValueError: A snapshot has no `pyproject.toml`, the model has no `uv.lock` (it would resolve against the live index and the hash would drift), or the model depends on a local package which is not a snapshotted source. RuntimeError: `uv` fails to lock, or a build backend cannot tell what an editable build needs. --- # simrun.store The run store: run records and the layout of a run on the shared drive. Layout: `/runs//tasks/`, the record in `/runs//run.json`. ## class `RunRecord(*, id: str, submitted_by: str, owner: str, agent: str | None, priority: simrun.workflow.Priority, group: str | None, workflow: simrun.workflow.Workflow, sources: list[simrun.snapshot.SourceProvenance], image: str | None, reserve: str | None, state: simrun.store.RunState, tasks: dict[str, simrun.store.TaskState], plans: dict[str, typing.Any], backend_name: str | None, created: datetime.datetime, finished: datetime.datetime | None, message: str | None, built: datetime.datetime | None = None, oom_kills: dict[str, list[int]] = , merge_pending: dict[str, datetime.datetime] = , pending_release: str | None = None, pending_cancel: bool = False, pending_logs: bool = False, local_commands: Literal['direct', 'docker'] | None = None) -> None` The record of a run. Attributes: id: Run id, `YYYYMMDD-HHMMSS-<6 hex>`. submitted_by: Submitting user. owner: Owner of the run. agent: Agent which submitted the run. priority: Priority in the queue. group: Group of the run. workflow: The workflow. sources: Provenance of the snapshots of the sources. image: Container image. reserve: Reservation. state: State of the run. tasks: State of every task. plans: The task plans as dicts. backend_name: Name of the backend workflow. created: Time of creation. built: Time the image of the run was ready and the submission of its workflow began, None while its environment is resolved and built. finished: Time of the end. message: A message, e.g. the failure. oom_kills: The attempts of the pods killed for memory per worker (the worker id, the task id of an unsplit task, `-merge` of a merge). merge_pending: The time the service first saw the coordinator of a split task which merges succeeded without the marker of its merge, by task id; the task waits for the marker until `MERGE_MARKER_GRACE_S` of `simrun.service.controller` after it. pending_release: A reserved node whose release failed, retried by the service. pending_cancel: Whether the workflow must still be cancelled, e.g. one which may have been created by a submission that failed. pending_logs: Whether the logs of the pods of an ended run must still be kept in the store (`simrun.service.logs`). local_commands: How the local backend runs the `command` tasks of the run: `direct` its args, `docker` in its image; None the choice of the backend. The Argo backend ignores it. ## class `RunState(*values)` State of a run. ## class `RunStore(root: pathlib.Path) -> None` The run store on the shared drive. ### `RunStore.iter_records(self) -> collections.abc.Iterator[simrun.store.RunRecord]` Iterate over the records of all runs, ordered by id. ### `RunStore.new_run_id(self) -> str` Return a new run id, `YYYYMMDD-HHMMSS-<6 hex>`. ### `RunStore.read(self, run_id: str) -> simrun.store.RunRecord` Read a record. Raises: KeyError: When the run does not exist. ### `RunStore.run_dir(self, run_id: str) -> pathlib.Path` Return the directory of a run. ### `RunStore.task_paths(self, run_id: str, task_id: str, inputs: dict[str, pathlib.Path], scratch: pathlib.Path) -> simrun.plugins.TaskPaths` Return the locations of a task of a run. ### `RunStore.write(self, record: simrun.store.RunRecord) -> None` Write a record atomically (temporary file and rename). ## class `RunSummary(*, id: str, state: simrun.store.RunState, group: str | None, submitted_by: str, agent: str | None, priority: simrun.workflow.Priority, tasks: dict[str, simrun.store.TaskState], created: datetime.datetime, finished: datetime.datetime | None, message: str | None) -> None` The state of a run without its workflow and plans. ## class `TaskState(*values)` State of a task of a run. --- # simrun.config The configuration of the cluster the runs are submitted to. ## class `ClusterConfig(*, namespace: str = 'simrun', local_queue: str = 'simrun', store_claim: str = 'simrun-store', store_mount: str = '/store', scratch_path: str = '/scratch', registry: str, registry_url: str | None = None, service_account: str = 'simrun-workflow', buildkit: str = 'tcp://buildkitd.simrun:1234', buildctl_image: str = 'moby/buildkit:v0.34.2-rootless', base_image: str = 'astral-sh/uv:0.13.7-python3.14-trixie-slim@sha256:e80c05a310400cfc90af00b5b7271ef9ad24816e3265e1162af8a678e1aad099', system_packages: list[typing.Annotated[str, StringConstraints(strip_whitespace=None, to_upper=None, to_lower=None, strict=None, min_length=None, max_length=None, pattern='^[a-z0-9][a-z0-9+.-]+$', ascii_only=None)]] = ['libexpat1'], grace_s: int = 60, ttl_after_finished_s: int = 86400, pod_start_s: Annotated[float, Gt(gt=0)] = 10.0) -> None` Where and how the pods of a run are placed in the cluster. Attributes: namespace: Namespace of the workflows and their pods. local_queue: Kueue LocalQueue of the pods. store_claim: PersistentVolumeClaim of the NFS run store. store_mount: Path the run store is mounted at in a pod. scratch_path: Local scratch, a hostPath on the local disk of a node, mounted at the same path in a pod. registry: Host of the image registry inside the network, as the nodes and BuildKit name it in image references. registry_url: URL of the API of the registry as the service reaches it, `http://` by default. service_account: Service account of the pods of the workflows, allowed to write the results of the Argo executor. buildkit: Address of the BuildKit daemon the builds of images run on. buildctl_image: Image of the job which sends a build to BuildKit. base_image: The base image of the images of the environments, an image with uv and the python of the service (`ServiceSettings.python`); a repository of `registry` (the mirror of an image of another registry) unless it names a registry host itself. system_packages: The Debian packages the images of the environments install on the base image: the shared libraries which wheels link but do not bundle, `libexpat1` for python-libsbml. grace_s: Seconds a worker has to stop after SIGTERM. ttl_after_finished_s: Seconds a finished workflow is kept. pod_start_s: About the seconds from the creation of a pod to its first unit; a split task gets a further pod only while every pod has at least this much work (`simrun.planner.plan_task`). ### `ClusterConfig.base_image_ref(self) -> str` The base image of the environments as the builds name it. `base_image` below `registry`, unless its first component is a registry host (it has a `.` or a `:`, or is `localhost`). --- # simrun.podspec The parts of a pod which every backend shares: image, shapes, command, environment. `simrun.argo` and `simrun.cwl` run the same commands with the same environment; they differ in where the run store and the scratch directory are (a mount, a path) and in how a worker is named. ## function `env_values(record: simrun.store.RunRecord, task: simrun.workflow.TaskSpec, task_type: simrun.plugins.TaskType, shape: simrun.planner.PodShape, store: str, scratch: str) -> dict[str, str]` The environment of the worker harness, see `simrun.worker`. Args: record: The run. task: The task. task_type: Its type. shape: The pod shape. store: The root of the store as the pod sees it. scratch: The scratch directory of the pod. Returns: The variables by name, `PYTHONPATH` the root of the model (`import_root`); the caller adds the ones of its backend. Raises: ValueError: The task stages a port which is no input. ## function `image(record: simrun.store.RunRecord, task: simrun.workflow.TaskSpec, task_type: simrun.plugins.TaskType) -> str` The image of a task: the one of its type, else the one of the run. Raises: ValueError: The type has no image and the run has none. ## function `import_root(record: simrun.store.RunRecord, store: str) -> str | None` The root of the model, which a task imports from first. As `python -m` in a checkout puts the checkout first on the path, a task of a model imports from the root of the model, e.g. `examples` at the root of a checkout of sbmlsim, which no installed package has. Args: record: The run, its first source the model. store: The root of the store as the pod sees it. Returns: The snapshot of the model, else its recorded working tree, None for a run without sources. ## function `merge_shape(shape: simrun.planner.PodShape, base_gb: float) -> simrun.planner.PodShape` The merge: one core and the memory of one worker process besides the base. ## function `number(value: float) -> str` A number of GiB for the environment, e.g. `0.5`. ## function `output(record: simrun.store.RunRecord, ref: str) -> str` The output a reference `.` names, relative to the store. ## function `task_dir(record: simrun.store.RunRecord, task_id: str, store: str) -> str` The directory of a task below the store root `store`. ## function `unsplit_argv(record: simrun.store.RunRecord, task: simrun.workflow.TaskSpec, task_type: simrun.plugins.TaskType, store: str, scratch: str) -> list[str]` The argv of an unsplit task, without its image. The command of a plugin type runs through `simrun-exec`, which installs the snapshots, sets the thread variables and writes a report; a `command` task runs its args. In the command template of a type and in the args of a `command` task, `{settings}` is the path of the settings file and `{out}` the output directory of the task, and a part with `{input}` is repeated once per input port of the task, sorted by port, with `{input}` replaced by `=`, the output the port reads as the pod sees it, or `/` for a staged port of a plugin type, the copy `simrun-exec` stages; without inputs the part is dropped. The local backend runs it without an image, which a run there does not have. Args: record: The run. task: The task. task_type: Its type. store: The root of the store as the pod sees it. scratch: The scratch directory of the pod, as in `env_values`. Returns: The argv. Raises: ValueError: The type is neither splittable nor has a command. ## function `unsplit_command(record: simrun.store.RunRecord, task: simrun.workflow.TaskSpec, task_type: simrun.plugins.TaskType, store: str, scratch: str) -> tuple[str, list[str]]` The image and argv of an unsplit task. The image of a `command` task is the one of its settings, the image of a plugin type the one of `image`; the argv is `unsplit_argv`. Args: record: The run. task: The task. task_type: Its type. store: The root of the store as the pod sees it. scratch: The scratch directory of the pod, as in `env_values`. Returns: The image and the argv. Raises: ValueError: The type is neither splittable nor has a command, or it has no image and the run has none. --- # simrun.argo Argo workflows from runs: one DAG, a scatter of workers per split task. A split task of `pods` pods is three DAG tasks of one worker template running `simrun-worker`: `-w0`, the coordinator (worker `-w0`, `SIMRUN_ROLE=base`, base priority), which returns once the ledger of the task is complete; `-extra` over `withSequence: {start: 1, count: pods - 1}` (workers `-w`, `SIMRUN_ROLE=extra`, extra priority; left out for one pod), which exit when nothing is pending. The coordinator merges the units itself once the ledger is complete (`SIMRUN_MERGE=1`, `TaskPlan.coordinator_merges`; only the coordinator acts on it), so extra workers still waiting for admission never hold up the merge. A plan recorded before that has a node `-merge` running `simrun-merge`, which depends on `-w0` only. An unsplit task is one pod running its command. A task waits for the tasks it reads from, a split one for their merge (its `-w0` when that merges). The extra workers continue on failure (`continueOn: {failed: true, error: true}`), so an extra worker which fails does not fail the run; the coordinator releases the claims of workers whose heartbeat is older than `SIMRUN_STALE_S` (`grace_s + 60`) and computes them. The workers carry the input parameters `worker`, `priority`, `task` and `role`. Extra workers may still be pending or running after the last merge of a run: the controller stops the whole workflow then and records the state of the run itself. Every pod carries the Kueue label of the LocalQueue, the WorkloadPriorityClass label and the Kubernetes `priorityClassName` (the coordinator, the merge and an unsplit task the base priority, the extra workers the extra priority), the annotation which requires one node per pod (topology aware scheduling), requests equal to limits, the run directory of the store mounted read-write and the snapshots read-only, a scratch directory on the local disk (`//` for a worker, `//merge` for a merge, `/` for an unsplit task), and the environment of the worker harness (`simrun.worker`). The label `simrun/group` is the group made a valid label value (`label_value`), the annotation `simrun/group` the group as given. The pods run as the service account `ClusterConfig.service_account`, which the wait container of the executor needs to write its results. The pods of a run with a reserved node select the node by its label `simrun/reserved=` and tolerate its taint of the same key and value whatever its effect (the backend taints with `NoExecute`). The Argo features this relies on (Argo Workflows v4.1.6): - `retryStrategy.expression` with `lastRetry.exitCode`, a string ("-1" when the pod has no exit code), and `retryPolicy: Always`, which must hold together to retry (docs/retries.md, "Conditional retries"; the type in docs/variables.md, "RetryStrategy"). The expression compares strings, since `"3" != 3` holds in expr. - `{{retries}}` and `lastRetry.*` in a container template with a retry strategy, substituted per attempt, `lastRetry.exitCode` "0" in the first attempt (docs/variables.md, "Container/Script Templates"; `executeTemplate` in workflow/controller/operator.go), also in `{{=...}}` expressions, which are left for the attempt until `lastRetry` is known (`variablesToCheck` in util/template/expression_template.go). So `podSpecPatch` sets the memory and `SIMRUN_MEMORY_PER_WORKER_GB` of an attempt after an exit 137 to the retry values, and every other attempt keeps the first values. - `podSpecPatch`, a strategic merge patch of the pod spec (containers and env merged by name), also for `terminationGracePeriodSeconds`, which a template has no field for; the emissary executor reads it after the patch (`createWorkflowPod` in workflow/controller/workflowpod.go). - `{{item}}` in the arguments of an expanded DAG task (`processItem` in workflow/controller/operator.go), and input parameters substituted anywhere in a template, its metadata and `priorityClassName` included (`SubstituteParams` in workflow/common/util.go). - `continueOn` of a task which no task depends on: the DAG ignores its failure (`assessDAGPhase` in workflow/controller/dag.go). ## function `label_value(text: str) -> str` A valid label value for free text. Lowercase, every character but `[a-z0-9-_.]` replaced by `-`, starting and ending alphanumeric, at most 63 characters; a value which had to be changed ends in a hash of the text, so different texts keep different values. Args: text: The text. Returns: The label value, `text` itself when it is one. ## function `to_argo(record: simrun.store.RunRecord, plans: dict[str, simrun.planner.TaskPlan], types: dict[str, simrun.plugins.TaskType], config: simrun.config.ClusterConfig) -> dict[str, typing.Any]` The Argo workflow of a run. The settings of every task are written to `/settings.json` before the workflow is submitted; the manifest does not carry them. Args: record: The run. plans: The plan of every task, by task id. types: The task types, by name. config: The cluster. Returns: The manifest of the workflow. Raises: ValueError: A task has no image or no command, or stages a port which is no input. ## function `workflow_name(run_id: str) -> str` The name of the workflow of a run. --- # simrun.cwl CWL v1.2 workflows from runs, executed with cwltool (or Toil on Slurm). The same run as `simrun.argo`, with the same commands and environment, as one CWL `Workflow`. The steps work on the run store by absolute path, the workflow input `store` (a directory every step sees: a local directory, or the NFS mount on Slurm), and communicate through it: the ledger and the outputs live in the store. The only CWL output of a step is its standard output, which exists to order the steps. A split task is three steps of command line tools: `-w0` (worker `-w0`, `SIMRUN_ROLE=base`), the coordinator, which returns once the ledger is complete; `-extra`, scattering over the worker indices `1..pods - 1` (workers `-w`, `SIMRUN_ROLE=extra`; left out for one pod), which exit when nothing is pending; and `-merge` after `-w0` only. With `--parallel` the coordinator and the extra workers run side by side; without it the steps run one after the other and the run still ends, since either one worker computes everything or the next finds nothing pending. A task waits for the merge of the tasks it reads from. A failed extra worker does not fail the workflow (`successCodes` of the step), as `continueOn` in Argo; the coordinator releases its claims and computes them, and a deterministic unit failure fails the workflow through `-w0`. The export targets a shared file system without containers (`cwltool --no-container`, Toil on Slurm with the store on NFS): `store` is a path, not a CWL `Directory`, and the image is only a `DockerRequirement` hint. Running the steps in containers is deferred to the Slurm plan. The scratch directory of a worker is `///` below the workflow input `scratch` (`//merge` for a merge, `/` for an unsplit task). The image is a `DockerRequirement` hint, so `--no-container` runs the commands of the environment. ## function `export_cwl(record: simrun.store.RunRecord, plans: dict[str, simrun.planner.TaskPlan], types: dict[str, simrun.plugins.TaskType], path: pathlib.Path, **options: Any) -> pathlib.Path` Write the CWL workflow of a run as YAML and validate it with cwl-utils. cwl-utils is the optional extra `cwl` (`pip install "simrun[cwl]"`), which also installs cwltool, the runner of the workflow. Args: record: The run. plans: The plan of every task, by task id. types: The task types, by name. path: The file to write. **options: The keyword arguments of `to_cwl`. Returns: The path. ## function `to_cwl(record: simrun.store.RunRecord, plans: dict[str, simrun.planner.TaskPlan], types: dict[str, simrun.plugins.TaskType], *, scratch: str = '/tmp/simrun-scratch', grace_s: int = 60) -> dict[str, typing.Any]` The CWL workflow of a run. The settings of every task are written to `/settings.json` before the workflow runs; the workflow does not carry them. Its inputs are `store` (the directory of the run store) and `scratch` (the local scratch, default `scratch`). Args: record: The run. plans: The plan of every task, by task id. types: The task types, by name. scratch: The default of the workflow input `scratch`. grace_s: Seconds a worker has to stop after SIGTERM. Returns: The workflow as a CWL document. Raises: ValueError: A task has no image or no command, or stages a port which is no input. --- # simrun.testing A backend in memory for tests of the service, advanced by the test. ## class `FakeBackend(nodes: list[simrun.planner.Node] | None = None, finish_after: float | None = None, images: collections.abc.Iterable[str] = ()) -> None` A backend whose workflows change only when the test says so. A submitted workflow is `Pending` without nodes. `start` sets every node running, `finish` every node succeeded, `fail` and `set_node` one node, `fail_attempt` and `oom_kill` fail attempts of a pod; `finish_after` finishes a workflow that many seconds after its submission. The attempts of a pod are numbered from 0 in the order they fail. A coordinator which merges and succeeds writes the marker of a merge into the output of its task in `store`, as the harness does, unless `merges` is False (the harness of simrun 0.1.0). The calls which change the cluster are recorded in `calls`. Attributes: manifests: The submitted manifests by name. subscribed: The callbacks registered with `subscribe`. calls: `("submit", name)`, `("cancel", name)`, `("stop", name)`, `("logs", name, task_id)`, `("reserve", node, run_id)`, `("unreserve", node, run_id)` and `("build_image", ref, files of the context)`. images: The images of the registry. pod_logs: The logs of the pods by workflow name and task id, see `add_log`. store: The store of the service, where a coordinator which merges writes the marker of its merge; None writes none. merges: Whether a coordinator which merges writes the marker. ### `FakeBackend.add_log(self, name: str, task_id: str, node: str, text: str, done: bool = True) -> None` Add the log of a pod of a task, the pod named `-`. ### `FakeBackend.build_image(self, context: pathlib.Path, ref: str) -> None` Record the build and add the image to the registry. ### `FakeBackend.cancel(self, name: str) -> None` Terminate a workflow. ### `FakeBackend.delete(self, name: str) -> None` Remove a workflow, as its time to live does. ### `FakeBackend.fail(self, name: str, node: str) -> None` Fail a node, given by its name or by a task id (its coordinator `-w0`). ### `FakeBackend.fail_attempt(self, name: str, worker: str, exit_code: int | None = None, oom_killed: bool = False) -> None` Fail the next attempt of a pod of a worker, merge or unsplit task. ### `FakeBackend.finish(self, name: str) -> None` Let every node of a workflow succeed. ### `FakeBackend.has_image(self, ref: str) -> bool` Whether the image was given or built. ### `FakeBackend.logs(self, name: str, task_id: str, tail_lines: int | None = None) -> list[simrun.service.backend.PodLog]` The logs added with `add_log`, the last `tail_lines` lines of each. ### `FakeBackend.nodes(self) -> list[simrun.planner.Node]` The nodes, with their reservations. ### `FakeBackend.oom_kill(self, name: str, worker: str, count: int = 1) -> None` Let the next attempts of a pod be killed by the out of memory killer. ### `FakeBackend.place(self, placement: simrun.service.backend.Placement) -> None` Place a pod, listed by `placements` until its workflow has ended. A placement of a pod placed before replaces it, e.g. once the pod is bound. ### `FakeBackend.placements(self) -> list[simrun.service.backend.Placement]` The pods placed for the workflows which have not ended. ### `FakeBackend.reserve(self, node: str, run_id: str) -> None` Reserve a node; KeyError for an unknown one. ### `FakeBackend.set_node(self, name: str, node: str, phase: str) -> None` Set the phase of one node. ### `FakeBackend.start(self, name: str) -> None` Set every node of a workflow running. ### `FakeBackend.status(self, name: str) -> simrun.service.backend.BackendStatus` The status of a workflow; KeyError for an unknown one. ### `FakeBackend.stop(self, name: str) -> None` Stop a workflow. ### `FakeBackend.submit(self, record: simrun.store.RunRecord, plans: dict[str, simrun.planner.TaskPlan], types: dict[str, simrun.plugins.TaskType], cluster: simrun.config.ClusterConfig) -> str` Accept the workflow of a run, pending without nodes; one which exists is kept. Raises: ValueError: The workflow cannot be built, e.g. a task has no image. ### `FakeBackend.subscribe(self, callback: collections.abc.Callable[[], None]) -> None` Remember the callback in `subscribed`; the fake never calls it. ### `FakeBackend.unreserve(self, node: str, run_id: str) -> None` Release a node reserved by the run; KeyError for an unknown node. ## function `dag_nodes(manifest: dict[str, typing.Any]) -> list[str]` The names of the DAG nodes of a workflow manifest, as Argo names them. A DAG task over a sequence is one node per item, `(:)`. --- # simrun.service.app The API of the service: people and agents submit, follow and compare runs. Every route of the API needs a bearer token of the tokens file (`simrun.service.auth`); only the files of the web app at `/ui/` (setting `web`) answer without one. The identity of the token decides the priority of its runs. A submission is validated, its resources resolved and its tasks planned on the nodes of the backend, and the record (`submitting`) and the settings of every task written; `POST /runs` returns then, and the environment, the image and the workflow follow in the background (`simrun.service.submission`). The controller (`simrun.service.controller`) runs the submissions and follows the workflows in a background task. Run it with `uvicorn --factory simrun.service.app:serve --timeout-graceful-shutdown 5`, which reads the `ServiceSettings` from the YAML file named by `SIMRUN_CONFIG`; the timeout ends the open event streams when the service stops. ## class `CleanupRequest(*, older_than_days: Annotated[float, Ge(ge=0)], group: str | None = None, failed_only: bool = False, dry_run: bool = False) -> None` Which ended runs a cleanup removes. Attributes: older_than_days: Runs which ended at least this many days ago. group: Only the runs of this group. failed_only: Only failed runs. dry_run: List what would be removed, remove nothing. ## class `CleanupResult(*, runs: list[str], snapshots: list[str], bytes_freed: int, dry_run: bool) -> None` What a cleanup or a deletion removed, paths relative to the store. ## class `CompareRequest(*, runs: list[str]) -> None` The runs to compare. ## class `HintView(*, memory_base_gb: float, memory_per_worker_gb: float, scratch_gb: float, gpus: int) -> None` The default resources of a task type. ## class `OutputFile(*, path: str, bytes: int) -> None` A file of the output of a task, its path relative to the store. ## class `PodUsage(*, run_id: str | None, task_id: str, pod: str, phase: str, cores: float, memory_gb: float) -> None` A pod of a run on a server. Attributes: run_id: The run of the pod, None when the pod belongs to no known run. task_id: The task of the pod. pod: The name of the pod. phase: The phase of the pod. cores: The cores it requests. memory_gb: The GiB of memory it requests. ## class `RunItem(*, id: str, state: simrun.store.RunState, group: str | None, submitted_by: str, agent: str | None, priority: simrun.workflow.Priority, tasks: dict[str, simrun.store.TaskState], created: datetime.datetime, finished: datetime.datetime | None, message: str | None, owner: str, workflow: str, progress: float | None = None) -> None` A run of the list of runs: its summary with its owner and workflow name. Attributes: owner: The owner of the run, of an agent the person it acts for. workflow: The name of the workflow. progress: The progress of an active run (submitting, queued, running) from 0 to 1, from the states of its tasks and their ledgers (`run_progress`); None for a run which ended. ## class `ServerUsage(*, name: str, cores: float, memory_gb: float, gpus: int, reserved_by: str | None, pods: list[simrun.service.app.PodUsage]) -> None` A server of the cluster with the pods of runs on it. Attributes: name: The name of the server. cores: The cores the scheduler gives the pods of the runs: the allocatable ones less the requests of other pods, e.g. of DaemonSets. memory_gb: The GiB of memory the scheduler gives the pods of the runs. gpus: The GPUs of the server. reserved_by: The run which reserved the server, None if free. pods: The pods of runs on the server, with their requests, which count against `cores` and `memory_gb`. ## class `ServiceSettings(*, store: pathlib.Path, db: pathlib.Path, tokens: pathlib.Path, owner: str, cluster: simrun.config.ClusterConfig, poll_s: float = 5.0, events_poll_s: float = 0.5, events_max_s: float = 300.0, default_image: str | None = None, simrun_wheel: pathlib.Path | None = None, python: str = '3.14', build_constraints: pathlib.Path | None = None, backend: str | None = None, web: pathlib.Path | None = , max_upload_bytes: int = 2147483648) -> None` The settings of the service. Attributes: store: Root of the run store. db: SQLite file of the resource observations and the index of the runs, on a local volume. tokens: YAML file of the tokens, see `simrun.service.auth.load_tokens`. owner: The person who owns the cluster; their runs have the top priority. cluster: The cluster. poll_s: Seconds between two reconciliations of the controller when no change of the backend woke it before. events_poll_s: Seconds between two checks of an event stream. events_max_s: Seconds after which an event stream ends and the client reconnects. default_image: Image of a run whose model has no `pyproject.toml`. simrun_wheel: The simrun wheel the images of the environments install. python: Python version of the images. build_constraints: The pinned versions of the build backends of the environments, the file of simrun (`simrun.environment.BUILD_CONSTRAINTS`) by default. backend: `module:callable` returning the backend, for `serve`. web: Directory of the built web app, served at `/ui/` without a token; the app itself talks to the API with the token of the user. The one of the installation by default (`simrun.local.web_dir`), None serves none. max_upload_bytes: The largest body of an upload of a snapshot; a larger one is refused with 413. ## class `SubmitRequest(*, workflow: simrun.workflow.Workflow, sources: list[simrun.snapshot.SourceProvenance] = , group: str | None = None, split: simrun.workflow.SplitMode | None = None, reserve: str | None = None, local_commands: Literal['direct', 'docker'] | None = None) -> None` A submission of a workflow. Attributes: workflow: The workflow. sources: The snapshots of the sources, the model first. group: Group of the run, e.g. a study. split: Split mode of every task, else the one of the task or the workflow. reserve: A node reserved for the run, else the one of the workflow. local_commands: How the local backend runs the `command` tasks: `direct` their args, `docker` in their images; None the choice of the backend. The Argo backend ignores it. ## class `TaskEstimate(*, resources: simrun.resources.ResolvedResources, plan: simrun.planner.TaskPlan, seconds: float | None) -> None` The resources, the plan and the expected duration of a task. Attributes: resources: The resolved resources. plan: The plan of the pods. seconds: The expected seconds of a split task: the median time of a unit in one process of past runs times the rounds of its units over the processes of its pods, None without past runs. ## class `TaskProgress(*, counts: simrun.ledger.LedgerCounts | None, first: dict[str, Any] | None = None, last: dict[str, Any] | None) -> None` The progress of a task: its ledger and the first and last lines of its progress file. The first line is the first unit done, whose time is when the task began to deliver units. ## class `TaskTypeView(*, name: str, settings_schema: dict[str, typing.Any], inputs: list[str], outputs: list[str], splittable: bool, uses_cores: bool, has_plan: bool, cost_setting: str | None, hint: simrun.service.app.HintView, description: str) -> None` A task type with the JSON Schema of its settings, for the editor. ## class `Usage(*, runs: dict[str, int], groups: dict[str, int]) -> None` Bytes of the run store per run and per group (`""` without a group). ## class `ValidationResult(*, ok: bool, errors: list[str]) -> None` The result of a validation: the errors of a workflow, none when valid. ## class `WorkerView(*, worker: str, node: str | None, state: Literal['running', 'done', 'failed'], attempts: int, processes: int | None, units_done: int, units_claimed: int, peak_memory_gb: float | None, memory_limit_gb: float | None, oom_killed: bool) -> None` A worker of a task, from its reports, its pod and its claims. Attributes: worker: The worker id. node: The server of the attempt shown: of the live pod while it runs, None while that pod is not bound, else of the last report. state: `running` while a pod of an attempt without a report lives (a retry after an eviction, a preemption, a lost node or a kill for memory, as a worker reports also when it is stopped), else from the last report: `failed` when it was killed for memory or failed (a unit or its command failed), else `done`. attempts: The attempts of the worker: one more than the highest attempt of its reports and of its live pod. processes: The processes of its pool, None before a report. units_done: The distinct units its `unit_done` lines in the progress file of the task name, live and over all its attempts. units_claimed: The units it holds claimed now. peak_memory_gb: The peak memory of the last report. memory_limit_gb: The memory limit of the last report. oom_killed: Whether the attempt of the last report was killed for memory. ## function `create_app(settings: simrun.service.app.ServiceSettings, backend: simrun.service.backend.Backend, types: dict[str, simrun.plugins.TaskType] | None = None) -> fastapi.applications.FastAPI` Create the app of the service. Args: settings: The settings. backend: The backend the workflows run on. types: The task types by name, the ones of the entry points by default. Returns: The app; its lifespan runs the controller every `settings.poll_s` seconds. ## function `disk_usage(path: pathlib.Path) -> int` The bytes of the files below `path`, every hard linked file once. ## function `freed_bytes(paths: collections.abc.Iterable[pathlib.Path]) -> int` The bytes removing `paths` frees: files of which no hard link stays. ## function `load_settings(path: pathlib.Path) -> simrun.service.app.ServiceSettings` Read the settings of the service from a YAML file. ## function `openapi_document() -> dict[str, typing.Any]` The OpenAPI document of the service, the input of the types of the frontend. The app is created on a fake backend in a temporary directory; the routes do not depend on either, so the document is the same on every machine. ## function `run_progress(tasks: dict[str, simrun.store.TaskState], counts: dict[str, simrun.ledger.LedgerCounts | None]) -> float` The progress of a run from 0 to 1, as the web app counts it. The mean over its tasks: a task which is over (done or skipped) counts whole, a task with a ledger the share of its units done, at most `UNFINISHED_TASK_MAX` (its merge and outputs are still to come), any other task nothing; so the progress is 1 only once every task is over. Args: tasks: The states of the tasks. counts: The ledger counts by task, None or missing without a ledger. Returns: The progress, 0 for a run without tasks. ## function `serve() -> fastapi.applications.FastAPI` The app of the settings in the YAML file named by `SIMRUN_CONFIG`. The backend is the `backend` of the settings, a `module:callable` called without arguments. Raises: RuntimeError: `SIMRUN_CONFIG` or the backend is not set. ## function `unknown_settings(model: type[pydantic.main.BaseModel], settings: dict[str, typing.Any]) -> list[str]` The keys of settings which are no field of their model, sorted. A field counts by its name and its aliases; a model which allows extra keys (`extra="allow"`) has none unknown. ## function `validate_workflow(workflow: simrun.workflow.Workflow, types: dict[str, simrun.plugins.TaskType]) -> list[str]` The errors of a workflow beyond its structure: types, settings and ports. A setting which is no field of the settings model of its type is an error, so a misspelled one is not silently dropped. An invalid setting is one error per field, `task '': : `, the location dotted. Args: workflow: The workflow. types: The task types by name. Returns: The errors, empty when the workflow is valid. --- # simrun.service.auth Identities of the callers of the service and the priorities of their runs. ## class `Identity(*, name: str, kind: Literal['person', 'agent'], owner: str) -> None` Who calls the service: a person or an agent acting for a person. Attributes: name: Name of the person or the agent. kind: `person` or `agent`. owner: The person responsible; a person's owner is the person. ## function `authenticate(tokens: dict[str, simrun.service.auth.Identity], token: str) -> simrun.service.auth.Identity | None` The identity of a token, compared in constant time; None if unknown. ## function `load_tokens(path: pathlib.Path) -> dict[str, simrun.service.auth.Identity]` Read the tokens of the service from a YAML mapping of token to identity. Args: path: The YAML file, e.g. `{: {name: mk, kind: person}}`. Returns: The identities by token. Raises: ValueError: The file is no mapping, or a token is empty. ## function `priority_of(identity: simrun.service.auth.Identity, owner: str) -> simrun.workflow.Priority` The priority of the runs of an identity. Args: identity: The submitting identity. owner: The owner of the cluster. Returns: `OWNER` for the owner and the owner's agents, `STANDARD` for other people, `AGENT` for their agents. --- # simrun.service.backend The backend the service runs workflows on: Argo Workflows, or a fake in tests. ## class `Backend(*args, **kwargs)` What the service needs of a cluster. ### `Backend.build_image(self, context: pathlib.Path, ref: str) -> None` Build the image of a build context and push it as `ref`. ### `Backend.cancel(self, name: str) -> None` Terminate a workflow and its pods. ### `Backend.has_image(self, ref: str) -> bool` Whether the registry has the image `ref`. ### `Backend.logs(self, name: str, task_id: str, tail_lines: int | None = None) -> list[simrun.service.backend.PodLog]` The logs of the pods of a task of a workflow, oldest first. Every attempt whose pod still exists; `tail_lines` limits each log to its last lines. ### `Backend.nodes(self) -> list[simrun.planner.Node]` The nodes of the cluster. ### `Backend.placements(self) -> list[simrun.service.backend.Placement]` The pods of the runs which have not terminated, with nodes and requests. ### `Backend.reserve(self, node: str, run_id: str) -> None` Reserve a node for a run. The node gets the taint `simrun/reserved=:NoExecute` and the label `simrun/reserved=`. ### `Backend.status(self, name: str) -> simrun.service.backend.BackendStatus` The status of a workflow. Raises: KeyError: The workflow does not exist (any more). ### `Backend.stop(self, name: str) -> None` Stop a workflow whose run is decided: pods still pending are removed. ### `Backend.submit(self, record: simrun.store.RunRecord, plans: dict[str, simrun.planner.TaskPlan], types: dict[str, simrun.plugins.TaskType], cluster: simrun.config.ClusterConfig) -> str` Submit a run and return the name of its workflow. Args: record: The run, with its image set when the backend `runs_images`. plans: The plan of every task by id. types: The task types by name. cluster: The configuration of the cluster. Returns: The name of the workflow of the run (`argo.workflow_name`). Raises: ValueError: The run cannot be submitted (e.g. a task has no image); the message is the one of the failed run. ### `Backend.subscribe(self, callback: collections.abc.Callable[[], None]) -> None` Register a callback the backend calls when the status of a run changed. The service registers the wake of its controller, so a backend which knows of a change makes it reconcile at once. ### `Backend.unreserve(self, node: str, run_id: str) -> None` Remove the taint and the label of the reservation of a run. A node reserved by another run (or by none) is left as it is, so a late release never removes the reservation of a later run. Raises: KeyError: The node does not exist. ## class `BackendStatus(*, phase: str, tasks: dict[str, str], failed: dict[str, list[simrun.service.backend.FailedAttempt]] = ) -> None` The state of a workflow in the backend. The service, not the backend, decides the state of a run from it (see `simrun.service.controller`). Attributes: phase: Phase of the workflow, e.g. `Pending`, `Running`, `Succeeded`. tasks: Phase of every node of the DAG by its name: `-w0`, `-extra(:)` and `-merge` of a split task, `` of an unsplit one; a node not yet created is missing. A node whose pod was evicted and whose retry waits is `Preempted` (`PREEMPTED_PHASE`), one whose pod failed otherwise `Retrying` (`RETRYING_PHASE`). failed: The attempts of pods which failed, in the order of their number: per worker id (`-w`), per task id of an unsplit task and per `-merge` of a merge. ## class `FailedAttempt(*, attempt: int, exit_code: int | None = None, oom_killed: bool = False) -> None` An attempt of a pod which failed. Attributes: attempt: The number of the attempt, from 0, as the pod has it in `SIMRUN_ATTEMPT` (Argo's `{{retries}}`) and in the name of its report. exit_code: The exit code of its main container, None without one (e.g. a pod deleted by an eviction of Kueue). oom_killed: Whether the main container was killed by the out of memory killer (the reason `OOMKilled`); an exit 137 alone is no evidence of it. ## class `Placement(*, workflow: str, dag_task: str, pod: str, node: str | None, phase: str, cores: float, memory_gb: float, worker: str | None = None, attempt: int | None = None) -> None` A pod of a run which has not terminated, with its node and requests. Attributes: workflow: The backend name of the run. dag_task: The DAG task of the pod: `-w0`, `-extra`, `-merge` or ``. pod: Name of the pod. node: The node the pod is bound to, `None` while it is not. phase: The phase of the pod. cores: The cores the pod requests. memory_gb: The GiB of memory the pod requests. worker: The worker id of the pod (`SIMRUN_WORKER`, the id of its ledger claims and report names), `None` for a pod without one, e.g. a merge. attempt: The attempt of the pod (`SIMRUN_ATTEMPT`, the number of its retries, the attempt of its report name), `None` when unknown. ## class `PodLog(*, node: str, pod: str, text: str, done: bool = True) -> None` The log of the main container of one pod (one attempt) of a task. Attributes: node: The node of the pod in the workflow, e.g. `squares-w0(1)`. pod: The name of the pod. text: The log. done: Whether the pod has terminated, so its log is complete. --- # simrun.service.argo_backend The backend of the service on a cluster: Argo Workflows, Kueue and BuildKit. A run is an Argo `Workflow` (`simrun.argo.to_argo`) in the namespace of the cluster; Kueue admits its pods (plain pods with a queue label) and preempts lower priorities. The status of a workflow is read from the nodes of its DAG (`workflow_status`): a node retried by Argo (a `Retry` node) is `Pending` while its attempt waits for admission, `Preempted` while the attempt after an eviction waits (a pod deleted by Kueue or lost, or a worker stopped on SIGTERM), `Retrying` while the attempt after another failure waits (a crash, a pod killed after its grace period), `Running` while its attempt runs; the failed attempts are listed per worker with their exit codes and whether the kernel killed them for memory (`OOMKilled`). The nodes are the ones labelled `simrun/worker=true` (the nodes of the ResourceFlavor of Kueue) without a taint which keeps pods off, a reservation of simrun aside; their cores and memory are the allocatable ones minus the wait container of the Argo executor and minus the requests of the pods on them which are no pods of runs (the DaemonSets of the cluster, e.g. of the NFS driver and of the monitoring), rounded down to whole cores and GiB, so a pod the planner sizes for a node fits it beside them. A reservation is the taint `simrun/reserved=:NoExecute`, which evicts the pods of other runs from the node, and the label `simrun/reserved=`, which the pods of the run select. An image is built by a Job of `buildctl`, which sends the build context (a ConfigMap of its files, owned by the Job) to the BuildKit daemon of the cluster and pushes the image to the registry with the credentials of the optional Secret `simrun-registry-auth`; whether the registry has an image is asked its API. A thread watches the workflows of the runs and wakes the controller on every change, so the state of a run follows Argo at once; the controller still polls, which covers a watch that is restarted. ## class `ArgoBackend(config: simrun.config.ClusterConfig, api: kubernetes.client.api_client.ApiClient | None = None) -> None` The backend of the service on a cluster of Argo Workflows and Kueue. ### `ArgoBackend.build_image(self, context: pathlib.Path, ref: str) -> None` Build the image of a flat build context with BuildKit and push it. Raises: ValueError: The context has a directory or is too large for a ConfigMap. RuntimeError: The build failed. TimeoutError: The build took longer than `BUILD_TIMEOUT_S`. ### `ArgoBackend.cancel(self, name: str) -> None` Terminate a workflow and its pods; KeyError when it does not exist. ### `ArgoBackend.close(self) -> None` Stop the watch of the workflows, at its next event at the latest. ### `ArgoBackend.has_image(self, ref: str) -> bool` Whether the registry has the image; asks the API of the registry. Raises: ValueError: The image is not in the registry of the cluster. RuntimeError: The registry answered neither found nor not found. ### `ArgoBackend.join_watch(self, timeout_s: float | None = None) -> None` Wait until the thread of the watch ended, after `close`. Args: timeout_s: The longest wait in seconds, None for no limit. ### `ArgoBackend.logs(self, name: str, task_id: str, tail_lines: int | None = None) -> list[simrun.service.backend.PodLog]` The logs of the main containers of the pods of a task, oldest first. Args: name: The workflow. task_id: The task. tail_lines: The last lines of every log, all without. Returns: A log per pod which still exists, every attempt, with whether the pod terminated; a log which cannot be read says why, e.g. that its pod has not started. At most `LOG_LIMIT_BYTES` of a log are read (the first of its last `tail_lines` lines). ### `ArgoBackend.nodes(self) -> list[simrun.planner.Node]` The worker nodes on which pods can run, with their reservations. Their capacity is the allocatable one less the requests of the pods bound to them which have not terminated and are no pods of runs, e.g. of DaemonSets; Kueue counts the pods of the runs itself. ### `ArgoBackend.placements(self) -> list[simrun.service.backend.Placement]` The pods of the runs which have not terminated, with nodes and requests. ### `ArgoBackend.reserve(self, node: str, run_id: str) -> None` Taint and label a node for a run; KeyError for an unknown node. The taint replaces a reservation the node had; the resource version of the node guards the list of taints against a concurrent change. ### `ArgoBackend.status(self, name: str) -> simrun.service.backend.BackendStatus` The status of a workflow; KeyError when it does not exist. ### `ArgoBackend.stop(self, name: str) -> None` Stop a workflow whose run is decided; KeyError when it does not exist. ### `ArgoBackend.submit(self, record: simrun.store.RunRecord, plans: dict[str, simrun.planner.TaskPlan], types: dict[str, simrun.plugins.TaskType], cluster: simrun.config.ClusterConfig) -> str` Create the workflow of a run; returns its name. A workflow of the name which exists already is taken as submitted: the name of the workflow of a run is fixed, so a resumed submission creates it once. Raises: ValueError: The workflow cannot be built, e.g. a task has no image. ### `ArgoBackend.subscribe(self, callback: collections.abc.Callable[[], None]) -> None` Call `callback` on every change of a workflow of a run. The first subscription starts the thread of the watch of the workflows, which every subscription shares. ### `ArgoBackend.unreserve(self, node: str, run_id: str) -> None` Remove the taint and the label of the reservation of a run. Raises: KeyError: The node does not exist. ## function `backend() -> simrun.service.argo_backend.ArgoBackend` The backend of the settings named by `SIMRUN_CONFIG`, for `serve`. ## function `manifest_url(ref: str, config: simrun.config.ClusterConfig) -> str` The URL of the manifest of an image of the registry of the cluster. Args: ref: The image, `/[:|@]`. config: The cluster. Returns: The URL of the manifest in the API of the registry. Raises: ValueError: The image is not in the registry of the cluster. ## function `node_from(node: dict[str, typing.Any], used: tuple[float, float] = (0.0, 0.0)) -> simrun.planner.Node | None` The node of the planner of a Kubernetes node, None when pods cannot run on it. Args: node: The node as the API serialises it. used: The cores and GiB the pods on the node request which are no pods of runs (e.g. of DaemonSets). Returns: The node with its allocatable cores and memory minus one executor container and minus `used`, rounded down, for the planner, the allocatable cores and memory minus `used` as the scheduler counts the requests of the pods of the runs, and its reservation; None for a node which is unschedulable or has a taint other than a reservation which keeps pods off. ## function `pod_requests(pod: dict[str, typing.Any]) -> tuple[float, float]` The cores and GiB a pod requests, as the scheduler counts them. The containers together, or an init container alone when it requests more (the init containers run before the containers, one at a time; a sidecar, an init container which keeps running, counts with the containers and the init containers after it), plus the overhead of the pod. Args: pod: The pod as the API serialises it. Returns: The cores and the GiB. ## function `workflow_status(workflow: dict[str, typing.Any]) -> simrun.service.backend.BackendStatus` The status of a workflow from its Argo object. Args: workflow: The workflow as the API returns it. Returns: The phase of the workflow, of every task node of its DAG by its name relative to the workflow, and the failed attempts of the pods per worker id, per task id of an unsplit task and per `-merge` of a merge. --- # simrun.service.local_backend The local backend: the pods of a run as processes on this machine. `LocalBackend` runs the DAG of a run without a cluster: a split task as its coordinator (`simrun-worker` with `SIMRUN_ROLE=base`, which merges itself), a plugin command through `simrun-exec` and a `command` task with its args directly or in its image with `docker run` (a container named after the run, the node and the attempt, killed with `docker kill` on a cancel). A task starts when the tasks it reads from succeeded and its cores fit into the free cores of the machine; the tasks of several runs take them in the order of their submission. The nodes and phases of its status are the ones of the Argo workflow of the run, so the controller decides the states of the tasks as on the cluster. Every process runs in its own process group with its stdout and stderr in `/logs/-.log`; while it runs, `-.pid` beside the log names it, so a service which starts after one which died without ending its processes kills them (`kill_orphans`). A run with snapshots gets a venv of its own (`simrun.service.local_env`), created when its first process starts and removed at its end: its snapshots are installed there, never into the environment of the service. A daemon thread per run starts the processes of the run, a thread per process waits for its end; the state lives under one condition, the callbacks of `subscribe` are called outside of it after every start and end of a process. ## class `LocalBackend(store: pathlib.Path, scratch: pathlib.Path, commands: Literal['direct', 'docker'] = 'direct', cores: int | None = None, memory_gb: float | None = None, envs: pathlib.Path | None = None) -> None` A backend which runs the tasks of a run as processes on this machine. Attributes: store: The root of the store. scratch: The root of the scratch directories of the processes. envs: The root of the venvs of the runs with snapshots. commands: How a `command` task runs unless its run says otherwise (`RunRecord.local_commands`): `direct` its args, `docker` in its image. cores: The cores of the machine the tasks share. memory_gb: The memory of the machine in GiB. hostname: The name of the machine, its one node. ### `LocalBackend.build_image(self, context: pathlib.Path, ref: str) -> None` Refused: the local backend runs no images of runs. ### `LocalBackend.cancel(self, name: str) -> None` End a run: SIGTERM to its processes, SIGKILL after `worker.GRACE_S`. A container gets the same signals with `docker kill`. No process of the run starts or is retried any more; KeyError for an unknown run. ### `LocalBackend.close(self) -> None` Refuse new runs, cancel every run and wait for the end of its processes. ### `LocalBackend.has_image(self, ref: str) -> bool` True: the local backend needs no image. ### `LocalBackend.kill_orphans(self) -> int` Kill the processes of runs which a service before left running. A service which died without ending its processes (SIGKILL, out of memory) leaves their pid files beside their logs. The process group of each is killed, and its container, unless its run is one of this backend or its leader is not the process of the file any more (another start time or command line: a reused pid); the files are removed. Returns: The process groups killed. ### `LocalBackend.logs(self, name: str, task_id: str, tail_lines: int | None = None) -> list[simrun.service.backend.PodLog]` The logs of the processes of a task, oldest first. ### `LocalBackend.nodes(self) -> list[simrun.planner.Node]` This machine. ### `LocalBackend.placements(self) -> list[simrun.service.backend.Placement]` A placement per running process. ### `LocalBackend.reserve(self, node: str, run_id: str) -> None` Refused: there is nothing to reserve locally. ### `LocalBackend.status(self, name: str) -> simrun.service.backend.BackendStatus` The phases of the nodes of a run; KeyError for an unknown run. ### `LocalBackend.stop(self, name: str) -> None` End a run whose state is decided, as `cancel`. ### `LocalBackend.submit(self, record: simrun.store.RunRecord, plans: dict[str, simrun.planner.TaskPlan], types: dict[str, simrun.plugins.TaskType], cluster: simrun.config.ClusterConfig) -> str` Start the run in a thread of its own; a run submitted before is kept. Raises: ValueError: A task stages a port which is no input, or its type is neither splittable nor has a command. RuntimeError: The backend is closed. ### `LocalBackend.subscribe(self, callback: collections.abc.Callable[[], None]) -> None` Call `callback` after every start and end of a process. ### `LocalBackend.unreserve(self, node: str, run_id: str) -> None` Refused: there is nothing to reserve locally. ## function `backend() -> simrun.service.local_backend.LocalBackend` The backend of the settings named by `SIMRUN_CONFIG`, for `serve`. The store is the one of the settings, the scratch `/../scratch`. --- # simrun.service.local_env The environment of a local run with snapshots: a venv of the run. A run of the local backend with snapshots (`--snapshot`) must not install them into the environment of the service, the one of the user. The backend creates a venv per run in `.simrun/envs//venv`, which sees the packages of the service through a path file (`BASE_PTH`), installs the snapshots into it as editable packages from shadow trees in `.simrun/envs//sources/` (`simrun.worker.shadow_tree`), and starts the harness with its python; a `command` task finds it first on `PATH`. The venv is removed at the end of the run. The snapshots are installed without their dependencies, which the environment of the service has, with `uv` (on `PATH` or the `uv` package), else with `pip`; their build backends come from the index or the cache of the installer, as for any build. ## class `RunEnvironmentError` The venv of a run cannot be created or the snapshots cannot be installed. ## function `base_site_dirs() -> list[str]` The site directories of this interpreter, the user site when enabled. ## function `create(directory: pathlib.Path, snapshots: list[str], log: , cancelled: collections.abc.Callable[[], bool] = at 0x7c8b17f78880>) -> pathlib.Path` Create the venv of a run and install its snapshots into it. Args: directory: The directory of the environment of the run, replaced. snapshots: The paths of the snapshots. log: Where the output of the installer goes. cancelled: Polled while the installer runs; when it holds, the installer is killed. Returns: The python of the venv. Raises: RunEnvironmentError: The venv cannot be created, there is no installer, or the installation failed or was cancelled. ## function `installer(python: pathlib.Path) -> list[str]` The command which installs into the venv of `python`, before its arguments. Raises: RunEnvironmentError: Neither uv nor pip is installed. --- # simrun.service.controller The controller: follows the workflows of the runs and records their end. The state of a run is the service's, not the one of Argo (Ruling 17b): a split task is done when its `-merge` node succeeded (when its coordinator `-w0` merges, `TaskPlan.coordinator_merges`, there is no such node and it is done when `-w0` succeeded and wrote `out/.simrun-merged`; without the marker it waits for it, running, up to `MERGE_MARKER_GRACE_S` from the first poll which saw the coordinator succeeded (`RunRecord.merge_pending`), and fails after it: the harness of its image is one which does not merge, from simrun 0.1.0) and failed when its `-w0` or its merge failed (after the retries of Argo), an extra worker never fails a task; an unsplit task is its node ``. When every task is done the run succeeded, when a task failed the run failed; in both cases the controller stops the workflow, so extra workers still waiting for admission go away, and releases the reserved node. A run whose workflow ended is not failed for its phase while a task waits for the marker of its merge. A pod is retried once with the retry memory after a kill for memory: a worker, a merge or an unsplit task killed for memory again with the retry memory fails the run (Ruling 35), whose message names the retry memory. A kill for memory needs the evidence of the out of memory killer (Ruling 39): the reason `OOMKilled` of the backend, or the report of an attempt which exited 137 counting kills of the out of memory killer in its cgroup; a pod killed after its grace period was evicted. The states of the tasks and the out of memory kills are stored in the record at every poll. When the workflow is gone (deleted by its time to live before the controller saw its end) the tasks are resolved from these states and from the store: a split task whose merge wrote `out/.simrun-merged` is done (Ruling 23), one which waits for the marker of its merge keeps the run open until the marker is there or its wait is over; only the tasks still undetermined fail. The reports of the workers of a task (of `simrun-exec` for an unsplit plugin task, by the task id) are recorded as resource observations once the state of the task is final, each marked in `tasks//recorded/` right after its observation; a kill for memory of an attempt without a report is an observation of the pod shape at its limit (Ruling 24). The logs of the pods of a task are kept in the run store once the task is final and its pods terminated, the ones which ended before the workflow is stopped or cancelled (`simrun.service.logs`); an ended run stays followed (`pending_logs`) until its logs are kept or its workflow is gone. A terminal record is written first; a release of the node or a cancellation of the workflow which fails stays in the record (`pending_release`, `pending_cancel`) and is retried at the next polls, which do nothing else for a terminal run (Ruling 25). No call of the backend is made while the record lock is held. The controller keeps no state in memory: every `reconcile` reads the records of the live runs of the index of the store (`simrun.service.index`), which the service rebuilds from the records at its start, so a restarted service continues where the one before stopped; a record which cannot be read is logged and skipped. ## class `Controller(store: simrun.service.index.IndexedRunStore, resources: simrun.resources.ResourceStore, backend: simrun.service.backend.Backend, types: dict[str, simrun.plugins.TaskType]) -> None` Maps the status of the backend onto the records of the runs. ### `Controller.cancel(self, run_id: str) -> simrun.store.RunRecord` Cancel a run: write it cancelled, then cancel its workflow, release its node. Raises: KeyError: The run does not exist. ValueError: The run has ended. ### `Controller.close(self) -> None` Stop taking submissions, without waiting for the running ones. A build may take an hour; the pod of the service is killed long before. A run left submitting is resumed at the next start of the service (`recover`), from its record. ### `Controller.in_flight(self, run_id: str) -> bool` Whether the submission of a run runs in this process. ### `Controller.reconcile(self) -> None` Bring the records of the live runs up to date with the backend. The live runs are the ones of the index which have not ended or have a release or a cancellation pending; an ended run is not read again. Runs missing from the index are indexed first (`IndexedRunStore.reindex`) and followed also while their index rows cannot be written. ### `Controller.recover(self) -> None` Settle what a stopped service left: called at the start of the service. The service is one process, so a run submitting since before the controller started has lost its submission: it is resumed from its record (`resume`, idempotent: an image which was built is found, a node reserved for the run is kept, a workflow which was created is taken as submitted). A submission created later may be in flight (the API accepts runs while this runs) and is left to `reconcile`, which never fails a submission running in this process. A record which is not terminal and has no workflow name was written before names were stored, and fails. The pending logs, releases and cancellations of terminal runs are retried. ### `Controller.settle(self, run_id: str) -> None` Retry the pending logs, release and cancellation of a terminal run. The logs are kept before the workflow is cancelled; a workflow which does not exist (any more) has no logs to keep. An unknown node counts as released and an unknown workflow as cancelled; a node is released only from the reservation of this run. A run which was deleted meanwhile is ignored. ### `Controller.start_submission(self, run_id: str, job: collections.abc.Callable[[str], None]) -> None` Run the submission of a run in the background, see `in_flight`. A job which raises fails a run which is still submitting; its workflow may exist, so it is cancelled. Args: run_id: The run, written `submitting`. job: Resolves, builds and submits the run; called with its id. ### `Controller.wait_for_wake(self, timeout_s: float) -> None` Block until woken or `timeout_s` passed; clears the wake. Args: timeout_s: The longest time to wait, in seconds. ### `Controller.wake(self) -> None` Make the control loop reconcile now instead of at its next poll. ### `Controller.write_if_unchanged(self, before: simrun.store.RunRecord, after: simrun.store.RunRecord) -> bool` Write `after` when the record on disk is still `before`. Returns: Whether `after` was written; not when the run changed or is gone. ## function `attempt_of(name: str) -> int` The attempt of a report file `-.json`, 0 without one. ## function `closed(record: simrun.store.RunRecord, state: simrun.store.RunState, tasks: dict[str, simrun.store.TaskState], message: str | None, **update: object) -> simrun.store.RunRecord` The terminal record: tasks not final are skipped, the node is to release. Args: record: The record. state: The terminal state. tasks: The states of the tasks. message: The message, e.g. why the run failed. **update: Further fields of the record. Returns: The record, ended now. ## function `has_merge_marker(task_dir: pathlib.Path) -> bool` Whether the output of a task has the marker of a merge, `out/.simrun-merged`. The directories are listed instead of looking up the marker: on NFS a listing revalidates the directory, while a lookup may answer from a cached negative entry of a lookup before the pod wrote the file. The task directory is listed first, so a cached negative entry of `out/` itself does not hide it either. Args: task_dir: The directory of the task in the run store. Returns: Whether the marker exists; not when `out/` does not exist. ## function `is_oom(report: simrun.worker.WorkerReport, attempt: int, kills: dict[str, list[int]]) -> bool` Whether a report is of a worker which was killed for memory. Only the evidence of the out of memory killer counts (Rulings 34, 39): a peak near the limit is no kill, the kernel reclaims the page cache there, and an exit 137 may be a kill after the grace period. Args: report: The report. attempt: Its attempt. kills: The attempts killed for memory per worker (`RunRecord.oom_kills`). Returns: True when the attempt is one killed for memory, or the report counts kills of the out of memory killer in its cgroup. ## function `task_plans(record: simrun.store.RunRecord) -> dict[str, simrun.planner.TaskPlan]` The plans of the tasks of a run from its record. A plan written before `coordinator_merges` existed reads it as False. ## function `task_states(workflow: simrun.workflow.Workflow, types: dict[str, simrun.plugins.TaskType], status: simrun.service.backend.BackendStatus, plans: dict[str, simrun.planner.TaskPlan], merge_state: collections.abc.Callable[[str], simrun.store.TaskState]) -> dict[str, simrun.store.TaskState]` The state of every task of a workflow from the phases of its DAG nodes. A task is running when one of its pods runs (or its coordinator is done and the merge waits), preempted when none runs and a pod evicted waits for its retry, retrying when none runs and a pod which failed otherwise waits for its retry, queued when it has pods of which none runs, waiting without pods. Args: workflow: The workflow of the run. types: The task types by name. status: The status of the workflow in the backend. plans: The plans of the tasks, by id; the state of a split task whose plan merges in the coordinator is `merge_state` once `-w0` succeeded. merge_state: The state of a task by id whose coordinator which merges succeeded: done with the marker of its merge, running while the marker is awaited, failed when the coordinator ran a harness which does not merge. Returns: The state of every task by id. --- # simrun.service.submission The submission of a run in the background: its environment, image and workflow. `POST /runs` validates and plans a run, writes its record `submitting` (with the deterministic name of its workflow and the settings of its tasks) and returns its id at once. The controller then runs `Submitter.run` in its pool of submissions (`Controller.start_submission`): it resolves the environment of the run, builds its image when the registry does not have it (once per image: a second submission of the same new environment waits for the first build and finds the image), records the image and the time it was ready (`built`, from which the grace of a submission counts), reserves the node of the run and submits its workflow; the run is queued then. A failure lands in the record: the run fails with a message, its node is released and a workflow a failed submission may have created is cancelled. A run cancelled while it is resolved or built is not submitted; one cancelled while its workflow is submitted has the workflow cancelled; the workflow of one cancelled and deleted meanwhile is cancelled as an orphan. ## class `SubmissionError` A submission which cannot go on; the message is the one of the record. ## class `Submitter(store: simrun.store.RunStore, backend: simrun.service.backend.Backend, types: dict[str, simrun.plugins.TaskType], controller: simrun.service.controller.Controller, cluster: simrun.config.ClusterConfig, python: str, simrun_wheel: pathlib.Path | None, build_constraints: pathlib.Path | None = None) -> None` Resolves, builds and submits the runs written `submitting`. ### `Submitter.environment(self, record: simrun.store.RunRecord) -> tuple[str | None, str | None]` The image of a run and the lock of its environment, built when missing. A run whose model has no `pyproject.toml` keeps the image it was written with (the default image of the service). Raises: SubmissionError: The environment cannot be resolved, the service has no simrun wheel to build with, or the build fails. ### `Submitter.run(self, run_id: str) -> None` Resolve, build and submit a run written `submitting`. Args: run_id: The run. ## function `has_model_project(store: pathlib.Path, sources: list[simrun.snapshot.SourceProvenance]) -> bool` Whether the model of a run (its first source) has a `pyproject.toml`. --- # simrun.service.index The index of the runs in SQLite (spec section 4), beside the run store. The store stays the source of truth: the index is rebuilt from the records at the start of the service and updated after every record the service writes (`IndexedRunStore.write`), the record first. It holds what the controller and the lists need without reading the records over NFS: the id, the owner, the agent, the group, the name of the workflow, the state, the times of creation and end, the pending release and cancellation, the snapshots and the summary of every run. The controller follows only the live runs (not ended, or with a release or a cancellation pending); `GET /runs`, a cleanup and the usage filter in the index. ## class `IndexedRun(summary: simrun.store.RunSummary, owner: str, workflow: str, pending: bool, snapshots: tuple[str, ...]) -> None` A run as the index has it. Attributes: summary: Its summary. owner: Its owner. workflow: The name of its workflow. pending: Whether a release of its node or a cancellation of its workflow is pending. snapshots: The snapshots of its sources, relative to the store. ## class `IndexedRunStore(root: pathlib.Path, index: simrun.service.index.RunIndex) -> None` The run store of the service: every record it writes is indexed too. A record whose index row could not be written (the index raised after the record was written) and a record which could not be read when the index was rebuilt are kept aside and indexed again at every `reindex`, which the controller calls at every poll; the ones which still cannot be read are logged then. ### `IndexedRunStore.forget(self, run_id: str) -> None` Remove a run from the index, once it was moved out of the runs. ### `IndexedRunStore.ids(self) -> list[str]` The ids of the runs in the store, sorted; hidden directories are no runs. ### `IndexedRunStore.rebuild(self) -> None` Rebuild the index from the records. A record which cannot be read is logged, left out and tried again at every `reindex`. ### `IndexedRunStore.reindex(self) -> list[str]` Index the runs whose index row is missing, from their records. A record which cannot be read is logged at every call, so a run the controller cannot follow is never silent. Only the index row is written, never the record file, so the caller holds the lock of the record writes around the call: a record is read and indexed without a write in between. Returns: The runs whose records were read but whose index rows still cannot be written, which the controller follows besides the live runs. ### `IndexedRunStore.write(self, record: simrun.store.RunRecord) -> None` Write a record, then index it. A failure of the index is logged and the run indexed at the next `reindex`: the record is written, so the caller goes on. ## class `RunIndex(db: pathlib.Path) -> None` The index of the runs in a SQLite file on a local volume. A connection is opened per call, so the index is safe to use from several threads of one process. ### `RunIndex.any_unended(self) -> bool` Whether a run has not ended, without reading its row. ### `RunIndex.live_ids(self) -> list[str]` The ids of the runs the controller follows, sorted. Runs which have not ended, and ended runs with a release of their node, a cancellation of their workflow or the logs of their pods pending. ### `RunIndex.put(self, record: simrun.store.RunRecord) -> None` Add or update a run. ### `RunIndex.rebuild(self, records: collections.abc.Iterable[simrun.store.RunRecord]) -> None` Replace the index by the runs of `records`. ### `RunIndex.remove(self, run_id: str) -> None` Remove a run. ### `RunIndex.runs(self, group: str | None = None, owner: str | None = None, agent: str | None = None, state: simrun.store.RunState | None = None) -> list[simrun.service.index.IndexedRun]` The runs with the given group, owner, agent and state, sorted by id. --- # simrun.service.logs The logs of the pods of a task, kept in the run store. The logs of a pod are gone with the pod: Kueue deletes a preempted pod, Argo deletes the pods of a workflow with it after its time to live, and stopping a workflow ends its pods. So the controller copies the logs of the main containers of the pods of a task into `tasks//logs/` once the task is final and its pods terminated, the ones which ended before it stops or cancels a workflow (the last `KEPT_TAIL_LINES` lines, at most what the backend reads of a log): a file `.log` per pod (every attempt the backend can still read) and `index.json`, the pods in their order, written last, which marks the logs as kept. An ended run is followed until its logs are kept or its workflow is gone (`RunRecord.pending_logs`); a run which never had a workflow keeps none. The API serves the kept logs first and the logs of the live pods otherwise. ## function `keep(task_dir: pathlib.Path, logs: list[simrun.service.backend.PodLog]) -> None` Keep the logs of the pods of a task in `task_dir/logs`. Args: task_dir: The directory of the task. logs: The logs of its pods, oldest first. ## function `kept(task_dir: pathlib.Path) -> bool` Whether the logs of a task are kept in its directory. ## function `read_kept(task_dir: pathlib.Path, tail_lines: int | None = None) -> list[simrun.service.backend.PodLog] | None` The kept logs of a task, the last `tail_lines` lines of each. Returns: The logs, None when they are not kept (yet). ## function `render(logs: list[simrun.service.backend.PodLog]) -> str` The logs of the pods of a task as one text, each pod under a header. --- # simrun.service.compare The comparison of runs: provenance and the summaries of their tasks side by side. ## class `CompareTable(*, columns: list[str], rows: list[dict[str, typing.Any]]) -> None` A table of runs. Attributes: columns: The columns in order: `run`, `group`, `agent`, `state`, then per source `.changed_files` and `.diff_sha256`, then every key of `out/summary.json` of every task as `.`. rows: One row per run, with every column (None where a run has no value). ## function `compare_runs(store: simrun.store.RunStore, run_ids: list[str]) -> simrun.service.compare.CompareTable` Compare runs by their provenance and the summaries their merges wrote. Args: store: The run store. run_ids: The runs, one row each in this order. Returns: The table. Raises: KeyError: A run does not exist. --- # Release notes The notes of every release, newest first. They are kept in `release-notes/` of the repository and are the text of the [GitHub releases](https://github.com/matthiaskoenig/simrun/releases). # Release notes for simrun 0.3.2 A release for the tests: the package is the same as 0.3.1. ## Fixes - The k3d tests try the download of their tools (k3d, kubectl) again after a server error or a network error, up to five times with a growing pause, instead of failing at once: a gateway timeout of GitHub failed the cluster job of the release 0.3.1. An error of the client, e.g. a missing file, still fails at once. ## Upgrading - Nothing to do. # Release notes for simrun 0.3.1 Faster images and a run on the k3d test cluster which takes about as long as on the machine itself: the smoke workflows run every analysis of sbmlsim and the HCTZ model of pkdb_models through simrun, and their timelines show where the time of a run goes. A retry with more memory, which never fitted its node, now does. ## Features - **Smoke workflows.** `tests/smoke/run.py` runs the analyses of sbmlsim (simulate, fit, identifiability, fisher, report, petab, scan, sensitivity, uncertainty) and the simulations, a fit and the build of the models of pkdb_models through simrun, from the checkouts of both, locally (`--local`) or on the local test system (`--k3d`). After every run it prints the timeline of its tasks (`tests/smoke/timeline.py`): the wait for the dependencies, the pod, its admission and its start, the preparation, the work, the merge, the cpu and the CPU throttling of every task, and the seconds of the environment and the image. On the same machine sbmlsim takes 78 s on k3d and 60 s locally, pkdb_models about 250 s both. See [Smoke workflows](https://matthiaskoenig.github.io/simrun/testing/#smoke-workflows). - **Faster images.** The images compile the bytecode of their packages, so a pod no longer compiles at every import, and keep the downloads of uv in a cache of BuildKit, whose state is now on a volume and outlives its pod. A cold image of sbmlsim builds in 47 s instead of 85 s, an image after a change of simrun in 6 s, and is 378 MiB instead of 583 MiB. - **Git in the images.** The image of an environment installs git when a dependency is locked from a git repository, e.g. sbmlsim from GitHub in the lock of pkdb_models. - **The reports of the workers.** A report carries when its harness started, the seconds it prepared (the install of the sources, the restart, the imports), the cpu of its units and, in a pod, the seconds its CPU quota held it back (`started`, `prepare_seconds`, `cpu_seconds`, `throttled_seconds`). The service reads the report of a newer harness and ignores the fields it does not know. The coordinator logs the seconds of its merge. ## Fixes - A retry with more memory waited for Kueue forever when its memory, e.g. 5 GiB times the retry factor, came out a hair above a whole GiB and was rounded up beyond the memory of every node: a pod is now sized to whole GiB without the noise of floating point. - A watch of the workflows which the API server or a proxy cut starts again at once with a note in the log, instead of with a warning after 5 s. - BuildKit asks for one core in k3d and may use four, so the controllers still fit the server node of a runner with four cores, and the setup of the k3d cluster fails at once on an error of the manifests instead of after ten minutes. ## Upgrading - Apply `deploy/cluster` again: BuildKit gets the volume `buildkitd-state` (local-path, 50 GiB) and a rolling update of one pod at a time. - Every model builds its image once more on its first run, as the Dockerfile changed. # Release notes for simrun 0.3.0 Faster workflows on the cluster and the first models of pkdb_models on it: a split task gets only the pods its work pays for, the service follows Argo through a watch instead of a poll, and the images of the environments have the system library which every SBML model needs. Locally, `simrun submit --local` runs every task type of sbmlsim and the projects at the root of a checkout, e.g. the tutorial workflow of sbmlsim. ## Features - **The split sized by the work.** A task of the split mode `fastest` gets no pod without a unit for each of its processes, and once the seconds of a unit are learned, a further pod only while every pod still has at least `pod_start_s` of work (new in `ClusterConfig`, 10 s). A pod which started after the others had taken the units only held cores. `demo.squares` learns per `sleep_s`. See [Split modes](https://matthiaskoenig.github.io/simrun/concepts/#split-modes). - **A watch of the workflows.** The service watches the workflows of the runs and reconciles a run as soon as Argo changes it; `poll_s` is the fallback. The event streams of the web app check the store every `events_poll_s` (new, 0.5 s) instead of every `poll_s`. Argo requeues a workflow after 1 s instead of 2 s, so the next task of a workflow starts about a second earlier. A workflow of two split tasks and a command takes 35 to 38 s on the k3d test cluster instead of 40 s; most of the rest is the start of the pods. - **System packages in the images.** The images of the environments install the Debian packages of `ClusterConfig.system_packages` on the base image, by default `libexpat1`, which the wheel of python-libsbml links. A fit of a pkdb_models model failed on the cluster without it. An image is named by its Dockerfile as well, so another base image or other system packages give a new image. - **Projects at the root of a checkout.** A task imports from the root of the model first, as `python -m` in its checkout: the projects at the root of a checkout which no package installs, e.g. the `examples` of sbmlsim, run through simrun, and the client plans with the same path (`SourceProvenance.path`, new). - **The settings form of the Editor.** It leaves out the fields the JSON Schema marks `readOnly`, which the plan of a type sets, e.g. `units`, and shows no placeholder `null`. The logs of a task in the inspector can wrap their lines at its width (**wrap lines**). - **Tables on 80 columns.** The tables of the command line have a rule under the header instead of a frame and keep every value whole: a long value wraps in its column, numbers are right-aligned. The list of runs has no columns for the agent and the time of creation, which the submitter and the run id carry. ## Fixes - An unsplit task of a plugin type, e.g. `sbmlsim.simulate` or `sbmlsim.report`, failed on the local backend with `needs the image of the run, and the run has none`: a local run has no image, and the argv of the task is now built without one (`podspec.unsplit_argv`). - A local task runs with the environment of the service activated, so the command of a plugin (`python -m ...`) finds its python also when the service runs from a virtual environment which is not activated. - A pod of a split task has no more processes than the task has units, so the other cores stay free for other tasks; an identifiability analysis of three parameters took every core of the machine before. - The manifest, the Dockerfile and the deployment documentation of the service named simrun 0.1.0; `bump-my-version` now updates them with the version. ## Upgrading - Apply `deploy/cluster` again: the controller of Argo gets `DEFAULT_REQUEUE_TIME=1s`. The Role of the service grants `watch` on workflows already. - Every model builds its image once more on its first run, as the images are named anew; the images of 0.2.0 lack libexpat. - `events_poll_s`, `pod_start_s` and `system_packages` are optional settings with defaults. # Release notes for simrun 0.2.0 Workflows without a cluster: `--local` runs a workflow on this machine through a local simrun service, with the same planner, harness, records and web app as on the cluster, at the speed of running the code directly. The cluster gets faster split tasks and a client which needs no mount of the store. ## Features - **The local mode.** `simrun submit fit.yaml --source model=. --local` runs on this machine, the same command without `--local` on the cluster. A local backend runs the tasks of a run as processes (a split task on the cores of the machine, several runs sharing them, a `command` with its args directly or in its image with `--docker`), and the service of a project lives in `.simrun/` next to the workflow: started on demand by any `--local` command, on `127.0.0.1` only, stopped after 30 idle minutes or with `simrun serve --local --stop`, and restarted when the python environment, the simrun version or the task types change. Sources are recorded (branch, commit, uncommitted files) without copying; `--snapshot` installs them into an environment of the run instead. A run of three tasks takes 4.8 s for 4.6 s of work. See [Local mode](https://matthiaskoenig.github.io/simrun/local/). - **The web app locally.** The service of a project serves the web app at `/ui/`, Executions and Editor (`simrun edit --local`); the wheel ships the built app. - **`submit --wait` and `simrun outputs`.** `submit --wait` follows a run with a live table per task and exits like `wait`, the default with `--local`; `simrun outputs RUN TASK [--to DIR]` lists or downloads the outputs of a task through the API. - **Snapshots through the API.** Without `SIMRUN_STORE` the client uploads the files of a working tree to the service, only those the store lacks, so a laptop on the VPN needs a URL and a token only. The snapshots and their hashes are those of 0.1.0. - **The coordinator merges.** A split task has no merge pod any more: its coordinator merges once its ledger is complete and checks the ledger every 0.5 s instead of 5 s, which saves several seconds per split task; a coordinator which succeeded without merging (an image with the harness of 0.1.0) fails its task after a grace of two minutes. - **The web app.** Why a task failed (the exit of its last pod, also in the message of the run), the phase instead of a dash (`starting`, `merging`), a worker whose unit or command failed shown as failed, one short name per server, the lines of the Argo executor hidden unless asked for and errors in red; in the Editor the planned memory per worker, a `+` handle to add an input of a `command`, the inputs of a `command` in its inspector, older closed sessions behind *Show older*. The documentation shows the views. ## Fixes - A run with the simrun repository as a source failed in every pod: the harness now restarts after installing the sources. - A failure which is not a stop is no longer retried a hundred times: only a pod which was stopped (no exit code, 137, 143) is retried, and the message of the run names the failed pod and its exit. - The `args` of a `command` get `{input}`, `{out}` and `{settings}` as a plugin command does; a `command` cannot stage its inputs. - The image of an environment is named also by the hash of the simrun wheel, so a new wheel of the same version builds a new image. - Argo requeues a workflow after 2 s instead of 10 s, so a run shows running soon after its pods start. ## Upgrading - Add `lookupcache=positive` to the `mountOptions` of the existing PersistentVolume of the store and restart the pods (`deploy/cluster/store.yaml` has it for new installs; see [Cluster deployment](https://matthiaskoenig.github.io/simrun/deployment/)). - Runs recorded by 0.2.0 carry `coordinator_merges` and `merge_pending`, which a service of 0.1.0 cannot read: the service cannot be rolled back to 0.1.0 once it recorded runs. - Runs of 0.1.0 still in flight keep their merge pod. # Release notes for simrun 0.1.0 The first release of `simrun`, a runner of simulation workflows on a small k3s cluster with Argo Workflows and Kueue, for people and agents ([sbmlsim#440](https://github.com/matthiaskoenig/sbmlsim/issues/440)). ## Features - **Workflows of tasks with ports.** A workflow is a YAML file of tasks of task types from plugins, whose inputs read the output ports (`out/`) of other tasks; the settings of a task are validated with the settings model of its type, so a misspelled setting is an error. The built-in type `command` runs a command in an image of its own. - **Split tasks on a ledger.** A task type with `count_units`, `compute_unit` and `merge` is split into units, which the workers of its pods claim from a ledger of atomic renames in the run store. A coordinator returns once the ledger is complete and computes the units of dead workers, extra workers spread over the idle servers (`fastest`, at most eight pods) or the task stays on one server (`single-server`), and the merge runs once every unit is done; the result does not depend on the number of pods. - **Priorities, preemption and take-back.** The runs of the owner of the cluster and the owner's agents, of the other people and of their agents have three priorities, each a base and a lower extra priority of Kueue, so a run of a higher priority preempts the runs of lower ones and a new run takes back the extra workers of the running runs of its own priority. A node can be reserved for one run. - **Learned resources.** The memory of a pod is learned from the anonymous memory of the cgroups of earlier runs of its task type, model and cost setting; a pod killed for memory is retried once with 1.5 times the memory per process, and `estimate` predicts the duration of a split task from the seconds of its units. - **Snapshots and environments.** The client snapshots the working trees of the model and its libraries, uncommitted changes included, into the run store with their provenance; the service locks their third party dependencies with uv, builds the image of the environment with BuildKit only when the registry does not have it, and the pods install the read-only snapshots editable at their start. - **Plans and provenance of tasks.** A task type can plan its settings before a run is submitted (`plan`, called by the client where the type is installed, also for `estimate` and `validate`); every task gets the provenance of the sources of its run (`provenance.json`, `SIMRUN_PROVENANCE`), and a `command` reads its inputs through `{input}`, a staged input from a local copy on the scratch of the node. - **The service, the command line and the MCP server.** A FastAPI service with bearer tokens of people and agents submits a run in the background and returns its id at once, follows the workflows with a stateless controller, keeps the logs of the pods and an index of the runs in SQLite beside the store; the command line `simrun` and the MCP server `simrun-mcp` of the agents submit, wait for, list, compare, cancel and clean up runs. - **Cluster deployment.** `deploy/cluster` is the cluster as a kustomization (Kueue v0.20.5, Argo Workflows v4.1.6, the run store on NFS, Zot, BuildKit and the service), `deploy/k3d` the overlay of the integration tests on k3d, which verify the design on a real cluster. The same run exports as a CWL workflow for cwltool (the extra `cwl`). - **The web app.** The service serves a web app (Vue 3, Vue Flow, Tailwind) at `/ui/`, live by server-sent events. *Executions*: the runs of people and agents with filters and groups and their progress, the capacity of the cluster, a strip of the servers with the cores and memory of every run, the canvas of a run with the state, units and workers of every task, and an inspector with the workers, outputs and live logs. *Editor*: `simrun edit ` on the machine with the code opens the workflow in the browser, which may run on another machine; a palette of the task types of the service, a canvas to add and connect tasks, the settings as a form of their JSON Schema, the resources with the estimate; Save writes the file next to the code, Run snapshots the working trees again; a file changed on disk is a conflict, not an overwrite; Rerun with changes opens a run in the Editor. - **A local test system.** `tests/cluster/local.py up` keeps the k3d cluster of the integration tests as a local system with the service on the host, for the command line, the MCP server and the web app (`examples/squares.yaml`). --- # Development Contributions are welcome. The repository is [matthiaskoenig/simrun](https://github.com/matthiaskoenig/simrun); development happens against the `develop` branch via pull requests. ## Branch model Two branches are permanent: - **`develop`** is the default branch and the branch everything is integrated into. The documentation on [matthiaskoenig.github.io/simrun](https://matthiaskoenig.github.io/simrun) is published from it. - **`main`** tracks the latest published release. It is fast-forwarded to the released commit by the `sync-main` job of the `CI-CD` workflow after the package went to PyPI, so `main` and the newest version on PyPI always agree. Nothing is developed on `main` and nothing is merged into it by hand. Work happens on short lived branches off `develop`, which GitHub deletes after the merge. Releases are tagged on `develop`, see [Release](#release). ## Pull requests Neither branch accepts a direct push, every change goes through a pull request against `develop`. This includes the maintainer, there is no bypass. A pull request can only be merged once the four required checks are green: | check | workflow | content | | --- | --- | --- | | `tests` | `ci-cd.yml` | the unit tests on python 3.14 and 3.15 on Linux and the plain install | | `ruff` | `lint.yml` | `ruff check` and `ruff format --check`, with the ruff of `uv.lock` | | `ty` | `lint.yml` | `ty check` | | `docs` | `docs.yml` | the strict zensical build including the API reference and the agent files | `tests` aggregates the test matrix and the job `plain install` into a single job, so the name of the required check stays the same when the matrix changes. `plain install` installs the package without extras, as `pip install simrun` does, imports every module and runs `simrun --help`: the test matrix has the extras of `dev`, which would hide a dependency the package needs and does not declare. It is the one job which does not use the lock, see [Dependencies](#dependencies). The tests on the k3d cluster, the job `cluster` (`cluster.yml`), take about half an hour and are no required check. They run on every push to `develop`, on a release, which waits for them, and when `CI-CD` is started by hand (`workflow_dispatch`). A pull request which changes what runs on the cluster gets the label `cluster`: the tests run once the label is added and with every push after it. Every job sets up its environment with the action `.github/actions/setup`: uv with its cache of the packages and the interpreters, the python of the job and `uv sync --locked` with the extras of the job, i.e. the versions of `uv.lock`. The key of the cache of uv has the os and the python version but not the job, so jobs on the same python would race to save it and warn `Unable to reserve cache`: only the jobs of the test matrix save the cache (`save-cache: "true"`), the others restore it. The jobs run the same commands as a local environment (`pytest`, `ty check`, `zensical build`), the tests against the package installed as a wheel (`uv sync --no-editable`). A run of a pull request is cancelled by the next push to it; a push to `develop` or `main` is never cancelled, a release waits for the run of its commit. The release builds the distributions without the cache (`cache: "false"`). Dependabot proposes its updates once a month. Further rules of a pull request: - conversations have to be resolved before the merge - an approval is dismissed when new commits are pushed - the history stays linear, i.e., a pull request is merged with squash or rebase; merge commits are disabled - the maintainer is the code owner of the repository (`.github/CODEOWNERS`) and is requested for review on every pull request. The rulesets themselves do not require an approval: on a personal repository a ruleset cannot ask for an approval only from somebody else, and requiring one would block the pull requests of the maintainer, who cannot approve their own [Auto-merge](https://docs.github.com/pull-requests/collaborating-with-pull-requests/incorporating-changes-from-a-pull-request/automatically-merging-a-pull-request) is enabled for the repository, so a pull request can be queued and is merged as soon as the checks pass. ### Repository policies { #repository-policies } The protection is implemented with [repository rulesets](https://docs.github.com/repositories/configuring-branches-and-merges-in-your-repository/managing-rulesets/about-rulesets). They are part of the repository in `.github/rulesets/` instead of only living in the web interface, so a change to a policy is reviewed like any other change: | ruleset | applies to | rules | | --- | --- | --- | | `develop.json` | `develop` | pull request required, the four checks above, resolved conversations, linear history, no force push, no deletion. **No bypass, for anybody.** | | `main.json` | `main` | linear history, no force push, no deletion, no bypass. The fast-forward of the release workflow needs none, only a force push would be rejected | | `tags.json` | all tags | a tag cannot be deleted or moved, so a release tag keeps pointing at what was released | Changing a policy means changing the json and applying it: ```bash .github/rulesets/apply.sh ``` The script is idempotent: it updates the rulesets which exist and creates the missing ones. It also sets the merge settings of the repository, i.e., auto-merge, delete branch on merge, and squash and rebase as the only merge methods. It needs the [github cli](https://cli.github.com) authenticated as a user with admin permission on the repository. ## Setup development environment Development needs [uv](https://docs.astral.sh/uv/) and a checkout of the repository: ```bash git clone https://github.com/matthiaskoenig/simrun.git cd simrun ``` A single sync creates the virtual environment in `.venv`, installs `simrun` into it in editable mode and adds the complete tooling, in the versions of `uv.lock`: ```bash uv sync --extra dev ``` The `dev` extra contains everything used below, i.e., pytest, ruff, ty, tox with tox-uv, pre-commit, zensical, bump-my-version and the extra `cwl`, so nothing has to be installed separately. The python version is taken from `.python-version` (currently 3.14). The tools are then run either with `uv run `, which uses the environment without activating it, or from the activated environment (`source .venv/bin/activate`). The commands in this document are written without the `uv run` prefix; prepend it if the environment is not activated. The last step installs the git hook: ```bash uv run pre-commit install # install the hook, once per checkout uv run pre-commit run --all-files # check the current state of the repository ``` From now on every commit is checked with ruff (lint and format) and ty, i.e., the same checks that run in continuous integration. On a commit only the changed files are looked at, `--all-files` checks the whole repository and is what a newly added hook should be tried with. The release manifests of Argo Workflows and Kueue in `deploy/cluster` are committed as released and excluded from the hooks. ## Dependencies The environment is resolved from `uv.lock`, which is committed. It pins every package of the environment, i.e. the dependencies, the extra `cwl` and the tools of `dev`, so a local `uv sync` and continuous integration install the same versions and a result can be reproduced; the image of the service installs simrun with the dependencies of the same lock. Every job which syncs through the action `.github/actions/setup` runs against the lock, and the `ruff` check installs the ruff version of the lock (`version-file: uv.lock`), so continuous integration and `uv run ruff` agree. Three things do not use the lock on purpose. The job `plain install` resolves from `pyproject.toml` as `pip install simrun` does, i.e. the newest releases within the bounds, so a release of a dependency which breaks simrun shows up there before the lock is moved to it. The tox environments resolve from `pyproject.toml` as well. The release builds and checks the package with `uvx hatch` and `uvx twine`. The lower bounds in `pyproject.toml` are what a user of the library installs against, the lock only records the versions the development runs with. After changing a dependency in `pyproject.toml` run `uv lock`: the action syncs with `uv sync --locked`, which fails when `pyproject.toml` and the lock disagree, so the change and the `uv lock` which records it go into the same pull request. `uv lock --upgrade` moves every package of the lock to its newest release, `uv lock --upgrade-package ` a single one. Dependabot keeps the lock current: once a month it opens one pull request which bumps the packages of the lock together, with the prefix `build`, and one which bumps the GitHub Actions, with the prefix `ci`. Its `versioning-strategy: lockfile-only` only changes `uv.lock`, the lower bounds in `pyproject.toml` are never raised by it and change by hand. The revisions of the ruff and ty hooks in `.pre-commit-config.yaml` are the versions of the lock, which dependabot does not update: when such a pull request moves ruff or ty, raise the revision of the hook to the same version. The web app in `frontend/` has a lock of its own, `frontend/package-lock.json`, which a third monthly pull request moves, also with the prefix `build`; it leaves TypeScript below 7, whose native compiler has no API for `vue-tsc` and `openapi-typescript`, and the types of node at the node of continuous integration, 24. ## Testing The tests are written with pytest, tox runs them against every supported python version (`py3.14`, `py3.15`, see `envlist` in `tox.ini`): ```bash tox r -e py3.14 # a single environment tox run-parallel # every supported python, before a pull request is opened ``` This needs the interpreters, which uv installs with `uv python install 3.14 3.15`. To run the tests directly against the development environment use `pytest`; the tests run in parallel, `-n auto` in the `addopts` of `pyproject.toml` gives pytest-xdist one worker per core, `-n 0` runs everything in one process, which the debugger needs. The tests on the k3d cluster are described in [Testing](testing.md#tests-on-a-k3d-cluster). ## Linting and formatting Linting and formatting use [ruff](https://docs.astral.sh/ruff/): ```bash ruff check # lint ruff format # format ``` The rules are in `.ruff.toml`; among them the docstring rules (`D`, google convention), which are enforced for the package and not for `tests/`, and the rules of logging calls (`G`): library code logs with lazy `%s` formatting rather than f-strings. ## Type checking Type checking is performed with [ty](https://docs.astral.sh/ty/): ```bash uv run ty check ``` ty resolves the imports in the environment of the project, which `uv sync --extra dev` creates; the `ty` check of continuous integration runs the same command and the pre-commit hook runs it on every commit. The configuration lives in `[tool.ty]` in `pyproject.toml`. Warnings are treated as errors, so the codebase is kept free of diagnostics. Suppress an unavoidable diagnostic with a rule specific `# ty: ignore[rule-name]` rather than a blanket comment. ## Documentation The documentation is built with [Zensical](https://zensical.org/), the static site generator of the Material for MkDocs authors. The sources are markdown files in `docs/`, the site is configured in `zensical.toml` in the repository root. Nothing rendered is committed: the site is built by the `documentation` workflow on every push and published to [matthiaskoenig.github.io/simrun](https://matthiaskoenig.github.io/simrun) from the `develop` branch. Build the site into `site/`, strict as continuous integration builds it, i.e. a warning fails the build: ```bash uv run zensical build --clean --strict ``` For writing, the preview rebuilds on save: ```bash uv run zensical serve ``` The API reference is rendered from the docstrings by [mkdocstrings](https://mkdocstrings.github.io/); a page in `docs/api/` only contains the module directive: ```markdown # ledger ::: simrun.ledger ``` Docstrings are therefore the place to document functions and classes, the markdown files provide the narrative around them. Adding a module to the reference means adding such a page, an entry to `nav` in `zensical.toml` and a row of `docs/api/index.md`. Markdown carries no hard line wraps: a paragraph, a list item or a table row is a single line. ### Files for agents { #files-for-agents } Agents and language models read markdown, not rendered html. `scripts/llms_txt.py` writes the files of the [llms.txt convention](https://llmstxt.org/) into the built site, i.e., [llms.txt](https://matthiaskoenig.github.io/simrun/llms.txt) as an annotated index of all pages, [llms-full.txt](https://matthiaskoenig.github.io/simrun/llms-full.txt) with the complete documentation in a single file, and the markdown of every page next to its html. The markdown of the API reference is generated from the docstrings with `inspect`, since the pages themselves only contain the mkdocstrings directive. ```bash uv run zensical build --clean --strict uv run python scripts/llms_txt.py ``` The `documentation` workflow runs both steps, so the files are regenerated with every push. `docs/robots.txt` points crawlers at the sitemap and at these files. ## Release A release is made from `develop`. Since `develop` only accepts pull requests, the release is prepared on a branch and tagged once that pull request is merged: 1. branch off `develop`: `git switch -c release/x.y.z develop` 2. write the release notes for the version in `release-notes/x.y.z.md` and add them to `docs/release-notes.md` 3. make sure everything passes: `tox run-parallel`, `ruff check`, `uv run ty check` 4. check the version bump: `uvx bump-my-version bump [major|minor|patch] --dry-run -vv` 5. bump the version: `uvx bump-my-version bump [major|minor|patch]`, which updates `src/simrun/__init__.py` and `CITATION.cff` and commits. It does not create the tag; a squash or rebase merge would rewrite the commit and leave the tag behind on a commit which is not part of `develop` 6. push the branch, open the pull request against `develop` and merge it once the checks are green 7. tag the merged commit on `develop` and push the tag: ```bash git switch develop git pull git tag x.y.z git push origin x.y.z ``` This starts the `CI-CD` workflow, which checks that the tag is the version of the package, builds the sdist and the wheel, publishes them to [PyPI](https://pypi.org/project/simrun/), creates the GitHub release from `release-notes/x.y.z.md` and fast-forwards `main` to the tagged commit. It does not run the test matrix again: the merge into `develop` ran it on the same commit, and the job `tested on develop` waits for that run and stops the release unless it passed, so the tag can be pushed right after the merge. Next to it the tests on the k3d cluster run on the tag, and the release waits for them. Check the version before pushing, a tag cannot be moved or deleted afterwards. The wheel and the sdist contain the built web app in `simrun/web`, which `simrun serve --local` serves: the job builds `frontend/` and copies `frontend/dist` to the gitignored `src/simrun/web` before the build (`artifacts` of `pyproject.toml`). A package built without it has no web app, and a checkout serves `frontend/dist` once built. To build the package with the web app locally: ```bash (cd frontend && npm ci && npm run build) rm -rf src/simrun/web && cp -r frontend/dist src/simrun/web uvx hatch build ``` 8. test the installation from PyPI in a fresh environment: ```bash uv venv --python 3.14 uv pip install simrun ``` 9. once Zenodo has archived the release, update the citation information, i.e., `date-released` and `doi` in `CITATION.cff` and the DOI badge and the citation of `README.md` and `docs/index.md`. `bump-my-version` only updates the version, not the date and the DOI, which are only known after the release. These changes go in through a pull request like everything else ### PyPI and Zenodo The package is published with [trusted publishing](https://docs.pypi.org/trusted-publishers/): the job `release` runs in the GitHub environment `pypi` with the permission `id-token: write`, and PyPI accepts the distributions of this workflow without a token. The publisher is registered on PyPI for the project `simrun`, the owner `matthiaskoenig`, the repository `simrun`, the workflow `ci-cd.yml` and the environment `pypi`. [Zenodo](https://zenodo.org) archives every GitHub release of the repository with the metadata of `.zenodo.json` and gives it a DOI; the first release creates the concept DOI, which stands for every version and goes into the badge and the citation.