Testing¶

The tests are written with pytest and live in tests/. Most of them are unit tests, which run anywhere; the tests in tests/cluster run on a k3d cluster and are deselected by default (-m 'not cluster' of the addopts of pyproject.toml).

The local mode needs no cluster: simrun submit examples/squares.yaml --local runs a workflow on this machine, and its tests (tests/test_local.py, tests/test_cli_local.py) are part of the unit tests.

Unit tests¶

pytest                                         # the unit tests, in parallel with pytest-xdist
pytest -n 0                                    # in one process, e.g. for --pdb
pytest tests/test_ledger.py                    # a single module
pytest tests/test_ledger.py::test_claim_order  # a single test
tox r -e py3.15                                # on one python in a fresh environment
tox run-parallel                               # on every supported python

The tests of the service run the API with an in-memory backend (simrun.testing.FakeBackend) which the test advances, so the controller, the submissions and the index are tested without a cluster; the tests of the environments and the snapshots call uv and git, the tests of the CWL export run cwltool. Continuous integration runs them on python 3.14 and 3.15 on Linux.

Tests on a k3d cluster¶

SIMRUN_K3D=1 uv run pytest -m cluster tests/cluster
SIMRUN_K3D=1 SIMRUN_K3D_KEEP=1 uv run pytest -m cluster tests/cluster  # keep the cluster

They need docker, node 24 (the web app of the image of the service), about 20 GiB of free disk and the network (the images of k3s, Kueue, Argo, BuildKit, python:3.14-slim-trixie and the uv base image). The session downloads k3d and kubectl into .tools/bin, creates the k3d cluster simrun-test (one server with the infrastructure; three agents of 6 GiB labelled simrun/worker=true) with the registry k3d-simrun-registry:5000 (localhost:5000 from the host), writes its kubeconfig to .tools/kubeconfig (the contexts of the user are not touched), applies deploy/k3d, builds the image of the runs without sources from the working tree (tests/cluster/Dockerfile) and pushes it, and copies the base image of the images of environments (ClusterConfig.base_image, from ghcr.io/astral-sh/uv) into the registry with its digest.

Each test runs the service in its process with an ArgoBackend and the store .tools/k3d/store, which k3d mounts into every node:

At the end the cluster, the registry, the store and the images are deleted (the files the pods of an environment wrote as root are given back to the user first), unless SIMRUN_K3D_KEEP=1; a kept cluster is used by the next session. The tests run in one worker of pytest-xdist, which shares the nodes, and take about nine minutes on a created cluster.

A local test system¶

The k3d cluster of the cluster tests also serves as a local test system: the service runs on the host against it, and runs are submitted with the command line or the MCP server as on the real cluster. It needs docker, about 20 GiB of free disk and the network, like the tests.

uv sync --extra dev
uv run python tests/cluster/local.py up      # about two minutes, reuses a cluster which exists

up creates the cluster simrun-test as the session of the cluster tests does (or reuses it), builds the image of the runs from the working tree and pushes it, and writes the settings and the tokens of the service into .tools/k3d/local. It ends with the commands of the next two steps. Start the service in one shell:

export KUBECONFIG=$PWD/.tools/kubeconfig
export SIMRUN_CONFIG=$PWD/.tools/k3d/local/settings.yaml
uv run uvicorn --factory simrun.service.app:serve --port 8000 --timeout-graceful-shutdown 5

and submit from another, e.g. the workflow examples/squares.yaml of A first run:

export SIMRUN_URL=http://localhost:8000
export SIMRUN_TOKEN=t-mk                       # the owner; t-mk-bot, t-alice and t-alice-bot test the priorities
export SIMRUN_STORE=$PWD/.tools/k3d/store      # the run store, mounted into every node

uv run simrun submit examples/squares.yaml
uv run simrun wait <run>
uv run simrun status <run>

To edit a workflow in the Editor of the web app, start the command which serves it in the same environment (on a copy, as a save writes the file):

cp examples/squares.yaml /tmp/squares.yaml
uv run simrun edit /tmp/squares.yaml           # prints the link of the Editor, Ctrl-C ends the session

The service also serves the web app at http://localhost:8000/ui/ (log in with t-mk): up points the setting web at frontend/dist when the app is built (npm ci && npm run build in frontend/, then up again). For the development of the app, run its dev server against this service, which reloads on every change of the code:

cd frontend && npm run dev                     # http://localhost:5173/ui/, the API proxied to http://localhost:8000

The service knows the task types installed in .venv; the runs without sources use the image built by up, a model submitted with --source gets an image of its environment built by BuildKit in the cluster. The workflows and pods are seen with .tools/bin/kubectl --kubeconfig .tools/kubeconfig -n simrun get workflows,pods.

After a change of the code of the pods, run up again, which builds a new image, and restart the service. The cluster is the one SIMRUN_K3D_KEEP=1 keeps, so the cluster tests and the local system share it. down deletes the cluster, the registry, the store and the settings:

uv run python tests/cluster/local.py down

Smoke workflows¶

The smoke workflows run the analyses of sbmlsim and pkdb_models through simrun, to find what breaks a real model before the migration of the models does: tests/smoke/sbmlsim.yaml every kind of analysis of the HCTZ example of sbmlsim (simulate, fit, identifiability, fisher, report, petab, scan, sensitivity, uncertainty) with the checkout of sbmlsim as its source, and tests/smoke/pkdb_models.yaml the simulations, a fit with the analyses which read it and the build of the models of the HCTZ model of pkdb_models, with the checkouts of pkdb_models and sbmlsim as its sources. They need checkouts of both (by default ~/git/sbmlsim and ~/git/pkdb_models):

uv run python tests/smoke/run.py                       # --local, on this machine (about 5 minutes)
uv run python tests/smoke/run.py --k3d                 # on the local test system, after `tests/cluster/local.py up`
uv run python tests/smoke/run.py --only sbmlsim --sbmlsim ../sbmlsim --pkdb-models ../pkdb_models

run.py installs this checkout of simrun and the two checkouts editable into an environment of its own, .tools/smoke/venv, so the client plans with the plugin of sbmlsim and the service knows its task types. Locally it uses the local service of .tools/smoke/local and stops it at the end; with --k3d it runs a service of that environment against the local test system on a free port, and the first run of each workflow builds the image of its environment. --local --k3d runs both, one after the other. After every run it prints the timeline of the run (tests/smoke/timeline.py, also on its own for any run of a store) and the load of the machine, which other work on it changes:

smoke-pkdb-models 20261108-235754-cc7e82 succeeded in 263 s
  environment and image 1 s, workflow 0 s
  task              start waited   dag admit start  prep   work merge    end    cpu procs thrott
  factory              +6      6     1     1     3     1      8     0    +15      9     1      0
  fit                  +6      6     1     1     3     2    102     1   +110    193     2      0
  simulate             +6      6     1     1     3     1     46     0    +53     19    11      0
  ...
  identifiability    +116      6     2     1     2     2     24    20   +163     26     1      0
  report             +171      9     4     1     2     1     87     0   +259     88     1      0

start and end are seconds from the creation of the run; waited the seconds from the end of the tasks a task reads (or the creation of the run) to the start of its harness, the submission and the start of its pod on the cluster, the free cores locally; prep the install of the sources, the restart and the imports; work its command or its units; merge the merge of a split task by its coordinator; cpu the cpu seconds of its processes; procs its processes. On k3d the pods split waited into dag (to the creation of the pod), admit (Kueue) and start (the image and Argo's init container), thrott is the time the CPU quota of its pods held it back, and environment and image the seconds the service took to resolve the environment and to build the image when it was missing. The smoke workflows read the checkouts as they are: for numbers to compare, point --sbmlsim and --pkdb-models at clones of fixed commits. The reports of the workers carry these times (started, prepare_seconds, cpu_seconds, throttled_seconds). The exit code is 1 when a run did not succeed.

Continuous integration¶

In continuous integration the job cluster runs the tests on the k3d cluster on a push to develop, on a release, when the workflow CI-CD is started by hand and on a pull request with the label cluster, see Development.