Skip to content

plugins

The plugin interface: task types, their resource hints and their discovery.

ResourceHint dataclass 

ResourceHint(
    memory_base_gb=2.0,
    memory_per_worker_gb=6.0,
    scratch_gb=0.0,
    gpus=0,
)

Default resources of a task type.

Attributes:

Name Type Description
memory_base_gb float

Base memory of a task in GiB.
memory_per_worker_gb float

Memory per worker in GiB.
scratch_gb float

Local scratch in GiB.
gpus int

Number of GPUs.

TaskPaths dataclass 

TaskPaths(store, run_dir, task_dir, out, inputs, scratch)

The locations a task works in.

Attributes:

Name Type Description
store Path

Root of the run store.
run_dir Path

Directory of the run.
task_dir Path

Directory of the task.
out Path

Output directory of the task, task_dir / "out".
inputs dict[str, Path]

Input port to the directory of the output it reads.
scratch Path

Local scratch directory.

TaskType dataclass 

TaskType(
    name,
    settings,
    inputs=(),
    outputs=("out",),
    hint=ResourceHint(),
    checkpoints=False,
    command=None,
    uses_cores=False,
    count_units=None,
    setup=None,
    compute_unit=None,
    merge=None,
    plan=None,
    image=None,
    cost_setting=None,
    description="",
)

A type of task, registered as an entry point of ENTRY_POINT_GROUP.

The callables are "module:function" strings, so a worker process resolves them by import, see resolve. Signatures: count_units(settings) -> int, setup(settings, paths) -> None (once per process), compute_unit(settings, index, paths) -> None, merge(settings, paths) -> None and plan(settings) -> settings, which the client calls before a submission when the type is installed locally (simrun.client.plan_tasks).

A unit may be computed more than once: by a worker which is evicted before it completes the unit, or by a restarted incarnation of a worker whose earlier one still runs. So compute_unit writes outputs which depend on the settings and the index only (deterministic), each written atomically (a temporary file in the same directory, then renamed), so a reader never sees a partial unit.

Attributes:

Name Type Description
name str

Name of the type, as used in a workflow.
settings type[BaseModel]

Pydantic model of the settings.
inputs tuple[str, ...]

Names of the input ports.
outputs tuple[str, ...]

Names of the output ports.
hint ResourceHint

Default resources.
checkpoints bool

Whether the task resumes from checkpoints.
command str | None

Argv template of unsplit tasks with {settings}, {out} and {input}: a part with {input} is repeated once per input port of the task, sorted by port, with <port>=<path> (<scratch>/<port> for a staged port), and dropped without inputs. It runs through simrun-exec (see simrun.worker.exec_main).
uses_cores bool

Whether the command of an unsplit task uses the cores of its pod (a pool of processes); else the pod is sized for one process.
count_units str | None

Reference to the function counting the units of work.
setup str | None

Reference to the function preparing a process.
compute_unit str | None

Reference to the function computing one unit.
merge str | None

Reference to the function merging the units.
plan str | None

Reference to the function which completes the settings of a task before it is submitted, e.g. with values derived from its model.
image str | None

Image of the task, None for the environment image of the run.
cost_setting str | None

Name of the setting which changes the cost of a task, e.g. the subset of the data of a fit; the resources are learned per value of it (simrun.resources.resource_key).
description str

What a task of the type does in a few words, the subtitle of the type in the palette of the Editor.

splittable property 

splittable

Whether the task can be split into pieces.

resolve 

resolve(ref)

Import the function a "module:function" string names.

Parameters:

Name Type Description Default
ref str

Reference of the form module:function.

 required

Returns:

Type Description
Callable[..., Any]

The function.

load_task_types 

load_task_types()

Load the task types of all entry points.

Returns:

Type Description
dict[str, TaskType]

The task types by name.

Raises:

Type Description
ValueError

A name is registered twice.