local_backend¶

The local backend: the pods of a run as processes on this machine.

LocalBackend runs the DAG of a run without a cluster: a split task as its coordinator (simrun-worker with SIMRUN_ROLE=base, which merges itself), a plugin command through simrun-exec and a command task with its args directly or in its image with docker run (a container named after the run, the node and the attempt, killed with docker kill on a cancel). A task starts when the tasks it reads from succeeded and its cores fit into the free cores of the machine; the tasks of several runs take them in the order of their submission. The nodes and phases of its status are the ones of the Argo workflow of the run, so the controller decides the states of the tasks as on the cluster. Every process runs in its own process group with its stdout and stderr in <task dir>/logs/<node>-<attempt>.log; while it runs, <node>-<attempt>.pid beside the log names it, so a service which starts after one which died without ending its processes kills them (kill_orphans).

A run with snapshots gets a venv of its own (simrun.service.local_env), created when its first process starts and removed at its end: its snapshots are installed there, never into the environment of the service.

A daemon thread per run starts the processes of the run, a thread per process waits for its end; the state lives under one condition, the callbacks of subscribe are called outside of it after every start and end of a process.

ENDED_KEPT `module-attribute` ¶

ENDED_KEPT = 64

The ended runs whose status the backend keeps, the latest; an older one is unknown (KeyError), which the controller resolves from its record.

LocalBackend ¶

LocalBackend(
    store,
    scratch,
    commands="direct",
    cores=None,
    memory_gb=None,
    envs=None,
)

A backend which runs the tasks of a run as processes on this machine.

Attributes:

Name	Type	Description
`store`		The root of the store.
`scratch`		The root of the scratch directories of the processes.
`envs`		The root of the venvs of the runs with snapshots.
`commands`		How a `command` task runs unless its run says otherwise (`RunRecord.local_commands`): `direct` its args, `docker` in its image.
`cores`		The cores of the machine the tasks share.
`memory_gb`		The memory of the machine in GiB.
`hostname`		The name of the machine, its one node.

Create the backend.

Parameters:

Name	Type	Description	Default
`store`	`Path`	The root of the store.	required
`scratch`	`Path`	The root of the scratch directories.	required
`commands`	`Commands`	How a `command` task runs by default.	`'direct'`
`cores`	`int \| None`	The cores of the machine, `os.process_cpu_count()` by default.	`None`
`memory_gb`	`float \| None`	The memory of the machine in GiB, its physical memory by default.	`None`
`envs`	`Path \| None`	The root of the venvs of the runs, `envs` beside `scratch` by default.	`None`

subscribe ¶

subscribe(callback)

Call callback after every start and end of a process.

submit ¶

submit(record, plans, types, cluster)

Start the run in a thread of its own; a run submitted before is kept.

Raises:

Type	Description
`ValueError`	A task stages a port which is no input, or its type is neither splittable nor has a command.
`RuntimeError`	The backend is closed.

status ¶

status(name)

The phases of the nodes of a run; KeyError for an unknown run.

cancel ¶

cancel(name)

End a run: SIGTERM to its processes, SIGKILL after worker.GRACE_S.

A container gets the same signals with docker kill. No process of the run starts or is retried any more; KeyError for an unknown run.

stop ¶

stop(name)

End a run whose state is decided, as cancel.

logs ¶

logs(name, task_id, tail_lines=None)

The logs of the processes of a task, oldest first.

nodes ¶

nodes()

This machine.

placements ¶

placements()

A placement per running process.

reserve ¶

reserve(node, run_id)

Refused: there is nothing to reserve locally.

unreserve ¶

unreserve(node, run_id)

Refused: there is nothing to reserve locally.

build_image ¶

build_image(context, ref)

Refused: the local backend runs no images of runs.

has_image ¶

has_image(ref)

True: the local backend needs no image.

close ¶

close()

Refuse new runs, cancel every run and wait for the end of its processes.

kill_orphans ¶

kill_orphans()

Kill the processes of runs which a service before left running.

A service which died without ending its processes (SIGKILL, out of memory) leaves their pid files beside their logs. The process group of each is killed, and its container, unless its run is one of this backend or its leader is not the process of the file any more (another start time or command line: a reused pid); the files are removed.

Returns:

Type	Description
`int`	The process groups killed.

backend ¶

backend()

The backend of the settings named by SIMRUN_CONFIG, for serve.

The store is the one of the settings, the scratch <store>/../scratch.