parallel¶

The process pools of sbmlsim.

Every pool of the package comes from here:

process_context is the start method of every pool, never fork of a process which runs threads;
pool(n) is a pool kept per process and number of workers, so that the start of the workers (the imports, roadrunner) is paid once per process and not once per run; start_pool(n) is a new pool which the caller stops, e.g. the pool of a fit, whose workers keep the initialized problem;
resolve_workers turns n_workers into the number of processes;
worker_cache keeps objects in a worker, e.g. the models of a scan or the initialized problem of a fit, and builds one only when its key is new.

Ctrl-C belongs to the process of the user: a terminal sends SIGINT to every process of the job, the workers ignore it and the parent stops a pool whose run it interrupts, see start_pool and stop. On POSIX a worker starts with SIGINT blocked, so Ctrl-C does not reach it while it still imports, see _SigintBlocked.

A pool starts worker processes which import the main module again (the start methods forkserver and spawn), so a script which starts a pool must do it behind the guard if __name__ == "__main__":. Without it the workers die while they start and the parent reports the guard, see GUARD_MESSAGE. A worker itself never starts a pool, a task which runs in a worker runs serially.

shared_payloads ¶

shared_payloads(sources)

Share immutable snapshots with local workers without repeated IPC copies.

The caller keeps this context until submitted work has finished or its workers have stopped. The temporary directory is private to this process. Equal snapshots share one file, independently of integrator settings.

Parameters:

Name	Type	Description	Default
`sources`	`Sequence[str]`	the snapshots, e.g. the SBML of the models of a run.	required

Yields:

Type	Description
`Generator[list[str]]`	The path of the file of every snapshot, in the order of `sources`; the
`Generator[list[str]]`	files are removed when the context exits, also after an error.

process_context ¶

process_context()

Get the multiprocessing context a pool of sbmlsim is created from.

A start method set with multiprocessing.set_start_method is used, else the default of the platform: forkserver on linux, spawn on macOS and windows. No default forks, the process of a simulation runs the threads of roadrunner and of the linear algebra, and a fork of a process with threads may deadlock in the child.

Every pool of sbmlsim comes from this module, see pool and start_pool, and the start method of the process is not fixed by the call (get_context() without a method would), because a start method which is set is read as one the user chose.

Returns:

Type	Description
`BaseContext`	The context.

in_worker ¶

in_worker()

Check whether this process is a child process of multiprocessing.

check_workers ¶

check_workers(n_workers)

Check a number of workers.

Parameters:

Name	Type	Description	Default
`n_workers`	`int \| None`	`None` or the number of processes.	required

Raises:

Type	Description
`ValueError`	if `n_workers` is less than 1.

resolve_workers ¶

resolve_workers(n_workers, n_tasks)

Get the number of processes of a run.

Parameters:

Name	Type	Description	Default
`n_workers`	`int \| None`	`1` runs serially in the calling process, a number is taken as given, `None` is the number of CPUs for `POOL_THRESHOLD` tasks or more and serial below, and serial in a worker process.	required
`n_tasks`	`int`	the number of tasks of the run, e.g. the points of a scan.	required

Returns:

Type	Description
`int`	The number of processes, `1` for a serial run.

Raises:

Type	Description
`ValueError`	if `n_workers` is less than 1.

start_pool ¶

start_pool(
    n_workers, preload=(), initializer=None, initargs=()
)

Start a pool whose workers answer, see pool for one which is kept.

The workers ignore SIGINT, Ctrl-C interrupts the parent only, see the module.

Parameters:

Name	Type	Description	Default
`n_workers`	`int`	the number of worker processes.	required
`preload`	`Sequence[str]`	modules the forkserver imports once for all workers; it takes effect only when the forkserver starts for the first time in this process.	`()`
`initializer`	`Callable[..., object] \| None`	a module level function every worker calls when it starts, e.g. to keep a queue to the parent, which a worker can only receive when it starts (a queue cannot be sent with a task).	`None`
`initargs`	`tuple[Any, ...]`	the arguments of the initializer.	`()`

Returns:

Type	Description
`ProcessPoolExecutor`	The pool, the caller stops it with `stop`.

Raises:

Type	Description
`RuntimeError`	in a worker process (no nested pools), or if the workers die while they start or do not start within `WORKER_STARTUP_TIMEOUT`; both are what a script without the guard does, see `GUARD_MESSAGE`.

pool ¶

pool(n_workers)

Get the pool of the process with a number of workers.

The pool is kept, so the workers start once per process; a pool which broke, e.g. because a worker died, is replaced. Every pool is stopped when the process ends, see shutdown.

Parameters:

Name	Type	Description	Default
`n_workers`	`int`	the number of worker processes.	required

Returns:

Type	Description
`ProcessPoolExecutor`	The pool.

Raises:

Type	Description
`RuntimeError`	see `start_pool`.

stop ¶

stop(executor)

Stop a pool: the pending tasks are cancelled and the workers end.

The workers are terminated, so a pool whose workers still run, e.g. after Ctrl-C or a timeout, stops at once. A kept pool is dropped, the next pool starts a new one.

The thread of the pool which manages its workers is the only one which waits for them to end, and stop waits for that thread: when it returns, the workers have ended and the pool knows it. A second thread which waited for a worker would race with it under the start method spawn on POSIX: the one which loses finds no process to wait for, and the worker looks alive for a moment after it ended.

shutdown ¶

shutdown()

Stop every kept pool of the process.

worker_cache ¶

worker_cache(key, factory)

Get an object of the worker process, built once per key.

The newest WORKER_CACHE_SIZE objects are kept.

Parameters:

Name	Type	Description	Default
`key`	`Hashable`	identifies the object, e.g. a model with the settings of its integrator.	required
`factory`	`Callable[[], T]`	builds the object when the key is new.	required

Returns:

Type	Description
`T`	The object of the key.