fit.runner¶

Module for running parameter optimizations.

The optimization runs either serial or in parallel. The parallel optimization runs in a pool of sbmlsim.parallel which belongs to the fit (worker_pool): one worker process per core, and every repeat of the fit goes to the worker which is free. The pool is stopped after the fit, so the workers do not keep the initialized problem, and it is started again, a few times, when a worker dies.

The OptimizationProblem is pickled and sent to the workers, so it must be picklable: every worker initializes it once and runs repeats on it. The start values are created by the runner, so a fit with a seed gives the same result for any number of workers.

The runner only optimizes. Its result carries the fitted parameters and the settings of the fit, and sbmlsim.fit.report.FitReport turns them into figures and reports, see sbmlsim.fit.parameters.

TotalTimeColumn ¶

Bases: ProgressColumn

The estimated total runtime, ~ H:MM:SS behind the elapsed time.

The estimate is estimate_total_time with the workers field of the task, so it is corrected for the runs a pool processes at once.

render ¶

render(task)

Render the estimate of a task.

FitPool `dataclass` ¶

FitPool(
    executor,
    token,
    problem,
    settings,
    n_cores=1,
    preload=(),
    restarts=0,
)

The workers of a fit: a pool and the problem its tasks run on.

Attributes:

Name	Type	Description
`executor`	`ProcessPoolExecutor`	the pool, see `sbmlsim.parallel.start_pool`; replaced when the pool broke and was started again, see `run`.
`token`	`str`	identifies the fit in the caches of the workers.
`problem`	`OptimizationProblem`	the problem, pickled into every task as its definition.
`settings`	`FitSettings`	the settings the workers initialize the problem with.
`n_cores`	`int`	the number of workers.
`preload`	`Sequence[str]`	the modules the forkserver imports, see `_preload`.
`restarts`	`int`	how often the pool was started again.

submit ¶

submit(function, task)

Run function(token, problem, settings, task) in a worker.

run ¶

run(function, tasks, timeout=None)

Run the tasks and hand out their results as they finish.

At most n_cores tasks are in flight, the next is submitted when one finishes, so a crash implicates only the tasks which were running. A worker which dies breaks the whole pool; it is started again and the tasks which were running, the suspects, are run again one at a time, so a task fails only if it breaks the pool while it runs alone. Then the remaining tasks run again at full width. After MAX_POOL_RESTARTS restarts the tasks which were running at a crash fail without a second run. A pool which is found broken when a task is submitted is started again the same way, with no suspect if nothing was running; after MAX_POOL_RESTARTS restarts with no suspect and no result in between the run ends there. If the pool cannot be started again, or the run ends, the tasks which did not run fail.

Parameters:

Name	Type	Description	Default
`function`	`Callable[..., Any]`	called in the worker as `function(token, problem, settings, task)`.	required
`tasks`	`Mapping[int, dict[str, Any]]`	the tasks by their key.	required
`timeout`	`float \| None`	seconds without any result after which the tasks which are running or waiting fail, `START_GRACE` s are added for the start; no limit if `None`.	`None`

Yields:

Type	Description
`tuple[int, Any]`	The key of a task and its result, or the exception which failed it.

resolve_n_cores ¶

resolve_n_cores(n_cores)

Resolve the number of worker processes.

Parameters:

Name	Type	Description	Default
`n_cores`	`int \| None`	requested number of workers, `None` uses all available cores but one.	required

Returns:

Type	Description
`int`	Number of workers, at least one and at most the number of available cores.

estimate_total_time ¶

estimate_total_time(elapsed, completed, total, workers=1)

Estimate the total runtime from the runs which are done.

The runs are handed to the workers in batches of workers, so the time of a batch is what the elapsed time measures: the estimate is the time per batch times the number of batches. With a single worker this is the mean time per run times the number of runs. The rate of the progress bar is not used, it is measured over a window of seconds and a run takes minutes.

Parameters:

Name	Type	Description	Default
`elapsed`	`float`	seconds since the start.	required
`completed`	`float`	runs which are done.	required
`total`	`float`	runs in total.	required
`workers`	`int`	runs which are processed at once.	`1`

Returns:

Type	Description
`float \| None`	The estimated total runtime in seconds, `None` before the first run
`float \| None`	is done.

optimization_progress ¶

optimization_progress(
    description, size, enabled=True, unit="runs", workers=1
)

Show the progress of the optimization runs on the console.

The bar shows the count, the elapsed time and the estimated total runtime, see TotalTimeColumn.

Parameters:

Name	Type	Description	Default
`description`	`str`	text in front of the progress bar.	required
`size`	`int`	total number of optimization runs.	required
`enabled`	`bool`	show the progress, a plain context without display if `False`.	`True`
`unit`	`str`	what is counted, behind the count.	`'runs'`
`workers`	`int`	runs which are processed at once, for the estimate.	`1`

Yields:

Type	Description
`Generator[Progress \| None]`	The progress with a single task, or `None` if it is disabled.

run_optimization ¶

run_optimization(
    problem,
    settings=None,
    size=5,
    algorithm=LEAST_SQUARE,
    seed=None,
    n_cores=1,
    serial=False,
    show_progress=True,
    timeout=None,
    runs_dir=None,
    sampling=UNIFORM,
    provenance=None,
    **kwargs,
)

Run the optimization of the problem.

The runner executes the given OptimizationProblem size times, every repeat starting from its own sample of the parameters, and returns the OptimizationResult with the fitted parameters and the settings of the fit.

Parameters:

Name	Type	Description	Default
`problem`	`OptimizationProblem`	problem to optimize (picklable); it does not have to be initialized, `run_optimization` initializes it to show the parameters and their coverage before the runs start.	required
`settings`	`FitSettings \| None`	settings of the fit, the defaults of `FitSettings` are used if none are given.	`None`
`size`	`int`	number of optimizations.	`5`
`algorithm`	`OptimizationAlgorithmType`	optimization algorithm to use.	`LEAST_SQUARE`
`seed`	`int \| None`	random seed (for sampling of the start values).	`None`
`n_cores`	`int \| None`	number of workers, `None` uses all available cores but one; one worker fits without starting a process.	`1`
`serial`	`bool`	run the optimization in a serial fashion (debugging), whatever `n_cores` says.	`False`
`show_progress`	`bool`	show the progress of the runs on the console.	`True`
`timeout`	`float \| None`	seconds a single optimization may run, no limit if `None`. A run which is out of time keeps the parameters it reached.	`None`
`runs_dir`	`Path \| None`	directory the single runs are written to while the fit runs, so a fit which is interrupted or crashes leaves the runs which finished; they are read back with `OptimizationResult.from_directory`.	`None`
`sampling`	`SamplingType`	sampling of the start values, see `sbmlsim.fit.sampling`.	`UNIFORM`
`provenance`	`FitProvenance \| None`	what the fit runs with, see `sbmlsim.fit.provenance`; it is stored with every run and with the result.	`None`
`kwargs`	`Any`	additional arguments for the optimizer, i.e., for `scipy.optimize.least_squares` or `scipy.optimize.differential_evolution`, e.g. xtol.	`{}`

Returns:

Type	Description
`OptimizationResult`	OptimizationResult with the fits of all repeats. A repeat which failed
`OptimizationResult`	is part of the result and carries its message. A fit which is
`OptimizationResult`	interrupted (Ctrl-C) after some repeats finished returns these.

Raises:

Type	Description
`ValueError`	if a bound or a start value does not suit the scale of its parameter or the algorithm, or if every worker of a parallel fit failed.
`TypeError`	if `kwargs` has an argument which the optimizer of the algorithm does not accept, e.g. one of the other optimizer.
`KeyboardInterrupt`	if the fit is interrupted (Ctrl-C) before a repeat finished, there is no result to keep.

worker_problem ¶

worker_problem(token, problem, settings)

Get the initialized problem of a fit in a worker process.

A worker resolves the data of the problem of a fit once and keeps it under the token of the fit, see sbmlsim.parallel.worker_cache, so the repeats and the scans of a profile a worker runs share one initialization. The workers would report the same messages about the data, once per core, so only their errors are shown; the runner reports the problem itself.

Parameters:

Name	Type	Description	Default
`token`	`str`	identifies the fit, see `FitPool`.	required
`problem`	`OptimizationProblem`	the problem, its definition as it is pickled into a task.	required
`settings`	`FitSettings`	the settings of the fit.	required

Returns:

Type	Description
`OptimizationProblem`	The initialized problem.

Raises:

Type	Description
`RuntimeError`	if the worker could not initialize the problem, with the error of the initialization.

worker_pool ¶

worker_pool(problem, settings, n_cores)

Start the workers of a fit and stop them after it.

The pool belongs to the fit and is not the kept pool of sbmlsim.parallel.pool: its workers keep the initialized problem, which a kept pool would keep after the fit.

Parameters:

Name	Type	Description	Default
`problem`	`OptimizationProblem`	the problem of the fit.	required
`settings`	`FitSettings`	the settings of the fit.	required
`n_cores`	`int`	the number of workers.	required

Yields:

Type	Description
`Generator[FitPool]`	The workers of the fit.

Raises:

Type	Description
`RuntimeError`	if the workers do not start, e.g. in a script without the guard, see `sbmlsim.parallel.start_pool`.

fit.runner¶

TotalTimeColumn ¶

render ¶

FitPool dataclass ¶

submit ¶

run ¶

resolve_n_cores ¶

estimate_total_time ¶

optimization_progress ¶

run_optimization ¶

worker_problem ¶

worker_pool ¶

FitPool `dataclass` ¶