verify¶

Verify that a change of a model changes only what it is meant to change.

A refactoring of a model, an update of sbmlsim or a refresh of the study data should leave the simulations of the model as they were, and a fix should change them only where it is meant to. The subcommands make this checkable:

dump runs the simulation experiments of a model in a checkout and stores the results, the datasets, the fit mappings and the figures in a pickle,
compare compares two dumps: the simulations on the matched rows of their output times, relative to the scale of every array (a time which only one dump has is a difference), everything else exactly; with --renames FILE the expected renames of a migration (metadata fields and members, added fields, texts such as legend keys, ids) and the expected differences are applied to the old dump first, see load_renames,
ranges lists the fit mappings of a dump whose data lie outside of the simulated range, e.g. data times beyond the end of the simulation,
fits builds the optimization problem of every fit definition of a model and runs a minimal fit,
definitions writes the fit definitions of a model as json: the settings, the parameters and the mappings of the collections, so that two checkouts are compared with diff,
settings writes what a run of the experiments of a model hands to sbmlsim, so that the settings of two checkouts are compared with diff,
png compares the PNG figures of two directories pixel by pixel,
sbml compares SBML models by their elements, ignoring the order and the annotations.

A model before and after a change, with tight tolerances so that the solver does not hide a difference:

git worktree add --detach ../base develop
# the experiments simulate the model at `MODEL_PATH`, which is generated:
# create it with the factory of each checkout right before its dump
(cd ../base && uv run python pkdb_models/models/hctz/models/factory.py)
uv run python pkdb_models/models/hctz/models/factory.py
uv run pkpdsim-verify dump --root ../base --model pkdb_models.models.hctz     --out base.pkl --atol 1e-16 --rtol 1e-12
uv run pkpdsim-verify dump --root . --model pkdb_models.models.hctz     --out new.pkl --order base.pkl --atol 1e-16 --rtol 1e-12
uv run pkpdsim-verify compare base.pkl new.pkl

--root is the directory which is put first on the python path, e.g. a checkout of the repository of the model, so that its packages are imported. --model is the dotted name of a model package, which defines its model object MODEL (pkpdsim.model.PKPDModel) which describes it; a package without a model object MODEL is a usage error (exit code 2). The command only needs the python packages of the environment it runs in, so a checkout which needs another version of sbmlsim is dumped with the python of an environment with that version, as long as it is one which pkpdsim supports.

Across versions of sbmlsim (a dump of pkpdsim 0.4 with sbmlsim 0.8 and one of 0.5 with 0.9) compare moves the time of every array first and matches the other dimensions by name, matches the rows of a repeated time from the last one, so that the state after a change, which is all sbmlsim 0.8 gave, is compared, and counts the rows of a time which only one dump has as differences (a new output time of a task is expected with the pattern of its variable time), and compares a figure of sbmlsim 0.8 with the one of sbmlsim 0.9 in the form of sbmlsim 0.9: the settings and the size it is rendered with, without the fields which only sbmlsim 0.9 has, which it lists (align_figures). The renames of a port are the ids which became SIds (sids) and the differences which are expected (expected).

Dump ¶

Bases: TypedDict

Everything dump stores of the experiments of a model.

ModelInfo `dataclass` ¶

ModelInfo(
    package,
    base_path,
    model_path,
    data_path,
    experiments,
    absolute_tolerance,
    relative_tolerance,
    run_experiments,
    fit_definitions,
)

What the verification needs of a model package.

RowMatch ¶

Bases: NamedTuple

The rows of the output of a task in two dumps which are compared.

Columns ¶

Bases: NamedTuple

A variable of a task in two dumps, one column per point of a scan.

The time is the first dimension, the points are the other dimensions of the variable, matched by name and flattened in the order of the old dump.

PointMatch ¶

Bases: NamedTuple

The rows of points of a task which are compared, see point_matches.

MetadataRenames ¶

Bases: TypedDict

The expected changes of the metadata of the fit mappings.

Renames ¶

Bases: TypedDict

The expected renames and differences between two dumps.

AlignedFigures ¶

Bases: NamedTuple

The figures of two dumps in the form in which they are compared.

describe ¶

describe(package)

Describe a model package by its model object MODEL.

Parameters:

Name	Type	Description	Default
`package`	`str`	the dotted name of the model package.	required

Returns:

Type	Description
`ModelInfo`	What the verification needs of the model.

Raises:

Type	Description
`ValueError`	if the package defines no model object.

registered_experiments ¶

registered_experiments(info)

Get the experiments of the registry of a model.

Returns:

Type	Description
`list[type[SimulationExperiment]] \| None`	The experiment classes, those of `GROUPS` first, without duplicates,
`list[type[SimulationExperiment]] \| None`	or `None` if the model has no registry.

defined_experiments ¶

defined_experiments(package, errors)

Get the experiments defined in the modules of experiments/ of a model.

Every subclass of SimulationExperiment defined in a module below <package>.experiments counts, except the modules base* and metadata and the classes from which another found class derives (the base experiments).

Parameters:

Name	Type	Description	Default
`package`	`str`	the dotted name of the model package.	required
`errors`	`dict[str, str]`	the traceback of every module which does not import is added here, as `import:<module>`.	required

Returns:

Type	Description
`list[type[SimulationExperiment]]`	The experiment classes in the order of module name and definition.

serialize ¶

serialize(obj, depth=0)

Serialize the attributes of an object (a figure) into json values.

References back to the experiment and to parents are replaced by their type, so that the specification of a figure only contains the figure.

metadata_names ¶

metadata_names(metadata)

Get the metadata of a fit mapping as {field: name}.

Enum members are stored by name, so a change of their values or of the enum type (StrEnum) does not count as a change.

describe_mapping ¶

describe_mapping(key, mapping)

Describe a fit mapping by its data, its observable, weight and metadata.

mapping_range ¶

mapping_range(mapping)

Get the range of the x values of the data and of the simulation of a mapping.

The data are converted into the unit of the simulation.

dump ¶

dump(args)

Run the experiments of a model and store everything in a pickle.

store_experiment ¶

store_experiment(result, sid, experiment)

Store the results, datasets, fit mappings and figures of a run experiment.

The time of a grid is a coordinate of a result of sbmlsim 0.9; it is stored as the variable time over ("time",), which the comparison reads.

figure_spec ¶

figure_spec(figure, experiment)

Get the specification of a figure, its attributes as json.

RESOLVED adds the settings of the experiment and of the figure (the figure_settings of the experiment overridden by the settings of the figure) and the width and height the figure is rendered with, as sbmlsim resolves them without the settings of a run. A dump of sbmlsim 0.8 has them as attributes of the figure (_width, _height and a setting such as legend_position), so that compare compares them across versions.

load ¶

load(path)

Load a dump.

time_axis ¶

time_axis(dims)

Get the position of the time (or scan point) dimension.

point_times ¶

point_times(dump, sid, task, dims, shape)

Get the output time of every value of a variable of a task, in its layout.

A grid of sbmlsim 0.9 has one vector of times (("time",)) for every point of a scan; sbmlsim 0.8 and the ragged layout of sbmlsim 0.9 (_point) have the times of every point, over the dimensions of the variable, padded with NaN to the longest point.

Parameters:

Name	Type	Description	Default
`dump`	`Dump`	the dump.	required
`sid`	`str`	the experiment.	required
`task`	`str`	the task.	required
`dims`	`Sequence[str]`	the dimensions of the variable.	required
`shape`	`tuple[int, ...]`	the shape of the variable.	required

Returns:

Type	Description
`ndarray \| None`	The times in the shape of the variable, or `None` if the task has no
`ndarray \| None`	times or they do not fit the variable.

matched_rows ¶

matched_rows(old_times, new_times)

Match the rows of the output times of a task in two dumps.

The times are matched relative to the largest time 1e-6. The rows of a time which an output has more than once (a change: the state before it, then the state after it) are matched from the last one, the k-th last old row with the k-th last new row, and the surplus rows of the output with more of them are left out. So the state after a change is compared across versions (sbmlsim 0.8 had only this one, sbmlsim 0.9 has the states before and after it), and two dumps of sbmlsim 0.9 compare every row of their common times.

Parameters:

Name	Type	Description	Default
`old_times`	`ndarray`	the output times of the old dump.	required
`new_times`	`ndarray`	the output times of the new dump.	required

Returns:

Type	Description
`RowMatch`	The rows which are compared and the rows which are not.

columns ¶

columns(old, new, key)

Get a variable of a task in two dumps with the times of every value.

Returns:

Type	Description
`Columns \| None`	The columns, or `None` if the dimensions of the variables differ or a
`Columns \| None`	dump has no times of the task which fit its variable.

task_columns ¶

task_columns(old, new, sid, task)

Get the output times of every point of a task in two dumps.

The times are those of a variable of the task (the first by name), so that the points are matched by the names of their dimensions as the variables are; a task with the times only compares their arrays.

Returns:

Type	Description
`Columns \| None`	The times as the columns (values and times), or `None`.

point_matches ¶

point_matches(old_times, new_times)

Match the rows of every point of a task by its own times.

The points of a grid have the same times and one match; a point of a ragged result is matched by its own times, without the NaN which pad it to the longest point. Points with the same times in both dumps share a match.

Parameters:

Name	Type	Description	Default
`old_times`	`ndarray`	the times of the old dump, (rows, points).	required
`new_times`	`ndarray`	the times of the new dump, (rows, points).	required

Returns:

Type	Description
`list[PointMatch]`	The matches.

relative_difference ¶

relative_difference(old, new, key)

Get the difference of a result array on the matched rows.

The time is moved first, the other dimensions are matched by name (a dimension of size 1 which only one array has is left out), and the rows of every point are matched by its own times (point_matches). The difference is max|new - old| / max|old|; other dimensions and NaN at different positions are an infinite difference. An array without a time is compared exactly; the variable time is compared as the output times of the points of the task.

equal ¶

equal(a, b)

Check that two values (arrays, lists, dicts) are equal, NaN equal to NaN.

load_renames ¶

load_renames(path)

Load the expected renames from a json file.

Raises:

Type	Description
`ValueError`	for a part which is not known.

rename_mapping ¶

rename_mapping(description, renames)

Apply the expected renames to the metadata of a mapping description.

rename_ids ¶

rename_ids(dump, rename)

Rename the ids of the tasks, datasets, figures and fit mappings of a dump.

Returns:

Type	Description
`Dump`	A new dump; the given one is not changed.

Raises:

Type	Description
`ValueError`	if two ids of a part have the same new id, or a mapping description cannot be renamed.

drop_figure_fields ¶

drop_figure_fields(dump, fields)

Leave out keys of the specifications of the figures, of the figure only.

A key is left out at the level of the figure, not inside its plots, curves or axes: a plot has a width and a height of its own.

Returns:

Type	Description
`Dump`	A new dump; the given one is not changed.

align_figures ¶

align_figures(old, new)

Bring a figure of sbmlsim 0.8 and the one of sbmlsim 0.9 into one form.

sbmlsim 0.9 moved the settings of a figure from its attributes (legend_position) into _settings and its size from _width and _height, which sbmlsim 0.8 computed when the figure was created, into width and height, which are None for the size of the panels at the time it is rendered; a plot gets a size from its figure only for an explicit one. A figure which is in both dumps, of sbmlsim 0.8 in the old and of sbmlsim 0.9 in the new one, is compared in the form of sbmlsim 0.9, with the settings and the size it is rendered with apart from the settings of the run (RESOLVED of the dump, see figure_spec), and without the size of a plot which its figure gave it. The selection of a Data, the dimensions of the lines of a Curve and the bands of a Plot (NEW_FIGURE_FIELDS) have no counterpart in sbmlsim 0.8: they are left out and listed. Two figures of one version are compared as they are.

Returns:

Type	Description
`AlignedFigures`	The two dumps (new ones, the given ones are not changed), the number of
`AlignedFigures`	figures compared across versions and the listed fields.

apply_renames ¶

apply_renames(dump, renames)

Apply the expected renames to a dump, the old one of a comparison.

The strings first, then the ids (ids, sids), then the metadata.

Returns:

Type	Description
`Dump`	A new dump; the given one is not changed.

Raises:

Type	Description
`ValueError`	if two keys or ids get the same new key or id, or a mapping description cannot be renamed.

compare ¶

compare(args)

Compare two dumps; the exit code is 1 if they differ beyond the threshold.

An expected difference (expected of the renames) is printed and not counted; an expected difference which does not occur is counted. A row at a time which only one dump has is a difference: one of the new dump is expected with the pattern of the variable time of its task (experiment/task/time, a changed output grid), one of the old dump always counts.

diff_lines ¶

diff_lines(a, b)

Describe the difference of two lists or dicts line by line.

The values of a dict are compared exactly, json values (the specification of a figure) by the paths of their elements which differ.

json_paths ¶

json_paths(a, b, path='')

Get the paths of the elements in which two json values differ.

A key which only one object has is named with its value, also a null one (a field which one version of sbmlsim added to a figure).

ranges ¶

ranges(args)

List the fit mappings whose data lie outside of the simulated range.

fits ¶

fits(args)

Run a minimal fit (one start, a few evaluations) of every fit definition.

definitions ¶

definitions(args)

Write the fit definitions of a model as json.

The settings, the parameters and the mappings of the collections of every fit definition, so that the definitions of two checkouts are compared with diff. The checkout directory is replaced by <root>.

capture_settings ¶

capture_settings(run_experiments, experiment, root)

Capture the settings a run of an experiment hands to sbmlsim.

The run is intercepted where it reaches sbmlsim: the arguments of the simulator (Simulator), of the runner (without the simulator) and of the run (without the output path; with the figure settings of the run), bound to the signatures, so positional and keyword calls agree, and every matplotlib rcParams entry which differs from the value before the run, at the time of the run. calls counts the intercepted calls. The simulator is the one created before the runner: the designs of an experiment (sbmlsim.simulation.sampling.parameters_of, local) create simulators of their own while the runner initializes it, which are neither recorded nor counted. Nothing is simulated and no report is written.

Parameters:

Name	Type	Description	Default
`run_experiments`	`Callable[..., object]`	the run of a model, `MODEL.run_experiments`.	required
`experiment`	`type[SimulationExperiment]`	the experiment class the run gets.	required
`root`	`Path`	the checkout; it is replaced by `<root>` in the paths.	required

Returns:

Type	Description
`dict[str, Any]`	The settings as json values.

Raises:

Type	Description
`RuntimeError`	if the run does not reach the simulator, the runner and the run of the experiments of sbmlsim.

settings ¶

settings(args)

Write the settings of a run of the experiments of a model as json.

png ¶

png(args)

Compare the PNG figures of two directories pixel by pixel.

The directories are searched recursively and the figures are paired by the path relative to the directory; with --sids an old figure is paired with the new figure of the SId of its file name (pkpdsim.experiment.to_sid).

Returns:

Type	Description
`int`	0 if all figures are identical, 1 if some differ or exist on one side
`int`	only, 2 if a directory does not exist, no figure was found or two old
`int`	figures have the same SId.

sbml_elements ¶

sbml_elements(path)

Get the elements of an SBML model which determine its simulation.

Returns:

Type	Description
`dict[str, object] \| None`	The elements by kind and id, `None` if the file has no SBML model (the
`dict[str, object] \| None`	layouts the factories write next to the models are no SBML).

sbml ¶

sbml(args)

Compare SBML models: two files, or the files of the same path in two dirs.

The directories are searched recursively and the files are paired by the path relative to the directory.

Returns:

Type	Description
`int`	0 if the models are equal, 1 if they differ or exist on one side only,
`int`	2 if a directory does not exist or no file was found.

parser ¶

parser()

Create the parser of the command line.

verify ¶

verify(args=None)

Run a subcommand.

Parameters:

Name	Type	Description	Default
`args`	`Sequence[str] \| None`	the command line arguments, `sys.argv[1:]` by default.	`None`

Returns:

Type	Description
`int`	The exit code, 1 if the subcommand found a difference.

main ¶

main()

The command pkpdsim-verify.

verify¶

Dump ¶

ModelInfo dataclass ¶

RowMatch ¶

Columns ¶

PointMatch ¶

MetadataRenames ¶

Renames ¶

AlignedFigures ¶

describe ¶

registered_experiments ¶

defined_experiments ¶

serialize ¶

metadata_names ¶

describe_mapping ¶

mapping_range ¶

dump ¶

store_experiment ¶

figure_spec ¶

load ¶

time_axis ¶

point_times ¶

matched_rows ¶

columns ¶

task_columns ¶

point_matches ¶

relative_difference ¶

equal ¶

load_renames ¶

rename_mapping ¶

rename_ids ¶

drop_figure_fields ¶

align_figures ¶

apply_renames ¶

compare ¶

diff_lines ¶

json_paths ¶

ranges ¶

fits ¶

definitions ¶

capture_settings ¶

settings ¶

png ¶

sbml_elements ¶

sbml ¶

parser ¶

verify ¶

main ¶

ModelInfo `dataclass` ¶