verify¶

Verify that a change of a model changes only what it is meant to change.

A refactoring of a model, an update of sbmlsim or a refresh of the study data should leave the simulations of the model as they were, and a fix should change them only where it is meant to. The subcommands make this checkable:

dump runs the simulation experiments of a model in a checkout and stores the results, the datasets, the fit mappings and the figures in a pickle,
compare compares two dumps: the simulations on the matched rows of their output times, a row at a time which only one dump has (the steps of the integrator of another run) with the other dump interpolated at its time, relative to the scale of every array (a row outside of the times of the other dump is a difference; an array whose interpolation cannot resolve the threshold is unresolved, not equal), everything else exactly; with --renames FILE the expected renames of a migration (metadata fields and members, added fields, texts such as legend keys, ids) and the expected differences are applied to the old dump first, see load_renames,
ranges lists the fit mappings of a dump whose data lie outside of the simulated range, e.g. data times beyond the end of the simulation,
fits builds the optimization problem of every fit definition of a model and runs a minimal fit,
definitions writes the fit definitions of a model as json: the settings, the parameters and the mappings of the collections, so that two checkouts are compared with diff,
settings writes what a run of the experiments of a model hands to sbmlsim, so that the settings of two checkouts are compared with diff,
png compares the PNG figures of two directories pixel by pixel,
sbml compares SBML models by their elements, ignoring the order and the annotations,
scorecard reads the credibility records of many models (credibility.json of run_credibility) into one table, pools their ratio pairs by a value of the metadata (the qualification of a scaling across drugs) and compares the table with a baseline, see pkpdsim.credibility.scorecard.

A model before and after a change, with tight tolerances so that the solver does not hide a difference:

git worktree add --detach ../base develop
# the experiments simulate the model at `MODEL_PATH`, which is generated:
# create it with the factory of each checkout right before its dump
(cd ../base && uv run python pkdb_models/models/hctz/models/factory.py)
uv run python pkdb_models/models/hctz/models/factory.py
uv run pkpdsim-verify dump --root ../base --model pkdb_models.models.hctz     --out base.pkl --atol 1e-16 --rtol 1e-12 --grid 1000
uv run pkpdsim-verify dump --root . --model pkdb_models.models.hctz     --out new.pkl --order base.pkl --atol 1e-16 --rtol 1e-12 --grid 1000
uv run pkpdsim-verify compare base.pkl new.pkl

--grid N outputs every simulation at N + 1 equidistant times from its start to its end and at the times of its changes (on_grid) instead of the steps of the integrator, which differ between two models or tolerances: both dumps share their times and compare matches them exactly. Without it the rows of a time which only one dump has are compared by interpolation, which on the steps of a real model resolves about 1e-4 to 1e-3 (unresolved arrays are reported, not passed); a migration compare therefore dumps both models with the same grid.

--root is the directory which is put first on the python path, e.g. a checkout of the repository of the model, so that its packages are imported. --model is the dotted name of a model package, which defines its model object MODEL (pkpdsim.model.PKPDModel) which describes it; a package without a model object MODEL is a usage error (exit code 2). The command only needs the python packages of the environment it runs in, so a checkout which needs another version of sbmlsim is dumped with the python of an environment with that version, as long as it is one which pkpdsim supports.

Across versions of sbmlsim (a dump of pkpdsim 0.4 with sbmlsim 0.8 and one of 0.5 with 0.9) compare moves the time of every array first and matches the other dimensions by name, matches the rows of a repeated time from the last one, so that the state after a change, which is all sbmlsim 0.8 gave, is compared, compares a row of a time which only one dump has with the other dump interpolated at its time (matched_rows, interpolation_envelope) and counts a row outside of the times of the other dump as a difference (a new output time of a task is expected with the pattern of its variable time), and compares a figure of sbmlsim 0.8 with the one of sbmlsim 0.9 in the form of sbmlsim 0.9: the settings and the size it is rendered with, without the fields which only sbmlsim 0.9 has, which it lists (align_figures). The renames of a port are the ids which became SIds (sids) and the differences which are expected (expected).

Dump ¶

Bases: TypedDict

Everything dump stores of the experiments of a model.

ModelInfo `dataclass` ¶

ModelInfo(
    package,
    base_path,
    model_path,
    data_path,
    experiments,
    absolute_tolerance,
    relative_tolerance,
    run_experiments,
    fit_definitions,
    jit_optimization=None,
)

What the verification needs of a model package.

RowMatch ¶

Bases: NamedTuple

The rows of the output of a task in two dumps which are compared.

InterpolationRange ¶

Bases: NamedTuple

The range of a trajectory at times, from the interpolants of its steps.

Columns ¶

Bases: NamedTuple

A variable of a task in two dumps, one column per point of a scan.

The time is the first dimension, the points are the other dimensions of the variable, matched by name and flattened in the order of the old dump.

PointMatch ¶

Bases: NamedTuple

The rows of points of a task which are compared, see point_matches.

ArrayDifference ¶

Bases: NamedTuple

The difference of an array of two dumps and the resolution it was found at.

MetadataRenames ¶

Bases: TypedDict

The expected changes of the metadata of the fit mappings.

Renames ¶

Bases: TypedDict

The expected renames and differences between two dumps.

AlignedFigures ¶

Bases: NamedTuple

The figures of two dumps in the form in which they are compared.

describe ¶

describe(package)

Describe a model package by its model object MODEL.

Parameters:

Name	Type	Description	Default
`package`	`str`	the dotted name of the model package.	required

Returns:

Type	Description
`ModelInfo`	What the verification needs of the model.

Raises:

Type	Description
`ValueError`	if the package defines no model object.

registered_experiments ¶

registered_experiments(info)

Get the experiments of the registry of a model.

Returns:

Type	Description
`list[type[SimulationExperiment]] \| None`	The experiment classes, those of `GROUPS` first, without duplicates,
`list[type[SimulationExperiment]] \| None`	or `None` if the model has no registry.

defined_experiments ¶

defined_experiments(package, errors)

Get the experiments defined in the modules of experiments/ of a model.

Every subclass of SimulationExperiment defined in a module below <package>.experiments counts, except the modules base* and metadata and the classes from which another found class derives (the base experiments).

Parameters:

Name	Type	Description	Default
`package`	`str`	the dotted name of the model package.	required
`errors`	`dict[str, str]`	the traceback of every module which does not import is added here, as `import:<module>`.	required

Returns:

Type	Description
`list[type[SimulationExperiment]]`	The experiment classes in the order of module name and definition.

serialize ¶

serialize(obj, depth=0)

Serialize the attributes of an object (a figure) into json values.

References back to the experiment and to parents are replaced by their type, so that the specification of a figure only contains the figure.

metadata_names ¶

metadata_names(metadata)

Get the metadata of a fit mapping as {field: name}.

Enum members are stored by name, so a change of their values or of the enum type (StrEnum) does not count as a change.

describe_mapping ¶

describe_mapping(key, mapping)

Describe a fit mapping by its data, its observable, weight and metadata.

mapping_range ¶

mapping_range(mapping)

Get the range of the x values of the data and of the simulation of a mapping.

The data are converted into the unit of the simulation.

on_grid ¶

on_grid(simulation, n)

Get a simulation with a common fixed output grid (dump --grid).

The output times are n + 1 equidistant times from the start to the end of the simulation and the times of its changes (and of the dimensions of a scan with a time), in the time unit of the simulation; the steps or times of the simulation are replaced, everything else stays. The output of the integrator differs between two runs whose model or tolerances differ; on this grid both dumps of a compare have the same times, which are matched exactly, without interpolation, so a compare resolves its threshold (a migration compare dumps both models with --grid).

A scan has its simulation on the grid and every simulation of a dimension of simulations, each on the grid of its own start, end and changes.

Parameters:

Name	Type	Description	Default
`simulation`	`Simulation \| Scan`	the simulation or the scan.	required
`n`	`int`	the number of intervals of the grid.	required

Returns:

Type	Description
`Simulation \| Scan`	The simulation or the scan on the grid, a new object.

Raises:

Type	Description
`ValueError`	for an object which is no simulation or scan, or a time which is a quantity of a simulation without a time unit.

dump ¶

dump(args)

Run the experiments of a model and store everything in a pickle.

store_experiment ¶

store_experiment(result, sid, experiment)

Store the results, datasets, fit mappings and figures of a run experiment.

The time of a grid is a coordinate of a result of sbmlsim 0.9; it is stored as the variable time over ("time",), which the comparison reads. The sex of the build of every task is stored for an experiment of pkpdsim (builds), so that a study which moves to the build of another sex is a difference of its own.

figure_spec ¶

figure_spec(figure, experiment)

Get the specification of a figure, its attributes as json.

RESOLVED adds the settings of the experiment and of the figure (the figure_settings of the experiment overridden by the settings of the figure) and the width and height the figure is rendered with, as sbmlsim resolves them without the settings of a run. A dump of sbmlsim 0.8 has them as attributes of the figure (_width, _height and a setting such as legend_position), so that compare compares them across versions.

load ¶

load(path)

Load a dump.

time_axis ¶

time_axis(dims)

Get the position of the time (or scan point) dimension.

point_times ¶

point_times(dump, sid, task, dims, shape)

Get the output time of every value of a variable of a task, in its layout.

A grid of sbmlsim 0.9 has one vector of times (("time",)) for every point of a scan; sbmlsim 0.8 and the ragged layout of sbmlsim 0.9 (_point) have the times of every point, over the dimensions of the variable, padded with NaN to the longest point.

Parameters:

Name	Type	Description	Default
`dump`	`Dump`	the dump.	required
`sid`	`str`	the experiment.	required
`task`	`str`	the task.	required
`dims`	`Sequence[str]`	the dimensions of the variable.	required
`shape`	`tuple[int, ...]`	the shape of the variable.	required

Returns:

Type	Description
`ndarray \| None`	The times in the shape of the variable, or `None` if the task has no
`ndarray \| None`	times or they do not fit the variable.

matched_rows ¶

matched_rows(old_times, new_times)

Match the rows of the output times of a task in two dumps.

The times are matched relative to the largest time (TIME_RESOLUTION). The rows of a time which an output has more than once (a change: the state before it, then the state after it) are matched from the last one, the k-th last old row with the k-th last new row, and the surplus rows of the output with more of them are left out. So the state after a change is compared across versions (sbmlsim 0.8 had only this one, sbmlsim 0.9 has the states before and after it), and two dumps of sbmlsim 0.9 compare every row of their common times.

The steps of the integrator differ between two runs whose model or tolerances differ, so most times of an adaptive output are in one dump only. A row at such a time is compared with the other output interpolated at its time (interpolated_old, interpolated_new) if the time lies between two rows of the other output (a segment between its changes, interpolation_envelope) and occurs once in its own output (the states before and after a change which the other output does not have are not one value of it); every other row of a time only one output has is unmatched.

Parameters:

Name	Type	Description	Default
`old_times`	`ndarray`	the output times of the old dump.	required
`new_times`	`ndarray`	the output times of the new dump.	required

Returns:

Type	Description
`RowMatch`	The rows which are compared and the rows which are not.

interpolation_envelope ¶

interpolation_envelope(times, values, at)

Get the range of the trajectory of an output at times, from its interpolants.

The rows of an adaptive output are the steps of the integrator, which are too far apart for one interpolant to reach the threshold of a comparison: between two runs of hctz whose steps differ (the relative tolerance 1e-9 and 1e-10), the linear interpolant is up to 4e-2 of the scale of an array off the other run, the monotone cubic one 4e-3, the cubic spline swings at a sharp transient (the concentration in the urine after it was emptied). Three interpolants of every segment between the changes of the output (the rows between two repeated times), the linear one, the monotone cubic one (PCHIP) and the cubic spline (not-a-knot), span the envelope of the trajectory, and their spread is the uncertainty of the interpolation: the range is the envelope widened by its width on both sides. A value within it is no difference which the steps of the output can tell, and the spread is the resolution which the interpolation reaches: it depends on the density of the steps only, about 1e-2 of the scale of an oral dose on 20 steps over 48 hr and up to 4e-2 on the steps of hctz, so compare reports an array whose resolution is coarser than its threshold as unresolved, never as equal. A column with a value which is not finite in a segment is interpolated linearly there.

Parameters:

Name	Type	Description	Default
`times`	`ndarray`	the output times, ascending, a time of a change twice.	required
`values`	`ndarray`	the values of the rows, (rows, columns).	required
`at`	`ndarray`	the times, each between two rows of a segment.	required

Returns:

Type	Description
`InterpolationRange`	The range and the spread of the interpolants, (times, columns) each.

columns ¶

columns(old, new, key)

Get a variable of a task in two dumps with the times of every value.

Returns:

Type	Description
`Columns \| None`	The columns, or `None` if the dimensions of the variables differ or a
`Columns \| None`	dump has no times of the task which fit its variable.

task_columns ¶

task_columns(old, new, sid, task)

Get the output times of every point of a task in two dumps.

The times are those of a variable of the task (the first by name), so that the points are matched by the names of their dimensions as the variables are; a task with the times only compares their arrays.

Returns:

Type	Description
`Columns \| None`	The times as the columns (values and times), or `None`.

point_matches ¶

point_matches(old_times, new_times)

Match the rows of every point of a task by its own times.

The points of a grid have the same times and one match; a point of a ragged result is matched by its own times, without the NaN which pad it to the longest point. Points with the same times in both dumps share a match.

Parameters:

Name	Type	Description	Default
`old_times`	`ndarray`	the times of the old dump, (rows, points).	required
`new_times`	`ndarray`	the times of the new dump, (rows, points).	required

Returns:

Type	Description
`list[PointMatch]`	The matches.

interpolated_pairs ¶

interpolated_pairs(found, match)

Get the rows of times only one dump has with the other dump at their times.

A row of the old dump is paired with the nearest value of the range of the interpolants of the new dump at its time (interpolation_envelope), a row of the new dump with the nearest value of the range of the old dump, so the difference of a pair is the distance from the range, zero within it. The old value comes first, the scale of a comparison, the spread of the interpolants at the row third, the resolution of the pair.

Parameters:

Name	Type	Description	Default
`found`	`Columns`	the variable of a task in the two dumps.	required
`match`	`PointMatch`	the match of the rows of points with the same times.	required

Returns:

Type	Description
`list[tuple[ndarray, ndarray, ndarray]]`	The old and the new values and the spread, (rows, points) each.

relative_difference ¶

relative_difference(old, new, key)

Get the difference of a result array on the matched rows.

The time is moved first, the other dimensions are matched by name (a dimension of size 1 which only one array has is left out), and the rows of every point are matched by its own times (point_matches); a row at a time which only one dump has is compared with the other dump interpolated at its time, its distance from the range of the interpolants (interpolated_pairs). The difference is max|new - old| / max|old|; other dimensions and NaN at different positions are an infinite difference. A numeric array without a time, e.g. the values at the end of the points of a scan, is compared on the same dimensions relative to its scale, any other array exactly; the variable time is compared as the output times of the points of the task.

relative_differences ¶

relative_differences(old, new, keys)

Get the difference of every result array of keys, see relative_difference.

The arrays of a task with the same output times are compared together, so that the interpolants of a segment are built once for all of them. An array is compared relative to its scale, but at least to SCALE_FLOOR of the largest scale of its variable in the tasks of its experiment among the keys: a variable which is zero in a task, e.g. the drug of a control without a dose, is the noise of the integrator there (1e-24 against 1e-4 of the doses in hctz), which differs with every other step. The resolution of an array is the largest spread of the interpolants of its interpolated rows (interpolation_envelope) relative to the same scale.

Parameters:

Name	Type	Description	Default
`old`	`Dump`	the old dump.	required
`new`	`Dump`	the new dump.	required
`keys`	`Iterable[tuple[str, str, str]]`	the arrays, (experiment, task, variable).	required

Returns:

Type	Description
`dict[tuple[str, str, str], ArrayDifference]`	The difference and the resolution of every array.

scaled_difference ¶

scaled_difference(pairs)

Get max|new - old| / max|old| over pairs of old and new arrays.

NaN at different positions is an infinite difference.

equal ¶

equal(a, b)

Check that two values (arrays, lists, dicts) are equal, NaN equal to NaN.

load_renames ¶

load_renames(path)

Load the expected renames from a json file.

Raises:

Type	Description
`ValueError`	for a part which is not known.

rename_mapping ¶

rename_mapping(description, renames)

Apply the expected renames to the metadata of a mapping description.

rename_ids ¶

rename_ids(dump, rename)

Rename the ids of the tasks, datasets, figures and fit mappings of a dump.

Returns:

Type	Description
`Dump`	A new dump; the given one is not changed.

Raises:

Type	Description
`ValueError`	if two ids of a part have the same new id, or a mapping description cannot be renamed.

drop_figure_fields ¶

drop_figure_fields(dump, fields)

Leave out keys of the specifications of the figures, of the figure only.

A key is left out at the level of the figure, not inside its plots, curves or axes: a plot has a width and a height of its own.

Returns:

Type	Description
`Dump`	A new dump; the given one is not changed.

align_figures ¶

align_figures(old, new)

Bring a figure of sbmlsim 0.8 and the one of sbmlsim 0.9 into one form.

sbmlsim 0.9 moved the settings of a figure from its attributes (legend_position) into _settings and its size from _width and _height, which sbmlsim 0.8 computed when the figure was created, into width and height, which are None for the size of the panels at the time it is rendered; a plot gets a size from its figure only for an explicit one. A figure which is in both dumps, of sbmlsim 0.8 in the old and of sbmlsim 0.9 in the new one, is compared in the form of sbmlsim 0.9, with the settings and the size it is rendered with apart from the settings of the run (RESOLVED of the dump, see figure_spec), and without the size of a plot which its figure gave it. The selection of a Data, the dimensions of the lines of a Curve and the bands of a Plot (NEW_FIGURE_FIELDS) have no counterpart in sbmlsim 0.8: they are left out and listed. Two figures of one version are compared as they are.

Returns:

Type	Description
`AlignedFigures`	The two dumps (new ones, the given ones are not changed), the number of
`AlignedFigures`	figures compared across versions and the listed fields.

apply_renames ¶

apply_renames(dump, renames)

Apply the expected renames to a dump, the old one of a comparison.

The strings first, then the ids (ids, sids), then the metadata.

Returns:

Type	Description
`Dump`	A new dump; the given one is not changed.

Raises:

Type	Description
`ValueError`	if two keys or ids get the same new key or id, or a mapping description cannot be renamed.

compare ¶

compare(args)

Compare two dumps; the exit code is 1 if they differ beyond the threshold.

An expected difference (expected of the renames) is printed and not counted; an expected difference which does not occur is counted. A row at a time which only one dump has is compared with the other dump interpolated at its time when the time lies between two rows of the other dump (the steps of the integrator of another run, matched_rows), and an array whose resolution there (the spread of the interpolants, ArrayDifference.resolution) is coarser than the threshold is unresolved on this grid, which counts as a difference unless expected names it; a row outside of the times of the other dump, or a change the other dump does not have, is a difference: one of the new dump is expected with the pattern of the variable time of its task (experiment/task/time, a changed output grid), one of the old dump always counts.

diff_lines ¶

diff_lines(a, b)

Describe the difference of two lists or dicts line by line.

The values of a dict are compared exactly, json values (the specification of a figure) by the paths of their elements which differ.

json_paths ¶

json_paths(a, b, path='')

Get the paths of the elements in which two json values differ.

A key which only one object has is named with its value, also a null one (a field which one version of sbmlsim added to a figure).

ranges ¶

ranges(args)

List the fit mappings whose data lie outside of the simulated range.

fits ¶

fits(args)

Run a minimal fit (one start, a few evaluations) of every fit definition.

definitions ¶

definitions(args)

Write the fit definitions of a model as json.

The settings, the parameters and the mappings of the collections of every fit definition, so that the definitions of two checkouts are compared with diff. The checkout directory is replaced by <root>.

capture_settings ¶

capture_settings(run_experiments, experiment, root)

Capture the settings a run of an experiment hands to sbmlsim.

The run is intercepted where it reaches sbmlsim: the arguments of the simulator (Simulator), of the runner (without the simulator) and of the run (without the output path; with the figure settings of the run), bound to the signatures, so positional and keyword calls agree, and every matplotlib rcParams entry which differs from the value before the run, at the time of the run. calls counts the intercepted calls. The simulator is the one created before the runner: the designs of an experiment (sbmlsim.simulation.sampling.parameters_of, local) create simulators of their own while the runner initializes it, which are neither recorded nor counted. Nothing is simulated and no report is written.

Parameters:

Name	Type	Description	Default
`run_experiments`	`Callable[..., object]`	the run of a model, `MODEL.run_experiments`.	required
`experiment`	`type[SimulationExperiment]`	the experiment class the run gets.	required
`root`	`Path`	the checkout; it is replaced by `<root>` in the paths.	required

Returns:

Type	Description
`dict[str, Any]`	The settings as json values.

Raises:

Type	Description
`RuntimeError`	if the run does not reach the simulator, the runner and the run of the experiments of sbmlsim.

settings ¶

settings(args)

Write the settings of a run of the experiments of a model as json.

png ¶

png(args)

Compare the PNG figures of two directories pixel by pixel.

The directories are searched recursively and the figures are paired by the path relative to the directory; with --sids an old figure is paired with the new figure of the SId of its file name (pkpdsim.experiment.to_sid).

Returns:

Type	Description
`int`	0 if all figures are identical, 1 if some differ or exist on one side
`int`	only, 2 if a directory does not exist, no figure was found or two old
`int`	figures have the same SId.

sbml_elements ¶

sbml_elements(path)

Get the elements of an SBML model which determine its simulation.

Returns:

Type	Description
`dict[str, object] \| None`	The elements by kind and id, `None` if the file has no SBML model (the
`dict[str, object] \| None`	layouts the factories write next to the models are no SBML).

sbml ¶

sbml(args)

Compare SBML models: two files, or the files of the same path in two dirs.

The directories are searched recursively and the files are paired by the path relative to the directory.

Returns:

Type	Description
`int`	0 if the models are equal, 1 if they differ or exist on one side only,
`int`	2 if a directory does not exist or no file was found.

scorecard_command ¶

scorecard_command(options)

Read the credibility records of many models, compare and qualify them.

Prints the scorecard (model, context of use, verdict, verified), the pooled qualification and the regressions against a baseline.

Returns:

Type	Description
`int`	0, 1 for a regression against the baseline, 2 for records or a baseline
`int`	which cannot be read.

parser ¶

parser()

Create the parser of the command line.

verify ¶

verify(args=None)

Run a subcommand.

Parameters:

Name	Type	Description	Default
`args`	`Sequence[str] \| None`	the command line arguments, `sys.argv[1:]` by default.	`None`

Returns:

Type	Description
`int`	The exit code, 1 if the subcommand found a difference.

main ¶

main()

The command pkpdsim-verify.

verify¶

Dump ¶

ModelInfo dataclass ¶

RowMatch ¶

InterpolationRange ¶

Columns ¶

PointMatch ¶

ArrayDifference ¶

MetadataRenames ¶

Renames ¶

AlignedFigures ¶

describe ¶

registered_experiments ¶

defined_experiments ¶

serialize ¶

metadata_names ¶

describe_mapping ¶

mapping_range ¶

on_grid ¶

dump ¶

store_experiment ¶

figure_spec ¶

load ¶

time_axis ¶

point_times ¶

matched_rows ¶

interpolation_envelope ¶

columns ¶

task_columns ¶

point_matches ¶

interpolated_pairs ¶

relative_difference ¶

relative_differences ¶

scaled_difference ¶

equal ¶

load_renames ¶

rename_mapping ¶

rename_ids ¶

drop_figure_fields ¶

align_figures ¶

apply_renames ¶

compare ¶

diff_lines ¶

json_paths ¶

ranges ¶

fits ¶

definitions ¶

capture_settings ¶

settings ¶

png ¶

sbml_elements ¶

sbml ¶

scorecard_command ¶

parser ¶

verify ¶

main ¶

ModelInfo `dataclass` ¶