sciml.testsuite¶

The cases of the PEtab SciML test suite on disk.

The test suite has three groups of cases, every case is a directory named by its number with a solutions.yaml:

group	case	compared
`ml_model_import`	`ModelImportCase`	the outputs of the forward pass
`initialization`	`InitializationCase`	the nominal values of the arrays
`sciml_problem_import`	`ProblemImportCase`	likelihood, simulations, gradient

The suite has no releases, so it is pinned by a commit. SciMLSuite is the directory of the groups with the download and the cache of the commit.

The arrays of the suite are in the PyTorch layout, which is the layout of sbmlsim: the axis orders input_order_py and output_order_py of a case name the axes of the arrays as they are stored, nothing is permuted.

CaseStatus ¶

Bases: StrEnum

The outcome of a case.

CaseResult `dataclass` ¶

CaseResult(
    group, cid, status, message="", max_difference=None
)

The outcome of a case.

Attributes:

Name	Type	Description
`group`	`str`	the group of the case.
`cid`	`str`	the number of the case, e.g. `001`.
`status`	`CaseStatus`	the outcome.
`message`	`str`	what failed, empty for a case which passes.
`max_difference`	`float \| None`	the largest absolute difference to the reference values, `None` when nothing was compared.

passed `property` ¶

passed

Check whether the case passes.

key `property` ¶

key

Get the key of the case in the baseline, e.g. ml_model_import/001.

ModelImportCase `dataclass` ¶

ModelImportCase(
    cid,
    path,
    net_file,
    inputs,
    parameters,
    outputs,
    input_order=list(),
    output_order=list(),
    dropout=None,
)

A case of the group ml_model_import.

A case is a network with combinations of inputs, arrays and the outputs the network has for them. The combination i is the input i, the arrays i and the output i.

Attributes:

Name	Type	Description
`cid`	`str`	the number of the case, e.g. `001`.
`path`	`Path`	the directory of the case.
`net_file`	`Path`	the NN YAML.
`inputs`	`list[list[Path]]`	the input files of every combination, one per input of the network.
`parameters`	`list[Path]`	the array file of every combination, empty for a network without arrays.
`outputs`	`list[Path]`	the output file of every combination.
`input_order`	`list[str]`	the axes of the inputs, e.g. `["C", "H", "W"]`.
`output_order`	`list[str]`	the axes of the outputs.
`dropout`	`int \| None`	the number of forward passes in training mode the reference values are the mean of, `None` for a case without dropout.

tolerance `property` ¶

tolerance

Get the absolute tolerance of the outputs.

from_directory `classmethod` ¶

from_directory(path)

Read a case from its directory.

Parameters:

Name	Type	Description	Default
`path`	`Path`	the directory of the case, named by its number.	required

Returns:

Type	Description
`ModelImportCase`	The case.

Raises:

Type	Description
`ValueError`	if the directory has no `solutions.yaml`, if the file does not list inputs and outputs, or if the inputs, the array files and the outputs are not listed for the same non-empty list of combinations.

network ¶

network(i)

Get the network with the arrays of a combination.

Parameters:

Name	Type	Description	Default
`i`	`int`	the index of the combination.	required

Returns:

Type	Description
`Network`	The network.

run ¶

run()

Evaluate the network for every combination and compare the outputs.

Returns:

Type	Description
`CaseResult`	The outcome of the case. It does not raise: a layer without an
`CaseResult`	implementation is `UNSUPPORTED` and any other error is `ERROR`.

InitializationCase `dataclass` ¶

InitializationCase(
    cid, path, problem_path, tolerance, parameter_files
)

A case of the group initialization.

A case is a PEtab SciML problem whose parameter table sets the nominal values of a part of a network, and the arrays the network has after the import.

Attributes:

Name	Type	Description
`cid`	`str`	the number of the case, e.g. `001`.
`path`	`Path`	the directory of the case.
`problem_path`	`Path`	the YAML of the problem.
`tolerance`	`float`	the absolute tolerance of the arrays.
`parameter_files`	`dict[str, Path]`	id of the network -> the file with its reference values.

from_directory `classmethod` ¶

from_directory(path)

Read a case from its directory.

Parameters:

Name	Type	Description	Default
`path`	`Path`	the directory of the case, named by its number.	required

Returns:

Type	Description
`InitializationCase`	The case.

Raises:

Type	Description
`ValueError`	if the directory has no `solutions.yaml`, or if the file has no tolerance or no reference files.

nominal ¶

nominal()

Import the networks of the problem with their nominal values.

The problem is read with the classes of petab which need no torch: the configuration, the parameter table and the mapping table. The rows of the parameter table which the mapping table resolves to the parameters of a network are the entries of nominal_parameters; the nominalValue array keeps the values of the array file.

Returns:

Type	Description
`dict[str, NetworkParameters]`	id of the network -> the arrays of the network.

Raises:

Type	Description
`NetworkImportError`	if the problem is not a PEtab SciML problem, a network is not in the format `YAML` or an array has no values.

expected ¶

expected()

Read the reference values of the arrays.

Returns:

Type	Description
`dict[str, NetworkParameters]`	id of the network -> the arrays of the network.

Raises:

Type	Description
`ValueError`	if a reference file has no arrays of its network.

run ¶

run()

Import the networks and compare their nominal values.

Returns:

Type	Description
`CaseResult`	The outcome of the case. It does not raise: a layer without an
`CaseResult`	implementation is `UNSUPPORTED` and any other error is `ERROR`.

ProblemImportCase `dataclass` ¶

ProblemImportCase(
    cid,
    path,
    problem_path,
    llh,
    log_posterior,
    simulation_files,
    gradient_files,
    tol_llh,
    tol_simulations,
    tol_grad,
)

A case of the group sciml_problem_import.

A case is a PEtab SciML problem with the log-likelihood, the simulations at the measurements and the gradient of the log-likelihood at the nominal values of its parameters. The reference values of the gradient are the central differences of five points of a simulation with the tolerances 1e-12.

Attributes:

Name	Type	Description
`cid`	`str`	the number of the case, e.g. `001`.
`path`	`Path`	the directory of the case.
`problem_path`	`Path`	the YAML of the problem.
`llh`	`float \| None`	the log-likelihood at the nominal values, `None` for a case with priors, which states `log_posterior`.
`log_posterior`	`float \| None`	the log-posterior at the nominal values, `None` for a case without priors.
`simulation_files`	`list[Path]`	the simulations at the measurement points.
`gradient_files`	`dict[str, Path]`	`mech` or the id of a network -> the gradient of the mechanistic parameters (TSV) or of the network (HDF5).
`tol_llh`	`float`	tolerance of the log-likelihood or the log-posterior.
`tol_simulations`	`float`	tolerance of the simulations.
`tol_grad`	`float`	tolerance of the gradient.

from_directory `classmethod` ¶

from_directory(path)

Read a case from its directory.

Parameters:

Name	Type	Description	Default
`path`	`Path`	the directory of the case, named by its number.	required

Returns:

Type	Description
`ProblemImportCase`	The case.

Raises:

Type	Description
`ValueError`	if the directory has no `solutions.yaml`, or if the file has no tolerance of the likelihood, the simulations or the gradient.

settings ¶

settings()

Get the settings the problem of the case is initialized with.

Returns:

Type	Description
`FitSettings`	Settings with the linear scale, a fixed grid and the tolerances
`FitSettings`	`PROBLEM_IMPORT_TOLERANCE`: two simulations on a variable grid
`FitSettings`	differ by more than a difference of the gradient resolves.

expected_simulations ¶

expected_simulations()

Read the reference values of the simulations.

Returns:

Type	Description
`DataFrame`	The rows of the simulation files with the columns `observableId`,
`DataFrame`	`experimentId`, `time` and `simulation`.

Raises:

Type	Description
`ValueError`	if the case has no simulation file, or if a file lacks a column.

expected_gradient ¶

expected_gradient()

Read the reference values of the gradient.

Returns:

Type	Description
`dict[str, float]`	id of the parameter -> derivative of the log-likelihood. The id of
`dict[str, float]`	an element of a network is its id in `sbmlsim`, see
`dict[str, float]`	`sbmlsim.sciml.network.element_id`. An array which the file
`dict[str, float]`	stores as an empty array is frozen and has no derivative.

Raises:

Type	Description
`ValueError`	if the case has no gradient of the mechanistic parameters, or if a file cannot be read.

run ¶

run()

Read the problem and compare its values with the reference values.

The log-likelihood, the simulations at the measurements and the gradient of the log-likelihood are compared, each with its tolerance. The models the problem is simulated with are written into a temporary directory.

Returns:

Type	Description
`CaseResult`	The outcome of the case. It does not raise: a problem with a gap
`CaseResult`	or a layer without an implementation is `UNSUPPORTED` and any
`CaseResult`	other error is `ERROR`.

round_trip ¶

round_trip(directory)

Write the problem of the case as PEtab SciML and read it back.

The problem is read twice from the case and once from its export: the first model roadrunner loads in a process differs by about 1e-9 from every later one, so the export is compared with the second read, bit for bit.

Parameters:

Name	Type	Description	Default
`directory`	`Path`	the directory the export and the derived models are written to.	required

Returns:

Type	Description
`list[str]`	What differs between the problem of the case and the problem
`list[str]`	which is read back, one line each: the parameters (id, start
`list[str]`	value, bounds, unit, scale, target), the hybridizations, the
`list[str]`	data, the kinds and the weights of the fit mappings, the
`list[str]`	predictions and the log-likelihood. What `petab` finds wrong
`list[str]`	with the export is listed too, or that it was not validated
`list[str]`	because torch is missing. A case without parameters, fit
`list[str]`	mappings or hybridizations is a difference, it compares nothing.
`list[str]`	Empty when the round trip is exact.

SciMLSuite `dataclass` ¶

SciMLSuite(path, commit)

The cases of a commit of the PEtab SciML test suite.

Attributes:

Name	Type	Description
`path`	`Path`	the directory which holds the directories of the groups, i.e. `test_cases` of the suite.
`commit`	`str`	the commit of the suite.

cache_path `staticmethod` ¶

cache_path(commit=SCIML_SUITE_COMMIT)

Get the directory a commit of the suite is unpacked into.

SBMLSIM_SCIML_SUITE_PATH overrides it. Otherwise it is sbmlsim/petab-sciml-testsuite/<commit> in the user cache.

Parameters:

Name	Type	Description	Default
`commit`	`str`	the commit of the suite.	`SCIML_SUITE_COMMIT`

Returns:

Type	Description
`Path`	The directory the groups of the commit live in.

cached `classmethod` ¶

cached(commit=SCIML_SUITE_COMMIT)

Get a commit of the suite if it is already on this machine.

Parameters:

Name	Type	Description	Default
`commit`	`str`	the commit of the suite.	`SCIML_SUITE_COMMIT`

Returns:

Type	Description
`SciMLSuite \| None`	The suite, or `None` if it was not downloaded yet.

load `classmethod` ¶

load(commit=SCIML_SUITE_COMMIT)

Get a commit of the suite, downloading it if it is not cached.

Parameters:

Name	Type	Description	Default
`commit`	`str`	the commit of the suite.	`SCIML_SUITE_COMMIT`

Returns:

Type	Description
`SciMLSuite`	The suite with its cases unpacked in the cache.

Raises:

Type	Description
`OSError`	if the commit cannot be downloaded.

case_ids ¶

case_ids(group)

Get the numbers of the cases of a group.

Parameters:

Name	Type	Description	Default
`group`	`str`	the group, i.e. the name of its directory.	required

Returns:

Type	Description
`list[str]`	The names of the case directories, sorted, empty for a group the
`list[str]`	suite does not have.

model_import_cases ¶

model_import_cases()

Iterate the cases of the group ml_model_import.

initialization_cases ¶

initialization_cases()

Iterate the cases of the group initialization.

problem_import_cases ¶

problem_import_cases()

Iterate the cases of the group sciml_problem_import.

run ¶

run()

Run the cases of the suite.

Returns:

Type	Description
`list[CaseResult]`	The results of the groups `ml_model_import`, `initialization` and
`list[CaseResult]`	`sciml_problem_import`, in this order.

read_solutions ¶

read_solutions(path)

Read the solutions.yaml of a case.

Parameters:

Name	Type	Description	Default
`path`	`Path`	the directory of the case.	required

Returns:

Type	Description
`dict[str, Any]`	The content of the file.

Raises:

Type	Description
`ValueError`	if the directory has no `solutions.yaml` or the file is not a mapping.

required ¶

required(solutions, key, path)

Get a value of the solutions.yaml of a case which must be there.

Parameters:

Name	Type	Description	Default
`solutions`	`dict[str, Any]`	the content of the file.	required
`key`	`str`	the key of the value.	required
`path`	`Path`	the directory of the case, for the message.	required

Returns:

Type	Description
`Any`	The value.

Raises:

Type	Description
`ValueError`	if the file has no value for the key.

read_array ¶

read_array(path, group)

Read the single array of an input or output file of a case.

Parameters:

Name	Type	Description	Default
`path`	`Path`	the HDF5 file.	required
`group`	`str`	`inputs` or `outputs`.	required

Returns:

Type	Description
`ndarray`	The array in the PyTorch layout, in double precision.

Raises:

Type	Description
`ValueError`	if the file does not hold exactly one array in the group or is not in the PyTorch layout.

compare_arrays ¶

compare_arrays(observed, expected, tolerance)

Compare an array with its reference values.

Parameters:

Name	Type	Description	Default
`observed`	`ndarray`	the values of `sbmlsim`.	required
`expected`	`ndarray`	the reference values.	required
`tolerance`	`float`	the absolute tolerance.	required

Returns:

Type	Description
`CaseStatus`	The outcome, the message and the largest absolute difference, `None`
`str`	when the shapes differ or a value is not finite.

parameter_key ¶

parameter_key(model_entity_id)

Get the key of an entry from a modelEntityId of the mapping table.

Parameters:

Name	Type	Description	Default
`model_entity_id`	`str`	the entity, e.g. `net1.parameters[layer1].weight`.	required

Returns:

Type	Description
`str \| None`	The key of `sbmlsim.sciml.parameters`, i.e. `net1` for
`str \| None`	`net1.parameters`, `net1.layer1` for `net1.parameters[layer1]` and
`str \| None`	`net1.layer1.weight` for `net1.parameters[layer1].weight`. `None` for
`str \| None`	an entity which is not the parameters of a network.

nothing_to_compare ¶

nothing_to_compare(problem)

Name what a problem lacks for a round trip to compare anything.

Parameters:

Name	Type	Description	Default
`problem`	`OptimizationProblem`	the problem read from a case.	required

Returns:

Type	Description
`list[str]`	One line for each of the parameters, fit mappings and hybridizations
`list[str]`	which the problem does not have.

sciml.testsuite¶

CaseStatus ¶

CaseResult dataclass ¶

passed property ¶

key property ¶

ModelImportCase dataclass ¶

tolerance property ¶

from_directory classmethod ¶

network ¶

run ¶

InitializationCase dataclass ¶

from_directory classmethod ¶

nominal ¶

expected ¶

run ¶

ProblemImportCase dataclass ¶

from_directory classmethod ¶

settings ¶

expected_simulations ¶

expected_gradient ¶

run ¶

round_trip ¶

SciMLSuite dataclass ¶

cache_path staticmethod ¶

cached classmethod ¶

load classmethod ¶

case_ids ¶

model_import_cases ¶

initialization_cases ¶

problem_import_cases ¶

run ¶

read_solutions ¶

required ¶

read_array ¶

compare_arrays ¶

parameter_key ¶

nothing_to_compare ¶

CaseResult `dataclass` ¶

passed `property` ¶

key `property` ¶

ModelImportCase `dataclass` ¶

tolerance `property` ¶

from_directory `classmethod` ¶

InitializationCase `dataclass` ¶

from_directory `classmethod` ¶

ProblemImportCase `dataclass` ¶

from_directory `classmethod` ¶

SciMLSuite `dataclass` ¶

cache_path `staticmethod` ¶

cached `classmethod` ¶

load `classmethod` ¶