testsuite.runner¶

Running a semantic case of the SBML Test Suite.

run_case reads the model of a case, simulates it as its settings ask and compares the results with the expected ones. It answers with a CaseResult rather than raising: a case whose model cannot be read or whose integration fails is a result of the suite like any other, and the report groups the cases by exactly these outcomes. A case which passes is simulated again on its loaded model, as every point of a scan after the first and every simulation of an experiment after the first are, and these simulations must equal the first, bitwise (#305): the same simulation again, the simulation after a shorter one, and a steady state before it, compared with that of a fresh model, see repeat_check.

CaseStatus ¶

Bases: StrEnum

The outcome of a case of the suite.

pass : the results are within the tolerances of the case.

tolerance : the case was simulated and its results are outside the tolerances, i.e. the simulation is wrong.

not_read : the model could not be loaded, e.g. an SBML construct which the simulator does not support.

simulation_error : the model was loaded and the integration failed.

missing_variable : the simulation ran and did not produce a variable the case compares, e.g. a symbol the simulator does not expose.

repeat : the results are within the tolerances, and a simulation of the loaded model after it differs from the first or fails, i.e. a simulation depends on the ones before it on the same model, see repeat_check.

not_read, simulation_error and missing_variable are failures of the simulator and not of its numerics, so a report separates them: they say something different about what is missing.

CaseResult `dataclass` ¶

CaseResult(
    cid,
    status,
    message="",
    duration=0.0,
    comparison=None,
    component_tags=frozenset(),
    test_tags=frozenset(),
    encoding="",
)

The result of running a case of the suite.

Attributes:

Name	Type	Description
`cid`	`str`	identifier of the case.
`status`	`CaseStatus`	outcome of the case.
`message`	`str`	what went wrong, empty for a case which passed.
`duration`	`float`	seconds the case took, including reading the model.
`comparison`	`CaseComparison \| None`	comparison with the expected results, `None` if the case was never simulated.
`component_tags`	`frozenset[str]`	SBML components of the case, for the report.
`test_tags`	`frozenset[str]`	what the case tests, for the report.
`encoding`	`str`	SBML encoding which was simulated, e.g. `l3v2`.

passed `property` ¶

passed

Check whether the case passed.

to_dict ¶

to_dict()

Convert to a dictionary of JSON serializable values.

case_simulation ¶

case_simulation(case, fraction=1.0, steady_state=False)

Get the simulation of a case, as its settings ask.

Parameters:

Name	Type	Description	Default
`case`	`SemanticCase`	the case.	required
`fraction`	`float`	the part of its duration, with as many of its steps.	`1.0`
`steady_state`	`bool`	integrate to a steady state before it, for at most the duration of the case; one which is not reached raises.	`False`

Returns:

Type	Description
`Simulation`	The simulation.

load_case_model ¶

load_case_model(case, simulator)

Load the model of a case with roadrunner, which the checks of a case use.

Parameters:

Name	Type	Description	Default
`case`	`SemanticCase`	the case.	required
`simulator`	`Simulator`	the simulator with the tolerances of the test suite, of the engine roadrunner.	required

Returns:

Type	Description
`RoadrunnerSBMLModel`	The model of the case.

Raises:

Type	Description
`TypeError`	if the simulator does not load the model in roadrunner.

simulate_case ¶

simulate_case(case, simulator, model, simulation=None)

Simulate a case with the selections and the output of its settings.

Parameters:

Name	Type	Description	Default
`case`	`SemanticCase`	the case.	required
`simulator`	`Simulator`	the simulator with the tolerances of the test suite.	required
`model`	`SBMLModel`	the model of the case, loaded by the simulator.	required
`simulation`	`Simulation \| None`	the simulation, the one of the settings of the case by default, see `case_simulation`.	`None`

Returns:

Type	Description
`TimecourseResult`	The values of the selections of the case at its output times, a `time`
`TimecourseResult`	column and one column per selection.

seed_event_order ¶

seed_event_order(model, seed=1)

Seed the random generator which orders simultaneous events.

libroadrunner executes simultaneous events of the same priority in a random order, drawn with the random generator of the model (from libroadrunner 2.11; before with std::rand of the C library), whose state advances with every draw: a second simulation of a model with such events draws other numbers than the first and differs in the last bits (cases 00952 and 01588). The generator is seeded before every simulation of a case, without a reset of the model, so a case gives the same results in every process and the repeat check compares simulations of the same order. The seed is also the one of the models roadrunner creates later in the process, i.e. of a regenerated model.

Parameters:

Name	Type	Description	Default
`model`	`RoadrunnerSBMLModel`	the loaded model.	required
`seed`	`int`	the seed.	`1`

repeat_difference ¶

repeat_difference(
    first,
    second,
    what="the second simulation of the loaded model",
)

Describe how a later simulation differs from the first, bitwise.

Parameters:

Name	Type	Description	Default
`first`	`TimecourseResult`	the first simulation.	required
`second`	`TimecourseResult`	the later simulation.	required
`what`	`str`	what the later simulation is, for the description.	`'the second simulation of the loaded model'`

Returns:

Type	Description
`str`	An empty string for equal results, `NaN` equal to `NaN`, else what
`str`	differs.

repeat_check ¶

repeat_check(case, simulator, model, first)

Simulate a case again on its loaded model and describe what differs.

Every point of a scan after the first of its chunk and every simulation of an experiment after the first runs on a model which was simulated before (#292, #303, #305). After the first simulation of a case, which passed:

the same simulation again;
a simulation of half the duration, then the simulation of the case, which integrates beyond it, so the pending events of the shorter one are discarded (a model with an event with a delay is generated again);
the simulation with a steady state before it, which integrates without end, compared with that simulation of a fresh model; both may fail to reach a steady state, then with the same error.

Each must equal the first, bitwise. The random generator of the model is seeded before every simulation, see seed_event_order. A regeneration draws from it as well, so a case of random event execution (the test tag RandomEventExecution) is compared with its expected results in 2 and within the tolerances of the case in 3 instead.

Parameters:

Name	Type	Description	Default
`case`	`SemanticCase`	the case.	required
`simulator`	`Simulator`	the simulator of the case.	required
`model`	`RoadrunnerSBMLModel`	the loaded model, simulated once.	required
`first`	`TimecourseResult`	the first simulation.	required

Returns:

Type	Description
`str`	An empty string, or what differs.

map_cases ¶

map_cases(function, cases, workers=None)

Apply a function to every case, in parallel processes.

The cases are independent of each other, so they are spread over a pool of processes. A case takes from milliseconds to seconds, so they are handed out in small chunks, which keeps every process busy until the end. The pool is the kept pool of sbmlsim.parallel; a run which is interrupted, e.g. by Ctrl-C, which the workers ignore, stops it.

Parameters:

Name	Type	Description	Default
`function`	`Callable[[SemanticCase], T]`	a module level function, the processes import it.	required
`cases`	`Sequence[SemanticCase]`	the cases.	required
`workers`	`int \| None`	number of processes, the cores available to this process by default; `1` applies the function in this process.	`None`

Returns:

Type	Description
`list[T]`	The answers of the function, in the order of the cases.

run_suite ¶

run_suite(suite, cids=None, workers=None)

Run the cases of a suite.

Parameters:

Name	Type	Description	Default
`suite`	`SemanticSuite`	the suite to run.	required
`cids`	`Iterable[str] \| None`	identifiers to run, all timecourse cases by default.	`None`
`workers`	`int \| None`	number of processes the cases are run in, the cores available to this process by default; `1` runs them in this process.	`None`

Returns:

Type	Description
`list[CaseResult]`	The results, in the order of the case identifiers.

run_case ¶

run_case(case)

Run a case of the suite and compare it with the expected results.

Every failure is a result: a model which cannot be read, an integration which fails and results outside the tolerances are the outcomes the report is grouped by, so nothing is raised.

Parameters:

Name	Type	Description	Default
`case`	`SemanticCase`	the case to run.	required

Returns:

Type	Description
`CaseResult`	The result of the case with its status and its comparison.

testsuite.runner¶

CaseStatus ¶

CaseResult dataclass ¶

passed property ¶

to_dict ¶

case_simulation ¶

load_case_model ¶

simulate_case ¶

seed_event_order ¶

repeat_difference ¶

repeat_check ¶

map_cases ¶

run_suite ¶

run_case ¶

CaseResult `dataclass` ¶

passed `property` ¶