baseline

The baseline of a test suite: the cases which do not pass, with their reasons.

The PEtab SciML test suite and the PEtab v2 test suite record the outcome of their cases in a JSON file, tests/data/<suite>_baseline.json:

{
  "suite_commit": "<commit>",
  "n_cases": 129,
  "n_passed": 128,
  "expected_failures": {
    "sbml/0001": {"status": "tolerance", "reason": "why it does not pass"}
  }
}

A run is compared with it in both directions: a case which passed and fails is a regression, a case which is listed and passes is a baseline which is out of date. scripts/sciml_testsuite.py and scripts/petab_testsuite.py are the command lines which write and check it.

Outcome ¶

Bases: Protocol

The outcome of a case, see the CaseResult of a suite.

key `property` ¶

key

Get the key of the case in the baseline.

passed `property` ¶

passed

Check whether the case passes.

status `property` ¶

status

Get the status, an enum whose value is the name in the baseline.

unexpected_outcomes ¶

unexpected_outcomes(results, baseline)

Get the cases which do not have the outcome the baseline records.

Parameters:

Name	Type	Description	Default
`results`	`Sequence[Outcome]`	the results of a run.	required
`baseline`	`dict[str, Any]`	the content of the baseline.	required

Returns:

Type	Description
`list[str]`	One line per case, `<key>: '<expected>' -> '<observed>'`, and one per
`list[str]`	case of the baseline which was not run.

write_baseline ¶

write_baseline(results, commit, path)

Write the cases which do not pass, keeping the reasons which are recorded.

A reason is kept while the case fails with the recorded status. A case whose status changed gets MISSING_REASON, the recorded reason describes the old status.