Skip to content

testsuite.results

Results of a suite run: per case and stage, JSON, regressions.

StageResult dataclass 

StageResult(status, message='', max_excess=None)

Outcome of one stage of one case.

CaseResult dataclass 

CaseResult(
    id,
    test_tags,
    component_tags,
    stages,
    name="",
    informative=None,
)

Outcome of one case.

SuiteResult dataclass 

SuiteResult(suite, version, cases=dict(), skipped=dict())

Outcome of a suite run.

counts 

counts(stage)

Number of cases per status of a stage.

to_json 

to_json(path)

Write the result as JSON, sorted and indented (deterministic).

max_excess is rounded to 3 significant digits: its exact value is an artifact of the solver and the machine it ran on, so keeping the full precision would churn thousands of lines on every regeneration.

from_json classmethod 

from_json(path)

Read a result written by to_json.

regressions 

regressions(old, new)

Stages which passed before and do not pass now, and missing cases.

Parameters:

Name Type Description Default
old SuiteResult

committed result.

 required
new SuiteResult

current result.

 required

Returns:

Type Description
list[str]

One line per regression, e.g. 00001 roundtrip: pass -> fail (message).

improvements 

improvements(old, new)

Stages which did not pass before and pass now.