Skip to content

credibility.scorecard

The credibility of many models at once: the scorecard and the pooling.

A repository of models assesses every model on its own; the scorecard reads their records (credibility.json) into one table, one row per model and context of use, and compares it with a stored baseline, so a change of the shared library or of the data which makes a verdict or a metric worse shows across all models at once (compare, a Regression each). The evidence extensions of a project add columns to the scorecard (sbmlsim.credibility.extensions.EvidenceExtension.scorecard_columns), judge them against the baseline themselves (EvidenceExtension.regressions) and pool their sections over the models (pool), e.g. the ratio pairs of all models by a value of the metadata of their numerator arm.

Regression dataclass 

Regression(*, model, context, what, baseline, current)

A context of use of a model which got worse than its baseline.

Attributes:

Name Type Description
model str

the model.
context str

the context of use.
what str

what got worse: missing, verdict, verified, uncertainty_robust, the name of an acceptance rule or a column of an extension.
baseline str

the value of the baseline, as text.
current str

the current value, as text.

load_records 

load_records(paths)

Read the records of assessments, by the model of their provenance.

Parameters:

Name Type Description Default
paths Iterable[Path]

records (credibility.json) or directories, which are searched for records.

 required

Returns:

Type Description
dict[str, CredibilityAssessment]

The assessments by model, sorted.

Raises:

Type Description
FileNotFoundError

if a path does not exist or no record is found.
ValueError

if two records are of one model.

scorecard 

scorecard(records, extensions=())

Get one row per model and context of use.

Parameters:

Name Type Description Default
records Mapping[str, CredibilityAssessment]

the assessments by model.

 required
extensions Sequence[EvidenceExtension]

the evidence extensions, whose columns follow the columns of the core.

 ()

Returns:

Type Description
DataFrame

The columns of SCORECARD_COLUMNS (verified empty without
DataFrame

verification, failed_checks the failed checks of the verification
DataFrame

comma separated, uncertainty_robust whether every rule is met with
DataFrame

the threshold probability over the parameter draws (False if one is
DataFrame

shown below it, empty if one is undecided or without the
DataFrame

propagation), band_fold the largest median fold width of the
DataFrame

prediction bands of the quantities, vpc_coverage and
DataFrame

vpc_agreement the coverage of the predictive check and its agreement
DataFrame

with the expected coverage, empty without the check), the columns of
DataFrame

the extensions (EvidenceExtension.scorecard_names) and per
DataFrame

acceptance rule its value, the rules of the extensions included.

compare 

compare(
    baseline, current, tolerance=TOLERANCE, extensions=()
)

Get the contexts of use which got worse than the baseline.

A context of the baseline which is missing is a regression, and so is a worse verdict (VERDICT_RANK), a verified context which is no longer verified, a context whose rules were robust over the parameter draws and no longer are, the value of a rule of both which got worse by more than the tolerance (a percentage by 100 * tolerance points, a GMFE by the share tolerance, an AFE by tolerance further from 1) and what every extension finds in its own columns (EvidenceExtension.regressions). A value which is no longer evaluable is worse, a new context or rule is none. The scorecards are compared by the names of their columns.

Parameters:

Name Type Description Default
baseline DataFrame

a scorecard, see scorecard, e.g. read from its TSV.

 required
current DataFrame

the current scorecard.

 required
tolerance float

the change of a value which is no regression.

 TOLERANCE
extensions Sequence[EvidenceExtension]

the evidence extensions, which judge their columns.

 ()

Returns:

Type Description
list[Regression]

The regressions, in the order of the baseline.

pool 

pool(records, extension, options=None)

Pool the sections of an extension over the contexts of use of the models.

Parameters:

Name Type Description Default
records Mapping[str, CredibilityAssessment]

the assessments by model.

 required
extension EvidenceExtension

the extension.

 required
options Mapping[str, Any] | None

the options of the pooling as the caller got them, e.g. {"field": "health"}; the defaults of the extension without.

 None

Returns:

Type Description
DataFrame | None

The pooled table of the extension (EvidenceExtension.pool), None
DataFrame | None

for an extension which pools nothing.