credibility.assessment¶

The assessment of the contexts of use of a model as plain data.

assess evaluates the adopted fits of a project (evaluate_fits) and assesses its declaration on their data points, assess_table does the same on a given table without a project. Per context of use:

The metrics of every evidence: the points, mappings and studies, criteria_metrics and the values of the acceptance rules of the context.
The acceptance: the rules on the union of the points of the decisive evidence (ContextOfUse.decisive_role, calibration for C1, else validation), each row of the table once, followed by the rules of the evidence extensions of the project (sbmlsim.credibility.extensions.ContextContribution.acceptance), each row with the name of its extension.
The coverage of every claimed value of the domain: validated with applicable points of validation evidence, calibrated with applicable points of calibration evidence only, uncovered without; a C1 context counts calibrated as covered. A point is applicable when the fold criteria can evaluate it, a positive and finite value and prediction (applicable): the points of a failed simulation or of zeros cover nothing. The values of the points outside the claimed domain are reported as unclaimed values, a point without the field has none.
The coverage of every quantity of interest: covered with an applicable decisive point of its observable inside the claimed domain (every claimed field of the point has a claimed value), uncovered without; a ratio quantity with a pairing by the points its extension derives (ContextContribution.quantities), a declared ratio without a pairing has no coverage.
The minimum of evaluable points, ContextOfUse.minimum_points.
The verification of the decisive predictions, when the assessment is verified (sbmlsim.credibility.verification): per check whether it passed, the largest relative difference of the alternative predictions and whether the acceptance changes with them, and the checks of the battery of the variants of the model (sbmlsim.credibility.battery), the same for every context.
The parameter uncertainty and the predictive check with the variability of the subjects of the decisive predictions, when they ran (sbmlsim.credibility.uncertainty, sbmlsim.credibility.variability).
The verdict, see verdict, and the factors of V&V 40 (FACTORS), the statements of the extensions combined with the ones of the core (sbmlsim.credibility.factors.combine_factor).
The sections of the extensions, which the extensions write and read (ContextResult.extensions, CredibilityAssessment.extensions).

The statement of the domain is recorded and shown, not assessed. The limitations are reported apart, with their metrics, and enter no acceptance. The assessment writes and reads JSON (to_json, from_json), with NaN as null; it reads the records of pkpdsim 0.17 to 0.19 as well, whose evidence of the extensions is read as their sections (sbmlsim.credibility.extensions.read_sections).

Verdict ¶

Bases: StrEnum

The verdict of the assessment of a context of use.

label `property` ¶

label

Get the verdict in words, e.g. supported with extrapolation.

Coverage ¶

Bases: StrEnum

The coverage of a claimed value of the domain of a context of use.

QuantityRecord ¶

Bases: TypedDict

A quantity of interest as plain data, see QuantityOfInterest.

Attributes:

Name	Type	Description
`id`	`str`	the id of the quantity.
`description`	`str`	what the quantity is.
`ratio`	`bool`	whether the quantity is a ratio between arms.
`coverage`	`str \| None`	`covered` with an applicable decisive point of the quantity inside the claimed domain, a ratio with a pairing with a point its extension derives, else `uncovered`; `None` for a ratio without a pairing and in a record of pkpdsim 0.17, which did not assess it.
`n_points`	`int`	the applicable decisive points of the quantity inside the claimed domain, the points the extension derives for a ratio with a pairing, 0 for a ratio without one.

AcceptanceRecord ¶

Bases: TypedDict

The result of an acceptance rule as plain data, see AcceptanceResult.

Attributes:

Name	Type	Description
`name`	`str`	the name of the rule.
`value`	`float`	its value, `NaN` if not evaluable.
`limit`	`str`	its limit in words.
`met`	`bool \| None`	whether it is met, `None` if not evaluable.
`extension`	`NotRequired[str]`	the name of the evidence extension of the rule, absent for a rule of the context, see `sbmlsim.credibility.extensions.ACCEPTANCE_EXTENSION`.

UnclaimedRecord ¶

Bases: TypedDict

The unclaimed validation points of a fit mapping.

Attributes:

Name	Type	Description
`fit`	`str`	the subset of the fit.
`experiment`	`str`	the experiment.
`mapping`	`str`	the key of the mapping.
`n`	`int`	the points.

EvidenceResult `dataclass` ¶

EvidenceResult(
    *,
    context,
    name,
    role,
    description,
    selection,
    n_points,
    n_mappings,
    n_studies,
    outliers_dropped,
    limitation_subtracted,
    metrics,
)

The metrics of an evidence of a context of use or of a limitation.

Attributes:

Name	Type	Description
`context`	`str`	the id of the context of use, `""` for a limitation of the model.
`name`	`str`	the name of the evidence.
`role`	`EvidenceRole`	the role of the evidence.
`description`	`str`	what the evidence is.
`selection`	`str`	what the evidence selects, see `describe_selection`.
`n_points`	`int`	the data points.
`n_mappings`	`int`	the fit mappings, by experiment and key.
`n_studies`	`int`	the studies, the experiments of the mappings.
`outliers_dropped`	`int`	the outlier points a validation dropped.
`limitation_subtracted`	`int`	the limitation points subtracted.
`metrics`	`dict[str, float]`	`criteria_metrics` and the values of the acceptance rules of the context, `NaN` if not evaluable.

to_dict ¶

to_dict()

Get the result as plain data, NaN as None.

from_dict `classmethod` ¶

from_dict(data)

Get the result from plain data, None as NaN.

CoverageEntry `dataclass` ¶

CoverageEntry(
    *, context, field, value, coverage, claimed, n_points
)

The coverage of a value of a field of the metadata by a context of use.

Attributes:

Name	Type	Description
`context`	`str`	the id of the context of use.
`field`	`str`	the field of the metadata, e.g. `health`.
`value`	`str`	the name of the value, e.g. `RENAL_IMPAIRMENT`.
`coverage`	`Coverage \| None`	the coverage of a claimed value, `None` for a value outside the claimed domain.
`claimed`	`bool`	whether the domain claims the value; the values of the points of the evidence outside the claim are not claimed.
`n_points`	`int`	the points with the value: the applicable points of the validation evidence for a validated value, of the calibration evidence for a calibrated one, every point of the evidence for a value outside the claim.

to_dict ¶

to_dict()

Get the entry as plain data.

from_dict `classmethod` ¶

from_dict(data)

Get the entry from plain data.

ContextResult `dataclass` ¶

ContextResult(
    *,
    id,
    title,
    question,
    level,
    influence,
    consequence,
    risk,
    rationale,
    extrapolation,
    domain_statement,
    quantities,
    acceptance,
    verdict,
    factors,
    n_decisive,
    n_evaluable,
    minimum_points,
    verification=list(),
    uncertainty=None,
    variability=None,
    extensions=dict(),
)

The assessment of a context of use.

Attributes:

Name	Type	Description
`id`	`str`	the id of the context.
`title`	`str`	the title.
`question`	`str`	the question.
`level`	`CouLevel`	the level on the ladder.
`influence`	`Level`	the influence of the model on the decision.
`consequence`	`Level`	the consequence of a wrong decision.
`risk`	`Level`	the model risk.
`rationale`	`str`	why this level, influence and consequence.
`extrapolation`	`str`	the argument for a domain without validation evidence.
`domain_statement`	`str`	what the domain claims beyond the metadata.
`quantities`	`list[QuantityRecord]`	the quantities of interest (`id`, `description`, `ratio`, `coverage`, `n_points`, see `QuantityRecord`).
`acceptance`	`list[AcceptanceRecord]`	the results of the acceptance rules (`name`, `value`, `limit`, `met`, see `AcceptanceRecord`), the rules of the context followed by the rules of the extensions, each with the name of its extension (`extension`); a record of pkpdsim 0.19 has its rule of the ratio pairs as the last row, of the extension `ratios`.
`verdict`	`Verdict`	the verdict.
`factors`	`list[FactorRecord]`	the factors of V&V 40 (`id`, `name`, `status`, `statement`, `step`), in the order of `FACTORS`.
`n_decisive`	`int`	the points of the decisive evidence, each row once.
`n_evaluable`	`int`	the decisive points which are applicable, see `applicable`.
`minimum_points`	`int`	the evaluable points the verdict needs, see `ContextOfUse.minimum_points`; 0 in a record of pkpdsim 0.17.
`verification`	`list[CheckRecord]`	the checks of the verification of the decisive predictions, the checks of the predictions followed by the checks of the battery; empty if the assessment was not verified.
`uncertainty`	`UncertaintyRecord \| None`	the parameter uncertainty of the decisive predictions, see `sbmlsim.credibility.uncertainty`; `None` for an assessment without the propagation and a record of pkpdsim 0.18 or earlier.
`variability`	`VariabilityRecord \| None`	the predictive check of the decisive points with the declared variability of the subjects, see `sbmlsim.credibility.variability`; `None` without it and in a record of pkpdsim 0.18 or earlier.
`extensions`	`dict[str, Any]`	the sections of the evidence extensions by their name, plain data which the extension of that name reads, see `sbmlsim.credibility.extensions`.

verified `property` ¶

verified

Get whether every check passed, None if not verified.

decisive_role `property` ¶

decisive_role

Get the role of the evidence which decides, see CouLevel.decisive_role.

uncovered_quantities `property` ¶

uncovered_quantities

Get the ids of the quantities of interest which are uncovered.

section ¶

section(name)

Get the section of an extension, None without one.

Parameters:

Name	Type	Description	Default
`name`	`str`	the name of the extension.	required

Returns:

Type	Description
`Any`	The section as plain data.

to_dict ¶

to_dict()

Get the result as plain data, NaN as None.

from_dict `classmethod` ¶

from_dict(data)

Get the result from plain data, None as NaN.

A record of pkpdsim 0.17 has no minimum of points, read as 0, and no coverage of the quantities, read as None with 0 points; a record of 0.18 or earlier has no verification, no uncertainty and no predictive check; a record of 0.19 has the evidence of its extensions as fields of its own, read as their sections, and the rule of its ratio pairs as a field, read as a row of the acceptance, see sbmlsim.credibility.extensions.read_sections and read_acceptance.

Provenance `dataclass` ¶

Provenance(
    *,
    model,
    title,
    fits,
    versions,
    commit,
    dirty,
    date,
    lock=None,
)

What an assessment was computed from.

Attributes:

Name	Type	Description
`model`	`str`	the name of the project of the model.
`title`	`str`	the title of the model.
`fits`	`list[str]`	the ids of the adopted fits.
`versions`	`dict[str, str]`	the versions of the packages, see `VERSIONED_PACKAGES`.
`commit`	`str \| None`	the commit of the repository of the model, `None` outside one.
`dirty`	`bool \| None`	whether the repository has changes, `None` outside one.
`date`	`str`	the date of the assessment, ISO 8601 with the time zone.
`lock`	`str \| None`	whether the evidence matched the lock of the declaration (`matched`), the lock was written with the assessment (`written`), or `None` without a lock, see `sbmlsim.credibility.lock`.

to_dict ¶

to_dict()

Get the provenance as plain data.

from_dict `classmethod` ¶

from_dict(data)

Get the provenance from plain data.

CredibilityAssessment `dataclass` ¶

CredibilityAssessment(
    *,
    provenance,
    contexts,
    evidence,
    coverage,
    limitations,
    unclaimed,
    notes,
    verification=None,
    cross_validations=list(),
    vpc=list(),
    extensions=dict(),
)

The assessment of the contexts of use of a model.

Attributes:

Name	Type	Description
`provenance`	`Provenance`	what the assessment was computed from.
`contexts`	`list[ContextResult]`	the result of every context of use, in the declared order.
`evidence`	`list[EvidenceResult]`	the metrics of every evidence of every context.
`coverage`	`list[CoverageEntry]`	the coverage of the domain of every context.
`limitations`	`list[EvidenceResult]`	the metrics of the limitations, of the model (context `""`) and of the contexts; they enter no acceptance.
`unclaimed`	`list[UnclaimedRecord]`	the validation points which no evidence and no limitation selects, per fit and mapping (`fit`, `experiment`, `mapping`, `n`).
`notes`	`str`	the notes of the declaration.
`verification`	`VerificationSummary \| None`	what the verification compared, `None` if the assessment was not verified.
`cross_validations`	`list[CrossValidation]`	the leave-one-study-out cross-validations of the adopted fits, see `sbmlsim.credibility.cross_validation`; empty without one and in a record of pkpdsim 0.18 or earlier.
`vpc`	`list[VpcPoint]`	every data point of the adopted fits with the band of its predictive check, see `sbmlsim.credibility.variability`; empty without the check and in a record of pkpdsim 0.18 or earlier.
`extensions`	`dict[str, Any]`	the sections of the evidence extensions in the assessment by their name, plain data, e.g. the provenance of the data of every study; a record of pkpdsim 0.19 has them as fields of its own, read as their sections.

context ¶

context(context_id)

Get the result of a context of use.

Parameters:

Name	Type	Description	Default
`context_id`	`str`	the id of the context.	required

Returns:

Type	Description
`ContextResult`	The result.

Raises:

Type	Description
`KeyError`	for an unknown context.

to_dict ¶

to_dict()

Get the assessment as plain data, NaN as None.

from_dict `classmethod` ¶

from_dict(data)

Get the assessment from plain data, None as NaN.

to_json ¶

to_json(path)

Write the assessment as JSON.

Parameters:

Name	Type	Description	Default
`path`	`Path`	the file.	required

from_json `classmethod` ¶

from_json(path)

Read an assessment written with to_json, or by pkpdsim 0.17 to 0.19.

Parameters:

Name	Type	Description	Default
`path`	`Path`	the file.	required

Returns:

Type	Description
`CredibilityAssessment`	The assessment.

describe_selection ¶

describe_selection(selection)

Describe what a selection of an evidence selects.

Parameters:

Name	Type	Description	Default
`selection`	`Selection`	the selection.	required

Returns:

Type	Description
`str`	The mapping keys by study for a `NamedFilter`, e.g.
`str`	`CredA: fm_dose; CredB: fm_dose, fm_zero`; the clauses of a
`str`	`MetadataFilter`, e.g. `route in [IV, PO] and health not in [HEALTHY]`;
`str`	`unclaimed validation data` for `Unclaimed`, followed by `within` and
`str`	the description of its filter; the qualified name otherwise.

applicable ¶

applicable(points)

Get the points the fold criteria can evaluate.

The predicate of the ratio criteria of sbmlsim (Fold, GMFE, AFE): a positive and finite value (DV) and prediction (IPRED).

Parameters:

Name	Type	Description	Default
`points`	`DataFrame`	the data points.	required

Returns:

Type	Description
`DataFrame`	The applicable points.

verdict ¶

verdict(
    n_decisive,
    acceptance,
    covered,
    extrapolation,
    *,
    level,
    n_evaluable,
    minimum,
    verified=None,
)

Get the verdict of a context of use.

In this order: no decisive point is insufficient evidence; a failed check of the verification is not verified, since the comparison of the predictions with the data is not to be trusted; a rule which is not met is not supported, also on few points; a rule which is not evaluable is insufficient evidence; fewer evaluable points than the minimum is insufficient evidence; every claimed value and every quantity with a coverage covered is supported, for C4 supported with extrapolation (a C4 context extrapolates without data of its condition by definition, its extrapolation argument is required); an extrapolation argument is supported with extrapolation; else insufficient evidence.

Parameters:

Name	Type	Description	Default
`n_decisive`	`int`	the points of the decisive evidence.	required
`acceptance`	`Sequence[AcceptanceResult]`	the results of the acceptance rules, of the context and of the extensions.	required
`covered`	`bool`	whether every claimed value and every quantity with a coverage is covered.	required
`extrapolation`	`str`	the extrapolation argument of the context.	required
`level`	`CouLevel`	the level of the context.	required
`n_evaluable`	`int`	the decisive points which are applicable.	required
`minimum`	`int`	the evaluable points the verdict needs.	required
`verified`	`bool \| None`	whether every check of the verification passed, `None` if the assessment is not verified.	`None`

Returns:

Type	Description
`Verdict`	The verdict.

assess_table ¶

assess_table(
    credibility,
    fits,
    provenance,
    verification=None,
    cross_validations=(),
    propagation=None,
    variability=None,
    *,
    extensions=(),
    prepared=None,
)

Assess the contexts of use of a declaration on the data points of the fits.

Parameters:

Name	Type	Description	Default
`credibility`	`Credibility`	the declaration.	required
`fits`	`FitTable`	the data points of the adopted fits with `DV`, `IPRED` and `DV_SD`, see `evaluate_fits`.	required
`provenance`	`Provenance`	what the table was computed from.	required
`verification`	`Verification \| None`	the alternative predictions of the table and the issues of the variants, see `sbmlsim.credibility.verification.verify`; the assessment is not verified without.	`None`
`cross_validations`	`Sequence[CrossValidation]`	the cross-validations of the adopted fits, which assess the factor model form.	`()`
`propagation`	`FitPropagation \| None`	the predictions at the draws of the parameters of the fits, see `sbmlsim.credibility.uncertainty.propagate_fits`; no context has an uncertainty record without.	`None`
`variability`	`PredictiveChecks \| None`	the predictive checks of the fits with the declared variability, see `sbmlsim.credibility.variability`; no context has a predictive check without.	`None`
`extensions`	`Sequence[EvidenceExtension]`	the evidence extensions of the project, in their order.	`()`
`prepared`	`Mapping[str, object] \| None`	what every extension prepared (`EvidenceExtension.prepare`) by its name; `None` for an extension without.	`None`

Returns:

Type	Description
`CredibilityAssessment`	The assessment.

Raises:

Type	Description
`SelectionError`	if the evidence selects invalid data, see `select`.

provenance ¶

provenance(project, source_commit=None, lock=None)

Get the provenance of an assessment of a project now.

Parameters:

Name	Type	Description	Default
`project`	`Project`	the project.	required
`source_commit`	`str \| None`	the commit of the source repository of the model, given by a release, which assesses the model in its release repository: the commit is recorded and whether its tree was clean is not known.	`None`
`lock`	`str \| None`	the state of the lock of the evidence, see `Provenance.lock`.	`None`

Returns:

Type	Description
`Provenance`	The versions of `VERSIONED_PACKAGES` and of the distributions of the
`Provenance`	extensions and the battery hook of the project, the commit and
`Provenance`	whether the root of the project has changes
`Provenance`	(`git status --porcelain -- .` in `Project.root`; `None` outside a
`Provenance`	repository, the dirty state `None` with a `source_commit`), the
`Provenance`	adopted fits, the date with the local time zone and the state of the
`Provenance`	lock.

assess ¶

assess(
    project,
    source_commit=None,
    *,
    verification=True,
    battery=True,
    cross_simulator=True,
    lock=None,
    uncertainty=None,
    variability=None,
    cross_validations=None,
)

Assess the contexts of use of a project on its adopted fits.

The fits are evaluated fresh on the current model and data, see evaluate_fits, and, with verification, again with the alternatives of the checks of sbmlsim.credibility.verification; the cross-validations of the adopted fits are read, and every evidence extension of the project prepares what it reads once (EvidenceExtension.prepare).

Parameters:

Name	Type	Description	Default
`project`	`Project`	the project with a declaration and adopted fits.	required
`source_commit`	`str \| None`	the commit of the source repository of a release, see `provenance`.	`None`
`verification`	`bool`	verify the decisive predictions; the factors of the verification are not assessed without.	`True`
`battery`	`bool`	run the battery of the checks of the variants with the verification, see `sbmlsim.credibility.battery`.	`True`
`cross_simulator`	`bool`	run the check `cross_simulator` of the battery, as the settings of the battery say; `False` switches it off.	`True`
`lock`	`Path \| None`	write the lock of the evidence (`sbmlsim.credibility.lock`) into this directory instead of comparing the evidence with the lock of the project; the lock of the package is never written.	`None`
`uncertainty`	`PropagationSettings \| None`	propagate the parameter uncertainty of the adopted fits to the decisive predictions with these settings, see `sbmlsim.credibility.uncertainty`; the factor model inputs is not assessed without.	`None`
`variability`	`VariabilitySettings \| None`	check the decisive points with the declared variability of the subjects (`Credibility.variability`) with these settings, nested in the draws of the parameters with `uncertainty`, see `sbmlsim.credibility.variability`.	`None`
`cross_validations`	`Mapping[str, CrossValidation] \| None`	the cross-validations of the adopted fits by their subset, the ones of the records of the project (`read_cross_validations`) without.	`None`

Returns:

Type	Description
`CredibilityAssessment`	The assessment.

Raises:

Type	Description
`ValueError`	if the project has no declaration or no adopted fits, or the settings of the variability are given for a declaration without a variability.
`KindMismatchError`	if a kind differs from the recorded one.
`SelectionError`	if the evidence selects invalid data.
`LockMismatchError`	if the evidence differs from the lock.

credibility.assessment¶

Verdict ¶

label property ¶

Coverage ¶

QuantityRecord ¶

AcceptanceRecord ¶

UnclaimedRecord ¶

EvidenceResult dataclass ¶

to_dict ¶

from_dict classmethod ¶

CoverageEntry dataclass ¶

to_dict ¶

from_dict classmethod ¶

ContextResult dataclass ¶

verified property ¶

decisive_role property ¶

uncovered_quantities property ¶

section ¶

to_dict ¶

from_dict classmethod ¶

Provenance dataclass ¶

to_dict ¶

from_dict classmethod ¶

CredibilityAssessment dataclass ¶

context ¶

to_dict ¶

from_dict classmethod ¶

to_json ¶

from_json classmethod ¶

describe_selection ¶

applicable ¶

verdict ¶

assess_table ¶

provenance ¶

assess ¶

label `property` ¶

EvidenceResult `dataclass` ¶

from_dict `classmethod` ¶

CoverageEntry `dataclass` ¶

from_dict `classmethod` ¶

ContextResult `dataclass` ¶

verified `property` ¶

decisive_role `property` ¶

uncovered_quantities `property` ¶

from_dict `classmethod` ¶

Provenance `dataclass` ¶

from_dict `classmethod` ¶

CredibilityAssessment `dataclass` ¶

from_dict `classmethod` ¶

from_json `classmethod` ¶