Skip to content

credibility.cross_validation

The leave-one-study-out cross-validation of an adopted fit.

A fit describes its training data; how well it predicts a study it has not seen is a question of the form of the model. The cross-validation refits the problem of an adopted fit once per training study without that study (its training mappings become validation mappings, held_out), starting from the adopted parameters, and predicts the held-out study with the refitted parameters. Compared with the predictions of the adopted fit for the same points, the held-out predictions show how much of the description of a study is fitted to it: a GMFE of the held-out points close to the one of the fitted points is a model whose form carries across studies, a much larger one a model whose parameters are fitted to every study.

cross_validate runs it, CrossValidation is its result, which write_cross_validation writes into an out in the layout of the records of the adopted fits (fits/<fit id>/cross_validation.json, sbmlsim.run.fit_kinds.FITS_DIR, never into the package), from where it is reviewed and committed next to the record of the adopted fit; the credibility assessment reads it there for the factor model form (read_cross_validations). It is run on demand, since it fits the problem once per training study. The format is the one of pkpdsim 0.19.

HeldOutStudy dataclass 

HeldOutStudy(
    *,
    study,
    n_points,
    gmfe_held_out,
    gmfe_fitted,
    within_2fold_held_out,
    within_2fold_fitted,
    cost,
)

The prediction of a training study by the fit without it.

Attributes:

Name Type Description
study str

the study, the id of its experiment.
n_points int

the points of its training mappings with a positive data value and positive predictions.
gmfe_held_out float

the GMFE of the predictions of the fit without the study.
gmfe_fitted float

the GMFE of the predictions of the adopted fit.
within_2fold_held_out float

the percentage of the held-out predictions within 2-fold.
within_2fold_fitted float

the same of the adopted fit.
cost float

the cost of the fit without the study.

to_dict 

to_dict()

Get the study as plain data, NaN as None.

from_dict classmethod 

from_dict(data)

Get the study from plain data.

CrossValidation dataclass 

CrossValidation(
    *,
    fit,
    subset,
    studies,
    gmfe_held_out,
    gmfe_fitted,
    seed,
    date,
)

The leave-one-study-out cross-validation of an adopted fit.

Attributes:

Name Type Description
fit str

the id of the adopted fit.
subset str

its fit problem, e.g. PK.
studies tuple[HeldOutStudy, ...]

the result of every held-out study.
gmfe_held_out float

the GMFE of all held-out points together.
gmfe_fitted float

the GMFE of the same points with the adopted fit.
seed int

the seed of the fits.
date str

the date of the cross-validation, ISO 8601 with the time zone.

n_points property 

n_points

Get the held-out points of all studies.

statement 

statement()

Describe the cross-validation in one sentence.

to_dict 

to_dict()

Get the cross-validation as plain data.

from_dict classmethod 

from_dict(data)

Get the cross-validation from plain data.

training_studies 

training_studies(definition)

Get the studies of a fit problem with a training mapping, sorted.

held_out 

held_out(definition, study)

Get a fit problem whose training mappings of a study are validation mappings.

Parameters:

Name Type Description Default
definition FitDefinition

the fit problem.

 required
study str

the study to hold out.

 required

Returns:

Type Description
FitDefinition

The problem without the study in its training data; the study is
FitDefinition

still simulated, as validation data, so its predictions are reported.

Raises:

Type Description
KeyError

if the study is no study of the problem.

cross_validate 

cross_validate(
    project,
    fit_id,
    studies=None,
    *,
    output_dir,
    seed=1234,
    n_cores=1,
)

Cross-validate an adopted fit by leaving out one training study at a time.

Every refit is a run of least squares from the adopted parameters (the sampling START) with the adopted fits of the other problems fixed, as a fit with a start does (sbmlsim.fit.project_fits).

Parameters:

Name Type Description Default
project Project

the project.

 required
fit_id str

an adopted fit of the project.

 required
studies Sequence[str] | None

the training studies to hold out, all by default.

 None
output_dir Path

the directory of the refits.

 required
seed int

the seed of the refits.

 1234
n_cores int

the workers of a refit.

 1

Returns:

Type Description
CrossValidation

The cross-validation.

Raises:

Type Description
ValueError

if the fit is not adopted or a study is no training study.

write_cross_validation 

write_cross_validation(result, out)

Write a cross-validation into an out, in the layout of the adopted fits.

Parameters:

Name Type Description Default
result CrossValidation

the cross-validation.

 required
out Path

the directory it is written into; never the package, the file is committed next to the record of its adopted fit after a review.

 required

Returns:

Type Description
Path

The file, fits/<fit id>/cross_validation.json of the out.

read_cross_validations 

read_cross_validations(project)

Read the cross-validations of the adopted fits of a project, by subset.

A cross-validation lies in the record of its adopted fit (Project.adopted_record).

Returns:

Type Description
dict[str, CrossValidation]

The cross-validation of every adopted fit which has one.