Skip to content

fit.evaluation

Evaluation of a fit: acceptance criteria and their aggregates.

The functions work on the table of the data points of a fit, FitMetrics.datapoints_df, i.e. on the columns DV (the measured value), IPRED (the prediction) and DV_SD (the standard deviation of the measured value, NaN without one), see sbmlsim.fit.metrics.

The points are evaluated per subset of the data with a Grouping: by the values of columns of the table, e.g. the tissue or the health of the metadata of the mappings, or by named subsets of fit mappings, which are selected with the MappingFilter of sbmlsim.fit.helpers a fit selects its data with.

A criterion decides per data point whether its prediction is acceptable, and it is applicable only to some points: a fold criterion to the points whose value and prediction are positive and finite, which have a ratio, an uncertainty criterion to the points with a standard deviation. A criterion is reported as the percentage of the applicable points it accepts.

GMFE, the geometric mean fold error 10**mean(|log10(f/y)|), is how far the predictions are from the data as a fold factor whatever the direction, AFE, the average fold error 10**mean(log10(f/y)), is the bias as a fold factor: 1 for no bias, above 1 for an overprediction.

The criterion of Guest et al. (2011) is defined for ratios, e.g. the ratio of the AUC of a victim drug with and without a perpetrator, and is the published one, delta = 1: guest_limits and within_guest. It is not a criterion of the data points of a fit: applied to an absolute value its limits depend on the unit of the value (an AUC of 50 in one unit is 0.05 in another).

Guest EJ, Aarons L, Houston JB, Rostami-Hodjegan A, Galetin A. Critique of the two-fold measure of prediction success for ratios: application for the assessment of drug-drug interactions. Drug Metab Dispos. 2011;39(2):170-173. https://doi.org/10.1124/dmd.110.036103

Criterion

Bases: Protocol

A criterion of the acceptance of the prediction of a data point.

name property 

name

Get the name of the criterion, the column of its percentage.

accepted 

accepted(points)

Get whether the criterion accepts the prediction of every point.

Parameters:

Name Type Description Default
points DataFrame

the data points, see the module.

 required

Returns:

Type Description
Series

1.0 for an accepted point, 0.0 for a rejected one and NaN
Series

where the criterion is not applicable, with the index of points.

Fold dataclass 

Fold(fold=2.0)

The prediction is within a fold factor of the data, y/fold <= f <= y*fold.

Attributes:

Name Type Description
fold float

the fold factor, larger than 1.

name property 

name

Get the name of the criterion, e.g. within_2fold.

accepted 

accepted(points)

Get whether the prediction of every point is within the fold factor.

The bounds are products, so that twofold is exact in floating point.

Uncertainty dataclass 

Uncertainty(k=2.0)

The prediction is within k standard deviations of the data.

Attributes:

Name Type Description
k float

the number of standard deviations.

name property 

name

Get the name of the criterion, e.g. within_2sd.

accepted 

accepted(points)

Get whether the prediction of every point is within k standard deviations.

Subset dataclass 

Subset(label, select)

A named subset of the fit mappings.

Attributes:

Name Type Description
label str

the name of the subset.
select MappingFilter

the filter of the mappings, called with the key and the fit mapping, see sbmlsim.fit.helpers.MappingFilter.

Grouping dataclass 

Grouping(name, columns=(), subsets=())

A grouping of the data points of a fit, by columns or by subsets.

Attributes:

Name Type Description
name str

the name of the grouping, the name of its figures and of its rows in the metrics.
columns tuple[str, ...]

the columns of the table of the data points; one group per combination of their values.
subsets tuple[Subset, ...]

the subsets of the fit mappings, one group per subset; the subsets may overlap.

by classmethod 

by(*columns)

Get the grouping by the values of columns, named after them.

Parameters:

Name Type Description Default
columns str

the columns of the table of the data points, e.g. the fields of the metadata of the mappings.

 ()

Returns:

Type Description
Grouping

The grouping, e.g. tissue_health for by("tissue", "health").

of_subsets classmethod 

of_subsets(name, *subsets)

Get the grouping by named subsets of the fit mappings.

Parameters:

Name Type Description Default
name str

the name of the grouping.

 required
subsets Subset

the subsets.

 ()

Returns:

Type Description
Grouping

The grouping.

groups 

groups(points, fit_mappings)

Get the groups of the data points with their labels.

Parameters:

Name Type Description Default
points DataFrame

the data points with the columns experiment and mapping and the columns of the grouping.

 required
fit_mappings MappingLookup

the fit mapping of an experiment and a mapping key, which the filter of a subset is called with.

 required

Returns:

Type Description
list[tuple[str, DataFrame]]

The label and the points of every group: by columns in the order
list[tuple[str, DataFrame]]

their values appear, the values joined by , and the value ""
list[tuple[str, DataFrame]]

labelled NOT_REPORTED; by subsets in the order of the subsets,
list[tuple[str, DataFrame]]

including a subset without points.

Raises:

Type Description
ValueError

if a column of the grouping is not a column of the points.

gmfe 

gmfe(points)

Get the geometric mean fold error 10**mean(|log10(f/y)|).

Parameters:

Name Type Description Default
points DataFrame

the data points, see the module.

 required

Returns:

Type Description
float

The fold error of the points with a ratio, nan without one.

afe 

afe(points)

Get the average fold error 10**mean(log10(f/y)), the bias as a fold factor.

Parameters:

Name Type Description Default
points DataFrame

the data points, see the module.

 required

Returns:

Type Description
float

The fold error of the points with a ratio, nan without one.

criteria_metrics 

criteria_metrics(points, criteria=DEFAULT_CRITERIA)

Get the fold errors and the percentage of the points every criterion accepts.

Parameters:

Name Type Description Default
points DataFrame

the data points, see the module.

 required
criteria Sequence[Criterion]

the criteria.

 DEFAULT_CRITERIA

Returns:

Type Description
dict[str, float]

GMFE, AFE and the percentage (0 to 100) of the applicable points
dict[str, float]

every criterion accepts by its name, nan for a criterion which is
dict[str, float]

applicable to no point.

guest_limits 

guest_limits(observed)

Get the limits of the criterion of Guest et al. for observed ratios.

With the observed ratio R, inverted when it is below 1, the limit is (1 + 2 (R - 1)) / R (delta = 1, as published) and a predicted ratio is accepted between R / limit and R * limit. The limit is 1 at R = 1 and tends to 2 for large ratios. The limits of a ratio below 1 are the inverted limits of its inverse.

Parameters:

Name Type Description Default
observed ArrayLike

the observed ratios, positive.

 required

Returns:

Type Description
ndarray

The lower and the upper limit of the predicted ratio of every
ndarray

observed ratio.

within_guest 

within_guest(observed, predicted)

Get the percentage of the predicted ratios within the limits of Guest et al.

Parameters:

Name Type Description Default
observed ArrayLike

the observed ratios, positive and finite.

 required
predicted ArrayLike

the predicted ratios; a ratio which is not finite, e.g. of a failed simulation, is not counted.

 required

Returns:

Type Description
float

The percentage (0 to 100) of the finite predicted ratios within the
float

limits of guest_limits, the limits included, nan without one.

Raises:

Type Description
ValueError

if an observed ratio is not positive and finite, or the lengths of the ratios differ.

group_metrics 

group_metrics(points, grouping, fit_mappings)

Get the metrics of every group and kind of fit mapping.

Parameters:

Name Type Description Default
points DataFrame

the data points, see Grouping.groups, with the columns kind, DV, IPRED, IRES, NRES and DV_SD.

 required
grouping Grouping

the grouping.

 required
fit_mappings MappingLookup

the fit mapping of an experiment and a mapping key.

 required

Returns:

Type Description
DataFrame

One row per group and kind, the kinds in the order of
DataFrame

EVALUATED_KINDS, a group without points of a kind has no row of it;
DataFrame

the columns grouping, group, kind, n, n_ratio (the points
DataFrame

with a ratio, which the fold criteria count), n_sd (the points with
DataFrame

a standard deviation, which the SD criteria count), NRMSE and the
DataFrame

metrics of criteria_metrics. There is no RMSE, a group mixes the
DataFrame

units of its observables; the NRMSE is scale free.

subset_agreement 

subset_agreement(points)

Get the agreement of a subset of the data points with its confidence.

The bias mean(log10(f/y)) and the half width 1.96 SD of the limits of agreement of the points with a ratio, and the half widths of the 95% confidence intervals of the bias, t SD / sqrt(n), and of a limit, t sqrt(3 SD^2 / n) (Bland and Altman 1999), with t the 97.5% quantile of the t distribution of n - 1 degrees of freedom: with few points a limit is uncertain, and the interval says how much.

Bland JM, Altman DG. Measuring agreement in method comparison studies. Stat Methods Med Res. 1999;8(2):135-160.

Parameters:

Name Type Description Default
points DataFrame

the data points, see the module.

 required

Returns:

Type Description
tuple[float, float, float, float] | None

The bias, the half width of the limits and the half widths of the
tuple[float, float, float, float] | None

intervals of the bias and of a limit, in decades; None for fewer
tuple[float, float, float, float] | None

than MIN_POINTS_SUBSET_AGREEMENT points with a ratio.