fit.metrics¶

Metrics of a fit.

The metrics are calculated for a parameter set on an initialized OptimizationProblem, i.e., from the data of the fit mappings and the predictions of the model for the parameters, see FitMetrics. The functions which do the arithmetic work on plain arrays and are used on their own as well.

The names of the columns follow the convention of population pharmacokinetics:

DV      the measured value (dependent variable)
PRED    prediction of the population parameters, i.e., the parameter set
        which is shared by all fit mappings
IPRED   prediction of the individual parameters, i.e., the parameter set of
        the fit mapping. Without individual parameters IPRED is PRED
RES     DV - PRED
IRES    DV - IPRED
NRES    IRES / mean(DV) of the fit mapping, the normalized residual, so the
        residuals of curves of different magnitude are comparable
IWRES   the residual of the cost of the optimization problem, i.e., IRES
        as the `ResidualType` of the settings defines it, weighted and
        with the loss function applied. The cost of the training data is
        `0.5 * sum(IWRES**2)`

The table of the data points has, after these, the columns a subset of the data is selected by: observable, the observable of the mapping, observation, the ObservationKind of the mapping (a timecourse, a value per simulation or values over a dimension), DV_SD, the standard deviation of the measured value (every error of a problem is one, NaN without one, also where the weights fill a missing error, see OptimizationProblem.y_errors_measured), N, the count of the data point, the number of subjects (NaN without one), x_unit, the unit of x (empty for a value per simulation, which has no x), unit, the unit of DV, the predictions, the residuals RES and IRES and DV_SD, both in the units of the model which a fit compares in and which pint reads, and one column per field of the metadata of the mappings, MappingMetaData.to_dict, empty for a mapping which does not have the field.

The data of a fit spans orders of magnitude, so the absolute metrics MSE, RMSE and R2 are dominated by the curves with the largest values and a curve of small values is missed by a factor of 20 at an absolute error which is still tiny. NRMSE, the root mean square of NRES, is the scale free metric which reads the same for every curve, and RMSE_w is the root mean square of IWRES, i.e., what the fit minimizes. within_2fold is the percentage of the data points whose prediction lies within twofold of the data, the criterion of pharmacokinetic predictions, which reads the same for every curve as well; it counts only the data points with a positive value and prediction, the others have no ratio.

Note the sign: the residuals of OptimizationProblem.residuals are prediction - data, the residuals here are data - prediction.

FitMetrics `dataclass` ¶

FitMetrics(
    problem,
    parameter_set,
    population_parameter_set=None,
    *,
    evaluation=None,
)

Metrics of a parameter set on an optimization problem.

The metrics are calculated from the data of the fit mappings and the predictions of the model, see the module docstring for the names.

Attributes:

Name	Type	Description
`problem`	`OptimizationProblem`	initialized optimization problem, it provides the data.
`parameter_set`	`ParameterSet`	parameters the predictions are calculated for, they give IPRED.
`population_parameter_set`	`ParameterSet \| None`	parameters shared by all fit mappings, they give PRED. Without them PRED is IPRED, which is the case of a deterministic fit of a single parameter set.
`evaluation`	`ParameterSet \| None`	a keyword only argument and no field, the complete residual data of the problem at the parameter set, i.e. `OptimizationProblem.residuals` with `complete_data=True`, which the metrics use instead of simulating the set again, e.g. the evaluation a report keeps of its sets.

n_parameters `property` ¶

n_parameters

Number of fitted parameters, the k of the AIC.

datapoints_df ¶

datapoints_df()

Get the table of the data points with their predictions.

Returns:

Type	Description
`DataFrame`	DataFrame with one row per data point and the columns `experiment`,
`DataFrame`	`mapping`, `kind`, `x`, `DV`, `PRED`, `IPRED`, `RES`, `IRES`,
`DataFrame`	`NRES`, `IWRES`, `observable`, `observation`, `DV_SD`, `N`,
`DataFrame`	`x_unit`, `unit` and the fields of the metadata, see the module.

mappings_df ¶

mappings_df()

Get the metrics of every fit mapping.

Returns:

Type	Description
`DataFrame`	DataFrame with one row per fit mapping and the columns `experiment`,
`DataFrame`	`mapping`, `kind`, `n`, `MSE`, `RMSE`, `NRMSE`, `RMSE_w`, `R2`,
`DataFrame`	`within_2fold`, the percentage of the points within twofold, and
`DataFrame`	the columns of `CRITERIA_COLUMNS`, see `criteria_metrics`.

observables_df ¶

observables_df()

Get the metrics of every observable and kind of fit mapping.

The data points of the mappings of one observable, e.g. the plasma concentration or the amount in the urine, of one kind together, so the RMSE of an observable is read next to the median standard deviation of its data, i.e. whether the residuals are within the uncertainty of the data (#325).

Returns:

Type	Description
`DataFrame`	DataFrame with one row per kind and observable, the kinds the
`DataFrame`	report evaluates, and the columns `kind`, `observable`, `n`,
`DataFrame`	`RMSE`, `NRMSE`, `median_sd` (the median of `DV_SD`, `nan` without
`DataFrame`	one), `GMFE`, `AFE` and the criteria, see `criteria_metrics`.

summary ¶

summary(kind=None)

Get the metrics over the data points of the problem.

MSE, RMSE, R2, AIC and BIC are the unweighted metrics of the data and the predictions, i.e., they are dominated by the fit mappings with the largest values. The two information criteria differ in how they penalize a parameter, 2 against ln(n), so the BIC prefers the smaller model of two which describe the data equally well. NRMSE is the root mean square of the residuals normalized by the mean of their curve, so every curve counts the same whatever its magnitude, and RMSE_w is the root mean square of the residuals of the cost, so a parameter set can have a larger RMSE and smaller weighted residuals than another one, which is what the weighting is for. within_2fold is the percentage of the data points with a positive value and prediction whose prediction is within twofold of the data, see within_fold, nan if no point has a ratio.

cost is the objective the optimization minimizes. It is defined on the training data alone, so it is only reported for the training data and is nan for the other kinds.

Parameters:

Name	Type	Description	Default
`kind`	`MappingKind \| None`	only the data points of the fit mappings of this kind, all data points if `None`.	`None`

Returns:

Type	Description
`dict[str, Any]`	Dictionary with the id of the parameter set, the `kind`, the number
`dict[str, Any]`	of data points `n`, the number of parameters `k`, the `cost`,
`dict[str, Any]`	`MSE`, `RMSE`, `NRMSE`, `RMSE_w`, `R2`, `within_2fold`, `AIC`,
`dict[str, Any]`	`BIC` and the columns of `CRITERIA_COLUMNS`, the fold errors
`dict[str, Any]`	`GMFE` and `AFE` and the percentages within 1.25 and 1.5 fold and
`dict[str, Any]`	within one and two standard deviations, see `criteria_metrics`.

Raises:

Type	Description
`ValueError`	if the problem has no data points of the kind.

summary_df ¶

summary_df()

Get the metrics per kind of fit mapping.

A fit is evaluated on its training data, on its validation data and on the outliers it dropped, so there is a row for every kind the problem has. There is no row over all data points: it pools the data a fit was fitted on with the data it dropped, which is not a number to read. summary() gives the metrics over all data points where they are wanted.

Returns:

Type	Description
`DataFrame`	DataFrame with one row per kind, see `summary`.

cost ¶

cost()

Get the cost of the parameter set, i.e., the objective of the fit.

report ¶

report()

Get the metrics as text, per kind of fit mapping and per mapping.

metadata_dict ¶

metadata_dict(metadata, mapping)

Get the metadata of a fit mapping as text by field.

The values of MappingMetaData.to_dict as text: None is empty, a member of an enum its name, anything else str of it. The fields are columns of the data points, see FitMetrics.datapoints_df, so a field must not be a column of them.

Parameters:

Name	Type	Description	Default
`metadata`	`MappingMetaData \| None`	the metadata of the mapping, `None` without.	required
`mapping`	`str`	the experiment and the key of the mapping, for the message.	required

Returns:

Type	Description
`dict[str, str]`	The values by field, empty without metadata.

Raises:

Type	Description
`ValueError`	if a field is named like a column of the data points.

sse ¶

sse(residuals)

Sum of Squared Errors (SSE) of the residuals.

Parameters:

Name	Type	Description	Default
`residuals`	`ArrayLike`	residuals of the fit.	required

Returns:

Type	Description
`float`	Sum of the squared residuals.

mse ¶

mse(residuals)

Mean Squared Error (MSE) of the residuals.

Parameters:

Name	Type	Description	Default
`residuals`	`ArrayLike`	residuals of the fit.	required

Returns:

Type	Description
`float`	Mean of the squared residuals.

Raises:

Type	Description
`ValueError`	if no residuals are given.

rmse ¶

rmse(residuals)

Root Mean Squared Error (RMSE) of the residuals.

Parameters:

Name	Type	Description	Default
`residuals`	`ArrayLike`	residuals of the fit.	required

Returns:

Type	Description
`float`	Square root of the mean squared error.

rmse_from_mse ¶

rmse_from_mse(mse)

Root Mean Squared Error (RMSE) from the mean squared error.

Parameters:

Name	Type	Description	Default
`mse`	`float`	mean squared error of the fit.	required

Returns:

Type	Description
`float`	Square root of the mean squared error.

Raises:

Type	Description
`ValueError`	if the mean squared error is negative.

aic ¶

aic(residuals, k)

Akaike Information Criterion (AIC) of the residuals.

Parameters:

Name	Type	Description	Default
`residuals`	`ArrayLike`	residuals of the fit.	required
`k`	`int`	number of fitted parameters.	required

Returns:

Type	Description
`float`	Akaike information criterion.

aic_from_mse ¶

aic_from_mse(mse, n, k)

Akaike Information Criterion (AIC) from the mean squared error.

The AIC is calculated for a least squares fit with normally distributed residuals, i.e., AIC = n * ln(MSE) + 2 * k up to an additive constant. Only differences of the AIC between models fitted on the same data are meaningful. A perfect fit, an MSE of 0, has the AIC -inf, the best possible value.

Parameters:

Name	Type	Description	Default
`mse`	`float`	mean squared error of the fit.	required
`n`	`int`	number of data points.	required
`k`	`int`	number of fitted parameters.	required

Returns:

Type	Description
`float`	Akaike information criterion, `-inf` for an MSE of 0.

Raises:

Type	Description
`ValueError`	if the mean squared error is negative or the number of data points is not positive.

bic ¶

bic(residuals, k)

Bayesian Information Criterion (BIC) of the residuals.

Parameters:

Name	Type	Description	Default
`residuals`	`ArrayLike`	residuals of the fit.	required
`k`	`int`	number of fitted parameters.	required

Returns:

Type	Description
`float`	Bayesian information criterion.

bic_from_mse ¶

bic_from_mse(mse, n, k)

Bayesian Information Criterion (BIC) from the mean squared error.

The BIC is calculated for a least squares fit with normally distributed residuals, i.e., BIC = n * ln(MSE) + k * ln(n) up to the same additive constant as the AIC, so only differences between models fitted on the same data are meaningful.

The BIC penalizes a parameter with ln(n) where the AIC penalizes it with 2, i.e. it prefers the smaller model as soon as there are more than seven data points and increasingly so with more of them.

Parameters:

Name	Type	Description	Default
`mse`	`float`	mean squared error of the fit.	required
`n`	`int`	number of data points.	required
`k`	`int`	number of fitted parameters.	required

Returns:

Type	Description
`float`	Bayesian information criterion, `-inf` for an MSE of 0, a perfect fit.

Raises:

Type	Description
`ValueError`	if the mean squared error is negative or the number of data points is not positive.

r_squared ¶

r_squared(y_observed, y_predicted)

Coefficient of determination (R²) of a prediction.

R² = 1 - SSE / SST with SST the total sum of squares of the data. The predictions of a non-linear model are not a linear regression of the data, so R² is not the square of a correlation and can be negative: a negative R² means the prediction is worse than the mean of the data.

R² of fewer than two data points is nan without a message, e.g. a value per simulation against the one row of a study group; a warning is logged for several data points without variance.

Parameters:

Name	Type	Description	Default
`y_observed`	`ArrayLike`	measured values.	required
`y_predicted`	`ArrayLike`	predicted values.	required

Returns:

Type	Description
`float`	Coefficient of determination, `nan` for fewer than two data points or
`float`	data without variance.

Raises:

Type	Description
`ValueError`	if the data and the prediction have different lengths.

fold_counts ¶

fold_counts(y_observed, y_predicted, fold=2.0)

Count the data points whose prediction is within a fold factor.

A data point is within the fold factor if 1/fold <= f / y <= fold for its value y and its prediction f, the bounds included. Only the data points with a positive and finite value and prediction have a ratio, the others are left out, i.e. the count is of the n_+ points with a ratio.

Parameters:

Name	Type	Description	Default
`y_observed`	`ArrayLike`	measured values.	required
`y_predicted`	`ArrayLike`	predicted values.	required
`fold`	`float`	the fold factor, larger than 1.	`2.0`

Returns:

Type	Description
`int`	The number of points within the fold factor and the number `n_+` of
`int`	the points with a ratio, the denominator of `within_fold`.

Raises:

Type	Description
`ValueError`	if the data and the prediction have different lengths or the fold factor is not larger than 1.

within_fold ¶

within_fold(y_observed, y_predicted, fold=2.0)

Percentage of the data points whose prediction is within a fold factor.

The percentage of the points of fold_counts, i.e. of the n_+ points with a positive and finite value and prediction, whose ratio f / y is within 1/fold and fold, the bounds included. The criterion of the predictions of pharmacokinetics is twofold, the default.

Parameters:

Name	Type	Description	Default
`y_observed`	`ArrayLike`	measured values.	required
`y_predicted`	`ArrayLike`	predicted values.	required
`fold`	`float`	the fold factor, larger than 1.	`2.0`

Returns:

Type	Description
`float`	Percentage (0 to 100) of the points with a ratio within the fold
`float`	factor, `nan` if no point has a ratio.

Raises:

Type	Description
`ValueError`	if the data and the prediction have different lengths or the fold factor is not larger than 1.

fit.metrics¶

FitMetrics dataclass ¶

n_parameters property ¶

datapoints_df ¶

mappings_df ¶

observables_df ¶

summary ¶

summary_df ¶

cost ¶

report ¶

metadata_dict ¶

sse ¶

mse ¶

rmse ¶

rmse_from_mse ¶

aic ¶

aic_from_mse ¶

bic ¶

bic_from_mse ¶

r_squared ¶

fold_counts ¶

within_fold ¶

FitMetrics `dataclass` ¶

n_parameters `property` ¶