Skip to content

fit.metrics

Metrics of a fit.

The metrics are calculated for a parameter set on an initialized OptimizationProblem, i.e., from the data of the fit mappings and the predictions of the model for the parameters, see FitMetrics. The functions which do the arithmetic work on plain arrays and are used on their own as well.

The names of the columns follow the convention of population pharmacokinetics:

DV      the measured value (dependent variable)
PRED    prediction of the population parameters, i.e., the parameter set
        which is shared by all fit mappings
IPRED   prediction of the individual parameters, i.e., the parameter set of
        the fit mapping. Without individual parameters IPRED is PRED
RES     DV - PRED
IRES    DV - IPRED
IWRES   IRES weighted with the weights of the optimization problem

Note the sign: the residuals of OptimizationProblem.residuals are prediction - data, the residuals here are data - prediction.

FitMetrics dataclass 

FitMetrics(
    problem, parameter_set, population_parameter_set=None
)

Metrics of a parameter set on an optimization problem.

The metrics are calculated from the data of the fit mappings and the predictions of the model, see the module docstring for the names.

Attributes:

Name Type Description
problem OptimizationProblem

initialized optimization problem, it provides the data.
parameter_set ParameterSet

parameters the predictions are calculated for, they give IPRED.
population_parameter_set ParameterSet | None

parameters shared by all fit mappings, they give PRED. Without them PRED is IPRED, which is the case of a deterministic fit of a single parameter set.

n_parameters property 

n_parameters

Number of fitted parameters, the k of the AIC.

datapoints_df 

datapoints_df()

Get the table of the data points with their predictions.

Returns:

Type Description
DataFrame

DataFrame with one row per data point and the columns experiment,
DataFrame

mapping, kind, x, DV, PRED, IPRED, RES, IRES and
DataFrame

IWRES.

mappings_df 

mappings_df()

Get the metrics of every fit mapping.

Returns:

Type Description
DataFrame

DataFrame with one row per fit mapping and the columns experiment,
DataFrame

mapping, kind, n, MSE, RMSE and R2.

summary 

summary(kind=None)

Get the metrics over the data points of the problem.

MSE, RMSE, R2 and AIC are the unweighted metrics of the data and the predictions, i.e., they are dominated by the fit mappings with the largest values. RMSE_w is the root mean square of the weighted residuals, so a parameter set can have a larger RMSE and smaller weighted residuals than another one, which is what the weighting is for.

cost is the objective the optimization minimizes. It is defined on the training data alone, so it is only reported for the training data and is nan for the other kinds.

Parameters:

Name Type Description Default
kind MappingKind | None

only the data points of the fit mappings of this kind, all data points if None.

 None

Returns:

Type Description
dict[str, Any]

Dictionary with the id of the parameter set, the kind, the number
dict[str, Any]

of data points n, the number of parameters k, the cost,
dict[str, Any]

MSE, RMSE, RMSE_w, R2 and AIC.

Raises:

Type Description
ValueError

if the problem has no data points of the kind.

summary_df 

summary_df()

Get the metrics per kind of fit mapping and over all data points.

A fit is evaluated on its training and on its validation data, so there is a row for every kind the problem has and, when it has more than one, a row all over all data points.

Returns:

Type Description
DataFrame

DataFrame with one row per kind, see summary.

cost 

cost()

Get the cost of the parameter set, i.e., the objective of the fit.

report 

report()

Get the metrics as text, per kind of fit mapping and per mapping.

sse 

sse(residuals)

Sum of Squared Errors (SSE) of the residuals.

Parameters:

Name Type Description Default
residuals ArrayLike

residuals of the fit.

 required

Returns:

Type Description
float

Sum of the squared residuals.

mse 

mse(residuals)

Mean Squared Error (MSE) of the residuals.

Parameters:

Name Type Description Default
residuals ArrayLike

residuals of the fit.

 required

Returns:

Type Description
float

Mean of the squared residuals.

Raises:

Type Description
ValueError

if no residuals are given.

rmse 

rmse(residuals)

Root Mean Squared Error (RMSE) of the residuals.

Parameters:

Name Type Description Default
residuals ArrayLike

residuals of the fit.

 required

Returns:

Type Description
float

Square root of the mean squared error.

rmse_from_mse 

rmse_from_mse(mse)

Root Mean Squared Error (RMSE) from the mean squared error.

Parameters:

Name Type Description Default
mse float

mean squared error of the fit.

 required

Returns:

Type Description
float

Square root of the mean squared error.

Raises:

Type Description
ValueError

if the mean squared error is negative.

aic 

aic(residuals, k)

Akaike Information Criterion (AIC) of the residuals.

Parameters:

Name Type Description Default
residuals ArrayLike

residuals of the fit.

 required
k int

number of fitted parameters.

 required

Returns:

Type Description
float

Akaike information criterion.

aic_from_mse 

aic_from_mse(mse, n, k)

Akaike Information Criterion (AIC) from the mean squared error.

The AIC is calculated for a least squares fit with normally distributed residuals, i.e., AIC = n * ln(MSE) + 2 * k up to an additive constant. Only differences of the AIC between models fitted on the same data are meaningful.

Parameters:

Name Type Description Default
mse float

mean squared error of the fit.

 required
n int

number of data points.

 required
k int

number of fitted parameters.

 required

Returns:

Type Description
float

Akaike information criterion.

Raises:

Type Description
ValueError

if the mean squared error or the number of data points is not positive.

r_squared 

r_squared(y_observed, y_predicted)

Coefficient of determination (R²) of a prediction.

R² = 1 - SSE / SST with SST the total sum of squares of the data. The predictions of a non-linear model are not a linear regression of the data, so R² is not the square of a correlation and can be negative: a negative R² means the prediction is worse than the mean of the data.

Parameters:

Name Type Description Default
y_observed ArrayLike

measured values.

 required
y_predicted ArrayLike

predicted values.

 required

Returns:

Type Description
float

Coefficient of determination, nan if the data has no variance.

Raises:

Type Description
ValueError

if the data and the prediction have different lengths.