fit.report¶

Report of a parameter fit.

Reporting is separate from optimizing: a FitReport is created from the definition of an OptimizationProblem, the FitSettings of the fit and one or more ParameterSets. It does not need an optimization to have been run in the same session, and several parameter sets can be compared in a single report, e.g., the fitted parameters against the initial values of the model or the results of two fits against each other.

The plots which describe an optimization run rather than a parameter set, i.e., the traces of the optimizers and the waterfall plot, are only created when the OptimizationResult of the run is passed as well.

FitReport ¶

FitReport(
    problem,
    settings,
    parameter_sets,
    opt_result=None,
    identifiability=None,
    fisher=None,
    show_titles=True,
    image_format="svg",
    mapping_figures=True,
)

Report of a fit for one or more parameter sets.

Creates the figures, the text report and the HTML report of a fit.

Construct the report.

The problem is initialized with the settings, which resolves the data of the fit mappings. A problem which is already initialized with the same settings is left alone.

Parameters:

Name	Type	Description	Default
`problem`	`OptimizationProblem`	definition of the optimization problem.	required
`settings`	`FitSettings`	settings the parameter sets were fitted with.	required
`parameter_sets`	`ParameterSets \| list[ParameterSet] \| ParameterSet`	one or more sets of parameters to report. The first set is the reference the others are compared against.	required
`opt_result`	`OptimizationResult \| None`	result of an optimization, adds the traces, the waterfall plot and the table of the runs.	`None`
`identifiability`	`IdentifiabilityResult \| None`	result of a profile likelihood analysis, adds the identifiability section with the profiles.	`None`
`fisher`	`FisherInformation \| None`	Fisher information of the parameters, adds its table of errors and intervals and the correlation of the parameters to the identifiability section.	`None`
`show_titles`	`bool`	add titles to the panels.	`True`
`image_format`	`str`	format of the figures.	`'svg'`
`mapping_figures`	`bool`	draw the two figures of every fit mapping, i.e. the data with the simulation and the residuals. They are one figure per mapping and per kind of panel and are almost the whole cost of a report, e.g. 70 of the 76 figures and 88% of the time for a problem with 35 mappings. A report which is only read for its tables and its overview figures is created without them, and its mapping cards carry their metrics alone.	`True`

reference_set `property` ¶

reference_set

First parameter set, the reference the others are compared against.

opt_result_required `property` ¶

opt_result_required

Result of the optimization, required for the plots of the runs.

Raises:

Type	Description
`ValueError`	if the report was created without a result.

from_optimization_result `staticmethod` ¶

from_optimization_result(
    problem, opt_result, size=1, with_model=False, **kwargs
)

Create the report of an optimization.

Parameters:

Name	Type	Description	Default
`problem`	`OptimizationProblem`	definition of the optimization problem.	required
`opt_result`	`OptimizationResult`	result of the optimization, it carries the settings.	required
`size`	`int`	number of fitted parameter sets to report, the best first.	`1`
`with_model`	`bool`	report the initial values of the model as the reference set as well, so that the figures and the tables compare the fit against the model it started from. The report shows the fitted parameters alone by default.	`False`
`kwargs`	`Any`	additional arguments of `FitReport`.	`{}`

Returns:

Type	Description
`FitReport`	The report of the fit.

mapping_title ¶

mapping_title(k)

Get the title of the plots of a fit mapping.

Data which is not fitted is marked, so that it is visible in the figures which curves the parameters were fitted on.

color ¶

color(pset)

Get the color of a parameter set.

studies ¶

studies()

Get the studies of the problem, i.e. its simulation experiments.

In the order the fit mappings of the problem name them, so the color of a study does not depend on which mappings a figure shows.

study_color ¶

study_color(study)

Get the color of a study, the same in every figure of the report.

set_marker ¶

set_marker(pset)

Get the marker of a parameter set in the figures colored by study.

x ¶

x(pset)

Get the values of a set in the parameter order of the problem.

metrics ¶

metrics(pset)

Get the metrics of a parameter set on the problem.

Parameters:

Name	Type	Description	Default
`pset`	`ParameterSet`	parameter set of the report.	required

Returns:

Type	Description
`FitMetrics`	The metrics of the set, see `sbmlsim.fit.metrics`.

metrics_df ¶

metrics_df()

Get the metrics of every parameter set, per kind of fit mapping.

A fit is evaluated on its training and on its validation data, so every parameter set has a row per kind.

metrics_mappings_df ¶

metrics_mappings_df()

Get the metrics of every fit mapping and parameter set.

datapoints_df ¶

datapoints_df()

Get the data points with their predictions for every parameter set.

residual_data ¶

residual_data(pset)

Get the complete residual data of the mappings for a parameter set.

Every evaluation simulates all fit mappings, so the results are cached for the plots and tables which use the same set.

points ¶

points(pset)

Get the data points of a parameter set with their kind.

The table of FitMetrics.datapoints_df, i.e. one row per data point with the measurement DV, the prediction IPRED and the kind of the fit mapping it belongs to. It is the source of the goodness of fit and the Bland-Altman plots and is cached, a data point costs a simulation.

point_kinds ¶

point_kinds()

Get the kinds of fit mapping the data points are shown in.

The kinds the problem has, in the order of EVALUATED_KINDS, i.e. the training data, the validation data and the outliers. There is no panel over all data points: it pools data a fit was fitted on with data it dropped, which is not a number to read.

create ¶

create(
    output_dir,
    name=None,
    show_report=False,
    mpl_parameters=None,
)

Create the complete report.

Writes the figures, the text report, the parameter sets and the HTML report into output_dir / name.

Parameters:

Name	Type	Description	Default
`output_dir`	`Path`	base directory of the reports.	required
`name`	`str \| None`	name of this report, the id of the optimization by default.	`None`
`show_report`	`bool`	open the HTML report in a web browser.	`False`
`mpl_parameters`	`dict[str, Any] \| None`	additional matplotlib rc parameters of the figures.	`None`

Returns:

Type	Description
`Path`	Path of the directory the report was written to.

parameters_report ¶

parameters_report()

Get the report of the parameter sets.

Reports the values of every set and the parameters which ended up close to one of their bounds.

html_context ¶

html_context(results_dir, name)

Collect everything the HTML report shows.

Parameters:

Name	Type	Description	Default
`results_dir`	`Path`	directory of the report, the files are relative to it.	required
`name`	`str`	name of the report.	required

Returns:

Type	Description
`dict[str, Any]`	The context of the `fit_report.html` template.

fit_info ¶

fit_info()

Get the key facts of the fit, the same the console reports.

html_report ¶

html_report(path, name=None)

Create the interactive HTML report of the fit.

The report is a single page with three sections: the overview of the fit, its results and the single fit mappings. It is rendered from the fit_report.html template and needs no network access.

Parameters:

Name	Type	Description	Default
`path`	`Path`	file to write.	required
`name`	`str \| None`	name of the report, the directory of `path` by default.	`None`

plot_fit ¶

plot_fit(output_dir)

Plot the data and the simulation of every parameter set per mapping.

plot_fit_residual ¶

plot_fit_residual(output_dir)

Plot data, prediction and residuals of every mapping.

The upper panels show the data, the interpolated prediction and the residuals, the lower panels the squared weighted residuals; the right panels are logarithmic.

panel_metrics ¶

panel_metrics(kind, plot)

Get the key metrics of a panel, one line per parameter set.

The goodness of fit shows how well the predictions describe the data of the kind: R², the scale free NRMSE and the RMSE_w of the cost of FitMetrics.summary; the absolute RMSE is not shown, the data spans orders of magnitude and it only reads the largest curves. The Bland-Altman plot shows the agreement of the points of the panel itself: the bias and the SD of log10(f(x)/y) as fold factors and the share of the points inside the limits of agreement of the training data, i.e. inside the band of agreement.

Parameters:

Name	Type	Description	Default
`kind`	`str`	kind of fit mapping of the panel.	required
`plot`	`str`	`goodness_of_fit` or `bland_altman`.	required

Returns:

Type	Description
`str`	The text of the box, the lines are prefixed with the id of the set
`str`	when several sets are compared.

plot_goodness_of_fit ¶

plot_goodness_of_fit(path)

Plot the predicted against the measured data points, per kind.

One panel per kind of fit mapping, i.e. the training data, the validation data and the outliers separately. The points scatter around the identity line when the model describes the data.

The band is the agreement of agreement, i.e. the bias and the limits bias ± 1.96 SD of the training data, which on logarithmic axes are lines parallel to the identity: a point inside the band is a prediction the fit agrees to. It is the same band the Bland-Altman plot draws, in the same styles, so the two figures are read the same way.

agreement ¶

agreement(pset)

Get the bias and the half width of the limits of agreement.

They are calculated on the training data alone, i.e. on the data the parameters were fitted on, and the Bland-Altman plot draws them in every panel: the limits are what the fit agrees to, and the validation data and the outliers are read against them. Calculating them per panel would give every subset its own reference and the panels could not be compared; pooling all data points would let the outliers, which are dropped exactly because they are far away, widen the limits.

Parameters:

Name	Type	Description	Default
`pset`	`ParameterSet`	parameter set of the report.	required

Returns:

Type	Description
`float`	The bias `mean(log10(f(x)/y))` and `1.96 * SD` of it, both in
`float`	decades. `10bias` and `10(bias ± half)` are the fold factors.

plot_bland_altman ¶

plot_bland_altman(path)

Plot the agreement of prediction and measurement, per kind.

A Bland-Altman plot of the ratio: the difference of the logarithms, log10(f(x)/y), over the geometric mean of the two. The data of a fit spans orders of magnitude, so the agreement is multiplicative and the limits are read as fold factors.

The bias and the limits of agreement bias ± 1.96 SD are those of agreement, i.e. of the training data, and they are the same in every panel, so the validation data and the outliers are read against what the fit agrees to. Every panel shows the limits even when its points are further out. It is the same band the goodness of fit draws, in the same styles: the identity there is no difference here.

Data points which are zero or negative have no logarithm and are left out, i.e. the plot shows the points a ratio is defined for.

plot_cost_bar ¶

plot_cost_bar(path)

Plot the cost and the weight of every curve, per set.

plot_residual_boxplot ¶

plot_residual_boxplot(path)

Plot the distribution of the squared weighted residuals per curve.

plot_cost_scatter ¶

plot_cost_scatter(path)

Plot the cost of every curve against the cost of the reference set.

plot_waterfall ¶

plot_waterfall(path)

Create waterfall plot for the fit results.

Plots the optimization runs sorted by cost.

plot_traces ¶

plot_traces(path)

Plot optimization traces.

Optimization time course of costs.

fit.report¶

FitReport ¶

reference_set property ¶

opt_result_required property ¶

from_optimization_result staticmethod ¶

mapping_title ¶

color ¶

studies ¶

study_color ¶

set_marker ¶

x ¶

metrics ¶

metrics_df ¶

metrics_mappings_df ¶

datapoints_df ¶

residual_data ¶

points ¶

point_kinds ¶

create ¶

parameters_report ¶

html_context ¶

fit_info ¶

html_report ¶

plot_fit ¶

plot_fit_residual ¶

panel_metrics ¶

plot_goodness_of_fit ¶

agreement ¶

plot_bland_altman ¶

plot_cost_bar ¶

plot_residual_boxplot ¶

plot_cost_scatter ¶

plot_waterfall ¶

plot_traces ¶

reference_set `property` ¶

opt_result_required `property` ¶

from_optimization_result `staticmethod` ¶