fit.report¶

Report of a parameter fit.

Reporting is separate from optimizing: a FitReport is created from the definition of an OptimizationProblem, the FitSettings of the fit and one or more ParameterSets. It does not need an optimization to have been run in the same session, and several parameter sets can be compared in a single report, e.g., the fitted parameters against the initial values of the model or the results of two fits against each other.

The plots which describe an optimization run rather than a parameter set, i.e., the traces of the optimizers and the waterfall plot, are only created when the OptimizationResult of the run is passed as well.

FitReport.create writes the report into a folder: the tables, the figures, the text report and index.html, a page of two tabs. The tab "Report" is the report document of the fit (sbmlsim.fit.report_document), i.e. its methods with their formulas, its results and its references, followed by the overview, the results and the identifiability of the fit; the tab "Fit mappings" is the figures of the cost and weight and of the residuals of every fit mapping, the table of the metrics of every fit mapping and a card per fit mapping. The same document is written as Typst (report.typ), compiled to report.pdf, and as LaTeX (report.tex, a fragment, and report_standalone.tex) with its references (references.bib, and references.csl, the style of the Typst bibliography). A PDF which cannot be compiled is stated in the HTML, the other outputs are written.

FitReport ¶

FitReport(
    problem,
    settings,
    parameter_sets,
    opt_result=None,
    identifiability=None,
    fisher=None,
    show_titles=True,
    image_format="svg",
    mapping_figures=True,
    context=None,
)

Report of a fit for one or more parameter sets.

Creates the figures, the text report and the HTML report of a fit.

Construct the report.

A copy of the problem definition is initialized with the settings. Its runtime and parameter sets belong to the report, so later fits or edits to the original inputs cannot change the report.

Parameters:

Name	Type	Description	Default
`problem`	`OptimizationProblem`	definition of the optimization problem.	required
`settings`	`FitSettings`	settings the parameter sets were fitted with.	required
`parameter_sets`	`ParameterSets \| list[ParameterSet] \| ParameterSet`	one or more sets of parameters to report. The first set is the reference the others are compared against.	required
`opt_result`	`OptimizationResult \| None`	result of an optimization, adds the traces, the waterfall plot and the table of the runs.	`None`
`identifiability`	`IdentifiabilityResult \| None`	result of a profile likelihood analysis, adds the identifiability section with the profiles.	`None`
`fisher`	`FisherInformation \| None`	Fisher information of the parameters, adds its table of errors and intervals and the correlation of the parameters to the identifiability section.	`None`
`show_titles`	`bool`	add titles to the figures: the title of a fit mapping above the panels of its figures, the title of a figure or of its panel for the other figures.	`True`
`image_format`	`str`	format of the figures.	`'svg'`
`mapping_figures`	`bool`	draw the two figures of every fit mapping, i.e. the data with the simulation and the residuals. They are one figure per mapping and per kind of panel and are almost the whole cost of a report, e.g. 70 of the 76 figures and 88% of the time for a problem with 35 mappings. A report which is only read for its tables and its overview figures is created without them, and its mapping cards carry their metrics alone.	`True`
`context`	`ReportContext \| None`	what the report document says about the fit beyond the fit, e.g. its title and its description; sbmlsim reports a fit with an empty context.	`None`

reference_set `property` ¶

reference_set

First parameter set, the reference the others are compared against.

display_units `property` ¶

display_units

Get the unit every table of the report shows for a parameter, by id.

It is the unit as the user defined it (FitParameter.unit), e.g. the unit id per_min of the model, and model for the unit of the model. The values carry the units of pint, e.g. 1/min, see OptimizationProblem.parameter_units, and so does the code Q(value, unit) of the optimal parameters of the text report, which runs with sbmlsim.Q.

opt_result_required `property` ¶

opt_result_required

Result of the optimization, required for the plots of the runs.

Raises:

Type	Description
`ValueError`	if the report was created without a result.

from_optimization_result `staticmethod` ¶

from_optimization_result(
    problem, opt_result, size=1, with_model=False, **kwargs
)

Create the report of an optimization.

Parameters:

Name	Type	Description	Default
`problem`	`OptimizationProblem`	definition of the optimization problem.	required
`opt_result`	`OptimizationResult`	result of the optimization, it carries the settings.	required
`size`	`int`	number of fitted parameter sets to report, the best first.	`1`
`with_model`	`bool`	report the initial values of the model as the reference set as well, so that the figures and the tables compare the fit against the model it started from. The report shows the fitted parameters alone by default.	`False`
`kwargs`	`Any`	additional arguments of `FitReport`.	`{}`

Returns:

Type	Description
`FitReport`	The report of the fit.

mapping_title ¶

mapping_title(k)

Get the title of the plots of a fit mapping.

Data which is not fitted is marked, so that it is visible in the figures which curves the parameters were fitted on.

color ¶

color(pset)

Get the color of a parameter set.

studies ¶

studies()

Get the studies of the problem, i.e. its simulation experiments.

In the order the fit mappings of the problem name them, so the color of a study does not depend on which mappings a figure shows.

study_color ¶

study_color(study)

Get the color of a study, the same in every figure of the report.

set_marker ¶

set_marker(pset)

Get the marker of a parameter set in the figures colored by study.

x ¶

x(pset)

Get the values of a set in the parameter order of the problem.

metrics ¶

metrics(pset)

Get the metrics of a parameter set on the problem.

Parameters:

Name	Type	Description	Default
`pset`	`ParameterSet`	parameter set, a set of the report is evaluated once, see `residual_data`.	required

Returns:

Type	Description
`FitMetrics`	The metrics of the set, see `sbmlsim.fit.metrics`.

metrics_df ¶

metrics_df()

Get the metrics of every parameter set, per kind of fit mapping.

A fit is evaluated on its training and on its validation data, so every parameter set has a row per kind.

metrics_mappings_df ¶

metrics_mappings_df()

Get the metrics of every fit mapping and parameter set.

datapoints_df ¶

datapoints_df()

Get the data points with their predictions for every parameter set.

residual_data ¶

residual_data(pset)

Get the complete residual data of the mappings for a parameter set.

Every evaluation simulates all fit mappings, so the results of a set of the report are cached for the plots and tables which use it. Any other set is evaluated as it is, see _stored.

points ¶

points(pset)

Get the data points of a parameter set with their kind.

The table of FitMetrics.datapoints_df, i.e. one row per data point with the measurement DV, the prediction IPRED and the kind of the fit mapping it belongs to. It is the source of the goodness of fit and the Bland-Altman plots and is cached for a set of the report, a data point costs a simulation; any other set is evaluated as it is, see _stored.

point_kinds ¶

point_kinds()

Get the kinds of fit mapping the data points are shown in.

The kinds the problem has, in the order of EVALUATED_KINDS, i.e. the training data, the validation data and the outliers. There is no panel over all data points: it pools data a fit was fitted on with data it dropped, which is not a number to read.

create ¶

create(
    output_dir,
    name=None,
    show_report=False,
    mpl_parameters=None,
)

Create the complete report.

Writes the figures, the text report, the parameter sets, the report document as Typst, PDF and LaTeX with its references and the HTML report into output_dir / name. A PDF which Typst cannot compile is logged and stated in the HTML report; report.typ is kept and the other outputs are written.

Parameters:

Name	Type	Description	Default
`output_dir`	`Path`	base directory of the reports.	required
`name`	`str \| None`	name of this report, the id of the optimization by default.	`None`
`show_report`	`bool`	open the HTML report in a web browser.	`False`
`mpl_parameters`	`dict[str, Any] \| None`	additional matplotlib rc parameters of the figures.	`None`

Returns:

Type	Description
`Path`	Path of the directory the report was written to.

document ¶

document(plots_dir)

Get the report document of the fit, its methods and its results.

Parameters:

Name	Type	Description	Default
`plots_dir`	`Path`	the folder of the plots of the report; a figure of the document is shown if its file is in the folder.	required

Returns:

Type	Description
`Document`	The document of `fit_document` with the context of the report.

parameters_report ¶

parameters_report()

Get the report of the parameter sets.

Reports the values of every set and the parameters which ended up close to one of their bounds.

twofold_statement ¶

twofold_statement()

Get the data points within twofold per kind, with their denominator.

The points of every kind the goodness of fit has a panel of, counted as within_2fold counts them, i.e. over the points with a positive value and prediction, which are the points the logarithmic axes draw, e.g. Within twofold of the data are 479 of the 523 training data points (91.6%) and 212 of the 243 validation data points (87.2%); only the data points with a positive value and prediction are counted. A kind without a point with a ratio is left out. With several parameter sets the counts of every set follow for <id>.

Returns:

Type	Description
`str`	The statement without a full stop, empty if no kind has a point
`str`	with a ratio.

plot_caption ¶

plot_caption(name)

Get the caption of a figure of the report.

The caption of PLOT_CAPTIONS; the goodness of fit states the percentage of the data points within twofold of every kind as well, see twofold_statement.

Parameters:

Name	Type	Description	Default
`name`	`str`	name of the figure, e.g. `goodness_of_fit`.	required

Returns:

Type	Description
`str`	The caption, without a full stop.

html_context ¶

html_context(
    results_dir, name, document=None, pdf_error=None
)

Collect everything the HTML report shows.

Parameters:

Name	Type	Description	Default
`results_dir`	`Path`	directory of the report, the files are relative to it.	required
`name`	`str`	name of the report.	required
`document`	`Document \| None`	the report document of the tab "Report", the `document` of the plots of `results_dir` by default.	`None`
`pdf_error`	`str \| None`	the reason the PDF of the document could not be compiled, if it could not.	`None`

Returns:

Type	Description
`dict[str, Any]`	The context of the `fit_report.html` template.

parameter_groups ¶

parameter_groups()

Get the arrays of the networks as groups of elements.

Returns:

Type	Description
`dict[str, tuple[str, ...]]`	label of the array -> the ids of all its elements, for
`dict[str, tuple[str, ...]]`	`bound_warnings` and `OptimizationResult.report`.

fit_info ¶

fit_info()

Get the key facts of the fit, the same the console reports.

html_report ¶

html_report(path, name=None, document=None, pdf_error=None)

Create the interactive HTML report of the fit.

The report is a single page with two tabs. "Report" is the report document with the methods and the results, followed by the overview of the fit, its results and its identifiability; "Fit mappings" is the figures of the cost and weight and of the residuals of every fit mapping, the table of the metrics of every fit mapping and a card per fit mapping. It is rendered from the fit_report.html template and needs no network access.

Parameters:

Name	Type	Description	Default
`path`	`Path`	file to write.	required
`name`	`str \| None`	name of the report, the directory of `path` by default.	`None`
`document`	`Document \| None`	the report document, see `html_context`.	`None`
`pdf_error`	`str \| None`	the reason the PDF could not be compiled, if it could not.	`None`

plot_fit ¶

plot_fit(output_dir)

Plot the data and the simulation of every parameter set per mapping.

plot_fit_residual ¶

plot_fit_residual(output_dir)

Plot data, prediction and residuals of every mapping.

The upper panels show the data, the interpolated prediction and the residuals, the lower panels the squared weighted residuals; the right panels are logarithmic.

panel_metrics ¶

panel_metrics(kind, plot)

Get the key metrics of a panel, one line per parameter set.

The goodness of fit shows how well the predictions describe the data of the kind: R², the scale free NRMSE and the RMSE_w of the cost of FitMetrics.summary and the percentage of the points within twofold, within_2fold; the absolute RMSE is not shown, the data spans orders of magnitude and it only reads the largest curves. The Bland-Altman plot shows the agreement of the points of the panel itself: the bias and the SD of log10(f(x)/y) as fold factors and the share of the points inside the limits of agreement of the training data, i.e. inside the band of agreement.

Parameters:

Name	Type	Description	Default
`kind`	`str`	kind of fit mapping of the panel.	required
`plot`	`str`	`goodness_of_fit` or `bland_altman`.	required

Returns:

Type	Description
`str`	The text of the box, the lines are prefixed with the id of the set
`str`	when several sets are compared.

plot_goodness_of_fit ¶

plot_goodness_of_fit(path)

Plot the predicted against the measured data points, per kind.

One panel per kind of fit mapping, i.e. the training data, the validation data and the outliers separately. The points scatter around the identity line when the model describes the data.

The band is the agreement of agreement, i.e. the bias and the limits bias ± 1.96 SD of the training data, which on logarithmic axes are lines parallel to the identity: a point inside the band is a prediction the fit agrees to. It is the same band the Bland-Altman plot draws, in the same styles, so the two figures are read the same way.

The dark red dotted lines are twofold, f = 2y and f = y/2: the percentage of the points between them is within_2fold of the metrics, which the box of a panel states, and the caption states the points with their denominator, see twofold_statement. The title of a panel states the points drawn, see _panel_title.

agreement ¶

agreement(pset)

Get the bias and the half width of the limits of agreement.

They are calculated on the training data alone, i.e. on the data the parameters were fitted on, and the Bland-Altman plot draws them in every panel: the limits are what the fit agrees to, and the validation data and the outliers are read against them. Calculating them per panel would give every subset its own reference and the panels could not be compared; pooling all data points would let the outliers, which are dropped exactly because they are far away, widen the limits.

Parameters:

Name	Type	Description	Default
`pset`	`ParameterSet`	parameter set of the report.	required

Returns:

Type	Description
`float`	The bias `mean(log10(f(x)/y))` and `1.96 * SD` of it, both in
`float`	decades. `10bias` and `10(bias ± half)` are the fold factors.

plot_bland_altman ¶

plot_bland_altman(path)

Plot the agreement of prediction and measurement, per kind.

A Bland-Altman plot of the ratio: the difference of the logarithms, log10(f(x)/y), over the geometric mean of the two. The data of a fit spans orders of magnitude, so the agreement is multiplicative and the limits are read as fold factors.

The bias and the limits of agreement bias ± 1.96 SD are those of agreement, i.e. of the training data, and they are the same in every panel, so the validation data and the outliers are read against what the fit agrees to. Every panel shows the limits even when its points are further out. It is the same band the goodness of fit draws, in the same styles: the identity there is no difference here.

Data points which are zero or negative have no logarithm and are left out, i.e. the plot shows the points a ratio is defined for.

plot_cost_bar ¶

plot_cost_bar(path)

Plot the cost and the weight of every curve, per set.

A curve is a row, in the order of the fit mappings from the top; the figure grows with the number of curves and the length of their labels, so that every label is readable, e.g. the 164 curves of a large fit.

plot_residual_boxplot ¶

plot_residual_boxplot(path)

Plot the distribution of the squared weighted residuals per curve.

A curve is a row, in the order of the fit mappings from the top.

plot_cost_scatter ¶

plot_cost_scatter(path)

Plot the cost of every curve against the cost of the reference set.

plot_waterfall ¶

plot_waterfall(path)

Create waterfall plot for the fit results.

Plots the optimization runs sorted by cost.

plot_traces ¶

plot_traces(path)

Plot optimization traces.

Optimization time course of costs.

fit.report¶

FitReport ¶

reference_set property ¶

display_units property ¶

opt_result_required property ¶

from_optimization_result staticmethod ¶

mapping_title ¶

color ¶

studies ¶

study_color ¶

set_marker ¶

x ¶

metrics ¶

metrics_df ¶

metrics_mappings_df ¶

datapoints_df ¶

residual_data ¶

points ¶

point_kinds ¶

create ¶

document ¶

parameters_report ¶

twofold_statement ¶

plot_caption ¶

html_context ¶

parameter_groups ¶

fit_info ¶

html_report ¶

plot_fit ¶

plot_fit_residual ¶

panel_metrics ¶

plot_goodness_of_fit ¶

agreement ¶

plot_bland_altman ¶

plot_cost_bar ¶

plot_residual_boxplot ¶

plot_cost_scatter ¶

plot_waterfall ¶

plot_traces ¶

reference_set `property` ¶

display_units `property` ¶

opt_result_required `property` ¶

from_optimization_result `staticmethod` ¶