Skip to content

petab_v2.benchmark

The problems of the PEtab benchmark collection.

The benchmark collection is a set of published parameter estimation problems, each in problems/<name>/v1 in PEtab v1 with the simulations of its measurements at the nominal parameters, simulations.tsv, and some also in problems/<name>/v2 in PEtab v2. sbmlsim reads PEtab v2 only: a problem is its encoding in PEtab v2 where the collection has one, otherwise it is converted from PEtab v1 with petab.v2.petab1to2; its simulations are converted with it, see sbmlsim.fit.petab_v2.conversion. The problem is read with PetabReader, simulated at the nominal values of its parameter table and compared with the simulations of the collection; where AMICI states the log-likelihood of the problem (tests/benchmark_models/benchmark_models.yaml of AMICI), it is compared too. The problems of EXCLUDED_PROBLEMS take too long and are left out.

status meaning
pass the simulations and the log-likelihood agree
tolerance a simulation or the log-likelihood is outside of its tolerance
conversion petab1to2 cannot convert the problem
error the problem cannot be read or simulated

The collection has no releases, so it is pinned by a commit, the references of AMICI by another. BenchmarkCollection is a commit of the collection with the download, the conversion and the cache.

BenchmarkStatus

Bases: StrEnum

The outcome of a problem.

BenchmarkResult dataclass 

BenchmarkResult(
    name,
    status,
    message="",
    max_difference=None,
    n_simulations=0,
    llh=None,
    llh_reference=None,
    timings=dict(),
)

The outcome of a problem.

Attributes:

Name Type Description
name str

the name of the problem.
status BenchmarkStatus

the outcome.
message str

what failed, or a note on what was not compared.
max_difference float | None

the largest absolute difference of a simulation to the collection, None when nothing was compared.
n_simulations int

the number of simulations which were compared.
llh float | None

the log-likelihood at the nominal values, None when it was not calculated.
llh_reference float | None

the log-likelihood of AMICI it was compared with.
timings dict[str, float]

the seconds it took to read, initialize, simulate and calculate the llh.

passed property 

passed

Check whether the problem passes.

key property 

key

Get the key of the problem in the baseline, its name.

to_dict 

to_dict()

Convert to a dictionary for serialization.

ProblemTimeout

Bases: Exception

A problem took longer than the time it was given.

BenchmarkProblem dataclass 

BenchmarkProblem(name, source_dir, v2_dir)

A problem of the benchmark collection.

Attributes:

Name Type Description
name str

the name of the problem.
source_dir Path

the directory of the problem in the collection, with its encodings v1 and, for some problems, v2.
v2_dir Path

the directory of the problem in PEtab v2 which is run: the encoding of the collection in PEtab v2 or the one converted from PEtab v1, with the simulations of the collection converted to PEtab v2, SIMULATIONS_FILE, see BenchmarkCollection.convert.

native property 

native

Check whether the collection has the problem in PEtab v2.

conversion_error property 

conversion_error

Get the file which holds why the conversion failed.

run 

run(llh_reference=None, settings=None, timeout=None)

Read, simulate and compare the problem.

Parameters:

Name Type Description Default
llh_reference float | None

the log-likelihood of AMICI, which is compared.

 None
settings FitSettings | None

the settings the problem is simulated with, by default the ones of PROBLEM_SETTINGS or BENCHMARK_SETTINGS.

 None
timeout float | None

the seconds the problem may take, a problem which takes longer is an error. The time is measured with the timer ITIMER_REAL of the process, so a time limit needs the main thread of a POSIX system, on Windows the problem runs without one and a warning says so; a call into roadrunner finishes before the problem stops.

 None

Returns:

Type Description
BenchmarkResult

The outcome.

BenchmarkCollection dataclass 

BenchmarkCollection(path, commit)

A commit of the benchmark collection, converted to PEtab v2.

Attributes:

Name Type Description
path Path

the directory with the problems of the collection in problems, the problems in PEtab v2 which are run in v2 and the references of AMICI.
commit str

the commit of the collection.

cache_path staticmethod 

cache_path(commit=BENCHMARK_COMMIT)

Get the directory a commit of the collection is unpacked into.

SBMLSIM_BENCHMARK_PATH overrides it. Otherwise it is sbmlsim/petab-benchmark/<commit> in the user cache.

Parameters:

Name Type Description Default
commit str

the commit of the collection.

 BENCHMARK_COMMIT

Returns:

Type Description
Path

The directory of the collection.

cached classmethod 

cached(commit=BENCHMARK_COMMIT)

Get a commit of the collection if it is already on this machine.

Parameters:

Name Type Description Default
commit str

the commit of the collection.

 BENCHMARK_COMMIT

Returns:

Type Description
BenchmarkCollection | None

The collection, or None if it was not downloaded yet.

load classmethod 

load(commit=BENCHMARK_COMMIT)

Get a commit of the collection, downloading and converting it.

The problems are downloaded once, converted once and the references of AMICI are fetched once.

Parameters:

Name Type Description Default
commit str

the commit of the collection.

 BENCHMARK_COMMIT

Returns:

Type Description
BenchmarkCollection

The collection in the cache.

Raises:

Type Description
OSError

if the commit or the references cannot be downloaded.

problem_names 

problem_names(excluded=False)

Get the names of the problems, sorted.

Parameters:

Name Type Description Default
excluded bool

whether the problems of EXCLUDED_PROBLEMS are listed, which are neither converted nor run by default.

 False

problem 

problem(name)

Get a problem of the collection by its name.

problems 

problems()

Iterate the problems of the collection.

references 

references()

Get the log-likelihoods of AMICI by the name of the problem.

convert 

convert(names=None)

Write the problems in PEtab v2 which are not written yet.

A problem is its encoding in PEtab v2 of the collection or, without one, its encoding in PEtab v1 converted with petab1to2; the simulations of the collection are converted from PEtab v1 in both cases, see sbmlsim.fit.petab_v2.conversion. A problem which cannot be converted gets the file v2/<name>.error with the reason, see BenchmarkProblem.conversion_error.

Parameters:

Name Type Description Default
names Sequence[str] | None

the problems, all of them by default.

 None

run 

run(names=None, progress=None)

Run the problems of the collection.

Parameters:

Name Type Description Default
names Sequence[str] | None

the problems, all of them by default.

 None
progress Callable[[BenchmarkResult], None] | None

called with the result of every problem.

 None

Returns:

Type Description
list[BenchmarkResult]

The results in the order of the names.