Skip to content

petab_v2.benchmark

The problems of the PEtab benchmark collection.

The benchmark collection is a set of published parameter estimation problems in PEtab v1, each in problems/<name>/v1 with the simulations of its measurements at the nominal parameters, simulations.tsv. A problem is converted to PEtab v2 with petab.v2.petab1to2 and read with PetabReader, simulated at the nominal values of its parameter table and compared with the simulations of the collection; where AMICI states the log-likelihood of the problem (tests/benchmark_models/benchmark_models.yaml of AMICI), it is compared too, unless an observable of the problem is on the scale log10, whose density the conversion changes.

status meaning
pass the simulations and the log-likelihood agree
tolerance a simulation or the log-likelihood is outside of its tolerance
conversion petab1to2 cannot convert the problem
error the problem cannot be read or simulated

The collection has no releases, so it is pinned by a commit, the references of AMICI by another. BenchmarkCollection is a commit of the collection with the download, the conversion and the cache.

BenchmarkStatus

Bases: StrEnum

The outcome of a problem.

BenchmarkResult dataclass 

BenchmarkResult(
    name,
    status,
    message="",
    max_difference=None,
    n_simulations=0,
    llh=None,
    llh_reference=None,
    timings=dict(),
)

The outcome of a problem.

Attributes:

Name Type Description
name str

the name of the problem.
status BenchmarkStatus

the outcome.
message str

what failed, or a note on what was not compared.
max_difference float | None

the largest absolute difference of a simulation to the collection, None when nothing was compared.
n_simulations int

the number of simulations which were compared.
llh float | None

the log-likelihood at the nominal values, None when it was not calculated.
llh_reference float | None

the log-likelihood of AMICI it was compared with.
timings dict[str, float]

the seconds it took to read, initialize, simulate and calculate the llh.

passed property 

passed

Check whether the problem passes.

key property 

key

Get the key of the problem in the baseline, its name.

to_dict 

to_dict()

Convert to a dictionary for serialization.

ProblemTimeout

Bases: Exception

A problem took longer than the time it was given.

BenchmarkProblem dataclass 

BenchmarkProblem(name, v1_dir, v2_dir)

A problem of the benchmark collection.

Attributes:

Name Type Description
name str

the name of the problem.
v1_dir Path

the directory of the problem in PEtab v1, with the measurements and the simulations of the collection.
v2_dir Path

the directory of the problem converted to PEtab v2.

conversion_error property 

conversion_error

Get the file which holds why the conversion failed.

run 

run(
    llh_reference=None,
    settings=BENCHMARK_SETTINGS,
    timeout=None,
)

Read, simulate and compare the problem.

Parameters:

Name Type Description Default
llh_reference float | None

the log-likelihood of AMICI, which is compared if it survives the conversion, see llh_is_comparable.

 None
settings FitSettings

the settings the problem is simulated with.

 BENCHMARK_SETTINGS
timeout float | None

the seconds the problem may take, a problem which takes longer is an error. The time is measured with the timer ITIMER_REAL of the process, so a time limit needs the main thread of a POSIX system, on Windows the problem runs without one and a warning says so; a call into roadrunner finishes before the problem stops.

 None

Returns:

Type Description
BenchmarkResult

The outcome.

BenchmarkCollection dataclass 

BenchmarkCollection(path, commit)

A commit of the benchmark collection, converted to PEtab v2.

Attributes:

Name Type Description
path Path

the directory with the problems of PEtab v1 in problems, the converted problems in v2 and the references of AMICI.
commit str

the commit of the collection.

cache_path staticmethod 

cache_path(commit=BENCHMARK_COMMIT)

Get the directory a commit of the collection is unpacked into.

SBMLSIM_BENCHMARK_PATH overrides it. Otherwise it is sbmlsim/petab-benchmark/<commit> in the user cache.

Parameters:

Name Type Description Default
commit str

the commit of the collection.

 BENCHMARK_COMMIT

Returns:

Type Description
Path

The directory of the collection.

cached classmethod 

cached(commit=BENCHMARK_COMMIT)

Get a commit of the collection if it is already on this machine.

Parameters:

Name Type Description Default
commit str

the commit of the collection.

 BENCHMARK_COMMIT

Returns:

Type Description
BenchmarkCollection | None

The collection, or None if it was not downloaded yet.

load classmethod 

load(commit=BENCHMARK_COMMIT)

Get a commit of the collection, downloading and converting it.

The problems are downloaded once, converted once and the references of AMICI are fetched once.

Parameters:

Name Type Description Default
commit str

the commit of the collection.

 BENCHMARK_COMMIT

Returns:

Type Description
BenchmarkCollection

The collection in the cache.

Raises:

Type Description
OSError

if the commit or the references cannot be downloaded.

problem_names 

problem_names()

Get the names of the problems, sorted.

problem 

problem(name)

Get a problem of the collection by its name.

problems 

problems()

Iterate the problems of the collection.

references 

references()

Get the log-likelihoods of AMICI by the name of the problem.

convert 

convert(names=None)

Convert the problems to PEtab v2 which are not converted yet.

A problem which cannot be converted gets the file v2/<name>.error with the reason, see BenchmarkProblem.conversion_error.

Parameters:

Name Type Description Default
names Sequence[str] | None

the problems, all of them by default.

 None

run 

run(names=None, progress=None)

Run the problems of the collection.

Parameters:

Name Type Description Default
names Sequence[str] | None

the problems, all of them by default.

 None
progress Callable[[BenchmarkResult], None] | None

called with the result of every problem.

 None

Returns:

Type Description
list[BenchmarkResult]

The results in the order of the names.

llh_is_comparable 

llh_is_comparable(v1_dir)

Check whether the log-likelihood survives the conversion to PEtab v2.

PEtab v2 has no scale log10 of an observable, petab1to2 writes such an observable as its logarithm, whose density differs.

Parameters:

Name Type Description Default
v1_dir Path

the directory of the problem in PEtab v1.

 required

Returns:

Type Description
bool

Whether no observable of the problem is on the scale log10.