data¶

The study data of a model, collected from the study folders of PK-DB.

The studies of PK-DB are curated in a checkout of pkdb_data: the folder studies holds a folder per substance and in it a folder per study. A model keeps the subset of the studies it uses in its own data/ folder, so its simulations run without the checkout; collect_tsv_files writes them there from the folder studies_dir() finds.

The experiments of the models read the tables of study format 1, one hidden .tsv file per source of a study (.Boulton1996a_Fig1.tsv, see pkpdsim.experiment.PKPDSimulationExperiment.load_dataframe). pkdb_data is curated in study format 2 ("format": 2 of the study.json, the tables subjects.tsv, interventions.tsv, timecourses_<source>.tsv, outputs_<source>.tsv, ...). A study of format 2 is read with the library of PK-DB (pkdb.prepare, the same reading and validation as pkdb check and the upload to the database), and its data are written as the tables of format 1 (study_tables), so that the experiments read them unchanged: a geometric mean in gmean (calculation geometric mean), a value below the limit of quantification as a row without a central value, an output with a label generated from its columns which is unique across the sources of the study, the interventions in the natural order of format 2 (GEM,SIM), the counts as integers, the time and the interventions of both axes of a scatter. What the curation should look at is a CollectionWarning (a label which is no SId, the same output in two sources). Beside the tables the collection writes the record of the study (StudyRecord, .<study>_study.json), the provenance of the data the simulations run on: the day the model first collected the study (which the temporal validation of pkpdsim.credibility.temporal reads), the day of the latest collection, the release of the study in PK-DB, its source digest, the commit of the checkout of pkdb_data and the sha256 of every table, which the credibility assessment compares with the tables the model holds. Format 2 has no collection intervals (no tstart and tend of format 1) and no columns of its own of a table, so neither have the tables. A study of format 1 is copied as it is. database_differences compares the studies with the ones uploaded to a PK-DB server; mean_of_individuals and sampling_times prepare the tables for an experiment.

PKDB_DATA `module-attribute` ¶

PKDB_DATA = 'PKDB_DATA'

The environment variable with the path of a checkout of pkdb_data.

StudyRecord `dataclass` ¶

StudyRecord(
    *,
    study,
    collected,
    dating=None,
    dating_commit=None,
    updated=None,
    released=None,
    pkdb_id=None,
    source_digest=None,
    pkdb_data_commit=None,
    pkdb_data_dirty=None,
    tables=dict(),
    versions=dict(),
)

Which data of a study a model holds, and since when.

The record lies beside the tables of the study in the data folder of the model (study_record_path), so it travels with the data into a release repository; write_study_tables and collect_tsv_files write it. It is the provenance of the data the simulations of the model run on: the version of the study in pkdb_data the tables were written from and the digest of every table, so a later change of a table by hand is found (StudyRecord.changed_tables).

Attributes:

Name	Type	Description
`study`	`str`	the name of the study.
`collected`	`date \| None`	the day the model first collected the study, kept by every later collection; `None` for a study collected before the record existed, whose day is not known.
`dating`	`str \| None`	how `collected` is known: `"collection"` for the day the collection wrote the first record, `"git"` for the day of the commit which first added the tables to the repository of the model (`scripts/backfill_study_records.py`), `None` without a day.
`dating_commit`	`str \| None`	the commit of the dating `"git"`.
`updated`	`date \| None`	the day of the latest collection, `None` if not known.
`released`	`date \| None`	the day PK-DB released the study (`release.date` of its `study.json`), `None` for a study which is not released.
`pkdb_id`	`str \| None`	the identifier of the release in PK-DB, `None` for a study which is not released.
`source_digest`	`str \| None`	the digest of the source files of the study of format 2 as PK-DB computes it (`source_digest` of `pkdb.prepare`), the version in the database; `None` for format 1 or if not known.
`pkdb_data_commit`	`str \| None`	the commit of the checkout of pkdb_data of the latest collection, `None` outside a git repository.
`pkdb_data_dirty`	`bool \| None`	whether the study folder had changes not committed in that checkout, `None` if not known.
`tables`	`Mapping[str, str]`	the sha256 of every table of the latest collection, by its file name; empty if not known.
`versions`	`Mapping[str, str]`	the versions of pkpdsim and pkdb which wrote the tables.

to_dict ¶

to_dict()

Get the record as plain data, the days in ISO format.

from_dict `classmethod` ¶

from_dict(data)

Get the record from plain data.

A record without a dating which has a day was written by the collection.

Raises:

Type	Description
`ValueError`	if a day is not in ISO format or the dating is invalid.

changed_tables ¶

changed_tables(folder)

Get the tables of the study which differ from the record.

Parameters:

Name	Type	Description	Default
`folder`	`Path`	the folder of the study in the data folder of the model.	required

Returns:

Type	Description
`list[str] \| None`	The file names of the tables whose sha256 differs, which the
`list[str] \| None`	record has and the folder not (`-<name>`) or the folder has and the
`list[str] \| None`	record not (`+<name>`), sorted; `None` for a record without the
`list[str] \| None`	digests of its tables.

CollectionWarning ¶

Bases: UserWarning

A finding of the collection of a study which the curation should look at.

The collection goes on: a curated label which is no SId, the same output in two sources of a study, a curator 0 in a curve of geometric means which is left out.

DatabaseDifference `dataclass` ¶

DatabaseDifference(study, sid, reason)

A study whose version in the database differs from its study folder.

Attributes:

Name	Type	Description
`study`	`Path`	the folder of the study.
`sid`	`str`	the study id of PK-DB, `<substance>/<name>`.
`reason`	`str`	why they differ, e.g. that the study is not in the database or that the digests of the source files differ.

table_digests ¶

table_digests(folder)

Get the sha256 of every table of a study in the data of a model.

Parameters:

Name	Type	Description	Default
`folder`	`Path`	the folder of the study in the data folder of the model.	required

Returns:

Type	Description
`dict[str, str]`	The sha256 of every file `.<study>_*.tsv`, by its file name.

study_record_path ¶

study_record_path(folder)

Get the path of the record of a study in the data of a model.

Parameters:

Name	Type	Description	Default
`folder`	`Path`	the folder of the study in the data folder of the model.	required

Returns:

Type	Description
`Path`	`folder/.<study>_study.json`.

read_study_record ¶

read_study_record(folder)

Read the record of a study in the data of a model.

Parameters:

Name	Type	Description	Default
`folder`	`Path`	the folder of the study in the data folder of the model.	required

Returns:

Type	Description
`StudyRecord \| None`	The record, `None` for a folder without one.

Raises:

Type	Description
`ValueError`	if the record is not readable.

studies_dir ¶

studies_dir()

Get the folder of the PK-DB studies, studies of a checkout of pkdb_data.

The checkout is the path of the environment variable PKDB_DATA; without the variable it is ../pkdb_data relative to the working directory, i.e. a checkout next to the repository a script runs in.

Returns:

Type	Description
`Path`	The absolute path of the folder `studies` of the checkout.

Raises:

Type	Description
`FileNotFoundError`	if the folder does not exist; the message names the environment variable and the default location.

study_format ¶

study_format(study_folder)

Get the study format of a study folder of pkdb_data.

Parameters:

Name	Type	Description	Default
`study_folder`	`Path`	the folder of a study.	required

Returns:

Type	Description
`int`	The `format` of its `study.json`; 1 for a study without one, or
`int`	without the field, which is study format 1.

Raises:

Type	Description
`ValueError`	if the `study.json` is no JSON object or its format is no integer.

prepare_study ¶

prepare_study(study_folder)

Read a study folder of format 2 with the library of PK-DB.

The folder is read and validated as pkdb check and the upload to the database do (pkdb.prepare), with the vocabulary of the checkout (vocabulary.lock.json of pkdb_data), else the one bundled with pkdb.

Parameters:

Name	Type	Description	Default
`study_folder`	`Path`	the folder of a study of format 2.	required

Returns:

Type	Description
`CanonicalStudy`	The study as PK-DB reads it.

study_tables ¶

study_tables(study_folder)

Get the data of a study of format 2 as the tables of format 1.

The tables are keyed by the source of format 1, the part of the file name after the study (Fig1 of .Boulton1996a_Fig1.tsv): a table per source of the measurements (the reported outputs and timecourses of outputs_<source>.tsv and timecourses_<source>.tsv, MEASUREMENT_COLUMNS), a table per source of the scatters of scatters_<source>.tsv (SCATTER_COLUMNS; <source>Scatter when the source also has measurements), and TabGroups (GROUP_COLUMNS), TabIndividuals (INDIVIDUAL_COLUMNS) and TabInterventions (INTERVENTION_COLUMNS) with the subjects, their characteristics and the interventions. The columns are those of format 1, so the experiments select their rows as before.

Parameters:

Name	Type	Description	Default
`study_folder`	`Path`	the folder of a study of format 2.	required

Returns:

Type	Description
`dict[str, DataFrame]`	The tables by source.

write_study_tables ¶

write_study_tables(
    study_folder, target_dir, *, keep_files=False
)

Write the data of a study of format 2 into the data folder of a model.

Every table of study_tables becomes target_dir/<study>/.<study>_<source>.tsv, the file PKPDSimulationExperiment.load_dataframe reads; the files of an earlier collection of the study are replaced, so a source which the study no longer has leaves no file behind. Other files of the folder stay.

The record of the study (StudyRecord, study_record_path) is written last: the day of the first collection (today for a new folder, the day of an earlier record, and None for a folder with tables of the study and no record, which was collected on an unknown day), today as the day of the latest collection, the release of study.json, the source digest of PK-DB, the commit of the checkout of pkdb_data and the sha256 of the tables.

Parameters:

Name	Type	Description	Default
`study_folder`	`Path`	the folder of a study of format 2.	required
`target_dir`	`Path`	the data folder of the model.	required
`keep_files`	`bool`	keep the tables `.<study>_*.tsv` of the folder which the collection does not write, e.g. tables a model keeps beside the ones of PK-DB; they are removed by default.	`False`

Returns:

Type	Description
`list[Path]`	The written files, the record last.

collect_tsv_files ¶

collect_tsv_files(
    source_dir,
    target_dir,
    filter_study=None,
    *,
    studies=(),
    keep_files=False,
)

Collect the data of the studies of a substance into the data folder of a model.

Every study folder of source_dir which passes filter_study, and every folder of studies, becomes a folder of the same name in target_dir with the data of the study as the tables of study format 1, and every written file is reported on the console. A study of format 2 is read with the library of PK-DB and written with write_study_tables; the hidden .tsv files of a study of format 1 are copied. The figures, the tables and the publication are left behind.

Parameters:

Name	Type	Description	Default
`source_dir`	`Path`	the folder of the studies of a substance, e.g. `studies_dir() / "albuterol"`.	required
`target_dir`	`Path`	the data folder of the model, created if it does not exist (its parent has to).	required
`filter_study`	`Callable[[str], bool] \| None`	decides by the name of a study folder of `source_dir` whether the study is collected; all studies by default.	`None`
`studies`	`Iterable[Path]`	further study folders, e.g. of the studies of another substance with a group of the model (`studies_dir() / "midazolam" / "Chung2006"`); collected without `filter_study`.	`()`
`keep_files`	`bool`	keep the tables of a study of format 2 in `target_dir` which the collection does not write (see `write_study_tables`).	`False`

Raises:

Type	Description
`ValueError`	if a study has a format which is neither 1 nor 2, see `study_format`, or two studies have the same name; nothing is written then.

mean_of_individuals ¶

mean_of_individuals(df, by=None)

Get the mean, SD and number of the individuals of a table at every time.

A curve of the means of a group which the publication does not report (the curation has the individuals only) is calculated from the rows of the individuals, the rows with a value, by by (the study, group, interventions, substance, tissue, measurement type and time by default). A missing value of a key is a value of its own.

Parameters:

Name	Type	Description	Default
`df`	`DataFrame`	the data of a table with the individuals.	required
`by`	`Sequence[str] \| None`	the columns which make a point of the curves.	`None`

Returns:

Type	Description
`DataFrame`	A row per point with the columns of `by`, `time_unit` and `unit` (of
`DataFrame`	the first individual), `mean`, `sd` (with one individual `NA`) and
`DataFrame`	`count`, sorted by substance and time.

sampling_times ¶

sampling_times(df, times, *, start=None, by='label')

Move the digitized times of the data to the sampling times of a protocol.

PK-DB has the times of a figure as digitized, some minutes off the samples, a marker drawn on the y axis at 0 or before. Within a curve (the rows with the same by) the points keep their order and go to distinct sampling times, the assignment with the least total shift.

Parameters:

Name	Type	Description	Default
`df`	`DataFrame`	the data of the curves, with the columns `time` and `by`.	required
`times`	`Sequence[float]`	the sampling times of the protocol in the time unit of the data.	required
`start`	`float \| None`	the points before it keep their digitized times (dense samples which the figure does not resolve); all points are moved by default.	`None`
`by`	`str`	the column of the curves.	`'label'`

Returns:

Type	Description
`DataFrame`	A copy of the data with the sampling times, sorted by time within a
`DataFrame`	curve.

Raises:

Type	Description
`ValueError`	if a curve has more points than sampling times.

database_differences ¶

database_differences(study_folders, client=None)

Compare studies of format 2 with their versions in a PK-DB database.

A study is read with prepare_study and fetched from the server (pkdb.Client().studies.get); the two agree when the digests of their source files (source_digest) are equal, i.e. the study folder is the one that was uploaded. The server and the credentials are those of the environment of pkdb.Client (PKDB_ENDPOINT, PKDB_API_KEY) unless a client is given.

Parameters:

Name	Type	Description	Default
`study_folders`	`Iterable[Path]`	the folders of the studies.	required
`client`	`Client \| None`	the client of the server, `pkdb.Client()` by default.	`None`

Returns:

Type	Description
`list[DatabaseDifference]`	The studies which differ, in the order of `study_folders`.

Raises:

Type	Description
`ClientError`	if the server cannot be asked, e.g. without the credentials of a private study; a study the server does not have (404) is a difference.

data¶

PKDB_DATA module-attribute ¶

StudyRecord dataclass ¶

to_dict ¶

from_dict classmethod ¶

changed_tables ¶

CollectionWarning ¶

DatabaseDifference dataclass ¶

table_digests ¶

study_record_path ¶

read_study_record ¶

studies_dir ¶

study_format ¶

prepare_study ¶

study_tables ¶

write_study_tables ¶

collect_tsv_files ¶

mean_of_individuals ¶

sampling_times ¶

database_differences ¶

PKDB_DATA `module-attribute` ¶

StudyRecord `dataclass` ¶

from_dict `classmethod` ¶

DatabaseDifference `dataclass` ¶