# pymetadata
> Python utilities for working with metadata and COMBINE archives
The complete documentation from https://matthiaskoenig.github.io/pymetadata, one section per page.
---
# pymetadata: python utilities for metadata and COMBINE archives
[](https://github.com/matthiaskoenig/pymetadata/actions/workflows/ci-cd.yml) [](https://matthiaskoenig.github.io/pymetadata) [](https://pypi.org/project/pymetadata/) [](https://pypi.org/project/pymetadata/) [](https://opensource.org/licenses/MIT) [](https://doi.org/10.5281/zenodo.5308801)
`pymetadata` is a collection of python utilities for working with metadata in the context of [COMBINE](https://co.mbine.org/) standards. The source code is available from [https://github.com/matthiaskoenig/pymetadata](https://github.com/matthiaskoenig/pymetadata).
## Background
Computational models in systems biology are rarely a single file. A study typically consists of one or more models (SBML, CellML), simulation experiments (SED-ML), figures, data files and a description of what the whole thing is about. Two problems follow from this:
**How do you ship such a study as one unit?** The [COMBINE archive](https://combinearchive.org/) (OMEX) answers this. It is a ZIP container with a `manifest.xml` listing every file and, for each file, the format it is in as an identifiers.org URI ([Bergmann et al. 2014](https://doi.org/10.1186/s12859-014-0369-z), [Bergmann et al. 2015](https://doi.org/10.2390/biecoll-jib-2015-261)). `pymetadata` reads, writes and validates these archives, see [COMBINE archives](omex.md).
**How do you say what the parts of a model mean?** A species named `glc` is meaningless to a machine. MIRIAM annotations attach a qualifier (*what is the relation?*) and a resource (*which database entry?*) to a model element, e.g., "this species **is** [CHEBI:17234](https://identifiers.org/CHEBI:17234)". `pymetadata` parses, normalizes and validates these annotations against the [identifiers.org](https://identifiers.org) registry and resolves additional information from the [Ontology Lookup Service](https://www.ebi.ac.uk/ols4), see [Annotations](annotations.md).
## Features
- **[COMBINE archives](omex.md)** — read and write OMEX archives, work with the `manifest.xml`, create archives from directories or single files, and read archives directly from a URL.
- **[Annotations](annotations.md)** — MIRIAM qualifiers (`BQB`, `BQM`), normalization of resources to identifiers.org compact identifiers, validation against the identifiers.org registry, and lookup of labels, descriptions, synonyms and cross references via OLS.
- **[Ontologies](annotations.md#ontology-terms)** — SBO, KISAO and PBPKO are shipped as python classes of terms, so a term is completed by the editor, checked at runtime and carries its label, definition and synonyms.
## Quickstart
Create a COMBINE archive from a model file, then read it back and resolve an entry to a file:
```python
from pathlib import Path
from pymetadata.omex import EntryFormat, ManifestEntry, Omex
# create an archive
omex = Omex()
omex.add_entry(
entry_path=Path("model.xml"),
entry=ManifestEntry(
location="./model.xml", format=EntryFormat.SBML_L3V2, master=True
),
)
omex.to_omex(Path("archive.omex"))
# read an archive; the context manager removes the temporary directory
with Omex.from_omex(Path("archive.omex")) as omex:
print(omex.manifest["./model.xml"].format)
# http://identifiers.org/combine.specifications/sbml.level-3.version-2
for entry in omex.entries_by_format("sbml"):
print(entry.location, omex.get_path(entry.location))
# ./model.xml /tmp/tmpb0m1xyz/model.xml
```
Annotate a model element with a MIRIAM qualifier and use ontology terms instead of strings:
```python
from pymetadata.core.annotation import RDFAnnotation
from pymetadata.core.miriam import BQB
from pymetadata.ontologies import SBO
annotation = RDFAnnotation(qualifier=BQB.IS, resource="CHEBI:17234")
print(annotation.resource_normalized)
# https://identifiers.org/CHEBI:17234
# a term is its identifier and knows what it means
print(SBO.SIMPLE_CHEMICAL, SBO.SIMPLE_CHEMICAL.label)
# SBO_0000247 simple chemical
```
If you have any questions or issues please [open an issue](https://github.com/matthiaskoenig/pymetadata/issues).
# How to cite
[](https://doi.org/10.5281/zenodo.5308801)
If you use `pymetadata` please cite the archived software on [Zenodo](https://doi.org/10.5281/zenodo.5308801):
> König, M. (2026). *pymetadata are python utilities for working with metadata* (Version 0.6.0) [Computer software]. Zenodo. https://doi.org/10.5281/zenodo.22639628
```bibtex
@software{konig_pymetadata,
author = {König, Matthias},
title = {pymetadata are python utilities for working with metadata},
year = {2026},
month = sep,
version = {0.6.0},
publisher = {Zenodo},
doi = {10.5281/zenodo.22639628},
url = {https://doi.org/10.5281/zenodo.22639628},
}
```
# License
- Source Code: [MIT](https://opensource.org/license/MIT)
- Documentation: [CC BY-SA 4.0](http://creativecommons.org/licenses/by-sa/4.0/)
# Funding
Matthias König is supported and by the German Research Foundation (DFG) within the Research Unit Programme FOR 5151 "QuaLiPerF (Quantifying Liver Perfusion-Function Relationship in Complex Resection - A Systems Medicine Approach)" by grant number 436883643 and by grant number 465194077 (Priority Programme SPP 2311, Subproject SimLivA).
Matthias König was supported by the Federal Ministry of Education and Research (BMBF, Germany) within the research network Systems Medicine of the Liver (LiSyM, grant number 031L0054).
© 2021-2026 Matthias König
---
# Installation
`pymetadata` requires python >= 3.11 and is available from [pypi](https://pypi.python.org/pypi/pymetadata). It is pure python without compiled dependencies, so the installation is the same on Linux, macOS and Windows.
## With uv
[uv](https://docs.astral.sh/uv/) is the recommended way to install the package. In a project it is added as a dependency, which resolves and locks it together with the rest of the environment:
```bash
uv add pymetadata
```
Into an existing virtual environment it is installed through the pip interface of uv:
```bash
uv venv
uv pip install pymetadata
```
## With pip
```bash
pip install pymetadata
```
## Development version
The current state of the `develop` branch is installed directly from GitHub:
```bash
uv add "pymetadata @ git+https://github.com/matthiaskoenig/pymetadata.git@develop"
```
or, with pip,
```bash
pip install git+https://github.com/matthiaskoenig/pymetadata.git@develop
```
To work on the repository itself, with the test and documentation tooling, see [Development](development.md).
## Logging
`pymetadata` does not configure logging. It logs to loggers below the `pymetadata` logger and leaves handlers, levels and formatting to the application, so the messages of the package stay under your control:
```python
import logging
logging.basicConfig(level=logging.INFO)
logging.getLogger("pymetadata").setLevel(logging.WARNING)
```
For scripts and interactive work the rich output of the package can be turned on explicitly:
```python
from pymetadata import log
log.enable_rich_logging()
```
## Cache
Some `pymetadata` features query web services: the [identifiers.org](https://identifiers.org) registry when annotations are validated, the [Ontology Lookup Service](https://www.ebi.ac.uk/ols4) when annotation information is resolved, and ChEBI and UniChem for substance cross references. Responses are cached on disk so that repeated lookups of the same term do not hit the network again.
Caching is controlled by two module level settings:
```python
import pymetadata
pymetadata.CACHE_USE # True by default
pymetadata.CACHE_PATH # ~/.cache/pymetadata by default
```
Both are read when a query is made, so they can be changed at any point after importing the package, e.g., to cache elsewhere or to always see the current state of a service:
```python
from pathlib import Path
import pymetadata
pymetadata.CACHE_PATH = Path("/tmp/pymetadata_cache")
pymetadata.CACHE_USE = False # query the services every time
```
### Cache duration
Cached content is refreshed once it is older than the cache duration of its service:
| content | duration | why |
| --- | --- | --- |
| OLS, ChEBI and UniChem responses | 30 days | they describe the terms of an ontology release, which changes with the release |
| identifiers.org registry | 24 hours | namespaces and their patterns are added and corrected continuously |
The durations are `CACHE_DURATION_ONTOLOGY` and `CACHE_DURATION_REGISTRY` in `pymetadata.cache`.
### Outdated content instead of a failure
If content has to be refreshed but the service cannot be reached, because there is no network or the service is down, the outdated content is used and a warning is logged:
```
Using the cache from 1080.0 h ago, it could not be refreshed:
/home/user/.cache/pymetadata/ols/....json (Service is not reachable for ...)
```
A query which was answered before therefore keeps working offline. Only a query which was never cached fails: `ChebiQuery.query` and `OLSQuery.query_ols` report the problem in their result, `UnichemQuery` and `Registry` raise a `WebserviceError`.
### The registry
The identifiers.org registry is cached independently of `CACHE_USE`. It is downloaded to `CACHE_PATH / "identifiers_registry.json"` and refreshed when the local copy is older than the cache duration:
```python
from pymetadata.webservices.registry import Registry
registry = Registry(cache_duration=24, cache=True)
namespace = registry.ns_dict["chebi"]
print(namespace.pattern)
# ^CHEBI:\d+$
```
Deleting `CACHE_PATH` is always safe; everything in it is re-downloaded on demand.
---
# COMBINE archives
A [COMBINE archive](https://combinearchive.org/) (OMEX, *Open Modeling EXchange format*) bundles everything belonging to a modeling study into a single ZIP file: models, simulation experiments, data, figures and documentation. What makes it more than a ZIP file is the `manifest.xml` at its root, which lists every file together with the format it is in:
```xml
```
Formats are identifiers.org URIs rather than file extensions, so a consumer knows that `./model.xml` is SBML without having to guess from the suffix. The `master` attribute marks the entry a tool should open first.
`pymetadata` maps this onto three classes:
| class | purpose |
| --- | --- |
| `Omex` | the archive itself; reading, writing and the files it contains |
| `Manifest` | the list of entries, i.e., the `manifest.xml` |
| `ManifestEntry` | a single file with its `location`, `format` and `master` flag |
## Reading an archive
An archive is read from a path, from a directory or directly from a URL:
```python
from pathlib import Path
from pymetadata.omex import Omex
omex = Omex.from_omex(Path("archive.omex"))
print(omex)
```
```python
omex = Omex.from_url(
"https://github.com/matthiaskoenig/canagliflozin-model/releases/download/0.7.0/canagliflozin_model.omex"
)
```
Reading extracts the archive into a temporary directory, so the content can be inspected without unpacking it by hand. `Omex.get_path(location)` returns the path of a single entry, which is what you pass on to a model reader:
```python
model_path = omex.get_path("./model.xml")
```
Encrypted archives can be opened by passing the password; writing encrypted archives is not supported:
```python
omex = Omex.from_omex(Path("archive.omex"), password=b"secret")
```
To check whether a file is a COMBINE archive at all, use `Omex.is_omex(path)`.
## Working with the manifest
The manifest behaves like a mapping keyed by location:
```python
print(len(omex.manifest)) # number of entries
print("./model.xml" in omex.manifest)
entry = omex.manifest["./model.xml"]
print(entry.format, entry.master)
```
Locations are normalized to relative paths starting with `./`. An entry given as `model/model1.xml` is stored as `./model/model1.xml`, so lookups are predictable.
Entries can be selected by format, which avoids matching format URIs by hand:
```python
for entry in omex.entries_by_format("sbml"):
print(entry.location)
```
`entries_by_format` understands `sbml`, `sedml` and `sbgn` across all their level and version variants; `ManifestEntry.is_sbml()`, `is_sedml()` and `is_sbgn()` answer the same question for a single entry.
## Creating an archive
An archive can be assembled entry by entry. Every file needs a `ManifestEntry` describing where it goes and what it is:
```python
from pathlib import Path
from pymetadata.omex import EntryFormat, ManifestEntry, Omex
omex = Omex()
omex.add_entry(
entry_path=Path("model.xml"),
entry=ManifestEntry(
location="./model.xml", format=EntryFormat.SBML_L3V2, master=True
),
)
omex.add_entry(
entry_path=Path("README.md"),
entry=ManifestEntry(location="./README.md", format=EntryFormat.MARKDOWN),
)
omex.to_omex(Path("archive.omex"))
```
Files are copied when they are added, so later changes to the source file do not affect the archive. Adding a second entry for an existing location replaces the first one and logs a warning.
For a directory that already has the intended layout, `from_directory` creates the archive in one step and guesses the format of every file from its suffix:
```python
omex = Omex.from_directory(Path("./study"))
omex.to_omex(Path("study.omex"))
```
If the directory contains a `manifest.xml`, the entries listed there are reused and only files missing from it are guessed. SED-ML files added this way get `master=True`, since they are the entry point of a simulation study.
## Formats
`EntryFormat` is an enum of the format URIs, covering the COMBINE specifications (SBML down to the level/version, SED-ML, CellML, SBGN, BioPAX, OMEX metadata, FROG results) and a large list of media types for everything else. Two helpers translate between suffixes and URIs:
```python
from pathlib import Path
from pymetadata.omex import Omex
Omex.guess_format(Path("model.xml")) # from the file suffix and content
Omex.lookup_format("sbml") # from a format key
```
`guess_format` looks at the start of `.xml` files to tell SBML, SED-ML, CellML and COPASI apart, so an `.xml` file is not classified as plain XML when it is in fact a model. For every other file the suffix decides.
## Cleaning up
Reading an archive extracts it into a temporary directory. Using the archive as a context manager removes that directory when the block is left, which matters when many archives are processed in one run:
```python
with Omex.from_omex(Path("archive.omex")) as omex:
model_path = omex.get_path("./model.xml")
...
```
## Writing back out
```python
omex.to_omex(Path("archive.omex")) # write a COMBINE archive
omex.to_directory(Path("./unpacked")) # extract, including the manifest.xml
```
`to_directory` writes the `manifest.xml` next to the files, so the result is a valid input for `Omex.from_directory` again.
## Examples
Runnable examples are in [`examples/omex`](https://github.com/matthiaskoenig/pymetadata/tree/develop/examples/omex) of the repository:
```bash
python examples/omex/omex.py # read, extract, create, write
python examples/omex/omex_from_url.py # read an archive from a url
```
The full API, generated from the docstrings, is in the [API reference](api/omex.md).
## References
The COMBINE archive and the OMEX format are described in:
> Bergmann FT, Adams R, Moodie S, Cooper J, Glont M, Golebiewski M, Hucka M, Laibe C, Miller AK, Nickerson DP, Olivier BG, Rodriguez N, Sauro HM, Scharm M, Soiland-Reyes S, Waltemath D, Yvon F, Le Novère N. **COMBINE archive and OMEX format: one file to share all information to reproduce a modeling project.** *BMC Bioinformatics.* 2014 Dec 14;15(1):369. doi: [10.1186/s12859-014-0369-z](https://doi.org/10.1186/s12859-014-0369-z), PMID: [25494900](https://pubmed.ncbi.nlm.nih.gov/25494900/)
> Bergmann FT, Rodriguez N, Le Novère N. **COMBINE Archive Specification Version 1.** *J Integr Bioinform.* 2015 Sep 4;12(2):261. doi: [10.2390/biecoll-jib-2015-261](https://doi.org/10.2390/biecoll-jib-2015-261), PMID: [26528559](https://pubmed.ncbi.nlm.nih.gov/26528559/)
Further resources:
- [COMBINE archive](https://combinearchive.org/) — specifications and tooling
- [COMBINE](https://co.mbine.org/) — the standards this archive format ties together
- [identifiers.org combine.specifications](https://registry.identifiers.org/registry/combine.specifications) — the format URIs used in the manifest
---
# Annotations
A model element on its own carries no meaning a machine can use. A species named `glc` could be glucose, a glucose transporter or a parameter someone abbreviated. [MIRIAM](https://identifiers.org/) annotations solve this by attaching two things to an element:
- a **qualifier** saying *how* the element relates to something else, e.g., "is", "is part of", "is version of"
- a **resource** pointing at a database entry, e.g., [CHEBI:17234](https://identifiers.org/CHEBI:17234)
Together they form a statement: *this species **is** the chemical entity CHEBI:17234*. `pymetadata` provides the qualifiers, parses resources written in any of the common notations, normalizes them, and validates them against the identifiers.org registry.
## Qualifiers
Biological qualifiers (`BQB`) relate an element to a biological entity, model qualifiers (`BQM`) describe the model itself:
```python
from pymetadata.core.miriam import BQB, BQM
BQB.IS # the element is the annotated entity
BQB.IS_VERSION_OF # the element is a version of the entity
BQB.IS_PART_OF # the element is part of the entity
BQB.HAS_TAXON # the entity occurs in the given taxon
BQM.IS_DESCRIBED_BY # the model is described by the resource, e.g., a publication
```
Choosing the right qualifier matters: `BQB.IS` on a species that is only *one form* of a compound is a stronger claim than the model supports, which is what `BQB.IS_VERSION_OF` is for.
## Resources
`RDFAnnotation` accepts the notations found in the wild and reduces them to a `collection` and a `term`:
```python
from pymetadata.core.annotation import RDFAnnotation
from pymetadata.core.miriam import BQB
for resource in [
"CHEBI:33699", # compact identifier
"chebi/CHEBI:33699", # collection and term
"https://identifiers.org/CHEBI:33699", # identifiers.org URL
"http://identifiers.org/chebi/CHEBI:33699", # legacy identifiers.org URL
"urn:miriam:chebi:CHEBI%3A33699", # deprecated MIRIAM URN
]:
annotation = RDFAnnotation(qualifier=BQB.IS, resource=resource)
print(annotation.resource_normalized)
# https://identifiers.org/CHEBI:33699
```
All five spellings normalize to the same compact identifier. Arbitrary URLs are also valid resources; they are kept as they are, because there is no registry entry to normalize them against:
```python
RDFAnnotation(qualifier=BQB.IS, resource="https://en.wikipedia.org/wiki/Cytosol")
```
`resource_normalized` returns `https://identifiers.org/:`. For namespaces which embed their prefix in the identifier (GO, CHEBI, SBO, BTO) the prefix is already part of the term and is not added a second time.
## Validation
Validation answers two questions: is the qualifier a real MIRIAM qualifier, and does the term match the pattern the identifiers.org registry defines for its collection?
```python
annotation = RDFAnnotation(qualifier=BQB.IS, resource="chebi/CHEBI:33699")
annotation.validate() # qualifier and term
annotation.check_miriam_term() # term against the registry pattern only
```
A term such as `chebi/CHEBI:X33699` fails, because the CHEBI pattern is `^CHEBI:\d+$`. The registry is downloaded once and cached, see [Installation](installation.md#cache).
## Resolving additional information
`RDFAnnotationData` takes an annotation and resolves what the identifier actually refers to.
Constructing it resolves the cross references: for every provider the identifiers.org registry lists for the collection, the provider's URL pattern is filled in with the term, giving one `CrossReference` per provider.
```python
from pymetadata.core.annotation import RDFAnnotation, RDFAnnotationData
from pymetadata.core.miriam import BQB
annotation = RDFAnnotation(qualifier=BQB.IS, resource="chebi/CHEBI:33699")
data = RDFAnnotationData(annotation)
print(data.url) # url of the first provider
print(data.xrefs) # one entry per identifiers.org provider
```
`query_ols()` then asks the [Ontology Lookup Service](https://www.ebi.ac.uk/ols4) for the term itself and fills in label, description and synonyms. Note that it also replaces `xrefs` with the cross references reported by OLS, which can be empty for a given term:
```python
data.query_ols()
print(data.label) # messenger RNA
print(data.description) # An RNA molecule that transfers the coding information ...
print(data.synonyms) # mRNA, ...
```
Both steps require network access. Responses are cached on disk by default, so repeated terms are not queried again; see [Cache](installation.md#cache).
## Ontology terms { #ontology-terms }
Ontology terms are usually passed around as strings, which means typos surface at runtime or not at all, and nothing tells you what a term means. `pymetadata` ships three ontologies as python classes generated from the ontology releases themselves:
| enum | ontology | terms |
| --- | --- | --- |
| `SBO` | Systems Biology Ontology | roles of model components |
| `KISAO` | Kinetic Simulation Algorithm Ontology | simulation algorithms and their parameters |
| `PBPKO` | PBPK Ontology | physiologically based pharmacokinetic modeling |
Every term is available under both its identifier and its name:
```python
from pymetadata.ontologies import SBO, PBPKO
SBO.SBO_0000247 # by id
SBO.SIMPLE_CHEMICAL # by name
PBPKO.BODYWEIGHT
SBO.get_name(SBO.SIMPLE_CHEMICAL)
# 'simple chemical'
```
`validate` accepts both the underscore and the colon notation and returns the enum member, which is the convenient way to accept terms from user input or from a file:
```python
SBO.validate("SBO:0000247") # SBO.SBO_0000247
SBO.validate("SBO_0000247") # SBO.SBO_0000247
SBO.validate("SBO_9999999") # raises AttributeError, the term does not exist
```
### What a term knows { #term-information }
A term is the identifier, i.e., a `str`, and carries what the ontology release says about it. This is the information which otherwise has to be looked up in [OLS](https://www.ebi.ac.uk/ols4), and it is available offline. Since every term is a documented attribute of its ontology, an editor completes `SBO.` and shows the definition of the term it suggests:
```python
term = SBO.SIMPLE_CHEMICAL
term.value # 'SBO_0000247', the enum is a str subclass
term.label # 'simple chemical'
term.definition # 'Simple, non-repetitive chemical entity.'
term.synonyms # ()
term.deprecated # False, the term is not obsolete
term.curie # 'SBO:0000247'
term.url # 'https://identifiers.org/SBO:0000247'
repr(term)
#
```
`get_term` resolves a term given as a string, and the ontology behaves like the enum it replaces, i.e., it can be iterated, asked for its length and indexed:
```python
from pymetadata.ontologies import KISAO
KISAO.get_term("KISAO:0000019").synonyms
# ('VODE', 'VODEPK', 'code value ordinary differential equation solver')
len(KISAO) # number of terms
KISAO["KISAO_0000019"] # lookup by identifier
"KISAO:0000019" in KISAO # True
[term for term in KISAO if term.deprecated]
```
A term stays a string, i.e., `SBO.SIMPLE_CHEMICAL == "SBO_0000247"` is true and serializing a term gives the identifier, so terms can be passed wherever the identifier is expected. The name in the ontology is `label`, `name` is the identifier.
The ontologies are generated with the internal `pymetadata.ontologies._ontology_builder`, which downloads the ontology in OWL format and writes a python module from it. See [Development](development.md#regenerating-the-ontologies) for how to update them to a newer ontology release.
---
# API reference
The API reference is generated from the docstrings of the package.
## pymetadata
Top level modules.
| module | description |
| --- | --- |
| [cache](cache.md) | Caching of the web service responses |
| [console](console.md) | Shared rich console |
| [log](log.md) | Logging of the package |
| [omex](omex.md) | COMBINE archive (OMEX) support |
## pymetadata.core
Core data structures: annotations with their qualifiers, creators and cross references.
| module | description |
| --- | --- |
| [core.annotation](core.annotation.md) | MIRIAM annotations |
| [core.creator](core.creator.md) | Creator information for models and archives |
| [core.miriam](core.miriam.md) | MIRIAM qualifiers |
| [core.xref](core.xref.md) | Cross references to database entries |
## pymetadata.webservices
The services queried for the information which does not ship with the package. They share one session and the cache.
| module | description |
| --- | --- |
| [webservices.chebi](webservices.chebi.md) | Substance information from ChEBI |
| [webservices.ols](webservices.ols.md) | Lookup of ontology terms in the Ontology Lookup Service |
| [webservices.registry](webservices.registry.md) | The identifiers.org registry |
| [webservices.unichem](webservices.unichem.md) | Substance cross references from UniChem |
| [webservices.webservice](webservices.webservice.md) | The shared HTTP session |
## pymetadata.ontologies
The ontology terms of SBO, KISAO and PBPKO, see [Annotations](../annotations.md#ontology-terms). The generated modules declare one attribute per ontology term and are not part of the reference, the behaviour of a term is in `term`.
| module | description |
| --- | --- |
| [ontologies.term](ontologies.term.md) | Ontology terms and the base class of the enums |
---
# pymetadata.cache
Caching of web service responses.
Queries to identifiers.org, OLS, ChEBI and UniChem are cached on disk so that
repeated lookups of the same term do not hit the network again. Caching is on
by default and controlled by `pymetadata.CACHE_USE` and `pymetadata.CACHE_PATH`,
which are read at query time.
A cached response is refreshed once it is older than the cache duration of its
service, see `CACHE_DURATION_ONTOLOGY` and `CACHE_DURATION_REGISTRY`. If the
refresh fails, because the service is unreachable or answers with an error, the
outdated content is used instead of failing the query and a warning is logged,
see `read_json_cache_fallback`. Working offline therefore keeps working with
whatever was cached before.
## class `DataclassJSONEncoder(*, skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, sort_keys=False, indent=None, separators=None, default=None)`
JSON encoder which serializes dataclasses via their `__dict__`.
### `DataclassJSONEncoder.default(self, o: Any) -> Any`
Serialize an object which json cannot serialize itself.
## function `cache_age(cache_path: pathlib.Path) -> float | None`
Get the age of a cache file in hours.
Args:
cache_path: path of the cache file
Returns:
The age in hours, or None if the file does not exist.
## function `read_json_cache(cache_path: pathlib.Path, max_age: float | None = None) -> dict`
Read a JSON cache file.
Args:
cache_path: path of the cache file
max_age: maximum age of the content in hours; older content is treated
as if it were not cached. Any age is accepted if None
Returns:
The cached content.
Raises:
IOError: if the cache file does not exist or is older than `max_age`
## function `read_json_cache_fallback(cache_path: pathlib.Path, reason: str) -> dict | None`
Read a cache file of any age, after the query which should refresh it failed.
The library prefers outdated content over no content, so that an
unreachable service does not fail a query which was answered before. The
age of the content is not checked and a warning names the reason, so that
the fallback is visible in the log.
Args:
cache_path: path of the cache file
reason: why the content could not be refreshed, e.g., the error of the
failed query
Returns:
The cached content, or None if nothing is cached.
## function `write_json_cache(data: dict, cache_path: pathlib.Path, json_encoder: type[json.encoder.JSONEncoder] | None = None) -> None`
Write a JSON cache file.
Missing parent directories are created.
Args:
data: data to serialize
cache_path: path of the cache file
json_encoder: encoder for objects json cannot serialize, e.g.
`DataclassJSONEncoder` for dataclasses
---
# pymetadata.console
Shared rich console.
The console is used for the output of scripts and examples; library code logs
instead of printing, see `pymetadata.log`.
```python
from pymetadata.console import console
console.print(omex)
console.rule("Section", style="white")
```
Importing this module has no side effects on the interpreter. To get rich
representations in an interactive session, install them explicitly with
`rich.pretty.install()`.
---
# pymetadata.log
Logging of the package.
`pymetadata` follows the convention for libraries: it only gets loggers and logs
to them, it does not configure logging. Handlers, levels and formatting are left
to the application, which keeps the messages of the package under the control of
whoever uses it.
Modules get their logger from the standard library with
```python
import logging
logger = logging.getLogger(__name__)
```
All loggers are therefore below the `pymetadata` logger, so an application
configures them in one place:
```python
import logging
logging.getLogger("pymetadata").setLevel(logging.WARNING)
```
For scripts and interactive work the rich formatting of the package can be
enabled explicitly, which is what the examples do:
```python
from pymetadata import log
log.enable_rich_logging()
```
## function `enable_rich_logging(level: int = 20, console: rich.console.Console | None = None) -> logging.Logger`
Log the messages of the package on a rich console.
This configures logging and is meant for scripts, examples and interactive
work. Applications should configure logging themselves instead of calling
this. Calling it repeatedly replaces the handler instead of adding a second
one.
Args:
level: level from which messages are logged
console: console to log on, the console of the package by default
Returns:
The `pymetadata` logger.
---
# pymetadata.omex
COMBINE archive (OMEX) support.
A COMBINE archive is a single file which bundles everything belonging to a
modeling project: models (SBML, CellML), simulation experiments (SED-ML), data,
figures and documentation. It is a ZIP container with a `manifest.xml` at its
root listing every file together with its format, given as an identifiers.org
URI rather than guessed from the file suffix.
This module provides three classes:
- `Omex`: the archive; reading, writing and access to its files
- `Manifest`: the entries of the archive, i.e., the `manifest.xml`
- `ManifestEntry`: a single file with `location`, `format` and `master`
Example:
Read an existing archive and list the SBML models it contains:
```python
from pathlib import Path
from pymetadata.omex import Omex
omex = Omex.from_omex(Path("archive.omex"))
for entry in omex.entries_by_format("sbml"):
print(entry.location, omex.get_path(entry.location))
```
Create an archive from single files:
```python
from pymetadata.omex import EntryFormat, ManifestEntry, Omex
omex = Omex()
omex.add_entry(
entry_path=Path("model.xml"),
entry=ManifestEntry(
location="./model.xml", format=EntryFormat.SBML_L3V2, master=True
),
)
omex.to_omex(Path("archive.omex"))
```
Encrypted archives can be read by passing a password; writing encrypted archives
is not supported. Manipulation of OMEX metadata is not supported.
References:
Bergmann FT, Adams R, Moodie S, Cooper J, Glont M, Golebiewski M, Hucka M,
Laibe C, Miller AK, Nickerson DP, Olivier BG, Rodriguez N, Sauro HM,
Scharm M, Soiland-Reyes S, Waltemath D, Yvon F, Le Novere N.
COMBINE archive and OMEX format: one file to share all information to
reproduce a modeling project. BMC Bioinformatics. 2014;15(1):369.
https://doi.org/10.1186/s12859-014-0369-z
Bergmann FT, Rodriguez N, Le Novere N. COMBINE Archive Specification
Version 1. J Integr Bioinform. 2015;12(2):261.
https://doi.org/10.2390/biecoll-jib-2015-261
## class `EntryFormat(*values)`
Format URIs used in the `manifest.xml`.
COMBINE specifications (SBML, SED-ML, CellML, SBGN, BioPAX, OMEX metadata,
FROG results) are identified by `http://identifiers.org/combine.specifications/*`
URIs, all other files by their media type via `https://purl.org/NET/mediatypes/*`.
Where a specification is versioned, both the generic and the level and
version specific term exist, e.g., `SBML` and `SBML_L3V2`.
## class `Manifest(*, entries: list[pymetadata.omex.ManifestEntry] = [ManifestEntry(location='.', format='http://identifiers.org/combine.specifications/omex', master=False), ManifestEntry(location='./manifest.xml', format='http://identifiers.org/combine.specifications/omex-manifest', master=False)]) -> None`
Content of the `manifest.xml`, i.e., the entries of an archive.
The manifest behaves like a mapping keyed by location and always contains
the two entries required by the specification: the archive itself (`.`) and
the manifest (`./manifest.xml`).
Attributes:
entries: the manifest entries
Example:
```python
print(len(omex.manifest))
print("./model.xml" in omex.manifest)
entry = omex.manifest["./model.xml"]
```
### `Manifest.add_entry(self, entry: pymetadata.omex.ManifestEntry) -> None`
Add an entry to the manifest.
The location is normalized to a relative path starting with `./`.
Duplicated locations are not checked, use `Omex.add_entry` to add
a file together with its entry.
Args:
entry: entry to add
### `Manifest.remove_entry_for_location(self, location: str) -> pymetadata.omex.ManifestEntry | None`
Remove entry for given location.
### `Manifest.to_manifest(self, manifest_path: pathlib.Path) -> None`
Write the manifest to a `manifest.xml` file.
Args:
manifest_path: path of the file to write
### `Manifest.to_manifest_xml(self) -> str`
Serialize the manifest to `manifest.xml` content.
Returns:
The XML of the manifest as a string.
## class `ManifestEntry(*, location: str, format: str, master: bool = False) -> None`
A single file of the archive, as listed in the `manifest.xml`.
Attributes:
location: location of the file in the archive, relative and starting
with `./`, e.g., `./models/model.xml`
format: format URI of the file, see `EntryFormat`
master: marks the entry a tool should open first, e.g., the SED-ML file
of a simulation study
Example:
```python
entry = ManifestEntry(
location="./model.xml", format=EntryFormat.SBML_L3V2, master=True
)
```
### `ManifestEntry.is_format(format_key: str, format: str) -> bool`
Check if a format URI matches a format key.
Args:
format_key: `sbml`, `sedml` or `sbgn`, which match all level and
version variants, or the name of an `EntryFormat`
format: format URI to check
Returns:
True if the format matches the key.
### `ManifestEntry.is_sbgn(self) -> bool`
Check if entry is SBGN.
### `ManifestEntry.is_sbml(self) -> bool`
Check if entry is SBML.
### `ManifestEntry.is_sedml(self) -> bool`
Check if entry is SED-ML.
## class `Omex() -> None`
COMBINE archive (OMEX), version 1.
The content of the archive is kept in a temporary directory, `manifest`
holds the corresponding entries. Use the `from_*` constructors to read an
archive and the `to_*` methods to write one:
| read | write |
| --- | --- |
| `Omex.from_omex` from an omex file | `Omex.to_omex` to an omex file |
| `Omex.from_url` from a url | `Omex.to_directory` to a directory |
| `Omex.from_directory` from a directory | |
An empty archive is filled with `Omex.add_entry`.
Attributes:
manifest: entries of the archive, i.e., the content of the `manifest.xml`
Example:
Using the archive as a context manager removes the temporary
directory when the block is left:
```python
with Omex.from_omex(Path("archive.omex")) as omex:
print(omex)
```
### `Omex.add_entry(self, entry_path: pathlib.Path, entry: pymetadata.omex.ManifestEntry) -> None`
Add a file to the archive.
The file is copied into the archive, i.e., later changes to the source
file do not affect the content of the archive. Adding a second entry for
an existing location replaces the first one and logs a warning.
Args:
entry_path: path of the file to add
entry: manifest entry describing location, format and master flag
Raises:
ValueError: if `entry_path` does not exist or is not a file
Example:
```python
omex.add_entry(
entry_path=Path("model.xml"),
entry=ManifestEntry(
location="./model.xml",
format=EntryFormat.SBML_L3V2,
master=True,
),
)
```
### `Omex.entries_by_format(self, format_key: str) -> list[pymetadata.omex.ManifestEntry]`
Get all entries of a given format.
Args:
format_key: `sbml`, `sedml` or `sbgn`, which match all level and
version variants, or the name of an `EntryFormat`
Returns:
List of matching entries, empty if the archive contains none.
Example:
```python
for entry in omex.entries_by_format("sbml"):
print(entry.location)
```
### `Omex.from_omex(omex_path: pathlib.Path, password: bytes | None = None) -> 'Omex'`
Read a COMBINE archive from a path.
The archive is extracted into a temporary directory; the entries are
taken from the `manifest.xml` of the archive.
Args:
omex_path: path of the omex file
password: password of an encrypted archive
Returns:
Omex with the content of the archive.
Raises:
ValueError: if the path does not exist or is not a file
Example:
```python
omex = Omex.from_omex(Path("archive.omex"))
```
### `Omex.from_url(omex_url: str, password: bytes | None = None) -> 'Omex'`
Read a COMBINE archive from a url.
The archive is downloaded to a temporary file and read from there.
Args:
omex_url: url of the omex file
password: password of an encrypted archive
Returns:
Omex with the content of the archive.
Raises:
requests.HTTPError: if the archive could not be downloaded
Example:
```python
omex = Omex.from_url(
"https://github.com/matthiaskoenig/canagliflozin-model/"
"releases/download/0.7.0/canagliflozin_model.omex"
)
```
### `Omex.get_path(self, location: str) -> pathlib.Path`
Get the path of an entry in the extracted archive.
Args:
location: location of the entry, e.g., `./model.xml`
Returns:
Path of the file in the temporary directory of the archive, which
can be passed on to a reader such as libsbml.
Raises:
KeyError: if no entry exists for the location
### `Omex.guess_format(path: pathlib.Path) -> str`
Guess the format URI of a file.
The start of `.xml` files is inspected to tell SBML, SED-ML, CellML and
COPASI apart; for every other file the suffix decides.
Args:
path: path of the file
Returns:
The format URI, or the URI for an unknown media type if the format
cannot be determined.
### `Omex.is_omex(omex_path: pathlib.Path) -> bool`
Check if the path is a COMBINE archive.
The file must be a zip archive containing a `manifest.xml`.
Args:
omex_path: path to check
Returns:
True if the path is a COMBINE archive.
Raises:
ValueError: if the path does not exist or is not a file
### `Omex.lookup_format(format_key: str) -> str`
Look up the format URI for a format key.
Args:
format_key: name of an `EntryFormat`, e.g., `sbml` or `csv`
Returns:
The format URI, or the URI for an unknown media type if the key
cannot be resolved.
### `Omex.remove_entry_for_location(self, location: str) -> pymetadata.omex.ManifestEntry | None`
Remove an entry and the corresponding file from the archive.
Args:
location: location of the entry, e.g., `./model.xml`
Returns:
The removed entry, or None if no entry exists for the location.
### `Omex.to_directory(self, output_dir: pathlib.Path) -> None`
Extract the archive to a directory.
The `manifest.xml` is written next to the files, so the result can be
read back with `Omex.from_directory`.
Args:
output_dir: directory to write to, created if it does not exist
Example:
```python
omex.to_directory(Path("./unpacked"))
```
### `Omex.to_omex(self, omex_path: pathlib.Path, password: str | None = None, compression: int = 8, compresslevel: int = 9) -> None`
Write the archive to an omex file.
The `manifest.xml` is generated from the entries of the archive. By
definition OMEX files are zip deflated.
Args:
omex_path: path of the omex file to write
password: unused, encrypted archives cannot be written yet
compression: zipfile compression algorithm
compresslevel: level of compression. Has no effect for `ZIP_STORED`
and `ZIP_LZMA`; 0-9 for `ZIP_DEFLATED` (see zlib) and 1-9 for
`ZIP_BZIP2` (see bz2). Larger values compress better.
Example:
```python
omex.to_omex(Path("archive.omex"))
```
---
# pymetadata.core.annotation
MIRIAM annotations.
An annotation combines a qualifier (`BQB`, `BQM`) with a resource pointing at a
database entry, forming the statement *this element **is** CHEBI:17234*.
`RDFAnnotation` parses the notations found in the wild, normalizes them to
identifiers.org compact identifiers and validates them against the
identifiers.org registry. `RDFAnnotationData` resolves what an identifier refers
to via the Ontology Lookup Service.
```python
from pymetadata.core.annotation import RDFAnnotation
from pymetadata.core.miriam import BQB
annotation = RDFAnnotation(qualifier=BQB.IS, resource="chebi/CHEBI:33699")
print(annotation.resource_normalized) # https://identifiers.org/CHEBI:33699
print(annotation.validate())
```
## class `ProviderType(*values)`
Resolver a resource was written for.
`IDENTIFIERS_ORG` and `BIOREGISTRY_IO` resources can be normalized and
validated, `NONE` marks an arbitrary url which is kept as it is.
## class `RDFAnnotation(qualifier: pymetadata.core.miriam.BQB | pymetadata.core.miriam.BQM, resource: str, validate: bool = True)`
RDFAnnotation class.
Basic storage of annotation information. This consists of the relation
and the resource.
The annotations can be attached to other objects thereby forming
triples which can be converted to RDF.
Resource can be either:
- `http(s)://identifiers.org/collection/term`, i.e., a identifiers.org URI
- `collection/term`, i.e., the combination of collection and term
- `http(s)://arbitrary.url`, an arbitrary URL
- urn:miriam:uniprot:P03023
- https://bioregistry.io/chebi:15996 urls via the bioregistry provider
### `RDFAnnotation.check_miriam_term(self) -> bool`
Check that term follows id pattern for collection.
Uses the Identifiers collection information.
### `RDFAnnotation.check_qualifier(qualifier: pymetadata.core.miriam.BQB | pymetadata.core.miriam.BQM) -> bool`
Check that the qualifier is a MIRIAM qualifier.
Args:
qualifier: qualifier to check
Returns:
True if the qualifier is a `BQB` or `BQM` term.
### `RDFAnnotation.from_tuple(t: tuple[pymetadata.core.miriam.BQB | pymetadata.core.miriam.BQM, str]) -> 'RDFAnnotation'`
Create an annotation from a `(qualifier, resource)` tuple.
### `RDFAnnotation.shorten_compact_term(term: str, collection: str) -> str`
Shorten the compact terms and return term.
If the namespace is not embedded in the term return the shortened term.
### `RDFAnnotation.to_dict(self) -> dict`
Convert the annotation to a dictionary.
### `RDFAnnotation.validate(self) -> bool`
Validate qualifier and term of the annotation.
Returns:
True if the qualifier is a MIRIAM qualifier and the term matches the
pattern of its collection.
## class `RDFAnnotationData(annotation: pymetadata.core.annotation.RDFAnnotation)`
An annotation with the information behind the identifier resolved.
Constructing the object resolves the cross references: for every provider
the identifiers.org registry lists for the collection, the url pattern is
filled in with the term. `query_ols` then adds label, description and
synonyms from the Ontology Lookup Service, and replaces `xrefs` with the
cross references reported by OLS.
Attributes:
url: url of the first provider of the collection
label: name of the term
description: definition of the term
synonyms: synonyms of the term
xrefs: cross references of the term
warnings: problems which do not invalidate the annotation
errors: problems which do
Example:
```python
data = RDFAnnotationData(RDFAnnotation(BQB.IS, "chebi/CHEBI:33699"))
data.query_ols()
print(data.label)
```
Raises:
ValueError: if the collection of the annotation is not in the registry
### `RDFAnnotationData.check_miriam_term(self) -> bool`
Check that term follows id pattern for collection.
Uses the Identifiers collection information.
### `RDFAnnotationData.check_qualifier(qualifier: pymetadata.core.miriam.BQB | pymetadata.core.miriam.BQM) -> bool`
Check that the qualifier is a MIRIAM qualifier.
Args:
qualifier: qualifier to check
Returns:
True if the qualifier is a `BQB` or `BQM` term.
### `RDFAnnotationData.from_tuple(t: tuple[pymetadata.core.miriam.BQB | pymetadata.core.miriam.BQM, str]) -> 'RDFAnnotation'`
Create an annotation from a `(qualifier, resource)` tuple.
### `RDFAnnotationData.query_ols(self) -> dict`
Resolve the term in the Ontology Lookup Service.
Fills in `label`, `description` and `synonyms`, and replaces `xrefs`
with the cross references reported by OLS. Requires network access;
errors are collected in `errors` instead of raising.
Returns:
The processed OLS response.
### `RDFAnnotationData.shorten_compact_term(term: str, collection: str) -> str`
Shorten the compact terms and return term.
If the namespace is not embedded in the term return the shortened term.
### `RDFAnnotationData.to_dict(self) -> dict[str, typing.Any]`
Convert the annotation to a dictionary.
### `RDFAnnotationData.validate(self) -> bool`
Validate qualifier and term of the annotation.
Returns:
True if the qualifier is a MIRIAM qualifier and the term matches the
pattern of its collection.
## function `get_ols_query() -> pymetadata.webservices.ols.OLSQuery`
Get the shared OLS query object, created on first use.
Returns:
The shared `OLSQuery` for the ontologies in `ONTOLOGIES`.
---
# pymetadata.core.creator
Creator information for models and archives.
## class `Creator(familyName: str, givenName: str, email: str, organization: str, site: str | None = None, orcid: str | None = None)`
A person credited with a model or archive.
Used in the SBML ModelHistory and in the metadata of other COMBINE formats.
Attributes:
familyName: family name of the creator
givenName: given name of the creator
email: email address
organization: affiliation of the creator
site: url of a personal or institutional website
orcid: ORCID of the creator, e.g., `0000-0003-1725-179X`
Example:
```python
creator = Creator(
familyName="König",
givenName="Matthias",
email="konigmatt@googlemail.com",
organization="Humboldt-University Berlin",
orcid="0000-0003-1725-179X",
)
```
---
# pymetadata.core.miriam
MIRIAM qualifiers.
A MIRIAM annotation combines a qualifier, saying how an element relates to
something else, with a resource pointing at a database entry. Biological
qualifiers (`BQB`) relate an element to a biological entity, model qualifiers
(`BQM`) describe the model itself.
```python
from pymetadata.core.annotation import RDFAnnotation
from pymetadata.core.miriam import BQB
RDFAnnotation(qualifier=BQB.IS, resource="CHEBI:17234")
```
Choosing the qualifier matters: `BQB.IS` states that the element *is* the
annotated entity, whereas `BQB.IS_VERSION_OF` is the weaker claim that it is one
form of it.
## class `BQB(*values)`
MIRIAM biological qualifier, relating an element to a biological entity.
The most common are `BQB.IS` (the element is the entity),
`BQB.IS_VERSION_OF` (the element is one form of the entity),
`BQB.IS_PART_OF` (the element is part of the entity) and `BQB.HAS_TAXON`
(the entity occurs in the given taxon).
## class `BQM(*values)`
MIRIAM model qualifier, relating a model to a resource.
Use `BQM.IS_DESCRIBED_BY` to link a model to the publication describing it,
and `BQM.IS_DERIVED_FROM` to link it to the model it was built from.
---
# pymetadata.core.xref
Cross references to database entries.
A cross reference points at one database entry for a term, e.g., the ChEBI web
page for `CHEBI:33699`. `RDFAnnotationData` creates one cross reference per
provider registered for a collection in the identifiers.org registry.
## class `CrossReference(name: str, accession: str, url: str) -> None`
A database cross reference.
Attributes:
name: name of the resource, e.g., `ChEBI`
accession: term in the resource, e.g., `CHEBI:33699`
url: url of the entry in the resource
### `CrossReference.to_dict(self) -> dict`
Convert the cross reference to a dictionary.
### `CrossReference.validate(self, warnings: bool = True) -> bool`
Check that the cross reference has a valid url.
Args:
warnings: log a warning for an invalid url
Returns:
True if the url is valid.
## function `is_url(url: str) -> bool`
Check if a string is a valid http(s) or ftp url.
Args:
url: string to check
Returns:
True if the string is a valid url.
---
# pymetadata.ontologies.term
Ontology terms with the information of the ontology release.
An ontology is a class with one attribute per term, e.g., `SBO`, and a term is
an `OntologyTerm`, i.e., a `str` which is the identifier of the term and carries
what the ontology says about it:
```python
from pymetadata.ontologies import SBO
term = SBO.SIMPLE_CHEMICAL
term == "SBO_0000247" # True, a term is the identifier
term.label # 'simple chemical'
term.definition # 'Simple, non-repetitive chemical entity.'
term.synonyms # ()
term.curie # 'SBO:0000247'
term.url # 'https://identifiers.org/SBO:0000247'
```
Every term exists under its identifier (`SBO.SBO_0000247`) and under its name
(`SBO.SIMPLE_CHEMICAL`), both are the same object. The generated modules declare
the terms with their definition as docstring, so that an editor shows it when
the term is completed, and register the data with `_register`.
The ontology classes behave like the enums they replace: they are iterable, a
term can be looked up with `SBO["SBO_0000247"]` or `SBO("SBO:0000247")`, and
`validate` normalizes the notations of an identifier.
## class `OntologyMeta`
Metaclass which gives an ontology the lookups of an enum.
The ontology is iterable, contains its terms and resolves an identifier with
`SBO["SBO_0000247"]` and `SBO("SBO:0000247")`.
## class `OntologyTerm(term: 'str | OntologyTerm') -> 'OntologyTerm'`
A term of an ontology.
The term is the identifier of the ontology term, i.e., it can be used
wherever the identifier is expected: `SBO.SIMPLE_CHEMICAL == "SBO_0000247"`
is True and serializing a term gives the identifier. The information of the
ontology release is available as attributes.
Attributes:
label: name of the term in the ontology, e.g., `simple chemical`
definition: definition of the term, `None` if the ontology has none
synonyms: alternative names of the term
deprecated: the term is obsolete and should not be used for annotation
---
# pymetadata.webservices.chebi
Substance information from ChEBI.
Queries the ChEBI web service for the information stored for a term, such as the
InChIKey, which can then be used to look up cross references with
`pymetadata.webservices.unichem`.
```python
from pymetadata.webservices.chebi import ChebiQuery
info = ChebiQuery.query("CHEBI:33699")
```
See .
## class `ChebiQuery()`
Queries against the ChEBI web service.
Responses are cached on disk for `CACHE_DURATION_ONTOLOGY` hours, see
`pymetadata.CACHE_USE`. If ChEBI cannot be reached, cached content is used
however old it is.
### `ChebiQuery.query(chebi: str, cache: bool | None = None, cache_path: pathlib.Path | None = None) -> dict`
Query the information stored for a ChEBI term.
Args:
chebi: ChEBI term, e.g., `CHEBI:33699`
cache: cache the response, defaults to `pymetadata.CACHE_USE`
cache_path: directory for cached responses, defaults to
`pymetadata.CACHE_PATH`
Returns:
The ChEBI information, empty if the term could not be resolved.
---
# pymetadata.webservices.ols
Lookup of ontology terms in the Ontology Lookup Service (OLS).
OLS resolves an ontology term to its label, description, synonyms and cross
references. `RDFAnnotationData` uses it to fill in what an annotation actually
refers to.
```python
from pymetadata.webservices.ols import ONTOLOGIES, OLSQuery
query = OLSQuery(ontologies=ONTOLOGIES)
info = query.query_ols(ontology="chebi", term="CHEBI:33699")
print(query.process_response(info)["label"])
```
`ONTOLOGIES` lists the ontologies used in most projects together with the IRI
pattern needed to build the term IRI OLS expects.
See .
## class `OLSOntology(name: str, iri_pattern: str | None = None) -> None`
An ontology available in OLS.
Attributes:
name: lowercase ontology id, e.g., `chebi`
iri_pattern: pattern of the term IRI with the placeholder `{$Id}`,
defaults to the OBO purl of the ontology
## class `OLSQuery(ontologies: list[pymetadata.webservices.ols.OLSOntology], cache_path: pathlib.Path | None = None, cache: bool | None = None)`
Queries against the Ontology Lookup Service.
Responses are cached on disk for `CACHE_DURATION_ONTOLOGY` hours, see
`pymetadata.CACHE_USE`. If OLS cannot be reached, cached content is used
however old it is.
Attributes:
ontologies: the queryable ontologies by name
cache_path: directory of the cached responses
cache: whether responses are cached
### `OLSQuery.get_iri(self, ontology: str, term: str) -> str`
Build the term IRI which OLS expects.
Args:
ontology: ontology id, e.g., `chebi`
term: term of the ontology, e.g., `CHEBI:33699`
Returns:
The IRI of the term, or an empty string for an unknown ontology.
### `OLSQuery.process_response(self, term: dict) -> dict[str, typing.Any]`
Reduce an OLS response to the information used for annotations.
Args:
term: OLS response from `query_ols`
Returns:
Dictionary with `label`, `description`, `synonyms` and `xrefs`.
### `OLSQuery.query_ols(self, ontology: str | None, term: str | None) -> dict`
Query OLS for a single term.
Args:
ontology: ontology id, e.g., `chebi`
term: term of the ontology, e.g., `CHEBI:33699`
Returns:
The OLS response, with `errors` and `warnings` describing problems
with the query.
---
# pymetadata.webservices.registry
The identifiers.org registry.
The registry defines, for every collection (`chebi`, `uniprot`, `taxonomy`, ...),
the pattern a valid term matches and the providers which resolve a term to a
web page. `pymetadata` uses it to validate annotations and to build cross
references.
The registry is downloaded once and cached in
`CACHE_PATH / "identifiers_registry.json"`, and refreshed when the local copy is
older than the cache duration of `CACHE_DURATION_REGISTRY` hours. Namespaces are
added and corrected continuously, so the registry is refreshed daily, much more
often than the ontology information of `pymetadata.webservices.ols`. If the
refresh fails, because identifiers.org is unreachable, the outdated copy is used
and a warning is logged.
```python
from pymetadata.webservices.registry import Registry
registry = Registry()
namespace = registry.ns_dict["chebi"]
print(namespace.pattern) # ^CHEBI:\d+$
```
See and
.
## class `Namespace(id: 'str | None', prefix: 'str | None', name: 'str', pattern: 'str', namespaceEmbeddedInLui: 'bool', description: 'str', mirId: 'str | None' = None, resources: 'list | None' = None, created: 'str | None' = None, modified: 'str | None' = None, sampleId: 'str | None' = None, deprecated: 'bool' = False, deprecationDate: 'str | None' = None) -> None`
A collection of the identifiers.org registry.
Attributes:
prefix: prefix of the collection, e.g., `chebi`
name: name of the collection
pattern: regular expression a valid term matches
namespaceEmbeddedInLui: whether the prefix is part of the term itself,
as for `CHEBI:33699` and `GO:0005829`
resources: providers which resolve terms of this collection
## class `Registry(cache_duration: 'float' = 24, cache: 'bool' = True)`
The identifiers.org registry, cached on disk.
The cached registry is refreshed once it is older than the cache duration.
If identifiers.org cannot be reached, the outdated copy is used however old
it is, so that validating annotations keeps working offline.
Attributes:
ns_dict: namespaces of the registry by prefix
registry_path: path of the cached registry
### `Registry.load_registry(registry_path: 'Path') -> 'dict[str, Namespace]'`
Load the registry from the cached file, downloading it if missing.
Args:
registry_path: path of the cached registry
Returns:
Namespaces of the registry by prefix.
### `Registry.namespaces_from_dict(data: 'dict[str, Any]') -> 'dict[str, Namespace]'`
Build the namespaces from the serialized registry.
Args:
data: content of the cached registry
Returns:
Namespaces of the registry by prefix.
### `Registry.update(self) -> 'dict[str, Namespace]'`
Download the registry and return the namespaces.
Returns:
Namespaces of the registry by prefix.
Raises:
WebserviceError: if the registry could not be downloaded
### `Registry.update_registry(registry_path: 'Path | None' = None) -> 'dict[str, Namespace]'`
Download the registry from the identifiers.org web service.
Namespaces without a prefix are skipped.
Args:
registry_path: path to cache the registry in, not cached if None
Returns:
Namespaces of the registry by prefix.
Raises:
WebserviceError: if the registry could not be downloaded
## class `Resource(id: 'int | None', providerCode: 'str', name: 'str', urlPattern: 'str', mirId: 'str | None', description: 'str', official: 'bool', sampleId: 'str | None', resourceHomeUrl: 'str | None', institution: 'dict', location: 'dict', deprecated: 'bool', deprecationDate: 'str', protectedUrls: 'bool' = False, renderProtectedLanding: 'bool' = False, authHelpUrl: 'str | None' = None, authHelpDescription: 'str | None' = None) -> None`
A provider which resolves terms of a collection.
A collection can have several providers; `urlPattern` contains the
placeholder `{$id}` which is replaced by the term to build the url of an
entry.
## function `get_registry() -> 'Registry'`
Get the shared registry, loading it on first use.
The registry is loaded lazily so that importing pymetadata does not query
the identifiers.org web service.
Returns:
The shared registry instance.
---
# pymetadata.webservices.unichem
Substance cross references from UniChem.
UniChem maps a structure, identified by its InChIKey, to the entries of many
chemistry databases, which gives cross references for a substance without
having to query every database separately.
```python
from pymetadata.webservices.unichem import UnichemQuery
query = UnichemQuery()
xrefs = query.query_xrefs_for_inchikey("AAOVKJBEBIDNHE-UHFFFAOYSA-N")
```
See .
## class `UnichemQuery(cache_path: pathlib.Path | None = None, cache: bool | None = None)`
Queries against the UniChem web service.
The sources of UniChem are retrieved once and shared by all instances.
Responses are cached on disk for `CACHE_DURATION_ONTOLOGY` hours, see
`pymetadata.CACHE_USE`. If UniChem cannot be reached, cached content is used
however old it is.
### `UnichemQuery.get_sources(self) -> dict[int, pymetadata.webservices.unichem.UnichemSource]`
Get the databases known to UniChem, from the cache or the service.
Returns:
The sources by their UniChem source id.
Raises:
WebserviceError: if the sources are neither cached nor retrievable
### `UnichemQuery.query_xrefs_for_inchikey(self, inchikey: str) -> list[pymetadata.core.xref.CrossReference]`
Get the cross references for a structure.
Args:
inchikey: InChIKey of the structure, e.g.,
`AAOVKJBEBIDNHE-UHFFFAOYSA-N`
Returns:
One cross reference per database which contains the structure.
## class `UnichemSource(sourceID: int, srcUrl: str, name: str, nameLabel: str, nameLong: str, UCICount: int, baseIdUrl: str, description: str, created: str, lastUpdated: str, srcDetails: str, srcReleaseDate: str, srcReleaseNumber: int, updateComments: str, private: bool) -> None`
Unichem source.
src_id (the src_id for this source),
src_url (the main home page of the source),
name (the unique name for the source in UniChem, always lower case),
name_long (the full name of the source, as defined by the source),
name_label (A name for the source suitable for use as a 'label' for the source within a web-page. Correct case setting for source, and always less than 30 characters),
description (a description of the content of the source),
base_id_url_available (an flag indicating whether this source provides a valid base_id_url for creating cpd-specific links [1=yes, 0=no]).
base_id_url (the base url for constructing hyperlinks to this source [append an identifier from this source to the end of this url to create a valid url to a specific page for this cpd], unless aux_for_url=1),
aux_for_url (A flag to indicate whether the aux_src field should be used to create hyperlinks instead of the src_compound_id [1=yes, 0=no]
---
# pymetadata.webservices.webservice
HTTP access to the web services.
The queried services (identifiers.org, OLS, ChEBI, UniChem) answer with a
transient error every now and then, e.g., a HTML error page with status 500
instead of the expected JSON. `get_session` provides a shared `requests.Session`
which retries these responses with an exponential backoff and applies a default
timeout, so that a single hiccup of a service does not fail the query.
```python
from pymetadata.webservices.webservice import get_json
data = get_json("https://www.ebi.ac.uk/unichem/rest/inchikey/...")
```
`get_json` raises a `WebserviceError` for everything which keeps a query from
answering, i.e., an unreachable service, an error response and a response which
is not JSON. A service which is down for longer than the retries answers with an
HTML error page, so the status code has to be checked before the response is
parsed; `get_json` does that and the callers fall back to their cache, see
`pymetadata.cache`.
## class `WebserviceError`
Raised when a web service query cannot be answered.
Covers the unreachable service, the error response and the response which
is not JSON, i.e., everything a caller handles the same way: fall back to
the cached content, see `pymetadata.cache.read_json_cache_fallback`.
## function `get_json(url: str) -> Any`
Query a url and return the parsed JSON response.
Args:
url: url to query
Returns:
The parsed JSON response.
Raises:
WebserviceError: if the service cannot be reached, answers with a
status other than 200, or does not answer with JSON
## function `get_session() -> requests.sessions.Session`
Get the shared session for the web service queries.
The session retries the transient responses in `RETRY_STATUS_CODES` and
times out after `TIMEOUT` seconds.
Returns:
The shared session, created on first use.
---
# Development
Contributions are welcome. The repository is [matthiaskoenig/pymetadata](https://github.com/matthiaskoenig/pymetadata); development happens against the `develop` branch via pull requests.
## Setup development environment
Development needs [uv](https://docs.astral.sh/uv/) and a checkout of the repository:
```bash
git clone https://github.com/matthiaskoenig/pymetadata.git
cd pymetadata
```
A single sync creates the virtual environment in `.venv`, installs `pymetadata` into it in editable mode and adds the complete tooling:
```bash
uv sync --extra dev
```
The `dev` extra contains everything used below, i.e., pytest, ruff, ty, tox, pre-commit, zensical and bump-my-version, and it pulls in the optional `ontology` extra, so nothing has to be installed separately. The python version is taken from `.python-version` (currently 3.14); to work against the oldest supported version instead use `uv sync --extra dev --python 3.11`, which replaces the environment.
The tools are then run either with `uv run `, which uses the environment without activating it, or from the activated environment:
```bash
source .venv/bin/activate # Linux and macOS
.venv\Scripts\activate # Windows
```
The commands in this document are written without the `uv run` prefix; prepend it if the environment is not activated.
The last step installs the git hook:
```bash
uv run pre-commit install # install the hook, once per checkout
uv run pre-commit run --all-files # check the current state of the repository
```
From now on every commit is checked with ruff (lint and format) and ty, i.e., the same checks that run in continuous integration. On a commit only the changed files are looked at, `--all-files` checks the whole repository and is what a newly added hook should be tried with.
## Testing
The tests are written with pytest, tox runs them against every supported python version.
The tox environments are named after the interpreter (`py3.11` to `py3.14`, see `envlist` in `tox.ini`), a single one is run with
```bash
tox r -e py3.14
```
and the complete matrix, including the `ty` environment, in parallel with
```bash
tox run-parallel
```
This needs the interpreters to be available, which uv installs with `uv python install 3.11 3.12 3.13 3.14`. Continuous integration runs the same environments as `uvx --with tox-uv tox -e py3.14`.
To run the tests directly against the development environment use
```bash
pytest # the full suite
pytest tests/test_omex.py # a single module
pytest tests/test_omex.py::test_entry_from_dict # a single test
```
Some tests query web services (identifiers.org, OLS, ChEBI, UniChem) and therefore require network access.
## Linting and formatting
Linting and formatting use [ruff](https://docs.astral.sh/ruff/):
```bash
ruff check # lint
ruff format # format
```
## Type checking
Type checking is performed with [ty](https://docs.astral.sh/ty/):
```bash
tox r -e ty
```
Or directly in the working tree:
```bash
uvx ty check
```
The configuration lives in `[tool.ty]` in `pyproject.toml`. Warnings are treated as errors, so the codebase is kept free of diagnostics. Suppress an unavoidable diagnostic with a rule specific `# ty: ignore[rule-name]` rather than a blanket comment.
## Documentation
The documentation is built with [Zensical](https://zensical.org/), the static site generator of the Material for MkDocs authors. The sources are markdown files in `docs/`, the site is configured in `zensical.toml` in the repository root. Nothing rendered is committed: the site is built by the `documentation` workflow on every push and published to [matthiaskoenig.github.io/pymetadata](https://matthiaskoenig.github.io/pymetadata) from the `develop` branch.
Build the site into `site/`:
```bash
uv run zensical build --clean
```
For writing, the preview rebuilds on save:
```bash
uv run zensical serve
```
The API reference is rendered from the docstrings by [mkdocstrings](https://mkdocstrings.github.io/); a page in `docs/api/` only contains the module directive:
```markdown
# omex
::: pymetadata.omex
```
Docstrings are therefore the place to document functions and classes, the markdown files provide the narrative around them. Adding a module to the reference means adding such a page and an entry to `nav` in `zensical.toml`.
### Files for agents { #files-for-agents }
Agents and language models read markdown, not rendered html. `scripts/llms_txt.py` writes the files of the [llms.txt convention](https://llmstxt.org/) into the built site, i.e., [llms.txt](https://matthiaskoenig.github.io/pymetadata/llms.txt) as an annotated index of all pages, [llms-full.txt](https://matthiaskoenig.github.io/pymetadata/llms-full.txt) with the complete documentation in a single file, and the markdown of every page next to its html (`/omex.md` for `/omex/`). The markdown of the API reference is generated from the docstrings with `inspect`, since the pages themselves only contain the mkdocstrings directive.
```bash
uv run zensical build --clean
uv run python scripts/llms_txt.py
```
The `documentation` workflow runs both steps, so the files are regenerated with every push. `docs/robots.txt` points crawlers at the sitemap and at these files. Zensical will provide agent context files itself at some point, then this script can go.
## Regenerating the ontologies { #regenerating-the-ontologies }
`pymetadata.ontologies.sbo`, `kisao` and `pbpko` are generated modules and should not be edited by hand. They are rendered from the ontology releases by `pymetadata.ontologies._ontology_builder`, which is internal tooling for maintainers rather than part of the public API, and therefore not in the API reference:
```bash
python -m pymetadata.ontologies._ontology_builder
```
This needs the optional `ontology` dependency (`pronto`), which is not installed with the package because the generated terms work without it. It is part of the development environment, so `uv sync --extra dev` covers it.
It downloads the OWL files of the packaged ontologies, stores them gzipped under `src/pymetadata/resources/ontologies/` (not part of the repository), and writes one python module per ontology: a class with one attribute per term, documented with the definition of the term so that editors show it, plus the information registered on the class; the behaviour comes from `OntologyTerm` in `pymetadata.ontologies.term`. The modules are written with plain python string building, there is no template engine. Adding an ontology means adding an `OntologyFile` entry and an entry to `ontology_patterns` with the id pattern of the ontology.
Run `ruff format` afterwards, since the rendered modules are not formatted.
## Release
A release is made from `develop`:
1. update the ontology modules, see [Regenerating the ontologies](#regenerating-the-ontologies), and commit the changes
2. write the release notes for the version in `release-notes/`
3. make sure everything passes: `tox run-parallel`, `ruff check`, `tox r -e ty`
4. check the version bump: `uvx bump-my-version bump [major|minor|patch] --dry-run -vv`
5. bump the version: `uvx bump-my-version bump [major|minor|patch]`, which updates `src/pymetadata/__init__.py` and `CITATION.cff`, commits and tags
6. `git push --tags`, which triggers the release workflow publishing to [pypi](https://pypi.org/project/pymetadata/), followed by `git push`
7. test the installation from pypi in a fresh environment:
```bash
uv venv --python 3.14
uv pip install pymetadata
```
8. once Zenodo has archived the release, update the citation information, i.e., `date-released` in `CITATION.cff` and the version, date and version DOI of the release in the citation of `README.md` and `docs/index.md`. `bump-my-version` only updates the version, not the date and the DOI, which are only known after the release