cache¶

Caching of web service responses.

Queries to identifiers.org, OLS, ChEBI and UniChem are cached on disk so that repeated lookups of the same term do not hit the network again. Caching is on by default and controlled by pymetadata.CACHE_USE and pymetadata.CACHE_PATH, which are read at query time.

A cached response is refreshed once it is older than the cache duration of its service, see CACHE_DURATION_ONTOLOGY and CACHE_DURATION_REGISTRY. If the refresh fails, because the service is unreachable or answers with an error, the outdated content is used instead of failing the query and a warning is logged, see read_json_cache_fallback. Working offline therefore keeps working with whatever was cached before.

The cache can be shared by processes which run at the same time, e.g., the workers of pytest-xdist. A cache file is written to a temporary file next to it, which then replaces it atomically, so a reader never sees a partially written file. On Windows a file which another process has open cannot be replaced, and a file which is being replaced cannot be opened; both are retried, see CACHE_IN_USE_ATTEMPTS. Writing the cache is best effort: the caller already has the content, so a cache file which cannot be written is a warning and not an error, and the cache file which is there is kept.

DataclassJSONEncoder ¶

Bases: JSONEncoder

JSON encoder which serializes dataclasses via their __dict__.

default ¶

default(o)

Serialize an object which json cannot serialize itself.

cache_file ¶

cache_file(directory, key)

Get the path of the cache file for a key.

The key, e.g., a term or an IRI, is percent-quoted into a single file name, so that it cannot contain path separators and leave the directory. A key whose file name would exceed CACHE_FILENAME_MAX_BYTES is replaced by its sha256 hash, which keeps the name within the limits of the file system.

Parameters:

Name	Type	Description	Default
`directory`	`Path`	directory of the cache files	required
`key`	`str`	what is cached, e.g., `CHEBI:2668`	required

Returns:

Type	Description
`Path`	The path of the JSON cache file inside `directory`.

Raises:

Type	Description
`ValueError`	if the key is empty or the file would be outside `directory`

cache_age ¶

cache_age(cache_path)

Get the age of a cache file in hours.

Parameters:

Name	Type	Description	Default
`cache_path`	`Path`	path of the cache file	required

Returns:

Type	Description
`float \| None`	The age in hours, or None if the file does not exist.

read_bytes_cache ¶

read_bytes_cache(cache_path)

Read a cache file.

The read is retried while another process replaces the file, which only fails on Windows.

Parameters:

Name	Type	Description	Default
`cache_path`	`Path`	path of the cache file	required

Returns:

Type	Description
`bytes`	The content of the file.

Raises:

Type	Description
`OSError`	if the file cannot be read

read_json_cache ¶

read_json_cache(cache_path, max_age=None)

Read a JSON cache file.

Parameters:

Name	Type	Description	Default
`cache_path`	`Path`	path of the cache file	required
`max_age`	`float \| None`	maximum age of the content in hours; older content is treated as if it were not cached. Any age is accepted if None	`None`

Returns:

Type	Description
`dict`	The cached content.

Raises:

Type	Description
`IOError`	if the cache file does not exist, is older than `max_age` or is corrupt. A corrupt file is removed, so that it is replaced by the next query.

read_json_cache_fallback ¶

read_json_cache_fallback(cache_path, reason)

Read a cache file of any age, after the query which should refresh it failed.

The library prefers outdated content over no content, so that an unreachable service does not fail a query which was answered before. The age of the content is not checked and a warning names the reason, so that the fallback is visible in the log.

Parameters:

Name	Type	Description	Default
`cache_path`	`Path`	path of the cache file	required
`reason`	`str`	why the content could not be refreshed, e.g., the error of the failed query	required

Returns:

Type	Description
`dict \| None`	The cached content, or None if nothing is cached.

write_bytes_cache ¶

write_bytes_cache(content, cache_path)

Write a cache file, best effort.

Missing parent directories are created. The content is written to a temporary file in the directory of the cache file, which then replaces the cache file atomically (os.replace): a reader never sees a partially written file, and of several processes which write the same cache file at the same time the last one wins with its complete file.

The caller has the content already, so a cache file which cannot be written is a warning and not an error, and the cache file which is there is kept. This covers a cache file which another process has open on Windows, where it cannot be replaced; the replace is retried for a while (see CACHE_IN_USE_ATTEMPTS) before the cache file of the other process is kept.

Parameters:

Name	Type	Description	Default
`content`	`bytes`	content of the file	required
`cache_path`	`Path`	path of the cache file	required

write_json_cache ¶

write_json_cache(data, cache_path, json_encoder=None)

Write a JSON cache file, best effort.

The data is serialized before anything is written, and the file is written with write_bytes_cache, i.e., atomically, and a cache file which cannot be written is a warning and not an error.

Parameters:

Name	Type	Description	Default
`data`	`dict`	data to serialize	required
`cache_path`	`Path`	path of the cache file	required
`json_encoder`	`type[JSONEncoder] \| None`	encoder for objects json cannot serialize, e.g. `DataclassJSONEncoder` for dataclasses	`None`

Raises:

Type	Description
`TypeError`	if the data cannot be serialized; nothing is written then