Skip to content

fairvis.data_io

Loading and validation of FAIR assessments and the indicators.

REQUIRED_COLUMNS module-attribute 

REQUIRED_COLUMNS = (
    "ID",
    "Short",
    "Type",
    "Category",
    "Priority",
    "Indicator",
    "Description",
    "Assessment details",
    "Assessment",
)

Columns of the template an assessment needs.

ASSESSMENT_VALUES module-attribute 

ASSESSMENT_VALUES = (0.0, 0.5, 1.0)

Allowed assessment values besides NA (not applicable, an empty cell).

NA_STRINGS module-attribute 

NA_STRINGS = frozenset(
    {
        "",
        "#N/A",
        "#N/A N/A",
        "#NA",
        "-1.#IND",
        "-1.#QNAN",
        "-NaN",
        "-nan",
        "1.#IND",
        "1.#QNAN",
        "<NA>",
        "N/A",
        "NA",
        "NULL",
        "NaN",
        "None",
        "n/a",
        "nan",
        "null",
    }
)

Texts of a cell read as NA, the defaults of pandas.read_excel.

METADATA_COLUMNS module-attribute 

METADATA_COLUMNS = (
    "Short",
    "Type",
    "Category",
    "Priority",
    "Indicator",
)

Columns describing an indicator; taken from the template, not from an assessment.

MAX_ROWS module-attribute 

MAX_ROWS = 1000

Rows read from an assessment; the template has one row per indicator, far fewer.

Bounds the work of parsing an upload.

MAX_COLUMNS module-attribute 

MAX_COLUMNS = 50

Columns read from an assessment; the template has 11.

A single cell in the last column of a sheet would otherwise make every row 16384 cells wide.

MAX_UNPACKED_BYTES module-attribute 

MAX_UNPACKED_BYTES = 4 * 1024 ** 2

Size of the unpacked content of an xlsx file, an assessment has less than 1 MB.

An xlsx file is a zip archive; the limit rejects a zip bomb before it is parsed.

assessment_class 

assessment_class(value)

Class of an assessment value.

Parameters:

Name Type Description Default
value object

assessment of an indicator, NaN for NA.

 required

Returns:

Type Description
int

0 for NA, 1 for 0.0, 2 for 0.5 and 3 for 1.0.

Raises:

Type Description
ValueError

if the value is none of NA, 0, 0.5 and 1.

template 

template()

The FAIR assessment template, the definition of the indicators.

Returns:

Type Description
DataFrame

the template as read from its xlsx file, one row per indicator.

template_indicator_ids 

template_indicator_ids()

Indicator ids of the template, in the order of the template.

Returns:

Type Description
tuple[str, ...]

the ID of every indicator of the template.

validate_assessment 

validate_assessment(df)

Validate the structure and the values of a raw assessment.

Parameters:

Name Type Description Default
df DataFrame

assessment as read from the xlsx file, before the ID index is set.

 required

Raises:

Type Description
ValueError

if a column of the template is missing, the indicators differ from the ones of the template or an assessment value is none of NA, 0, 0.5 and 1.

read_first_sheet 

read_first_sheet(source)

Read the first sheet of an xlsx file, at most MAX_ROWS rows and MAX_COLUMNS.

The first row is the header; empty rows are skipped, empty cells are NaN.

Parameters:

Name Type Description Default
source Path | BinaryIO

path of the xlsx file or an uploaded file.

 required

Returns:

Type Description
DataFrame

the rows of the sheet with the header as column names.

Raises:

Type Description
ValueError

if the file is no readable xlsx file, unpacks to more than MAX_UNPACKED_BYTES or has more than MAX_ROWS rows.

load_assessment 

load_assessment(source)

Load an assessment from an xlsx file.

Only the assessment comes from the file; the columns describing the indicators, see METADATA_COLUMNS, are the ones of the template.

Parameters:

Name Type Description Default
source Path | BinaryIO

path of the xlsx file or an uploaded file.

 required

Returns:

Type Description
DataFrame

assessment indexed by the indicator ID, without the description columns
DataFrame

and without columns lacking a header (named Unnamed: <n>).

Raises:

Type Description
ValueError

if the file is no readable xlsx file, unpacks to more than MAX_UNPACKED_BYTES, has more than MAX_ROWS rows or is no valid assessment.

load_model_assessments 

load_model_assessments(assessment_dir=ASSESSMENTS_PATH)

Load all model assessments of a directory.

Parameters:

Name Type Description Default
assessment_dir Path

directory with one xlsx file per model.

 ASSESSMENTS_PATH

Returns:

Type Description
dict[str, DataFrame]

assessments by model id (the file name without suffix), sorted by model id.

load_indicators 

load_indicators(df_models)

Load the indicators and count their assessments over all models.

Parameters:

Name Type Description Default
df_models dict[str, DataFrame]

assessments by model id, see load_model_assessments.

 required

Returns:

Type Description
DataFrame

indicators indexed by ID; the column Assessment holds the number of
DataFrame

models with NA, 0, 0.5 and 1 for the indicator.

Raises:

Type Description
ValueError

if an assessment has other indicators than the template.