HED annotation and validation in ndx-hed

This document describes how HED (Hierarchical Event Descriptors) annotations are represented in NWB tables, the rules ndx-hed follows, and how the validator works. It reflects the 1.0.0 design (PyNWB 4.0.0, where EventsTable, MeaningsTable, and related types are part of NWB core).

Representing HED in a table

There are three ways a column contributes HED to a table, mirroring the two BIDS sidecar forms:

• Per-row HED: a HedTags column stores one HED string per row.

• Value HED: a HedValueVector column stores one HED template containing a single # placeholder; the placeholder is replaced by each row’s value to produce that row’s annotation.

• Categorical HED: a plain column of category labels is annotated by a MeaningsTable, which maps each distinct value to a meaning and (optionally) a HED string held in a HedTags column named HED inside the MeaningsTable.

MeaningsTable is a core hdmf type. ndx-hed does not define it; ndx-hed carries categorical HED by adding a HedTags column to it.

HED column rules

The following rules govern how HED may appear in an NWB table.

• Rule R1: A HedTags column must be named HED.

• Rule R2: A DynamicTable has at most one HedTags column.

• Rule R3: A HedTags column inside a MeaningsTable provides categorical (per-value) HED for the column that the MeaningsTable annotates.

• Rule R4: A HedTags column in any other DynamicTable provides per-row HED for that table.

• Rule R5: A HedValueVector must not appear in a MeaningsTable.

• Rule R6: A column named HED, in a DynamicTable or in a MeaningsTable, must be a HedTags. The name is what marks the column as HED (R3, R4), so a VectorData or a HedValueVector named HED is an error.

• Rule R7: The values of a HedValueVector must satisfy the value class of the # tag of its template. A # with no value class is textClass. When the template is Def/Name/#, the class is that of the placeholder tag inside the definition.

• Rule R8: A MeaningsTable HED string or a HedValueVector template may reference another HED-annotated column of the same table, or HED for the table’s per-row column, with the curly-brace notation of the HED specification (section 3.2.9.3); all rules of that section apply. A per-row HED cell may not contain braces.

How each rule is enforced

Rule	How it is enforced
R1	HedTags.__init__ raises ValueError if the name is not HED (which is also the default).
R2	Follows from R1: column names within a table are unique, so two HED columns cannot coexist.
R3	The MeaningsTable’s HED column is consumed as categorical HED when assembling the annotated table.
R4	The table’s HED column is assembled as its per-row HED.
R5	HedNWBValidator.validate_table reports MEANINGS_VALUE_VECTOR_INVALID and stops validating that table.
R6	HedNWBValidator.validate_table reports HED_COLUMN_TYPE_INVALID and stops validating that table.
R7	HedNWBValidator.validate_table checks the column against the class (a numeric column under a text class, or an integer column under a numeric or name class, passes on its dtype alone; a float column under a numeric class is scanned once for infinities; otherwise each distinct value is checked as the # tag alone with the value in place, so only that tag’s units and value class matter, and a value that passes is parsed as the substituted tag so one that breaks HED syntax is rejected too) and stops on any error.
R8	hedtools Sidecar.validate in step 4 of validate_table checks the references (unknown target, self or chained reference, nesting) and validates each string with the references removed; a brace in a per-row HED cell is CHARACTER_INVALID. The value check (R7) and validate_value_vector never see a reference; validate_vector validates per-row strings, where a brace is CHARACTER_INVALID.

Notes:

• HedValueVector may have any column name and is identified by its neurodata type, not by name.

• MEANINGS_VALUE_VECTOR_INVALID and HED_COLUMN_TYPE_INVALID are ndx-hed codes, registered with hedtools so the issues have the same shape as hedtools’ own (code, message, severity, and the ec_* context keys). R7 errors are hedtools codes: VALUE_INVALID, CHARACTER_INVALID, or a units code from hedtools’ validate_units on the one tag, or a syntax code such as PARENTHESES_MISMATCH or PLACEHOLDER_INVALID from the basic checks on the substituted tag.

• The HedValueVector schema documentation previously stated “Always has the name HED”; that was incorrect and has been corrected.

How validation works

HED is not validated only tag-by-tag. Its meaning depends on the other HED in the same row, and for time-anchored tables on the other rows. HedNWBValidator.validate_table is the table validator; HedNWBValidator.validate_file finds the file’s HedLabMetaData and calls validate_table for every DynamicTable in the file except a MeaningsTable, whose HED is validated with the table it annotates. Passing a MeaningsTable to validate_table raises ValueError.

The steps of validate_table

“Stops” means the issues found so far for that table are returned; the other tables of a file are still validated.

1. Structural rules R5 and R6 on the table and its MeaningsTables. An issue stops.

2. A table with no HedTags column, no HedValueVector column, and no MeaningsTable with a HED column has nothing to validate and returns no issues without being read. A file’s large HED-free tables (a Units table) cost nothing.

3. The table’s column metadata becomes a BIDS sidecar dict with get_json_hed_dict: the HedValueVector templates, the categorical Levels and HED of each MeaningsTable, and the definitions of the HedLabMetaData under the definitions key. No table data is read.

4. hedtools’ staged tabular validation (SpreadsheetValidator.validate) runs once on the table, through a column source (hedtools’ ColumnSource protocol) that answers with the column names and with the distinct values of one column at a time, so a column is read only when a stage needs it and never cell by cell. A TimestampVectorData column is named onset, as get_bids_dataframe names it. Each stage stops on an error; warnings do not stop.

   • The sidecar, with hedtools’ sidecar validator. Only this stage performs the brace-structure and column-reference checks (self, nested, invalid, or malformed {column} references) and validates the HED of every categorical level, used or not. A sidecar-level error would be repeated on every row that uses it, and the row annotations assembled from it cannot be trusted.

   • The column values. Rule R7 on every HedValueVector column: each distinct value is checked as the template’s # tag with the value in place, then the substituted tag is parsed for the basic syntax checks. A column whose dtype settles the question is not read: a numeric column passes textClass, an integer column passes numericClass and nameClass, and a float column under numericClass is scanned once for infinities, which are not numeric values. Each categorical column’s distinct values are checked against the levels of its MeaningsTable (SIDECAR_KEY_MISSING, a warning).

   • The HED column: each distinct string once, with the basic (single-string) checks.

   • The assembly gate:

     • assemble=True (the default): the whole table is read into a BIDS dataframe with get_bids_dataframe (a TimestampVectorData column becomes onset; missing cells become n/a), only now, and hedtools assembles each row’s HED from its HED cell, its categorical HED, and its value templates and validates the row as a whole. If the table has an onset column it is a timeline and HED’s temporal constructs (onset, offset, duration, delay scopes that span rows) are validated over the rows in time order; otherwise each row is validated on its own.

     • assemble=False: the table is never converted to a dataframe. Instead the group-level checks run on each distinct HED string by itself (an Onset without its Def, a tag repeated within the cell, an empty group); the required-tag check, which needs every tag of the row, waits for assembly. Nothing that needs the other tags of a row or the other rows is checked: a tag repeated between the HED column and a categorical annotation, or an Offset without its Onset, is reported only with assembly.

Where an issue points

Every issue carries the table name in ec_table_name; through validate_file it also carries the file identifier in ec_filename.

• A sidecar issue (a HedValueVector template, a MeaningsTable HED string, a definition) carries the column in ec_sidecarColumnName and, for categorical HED, the value in ec_sidecarKeyName. It has no row, because a sidecar error is not a row error.

• A row issue carries the column in ec_column and the row in ec_row, which is the table row index in both modes: the first data row is 0. (hedtools itself numbers the lines of a BIDS TSV file, header first; validate_table converts.)

• The values of a HedValueVector and the strings of the HED column are validated by distinct value in both modes: each distinct annotation is validated once and reported at the first row holding it, with the number of rows that hold it in row_count (hedtools’ ROW_COUNT_KEY, re-exported by ndx_hed.utils.hed_nwb_validator). The same misspelled tag on ten thousand rows is one issue. Both checks are hedtools’ column stages, run before the assembly gate in both modes. Issues from the assembled rows (group-level and temporal checks) are per row.

get_printable_issue_string renders these as “Errors in table ‘’”, “Issues in column ”, “Issues in row ”, “Column ‘’:”, and “Key: ”.

Using the table validator from nwbinspector

An inspector check receives one object. For a DynamicTable it finds the schema through the file and validates the table without assembly:

nwbfile = table.get_ancestor(data_type="NWBFile")
hed_metadata = nwbfile.lab_meta_data.get("hed_schema") if nwbfile is not None else None
if isinstance(hed_metadata, HedLabMetaData):
    issues = HedNWBValidator(hed_metadata).validate_table(table, assemble=False)

Constructing a validator per table costs two attribute reads; the schema was loaded when the HedLabMetaData was built.

Single-column helpers

validate_vector (a HedTags column) validates one column in isolation: each distinct string once, with the basic and the group-level checks (hedtools validate_hed_column with full_string=True), reported at the first row holding it with row_count. It is for a per-row HED column, which may not contain braces, so a MeaningsTable HED column belongs to validate_table of the table it annotates. validate_value_vector (a HedValueVector column) validates the template on its own with any {column} references removed, then checks each distinct value as the # tag with the value in place through hedtools validate_value_column, exactly as rule R7 in validate_table, reporting the first row and row_count; a column whose dtype settles its value class is not read. Both carry the column in ec_column. Neither helper sees another column or another row, and neither checks a reference; that is validate_table’s sidecar stage.

Public API summary

Function	Direction / purpose
extract_meanings(sidecar)	BIDS sidecar to a meanings dictionary.
get_events_table(name, description, df, meanings)	BIDS dataframe plus meanings to an EventsTable.
get_bids_tabular(table, hed_metadata=None)	Any DynamicTable to a BIDS (dataframe, sidecar) pair: get_bids_dataframe plus get_json_hed_dict. Formerly get_bids_events.
get_bids_dataframe(table)	Any DynamicTable to its BIDS dataframe (onset, missing cells as n/a); reads every column.
get_json_hed_dict(table, hed_metadata=None)	Any DynamicTable to just its BIDS JSON sidecar dict (no dataframe, no pandas, no table data read).
get_levels_and_hed(meanings_table)	A MeaningsTable to its (Levels, HED) sidecar dictionaries.
HedNWBValidator.validate_file(nwbfile, assemble=True)	Metadata check, then validate_table on every table in the file.
HedNWBValidator.validate_table(table, assemble=True)	Structural rules, sidecar, value classes, then assembled (row and temporal) validation, or with assemble=False cell by cell.
HedNWBValidator.validate_vector / validate_value_vector	One column in isolation.

Definitions in the exported sidecar

In NWB, HED definitions live in the HedLabMetaData object, which is the only place extra definitions may come from. BIDS instead carries them in the sidecar, in an entry that names no column. get_json_hed_dict (and get_bids_tabular) therefore take an optional HedLabMetaData; when one is supplied, its definitions are exported as

{"definitions": {"HED": {"defList": "(Definition/go,(Sensory-event))"}}}

so the sidecar is self-contained and any Def/ references in the table resolve from it alone. Without the metadata no such entry is written, and Def/ references in the exported sidecar are unresolvable (DEF_INVALID). HedNWBValidator always passes its own HedLabMetaData, so hedtools sees the definitions exactly as a BIDS validator would; for a namespaced schema (ts:8.5.0) the exported Definition tag carries the namespace prefix.

Relationship to BIDS

ndx-hed can represent everything BIDS can, plus arrangements BIDS does not (for example a table with no onset). Construction from BIDS builds an EventsTable (get_events_table); serialization to a BIDS-style tabular form works on any table (get_bids_tabular).