# HED annotation and validation in ndx-hed

This document describes how HED (Hierarchical Event Descriptors) annotations are represented in NWB tables, the rules ndx-hed follows, and how the validator works. It reflects the 1.0.0 design (PyNWB 4.0.0, where EventsTable, MeaningsTable, and related types are part of NWB core).

## Representing HED in a table

There are three ways a column contributes HED to a table, mirroring the two BIDS sidecar forms:

- Per-row HED: a `HedTags` column stores one HED string per row.
- Value HED: a `HedValueVector` column stores one HED template containing a single `#` placeholder; the placeholder is replaced by each row's value to produce that row's annotation.
- Categorical HED: a plain column of category labels is annotated by a `MeaningsTable`, which maps each distinct value to a meaning and (optionally) a HED string held in a `HedTags` column named `HED` inside the MeaningsTable.

`MeaningsTable` is a core hdmf type. ndx-hed does not define it; ndx-hed carries categorical HED by adding a `HedTags` column to it.

## HED column rules

The following five rules govern how HED columns may appear in an NWB table.

- Rule R1: A `HedTags` column must be named `HED`.
- Rule R2: A `DynamicTable` has at most one `HedTags` column.
- Rule R3: A `HedTags` column inside a `MeaningsTable` provides categorical (per-value) HED for the column that the MeaningsTable annotates.
- Rule R4: A `HedTags` column in any other `DynamicTable` provides per-row HED for that table.
- Rule R5: A `HedValueVector` must not appear in a `MeaningsTable`.

### How each rule is enforced

| Rule | How it is enforced                                                                                           |
| ---- | ------------------------------------------------------------------------------------------------------------ |
| R1   | `HedTags.__init__` raises `ValueError` if the name is not `HED` (which is also the default).                 |
| R2   | Follows from R1: column names within a table are unique, so two `HED` columns cannot coexist.                |
| R3   | The MeaningsTable's `HED` column is consumed as categorical HED when assembling the annotated table.         |
| R4   | The table's `HED` column is assembled as its per-row HED.                                                    |
| R5   | `HedNWBValidator.validate_file` raises `ValueError` if a `MeaningsTable` contains a `HedValueVector` column. |

Notes:

- `HedValueVector` may have any column name and is identified by its neurodata type, not by name.
- The `HedValueVector` schema documentation previously stated "Always has the name HED"; that was incorrect and has been corrected.

## How validation works

HED is not validated only tag-by-tag. Its meaning depends on the other HED in the same row, and for time-anchored tables on the other rows. There are two validation modes.

### Two modes: timeline and non-timeline

- Non-timeline: rows are not anchored to time. Each row's complete HED annotation is assembled from all of that row's columns and validated independently of the other rows.
- Timeline: rows are anchored to time (there is an `onset` column). In addition to per-row assembly, HED's temporal constructs (for example onset, offset, duration, and delay scopes that span rows) are validated over the events in time order.

The mode is determined automatically. A time-anchored table (for example an `EventsTable`, whose `timestamp` column is exported as `onset`) is validated as a timeline; a table with no `onset` column is validated as non-timeline.

### Assembled validation

`HedNWBValidator.validate_file` validates every `DynamicTable` except `MeaningsTable` using assembled validation, in two steps:

1. The table is converted to a BIDS-style dataframe and JSON sidecar with `get_bids_tabular`.
2. The sidecar (column metadata: value templates and categorical Levels/HED) is validated first with `Sidecar.validate`, then the assembled table is validated with `TabularInput.validate`. `TabularInput.validate` combines each row's per-row HED, categorical HED, and value-template HED into a single annotation before validating, and applies temporal validation when an `onset` column is present.

Both steps are required. `TabularInput.validate` does NOT re-run the sidecar's brace-structure and column-reference checks (self, nested, invalid, or malformed `{column}` references); those are only performed by `Sidecar.validate`. If the sidecar were validated only through the assembled table, those structural problems would be missed and would instead surface as misleading downstream errors (for example a stray `{` reported as `CHARACTER_INVALID`). Therefore the sidecar is validated explicitly first; if it has errors, they are returned and the assembled-table step is skipped so that downstream noise is not produced. (A consequence is that errors found in the sidecar step are reported by tag and message but without table/column/row context; that context is only available from the assembled-table step.)

A `MeaningsTable` is not validated on its own. Its categorical HED is validated as part of the table whose column it annotates. A `MeaningsTable` is only checked against rule R5.

### Per-column helpers

`validate_table`, `validate_vector`, and `validate_value_vector` remain available for validating a single table or a single column in isolation (per-column checks). These do not perform row assembly or temporal validation; use `validate_file` (or `validate_events` for a single EventsTable) for full assembled validation.

## Public API summary

| Function                                                                   | Direction / purpose                                                                   |
| -------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- |
| `extract_meanings(sidecar)`                                                | BIDS sidecar to a meanings dictionary.                                                |
| `get_events_table(name, description, df, meanings)`                        | BIDS dataframe plus meanings to an `EventsTable`.                                     |
| `get_bids_tabular(table)`                                                  | Any `DynamicTable` to a BIDS `(dataframe, sidecar)` pair. Formerly `get_bids_events`. |
| `HedNWBValidator.validate_file(nwbfile)`                                   | Assembled validation of every table in a file.                                        |
| `HedNWBValidator.validate_events(events)`                                  | Assembled validation of a single `EventsTable`.                                       |
| `HedNWBValidator.validate_table / validate_vector / validate_value_vector` | Per-column validation of a table or column.                                           |

## Relationship to BIDS

ndx-hed can represent everything BIDS can, plus arrangements BIDS does not (for example a table with no `onset`). Construction from BIDS builds an `EventsTable` (`get_events_table`); serialization to a BIDS-style tabular form works on any table (`get_bids_tabular`).