HED online tools user guide¶

This guide provides comprehensive instructions for using the HED web-based tools and REST API services for validation, transformation, and analysis of HED annotations.

Quick links¶

Table of contents¶

Getting started
Working with events files
Working with sidecars
Working with spreadsheets
Working with HED strings
Working with HED schemas
Using RESTful services
Best practices
Troubleshooting

Getting started¶

Accessing HED online tools¶

HED web-based tools are available directly through a browser at https://hedtools.org/hed. These tools are free to use and don’t require a login. Two versions are available:

Production: https://hedtools.org/hed - Stable release version
Development: https://hedtools.org/hed_dev - Latest development features

Tool organization¶

The HED browser-based tools are organized into the following pages:

Tool	URL	Description
Events	link	Validation, summary, search, and generation tools for tabular (.tsv) files
Sidecars	link	Validation, transformation, extraction, and merging tools for JSON sidecars
Spreadsheets	link	Validation and transformation tools for spreadsheet files
Strings	link	Quick validation and transformation of individual HED strings
Schemas	link	Validation, conversion, and comparison tools for HED schemas

Common features¶

Schema selection¶

Most tools require you to specify a HED schema version. You can:

Select a standard version: Use the pull-down menu to choose from official HED releases
Upload a custom schema: Select “Other” and upload your own HED schema file

The tool automatically caches standard schemas for faster processing.

Tag formats¶

HED supports three tag formats:

Short form: Only the leaf node (e.g., Red)
Long form: Full path from root (e.g., Property/Sensory-property/.../Red)
Intermediate form: Partial path (e.g., Color/Red)

All HED tools handle any combination of these formats. The short form is recommended for annotation.

Definition expansion¶

Many tools have an Expand defs option that replaces Def/xxx tags with expanded definitions in the form (Def-expand/xxx, yyy) where yyy is the actual definition content.

Tabular formats¶

Tabular files are BIDS-style tab-separated value (.tsv) files with a header line giving column names. These columns map to metadata in accompanying JSON sidecars. HED handles two types of tabular files – files that have an onset column (timeline files) and those that don’t (descriptive files). The BIDS _events.tsv files are the most common timeline files. An example of a descriptive file is the BIDS participants.tsv file.

Limiting errors¶

Most of the validation actions have an option to limit the errors. Checking this option results in the error report only reporting two of each type of error. This is useful because often the same error is repeated for each error of the tabular file and fixing it once (usually a sidecar problem), fixes them all. Once you correct all the errors with the limit errors on and the check for warnings off, you should try again with the limit errors off and the check for warnings on to make sure that you have understood and corrected all issues.

Working with tabular files¶

Events files are BIDS-style tab-separated value (.tsv) files with a header line giving column names. These columns map to metadata in accompanying JSON sidecars.

Validate an events file¶

Validates HED annotations in a BIDS events file or other tabular file. An excellent annotation strategy is to focus on debugging a single annotation file supported by a JSON sidecar located at the highest possible level. For _events.tsv, a JSON sidecar at the top level of a BIDS dataset applies to all _events.tsv files in the dataset – so debugging one file and its sidecar essentially debugs them all.

Steps:

Select the Validate action
Enable Check for warnings if you want to include warnings (don’t on the first pass)
Enable Limit errors reported so that at most 2 of each type of error appear in output (only at first)
Upload a tabular file (.tsv)
Optionally upload a JSON sidecar file (.json)
Select the HED version
Click the Process button

Returns: A downloadable .txt file of error messages if errors are found.

Note: This tool validates a single tabular file. For full BIDS dataset validation with inherited sidecars and library schemas, use the JavaScript HEDTools which has a browser-based validator that works on the entire dataset. However, from a practical viewpoint, it is better to debug a single file first.

Check HED quality¶

Evaluates the HED annotations in a BIDS events file or other tabular file for correct semantic usage.

Steps:

Select the Check HED quality action
Disable Show detailed errors (only at first to get a sense of the quality complaints)
Enable Limit errors reported so that at most 2 of each type of error appear in output (only at first)
Upload a tabular file (.tsv)
Optionally upload a JSON sidecar file (.json)
Select the HED version
Click the Process button

Returns: A downloadable .txt file of quality messages if errors are found.

Assemble HED annotations¶

Produces a file with fully assembled HED annotations for each event, combining sidecar and column values.

Steps:

Select the Assemble HED annotations action
Enable Append assembled if you want a HEDAssembled column added to the tabular file.
Enable Include context if you want the `Event-context group with ongoing events included
Enable Replace defs if you to replace the definitions with their contents
Enable Remove condition and task if you want condition variables and task groups removed.
Upload a tabular file (.tsv)
Optionally upload a JSON sidecar file (.json)
Select the HED version
Click the Process button

Returns: A downloadable .tsv file with two columns: onset times and assembled HED strings (unless the Append assembled option is selected).

Search HED¶

Search for events matching specific HED criteria within an events file.

Steps:

Select the Search HED action
Enable Append assembled if you want a HEDAssembled column added to the tabular file.
Enable Include context if you want the `Event-context group with ongoing events included
Enable Replace defs if you to replace the definitions with their contents
Enable Remove condition and task if you want condition variables and task groups removed.
Upload a tabular file (.tsv)
Optionally upload a JSON sidecar file (.json)
Enter the search query
Select the HED version
Click the Process button

Returns: A downloadable .tsv file containing only the rows matching your search criteria.

Generate sidecar template¶

Creates a JSON sidecar template from tabular file, ready to be filled with HED annotations.

Steps:

Select the Generate JSON sidecar template action
Upload a tabular file (.tsv)
Review the column list that appears
In the left checkboxes, select columns to include in the template
In the right checkboxes, select columns requiring annotations for each individual unique column value
Click the Process button

Returns: A downloadable .json sidecar template file.

Tip: This action generates a template for an individual tabular file. This is sufficient for most datasets, provided their tabular files are similarly structured. The Python HEDTools hedpy extract-sidecar command line tool and the extract_json_template.ipynb Jupyter notebook for generating a consolidated template from all of the tabular files of a particular type.

Execute remodel script¶

Apply table remodeling operations to transform or summarize events files without programming.

Steps:

Select the Execute remodel script action
Enable Include summaries if you want summary output
Upload a tabular file (.tsv)
Optionally upload a JSON sidecar file (.json)
Upload the remodel instructions file (.json)
Select the HED version
Click the Process button

Returns: A zip archive containing transformed tabular and summaries.

Tip: The remodel instructions file contains a series of commands in JSON format that are designed to be run to transform and summarize every designated tabular file in a dataset (e.g., all of the _events.tsv files). The online tools only run the remodeling on a single tabular file. This is very useful for validating the instruction file and assuring the transformations are what is desired.

See the Table remodeler documentation for details on creating remodel scripts.

Working with sidecars¶

BIDS JSON sidecars (files ending in xxx.json) contain metadata and HED annotations for the designated files (ending in xxx.tsv). See the BIDS Inheritance Principle for a description of how the association is made. A convenient aspect of BIDS inheritance is that users can annotate all of the files of a single type (say _events.tsv) by placing a single (_events.json) file with the annotations at the top level of the BIDS dataset directory tree.

Validate a sidecar¶

Validates HED annotations in a JSON sidecar file.

Steps:

Select the Validate action
Enable Check for warnings if you want warnings (don’t do it to start – get the errors out first)
Upload a JSON sidecar file (.json)
Select the HED version
Click the Process button

Returns: A downloadable .txt file of error messages if errors are found.

Note: For sidecars with external definitions or library schemas, use the Python tools for complete validation.

Convert sidecar to long¶

Converts all HED tags in a sidecar to long form (full paths).

Steps:

Select the Convert to long action
Enable Expand defs if you want expanded definitions
Upload a JSON sidecar file (.json)
Select the HED version
Click the Process button

Returns: A downloadable converted .json sidecar file.

Convert sidecar to short¶

Converts all HED tags in a sidecar to short form (leaf nodes only).

Steps:

Select the Convert to short action
Enable Expand defs if you want expanded definitions
Upload a JSON sidecar file (.json)
Select the HED version
Click the Process button

Returns: A downloadable converted .json sidecar file.

Extract spreadsheet from sidecar¶

Exports sidecar content to a 4-column spreadsheet for easier editing.

Steps:

Select the Extract HED spreadsheet action
Upload the JSON sidecar file (.json)
Click the Process button

Returns: A downloadable .tsv spreadsheet with columns: column_name, column_value, description, HED.

See the BIDS Annotation Quickstart for a tutorial on using this spreadsheet.

Merge spreadsheet with sidecar¶

Imports HED annotations from a 4-column spreadsheet back into a JSON sidecar.

Steps:

Select the Merge HED spreadsheet action
Toggle Include description to include descriptions as HED tags
Upload the target JSON sidecar file (.json)
Upload the spreadsheet (.tsv or .xlsx)
Click the Process button

Returns: A downloadable merged .json sidecar file.

Working with spreadsheets¶

Spreadsheet tools support general-purpose HED annotation in Excel (.xlsx) or tab-separated (.tsv) format, useful when developing annotations outside of specific datasets.

Validate a spreadsheet¶

Validates HED tags in spreadsheet columns.

Steps:

Select the Validate action
Enable Check warnings if you want warnings (usually not to start)
Upload a spreadsheet file (.tsv or .xlsx) - this will trigger worksheet and column name display
Select a worksheet (if Excel file with multiple sheets)
Check boxes next to columns containing HED tags
Upload external definitions (.json file) if needed
Select the HED version
Click the Process button

Returns: A downloadable .txt file of error messages if errors are found.

Convert spreadsheet to long¶

Converts all HED tags in selected columns to long form.

Steps:

Select the Convert to long action
Enable Expand defs if desired
Upload a spreadsheet file (.tsv or .xlsx) - this will trigger worksheet and column name display
Select a worksheet (if Excel file with multiple sheets)
Check boxes next to columns containing HED tags
Upload external definitions (.json file) if needed
Select the HED version
Click the Process button

Returns: A downloadable spreadsheet with converted HED tags.

Convert spreadsheet to short¶

Converts all HED tags in selected columns to short form.

Steps:

Select the Convert to short action
Enable Expand defs if desired
Upload a spreadsheet file (.tsv or .xlsx) - this will trigger worksheet and column name display
Select a worksheet (if Excel file with multiple sheets)
Check boxes next to columns containing HED tags
Upload external definitions (.json file) if needed
Select the HED version
Click the Process button

Returns: A downloadable spreadsheet with converted HED tags.

Working with HED strings¶

String tools provide quick validation and conversion for individual HED annotations.

Validate a HED string¶

Validates a single HED string containing tags and groups.

Steps:

Select the Validate action
Toggle Check warnings if you want warnings
Type or paste your HED string into the text box
Upload external definitions (.json file) if needed
Select the HED version
Click the Process button

Returns: Error messages displayed in the Results box under the definition upload.

Convert string to long¶

Converts a HED string to long form (full paths).

Steps:

Select the Convert to long action
Type or paste your HED string into the text box
Select the HED version
Click the Process button

Returns: Converted string or error messages in the Results box.

Convert string to short¶

Converts a HED string to short form (leaf nodes only).

Steps:

Select the Convert to short action
Type or paste your HED string into the text box
Select the HED version
Click the Process button

Returns: Converted string or error messages in the Results box.

Search HED strings¶

Search a HED string.

Steps:

Select the Search a HED string action
Type or paste your HED string into the HED string box
Type or paste your query into the HED search query box
Select the HED version
Click the Process button

Returns: Converted string or error messages in the Results box.

Working with HED schemas¶

Schema tools help HED schema developers validate, convert, and compare schemas.

View standard schemas using the HED Schema Browser.

Validate a schema¶

Checks schema syntax and HED-3G compliance.

Steps:

Select the Validate action
Toggle Check warnings if you want warnings
Enter a source schema URL or upload a schema file (.xml, .mediawiki, or .json) or upload a folder containing the .tsv files for a schema
Click the Process button

Returns: A downloadable .txt file of error messages if errors are found.

Convert a schema¶

Converts between any schema format and all the formats:

Steps:

Select the Convert schema action
Enter a source schema URL or upload a schema file (.xml, .mediawiki, or .json) or upload a folder containing the .tsv files for a schema
Click the Process button

Returns: A downloadable zip file contained the converted schema file in all 4 formats.

Note: The conversion produces new versions of the schema in all 4 formats. This makes sure that the input schema can be converted correctly to all 4 formats.

Compare schemas¶

Shows differences between two schema versions.

Steps:

Select the Compare schemas action
Enter a source schema URL or upload a schema file (.xml, .mediawiki, or .json) or upload a folder containing the .tsv files for a schema
Enter a comparison schema URL or upload a schema file (.xml, .mediawiki, or .json) or upload a folder containing the .tsv files for a schema
Click the Process button

Returns: A downloadable .txt file with schema differences in an organized format.

Using RESTful services¶

HED provides REST API services for programmatic access to all tools.

Service setup¶

Send HTTP GET request to obtain CSRF token: https://hedtools.org/hed/services
Extract CSRF token and cookie from response
Send HTTP POST requests with token to: https://hedtools.org/hed/services_submit

See the web services demos in the MATLAB HEDTools for implementation examples.

Request format¶

All requests are JSON dictionaries with a service parameter and additional parameters.

Example: Validating a JSON sidecar:

{
    "service": "sidecar_validate",
    "schema_version": "8.4.0", 
    "json_string": "json file text",
    "check_for_warnings": "on"
}

Available services¶

Service	Description
`get_services`	Returns list of available services
`events_validate`	Validate BIDS events file
`events_assemble`	Assemble HED annotations
`events_extract`	Extract sidecar template
`sidecar_validate`	Validate JSON sidecar
`sidecar_to_long`	Convert sidecar to long form
`sidecar_to_short`	Convert sidecar to short form
`spreadsheet_validate`	Validate spreadsheet
`spreadsheet_to_long`	Convert spreadsheet to long form
`spreadsheet_to_short`	Convert spreadsheet to short form
`strings_validate`	Validate HED strings
`strings_to_long`	Convert strings to long form
`strings_to_short`	Convert strings to short form

Service parameters¶

Common parameters:

service (string): Service name
schema_version (string): HED version (e.g., “8.4.0”)
schema_string (string): Custom schema as XML string
check_for_warnings (boolean): Include validation warnings
expand_defs (boolean): Expand definition tags
events_string (string): BIDS events file content
json_string (string): JSON sidecar content
spreadsheet_string (string): Spreadsheet content
hed_strings (list): List of HED strings

Service responses¶

All services return JSON with:

{
    "service": "service_name",
    "results": {
        "command": "command_executed",
        "command_target": "data_type",
        "data": "result_or_errors",
        "schema_version": "8.4.0",
        "msg_category": "success|warning|failure",
        "msg": "explanation"
    },
    "error_type": "",
    "error_msg": ""
}

Empty error_type and error_msg: Operation completed otherwise indication of server failure
msg_category: Indicates validation result (success/warning/failure)
data: Contains processed result or validation errors

Best practices¶

For annotation development¶

Start with templates: Generate sidecar templates from events or other tabular files
Use spreadsheet workflow: Extract to spreadsheet, annotate, merge back
Validate frequently: Check annotations after each change
Test with warnings: Enable warning checks to catch potential issues after eliminating errors
Always use short form: Shorter tags are easier to read and maintain

For schema development¶

Validate before converting: Always validate schemas before format conversion
Use MEDIAWIKI for editing: Easier to read and modify than other formats
Compare versions: Use comparison tool to track changes between versions
Check warnings: Schema warnings can indicate potential issues

For programmatic access¶

Cache CSRF tokens: Reuse tokens across multiple requests in a session
Handle errors gracefully: Check both service errors and validation results
Use specific versions: Always specify schema versions (ideally the latest) for reproducibility

Troubleshooting¶

Common validation errors¶

“Invalid HED tag”

Tag not found in specified schema version
Check for typos in tag names
Verify you’re using the correct schema version

“Comma and parentheses errors”

Missing or extra commas in HED string
Tags in a group must be comma-separated
Groups must be comma-separated
Parentheses must match

“Placeholder errors”

Placeholder can only be used in sidecars and definitions
Only one placeholder per string

File format issues¶

“Cannot parse file”

Ensure TSV files use tabs, not spaces or commas
Check for proper UTF-8 encoding
Verify JSON files are valid JSON

“Column not found”

A sidecar can have annotations for columns that are not in the .tsv
A .tsv file that has a column not deliberately skipped and is not defined in the sidecar generates a warning
Column names are case-sensitive

Schema issues¶

“Cannot load schema”

Check internet connection for downloading standard schemas
For custom schemas, verify XML/mediawiki format
Ensure schema file is complete and well-formed

Service issues¶

“CSRF token error”

Obtain fresh token from services endpoint
Include token in request headers
Check that cookie is preserved across requests

“Service timeout”

Large files may timeout on web interface
Consider using Python HEDTools for large datasets ir the JavaScript HEDTools for browser-based validation
Break operations into smaller batches

HED online tools user guide¶

Quick links¶

Table of contents¶

Getting started¶

Accessing HED online tools¶

Tool organization¶

Common features¶

Schema selection¶

Tag formats¶

Definition expansion¶

Tabular formats¶

Limiting errors¶

Working with tabular files¶

Validate an events file¶

Check HED quality¶

Assemble HED annotations¶

Search HED¶

Generate sidecar template¶

Execute remodel script¶

Working with sidecars¶

Validate a sidecar¶

Convert sidecar to long¶

Convert sidecar to short¶

Extract spreadsheet from sidecar¶

Merge spreadsheet with sidecar¶

Working with spreadsheets¶

Validate a spreadsheet¶

Convert spreadsheet to long¶

Convert spreadsheet to short¶

Working with HED strings¶

Validate a HED string¶

Convert string to long¶

Convert string to short¶

Search HED strings¶

Working with HED schemas¶

Validate a schema¶

Convert a schema¶

Compare schemas¶

Using RESTful services¶

Service setup¶

Request format¶

Available services¶

Service parameters¶

Service responses¶

Best practices¶

For annotation development¶

For schema development¶

For programmatic access¶

Troubleshooting¶

Common validation errors¶

File format issues¶

Schema issues¶

Service issues¶

Getting help¶