Tools

Utility functions and data processing tools for HED operations.

Analysis tools

EventManager

class EventManager(input_data, hed_schema, extra_defs=None)

Bases: object

Manager of events of temporal extent.

static compress_strings(list_to_compress)

Compress a list of lists of strings into a single str with comma-separated elements.

Parameters:

list_to_compress (list) – List of lists of HED str to turn into a list of single HED strings.

Returns:

List of same length as list_to_compress with each entry being a str.

Return type:

list

get_type_defs(types)

Return a list of definition names (lower case) that correspond to any of the specified types.

Parameters:

types (list or None) – List of tags that are treated as types such as ‘Condition-variable’

Returns:

List of definition names (lower-case) that correspond to the specified types

Return type:

list

str_list_to_hed(str_list)

Create a HedString object from a list of strings.

Parameters:

str_list (list) – A list of strings to be concatenated with commas and then converted.

Returns:

The converted list.

Return type:

Union[HedString, None]

unfold_context(remove_types=None)

Unfold the event information into a tuple based on context.

Parameters:

remove_types (list or None) – List of types to remove. If None, defaults to empty list.

Returns:

Union[list(str), HedString]: The information without the events of temporal extent. Union[list(str), HedString, None]: The onsets of the events of temporal extent. Union[list(str), HedString, None]: The ongoing context information.

Return type:

tuple[Union[list(str), HedString], Union[list(str), HedString, None], Union[list(str), HedString, None]]

HedTagManager

class HedTagManager(event_manager, remove_types=None)

Bases: object

Manager for the HED tags from a columnar file.

get_hed_obj(hed_str, remove_types=False, remove_group=False)

Return a HED string object with the types removed.

Parameters:

• hed_str (str) – Represents a HED string.

• remove_types (bool) – If False (the default), do not remove the types managed by this manager.

• remove_group (bool) – If False (the default), do not remove the group when removing a type tag, otherwise remove its enclosing group.

get_hed_objs(include_context=True, replace_defs=False)

Return a list of HED string objects of same length as the tabular file.

Parameters:

• include_context (bool) – If True (default), include the Event-context group in the HED string.

• replace_defs (bool) – If True (default=False), replace the Def tags with Definition contents.

Returns:

list - List of HED strings of same length as tabular file.

HedTypeManager

class HedTypeManager(event_manager)

Bases: object

Manager for type factors and type definitions.

add_type(type_name)

Add a type variable to be managed by this manager.

Parameters:

type_name (str) – Type tag name of the type to be added.

get_factor_vectors(type_tag, type_values=None, factor_encoding='one-hot')

Return a DataFrame of factor vectors for the indicated HED tag and values.

Parameters:

• type_tag (str) – HED tag to retrieve factors for.

• type_values (list or None) – The values of the tag to create factors for or None if all unique values.

• factor_encoding (str) – Specifies type of factor encoding (one-hot or categorical).

Returns:

DataFrame containing the factor vectors as the columns.

Return type:

Union[pd.DataFrame, None]

get_type(type_tag)

Returns the HedType variable associated with the type tag.

Parameters:

type_tag (str) – HED tag to retrieve the type for.

Returns:

the values associated with this type tag.

Return type:

Union[HedType, None]

get_type_def_names(type_var)

Return the definitions associated with a particular type tag.

Parameters:

type_var (str) – The name of a type tag such as Condition-variable.

Returns:

Names of definitions that use this type.

Return type:

list

get_type_tag_factor(type_tag, type_value)

Return the HedTypeFactors a specified value and extension.

Parameters:

• type_tag (str or None) – HED tag for the type.

• type_value (str or None) – Value of this tag to return the factors for.

summarize_all(as_json=False)

Return a dictionary containing the summaries for the types managed by this manager.

Parameters:

as_json (bool) – If False (the default), return as an object otherwise return as a JSON string.

Returns:

Dictionary with the summary.

Return type:

Union[dict, str]

property types

Return a list of types managed by this manager.

Returns:

Type tags names.

Return type:

list

TabularSummary

class TabularSummary(value_cols=None, skip_cols=None, name='', categorical_limit=None)

Bases: object

Summarize the contents of columnar files.

extract_sidecar_template() → dict

Extract a BIDS sidecar-compatible dictionary.

Returns:

A sidecar template that can be converted to JSON.

Return type:

dict

static extract_summary(summary_info) → TabularSummary

Create a TabularSummary object from a serialized summary.

Parameters:

summary_info (dict or str) – A JSON string or a dictionary containing contents of a TabularSummary.

Returns:

contains the information in summary_info as a TabularSummary object.

Return type:

TabularSummary

static get_columns_info(dataframe, skip_cols=None) → dict[str, dict]

Extract unique value counts for columns.

Parameters:

• dataframe (DataFrame) – The DataFrame to be analyzed.

• skip_cols (list) – List of names of columns to be skipped in the extraction.

Returns:

A dictionary with keys that are column names (strings) and values that

are dictionaries of unique value counts.

Return type:

dict[str, dict]

get_number_unique(column_names=None) → dict

Return the number of unique values in columns.

Parameters:

column_names (list, None) – A list of column names to analyze or all columns if None.

Returns:

Column names are the keys and the number of unique values in the column are the values.

Return type:

dict

get_summary(as_json=False) → dict | str

Return the summary in dictionary format.

Parameters:

as_json (bool) – If False, return as a Python dictionary, otherwise convert to a JSON dictionary.

Returns:

A dictionary containing the summary information or a JSON string if as_json is True.

Return type:

Union[dict, str]

static make_combined_dicts(file_dictionary, skip_cols=None) → tuple[TabularSummary, dict[str, TabularSummary]]

Return combined and individual summaries.

Parameters:

• file_dictionary (FileDictionary) – Dictionary of file name keys and full path.

• skip_cols (list) – Name of the column.

Returns:

• A combined summary of all files in the dictionary.

• A dictionary where keys are file names and values are individual TabularSummary objects.

Return type:

tuple[TabularSummary, dict[str, TabularSummary]]

update(data, name=None)

Update the counts based on data (DataFrame, filename, or list of filenames).

Parameters:

• data (DataFrame, str, or list) – DataFrame containing data to update.

• name (str) – Name of the summary.

update_summary(tab_sum)

Add TabularSummary values to this object.

Parameters:

tab_sum (TabularSummary) – A TabularSummary to be combined.

Notes

• The value_cols and skip_cols are updated as long as they are not contradictory.

• A new skip column cannot be used.

HedType

class HedType(event_manager, name, type_tag='condition-variable')

Bases: object

Manager of a type variable and its associated context.

get_summary()

get_type_def_names()

Return the type defs names

get_type_factors(type_values=None, factor_encoding='one-hot')

Create a dataframe with the indicated type tag values as factors.

Parameters:

• type_values (list or None) – A list of values of type tags for which to generate factors.

• factor_encoding (str) – Type of factor encoding (one-hot or categorical).

Returns:

Contains the specified factors associated with this type tag.

Return type:

pd.DataFrame

static get_type_list(type_tag, item)

Find a list of the given type tag from a HedTag, HedGroup, or HedString.

Parameters:

• type_tag (str) – a tag whose direct items you wish to remove

• item (HedTag or HedGroup) – The item from which to extract condition type_variables.

Returns:

List of the items with this type_tag

Return type:

list

get_type_value_factors(type_value)

Return the HedTypeFactors associated with type_name or None.

Parameters:

type_value (str) – The tag corresponding to the type’s value (such as the name of the condition variable).

Returns:

Union[HedTypeFactors, None]

get_type_value_level_info(type_value)

Return type variable corresponding to type_value.

Parameters:

type_value (str)

Returns:

get_type_value_names()

property total_events

property type_variables

FileDictionary

class FileDictionary(collection_name, file_list, key_indices=(0, 2), separator='_')

Bases: object

A file dictionary keyed by entity pair indices.

Notes

• The entities are identified as 0, 1, … depending on order in the base filename.

• The entity key-value pairs are assumed separated by ‘_’ unless a separator is provided.

create_file_dict(file_list, key_indices, separator)

Create new dict based on key indices.

Parameters:

• file_list (list) – Paths of the files to include.

• key_indices (tuple) – A tuple of integers representing order of entities for key.

• separator (str) – The separator used between entities to form the key.

property file_dict

Dictionary of path values in this dictionary.

property file_list

List of path values in this dictionary.

get_file_path(key)

Return file path corresponding to key.

Parameters:

key (str) – Key used to retrieve the file path.

Returns:

File path.

Return type:

str

iter_files()

Iterator over the files in this dictionary.

Yields:

- str – Key into the dictionary. - file: File path.

key_diffs(other_dict)

Return symmetric key difference with another dict.

Parameters:

other_dict (FileDictionary)

Returns:

The symmetric difference of the keys in this dictionary and the other one.

Return type:

list

property key_list

Keys in this dictionary.

static make_file_dict(file_list, key_indices=(0, 2), separator='_')

Return a dictionary of files using entity keys.

Parameters:

• file_list (list) – Paths to files to use.

• key_indices (tuple) – Positions of entities to use for key.

• separator (str) – Separator character used to construct key.

Returns:

Key is based on key indices and value is a full path.

Return type:

dict

static make_key(key_string, indices=(0, 2), separator='_')

Create a key from specified entities.

Parameters:

• key_string (str) – The string from which to extract the key (usually a filename or path).

• indices (tuple) – Positions of entity pairs to use as key.

• separator (str) – Separator between entity pairs in the created key.

Returns:

The created key.

Return type:

str

property name

Name of this dictionary.

output_files(title=None)

Return a string with the output of the list.

Parameters:

title (None, str) – Optional title.

Returns:

The dictionary in string form.

Return type:

str

BIDS tools

BidsDataset

class BidsDataset(root_path, schema=None, suffixes=<object object>, exclude_dirs=<object object>)

Bases: object

A BIDS dataset representation primarily focused on HED evaluation.

root_path

Real root path of the BIDS dataset.

Type:

str

schema

The schema used for evaluation.

Type:

HedSchema or HedSchemaGroup

file_groups

A dictionary of BidsFileGroup objects with a given file suffix.

Type:

dict

get_file_group(suffix)

Return the file group of files with the specified suffix.

Parameters:

suffix (str) – Suffix of the BidsFileGroup to be returned.

Returns:

The requested tabular group.

Return type:

Union[BidsFileGroup, None]

get_summary()

Return an abbreviated summary of the dataset.

validate(check_for_warnings=False, schema=None)

Validate the dataset.

Parameters:

• check_for_warnings (bool) – If True, check for warnings.

• schema (HedSchema or HedSchemaGroup or None) – The schema used for validation.

Returns:

List of issues encountered during validation. Each issue is a dictionary.

Return type:

list

BidsFile

class BidsFile(file_path)

Bases: object

A BIDS file with entity dictionary.

file_path

Real path of the file.

Type:

str

suffix

Suffix part of the filename.

Type:

str

ext

Extension (including the .).

Type:

str

entity_dict

Dictionary of entity-names (keys) and entity-values (values).

Type:

dict

Notes

• This class may hold the merged sidecar giving metadata for this file as well as contents.

clear_contents()

Set the contents attribute of this object to None.

property contents

Return the current contents of this object.

get_entity(entity_name)

Return the entity value for the specified entity.

Parameters:

entity_name (str) – Name of the BIDS entity, for example task, run, or sub.

Returns:

Entity value if any, otherwise None.

Return type:

Union[str, None]

get_key(entities=None)

Return a key for this BIDS file given a list of entities.

Parameters:

entities (tuple) – A tuple of strings representing entities.

Returns:

A key based on this object.

Return type:

str

Notes

If entities is None, then the file path is used as the key.

set_contents(content_info=None, overwrite=False)

Set the contents of this object.

Parameters:

• content_info (Any) – JSON dictionary The contents appropriate for this object.

• overwrite (bool) – If False and the contents are not empty, do nothing.

Notes

• Do not set if the contents are already set and no_overwrite is True.

BidsFileGroup

class BidsFileGroup(root_path, file_list, suffix='events')

Bases: object

Container for BIDS files with a specified suffix.

suffix

The file suffix specifying the class of file represented in this group (e.g., events).

Type:

str

sidecar_dict

A dictionary of sidecars associated with this suffix .

Type:

dict

datafile_dict

A dictionary with values either BidsTabularFile or BidsTimeseriesFile.

Type:

dict

sidecar_dir_dict

Dictionary whose keys are directory paths and values are list of sidecars in the corresponding directory.

Type:

dict

static create_file_group(root_path, file_list, suffix)

summarize(value_cols=None, skip_cols=None)

Return a BidsTabularSummary of group files.

Parameters:

• value_cols (list) – Column names designated as value columns.

• skip_cols (list) – Column names designated as columns to skip.

Returns:

A summary of the number of values in different columns if tabular group.

Return type:

Union[TabularSummary, None]

Notes

• The columns that are not value_cols or skip_col are summarized by counting

the number of times each unique value appears in that column.

validate(hed_schema, extra_def_dicts=None, check_for_warnings=False)

Validate the sidecars and datafiles and return a list of issues.

Parameters:

• hed_schema (HedSchema) – Schema to apply to the validation.

• extra_def_dicts (DefinitionDict) – Extra definitions that come from outside.

• check_for_warnings (bool) – If True, include warnings in the check.

Returns:

A list of validation issues found. Each issue is a dictionary.

Return type:

list

validate_datafiles(hed_schema, extra_def_dicts=None, error_handler=None)

Validate the datafiles and return an error list.

Parameters:

• hed_schema (HedSchema) – Schema to apply to the validation.

• extra_def_dicts (DefinitionDict) – Extra definitions that come from outside.

• error_handler (ErrorHandler) – Error handler to use.

Returns:

A list of validation issues found. Each issue is a dictionary.

Return type:

list

Notes: This will clear the contents of the datafiles if they were not previously set.

validate_sidecars(hed_schema, extra_def_dicts=None, error_handler=None)

Validate merged sidecars.

Parameters:

• hed_schema (HedSchema) – HED schema for validation.

• extra_def_dicts (DefinitionDict) – Extra definitions.

• error_handler (ErrorHandler) – Error handler to use.

Returns:

A list of validation issues found. Each issue is a dictionary.

Return type:

list

BidsSidecarFile

class BidsSidecarFile(file_path)

Bases: BidsFile

A BIDS sidecar file.

clear_contents()

Set the contents attribute of this object to None.

property contents

Return the current contents of this object.

get_entity(entity_name)

Return the entity value for the specified entity.

Parameters:

entity_name (str) – Name of the BIDS entity, for example task, run, or sub.

Returns:

Entity value if any, otherwise None.

Return type:

Union[str, None]

get_key(entities=None)

Return a key for this BIDS file given a list of entities.

Parameters:

entities (tuple) – A tuple of strings representing entities.

Returns:

A key based on this object.

Return type:

str

Notes

If entities is None, then the file path is used as the key.

static is_hed(json_dict)

Return True if the json has HED.

Parameters:

json_dict (dict) – A dictionary representing a JSON file or merged file.

Returns:

True if the dictionary has HED or HED_assembled as a first or second-level key.

Return type:

bool

is_sidecar_for(obj)

Return True if this is a sidecar for obj.

Parameters:

obj (BidsFile) – A BidsFile object to check.

Returns:

True if this is a BIDS parent of obj and False otherwise.

Return type:

bool

Notes

• A sidecar is a sidecar for itself.

static merge_sidecar_list(sidecar_list, name='merged_sidecar.json')

Merge a list of sidecars into a single sidecar.

Parameters:

• sidecar_list (list) – A list of Sidecar objects.

• name (str) – The name of the merged sidecar.

Returns:

A sidecar constructed from the merged list.

Return type:

Union[Sidecar, None]

set_contents(content_info=None, name='unknown', overwrite=False)

Set the contents of the sidecar.

Parameters:

• content_info (dict, or None) – If None, create a Sidecar from the object’s file-path.

• name (str) – The name of the sidecar.

• overwrite (bool) – If True, overwrite contents if already set.

Notes

• The handling of content_info is as follows:

  • None: This object’s file_path is used.

  • dict: This is interpreted as a JSON dictionary.

BidsTabularFile

class BidsTabularFile(file_path)

Bases: BidsFile

A BIDS tabular file including its associated sidecar.

clear_contents()

Set the contents attribute of this object to None.

property contents

Return the current contents of this object.

get_entity(entity_name)

Return the entity value for the specified entity.

Parameters:

entity_name (str) – Name of the BIDS entity, for example task, run, or sub.

Returns:

Entity value if any, otherwise None.

Return type:

Union[str, None]

get_key(entities=None)

Return a key for this BIDS file given a list of entities.

Parameters:

entities (tuple) – A tuple of strings representing entities.

Returns:

A key based on this object.

Return type:

str

Notes

If entities is None, then the file path is used as the key.

set_contents(content_info=None, overwrite=False)

Set the contents of this tabular file (a TabularInput object). It’s sidecar should already be set.

Parameters:

• content_info (None) – This always uses the internal file_path to create the contents.

• overwrite (bool) – If False (The Default), do not overwrite existing contents if any.

set_sidecar(sidecar)

Set the sidecar for this tabular file.

Parameters:

sidecar (Sidecar) – The sidecar for this tabular file.

BIDS utilities

get_candidates(source_dir, tsv_file_dict)

get_merged_sidecar(root_path, tsv_file)

get_schema_from_description(root_path)

group_by_suffix(file_list)

Group files by suffix.

Parameters:

file_list (list) – List of file paths.

Returns:

Dictionary with suffixes as keys and file lists as values.

Return type:

dict

matches_criteria(json_file_dict, tsv_file_dict)

parse_bids_filename(file_path)

Split a filename into BIDS-relevant components.

Parameters:

file_path (str) – Path to be parsed.

Returns:

Dictionary with keys ‘basename’, ‘suffix’, ‘prefix’, ‘ext’, ‘bad’, and ‘entities’.

Return type:

dict

Notes

• Splits into BIDS suffix, extension, and a dictionary of entity name-value pairs.

update_entity(name_dict, entity)

Update the dictionary with a new entity.

Parameters:

• name_dict (dict) – Dictionary of entities.

• entity (str) – Entity to be added.

walk_back(root_path, file_path)