hyped.core.nodes.collect module

This module defines the CollectNode.

The CollectNode class is a specialized node that gathers and structures data from input references based on a predefined lookup structure. This structure can be a dictionary, list, tuple, or individual feature, and the node ensures that the output matches the specified data types as defined in the data flow graph.

class hyped.core.nodes.collect.CollectNode(config: None | U = None, **kwargs)

Bases: BaseNode[CollectNodeConfig]

Node for collecting and structuring inputs within a data flow graph.

CollectNode aggregates data from multiple inputs according to a defined structure specified in its configuration. It transforms the collected data into the desired output format, supporting nested mappings and sequences.

build_output_type(input_dtypes: dict[str, DType]) → tuple[DType, dict[str, DType]]

Construct the output data type based on the input structure.

This method recursively determines the type of each element in the structure specified by lookup, building complex types like mappings and sequences as needed. It leverages input references to map strings in lookup to actual data types from the graph.

Parameters:

input_dtypes (dict[str, DType]) – A dictionary mapping input names to their data types.

Returns:

A tuple containing the constructed output type and a

lookup for inputs that need to be casted to a different data type before collection.

Return type:

tuple[DType, dict[str, DType]]

call(*args: Any, **kwargs: Any) → Any

Raises an error, as direct calls are not supported for this node.

collect(ctx: RunContext, inputs: dict[str, Array]) → Array

Collect and arrange input data according to the node’s structure.

Recursively gathers data from the input arrays, organizing it into a nested format as specified in lookup. The method supports nested mappings and sequences, building them using PyArrow structures like StructArray and FixedSizeListArray.

Parameters:

• ctx (RunContext) – The execution context for the current operation, including output type.

• inputs (dict[str, pa.Array]) – Input data arrays indexed by the input names specified in lookup.

Returns:

A single PyArrow array that represents the collected data, structured according to lookup.

Return type:

pa.Array

Raises:

TypeError – If the structure or types in lookup are not compatible with the inputs.

property signature: Any

Raises an error, as signature is not supported for this node.

class hyped.core.nodes.collect.CollectNodeConfig(*, lookup: dict | list | tuple | str)

Bases: BaseNodeConfig

Configuration for CollectNode.

lookup: dict | list | tuple | str

Nested collect structure.

The structure has string identifiers matching the input arguments at the lowest level.

model_config = {'extra': 'forbid', 'validate_default': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].