hyped.core.builder module

This module implements the Data Flow Graph Builder.

class hyped.core.builder.DataFlowGraphBuilder(graph: None | DataFlowGraph = None)[source]

Bases: AbstractDataFlowGraphBuilder

The Data Flow Graph Builder.

The builder encapsulates higher-level logic for constructing cohesive data flow graphs. It abstracts away the low-level details of input preparation, type conversions, and intermediate data representation.

For instance, when adding a compute node, the builder may also model any necessary input preparation steps, such as ensuring that constant inputs are represented as constant nodes in the graph.

attach_graph_to_node(node_id: str, graph: DataFlowGraph) → DataFlowGraph[source]

Attach a data flow graph to a specific node within the current graph.

This method embeds a separate data flow graph (graph) into the current graph by attaching the source node of the graph to the node identified by node_id in the current graph. During this process, the nodes and edges of the graph are integrated into the current graph, preserving their relationships, configurations and node ids.

Parameters:

  • node_id (NodeId) – The ID of the target node in the current graph to which the source node of the graph will be attached.

  • graph (DataFlowGraph) – The data flow graph to be attached to the current graph.

Returns:

The updated graph after the attachment.

Return type:

DataFlowGraph

Raises:
cast(ref: ConcreteReference, dtype: DType, node_id: None | str = None) → ConcreteReference[source]

Add a cast node to the data flow graph.

This method adds a cast node, which transforms data from one type to another, ensuring compatibility between nodes or preparing the data for specific processing requirements. The method validates type compatibility and computes the resulting type if the cast is feasible.

Parameters:

  • ref (ConcreteReference) – A reference to the input data to be cast.

  • dtype (DType) – The target type to which the input data should be cast.

  • node_id (None | NodeId) – An optional unique identifier for the node. If not provided, a random UUID is generated. Defaults to None.

Returns:

A reference to the added cast node, allowing access to its output.

Return type:

ConcreteReference

Raises:

RuntimeError – If the source and target types are incompatible or if other casting constraints are violated.

collect(collect: dict[str, dict[str, NestedType] | list[NestedType] | tuple[NestedType] | T] | list[dict[str, NestedType] | list[NestedType] | tuple[NestedType] | T] | tuple[dict[str, NestedType] | list[NestedType] | tuple[NestedType] | T] | ConcreteReference | Any, dtype: None | DType = None, node_id: None | str = None) → ConcreteReference[source]

Adds a collect node to the data flow graph.

A collect node combines features from other nodes, organizing them into a nested structure as specified by the collect argument. It enables combining outputs from multiple nodes into a single, structured output. Additionally, the function processes the nested structure to check for constants to add to the graph before collecting.

Parameters:

  • collect (NestedType[ConcreteReference]) – A nested structure (e.g., lists, dictionaries) containing ConcreteReference objects that specify the features to collect.

  • dtype (None | DType) – The expected data type of the collect structure. If None, the function attempts to infer the type where possible.

  • node_id (None | NodeId) – A unique identifier for the node. If None, a random UUID is generated. Defaults to None.

Returns:

A reference to the newly created collect node.

Return type:

ConcreteReference

compute(obj: BaseNode, inputs: dict[str, ConcreteReference | Any], node_id: None | str = None) → ConcreteReference[source]

Add a compute node to the data flow graph.

This method adds a compute node, which performs a transformation or computation on input features, to the data flow graph. The node type is dynamically determined based on the class of the provided node object, and appropriate edges are created to define the data flow.

Parameters:

  • obj (BaseNode) – The compute node object, which defines the transformation logic. Supported subclasses include BaseDataProcessor, BaseDataAggregator, and BaseDataAugmentor.

  • inputs (dict[str, ConcreteReference]) – A mapping of input names to references for input features consumed by this compute node.

  • node_id (None | NodeId) – An optional unique identifier for the node. If not provided, a random UUID is generated.

Returns:

A reference instance to the added compute node, allowing access to its output features.

Return type:

ConcreteReference

Raises:

  • AssertionError – If the node object is invalid or its type cannot be determined.

  • RuntimeError – If any input reference do not belong to this data flow.

const(value: Any, dtype: DType, node_id: None | str = None) → ConcreteReference[source]

Adds a constant node to the data flow graph.

This function creates a node representing a constant value in the graph. The value is wrapped in a PyArrow array of the specified data type.

Parameters:

  • value (Any) – The constant value to be added to the graph.

  • dtype (DType) – The data type of the constant value.

  • node_id (None | NodeId) – A unique identifier for the node. If None, a random UUID is generated. Defaults to None.

Returns:

A reference to the newly created constant node.

Return type:

ConcreteReference

property graph: DataFlowGraph

The data flow graph.

source(dtype: DType, node_id: str | None = None) → ConcreteReference[source]

Add a the source node to the graph.

Parameters:

  • dtype (DType) – The data type representing the source features.

  • node_id (None | NodeId) – The id of the node, defaults to a random uuid.

Returns:

A reference object to the source node.

Return type:

ConcreteReference

trace(ref: ConcreteReference, tgt: str, node_id: None | str = None) → ConcreteReference[source]

Adds a trace node to the data flow graph.

The trace node follows the shortest path through the partition graph from the source partition of the provided reference to the specified target partition. It transforms the values associated with the reference according to the trace indices of the path.

If the source and target partitions are the same, no trace node is added, and the reference is returned as-is.

Parameters:

  • ref (ConcreteReference) – A reference to the value or node whose partition needs to be traced.

  • tgt (PartitionId) – The identifier of the target partition in the partition graph.

  • node_id (None | NodeId) – A unique identifier for the trace node. If None, a random UUID is generated. Defaults to None.

Returns:

A reference to the newly created trace node, or the original

reference if no trace is required.

Return type:

ConcreteReference