hyped.core.graph module

Defines the structure of data flow graphs and their components.

This module provides the DataFlowGraph class and related components, which represent the structure of a data processing workflow. The graph consists of nodes (data processors) and edges (data flow between processors).

class hyped.core.graph.DataFlowGraph(*args, backend=None, **kwargs)

Bases: MultiDiGraph, AbstractDataFlowGraph

A multi-directed graph representing a data flow of data processors.

This class is used internally to define a directed acyclic graph (DAG) where nodes represent data processors of BaseDataProcessor type, and edges define the data flow between these processors.

class GraphAttribute(value)

Bases: str, Enum

Enum representing properties of the data flow graph.

SRC_NODE_ID = 'src_node_id'

Property representing the source node ID.

This property identifies the source node ID of an edge in the graph.

class NodeAttribute(value)

Bases: str, Enum

Enum representing properties of a node in the data flow graph.

DEPTH = 'depth'

Represents the depth of the node within the data flow graph.

Type: int

This property indicates the level of the node in the graph, with the root node having a depth of 0. It is used to understand the hierarchical position of the node relative to other nodes in the data flow.

IN_FEATURE_TYPE = 'in_feature_type'

Represents the input features associated with the node.

Type: MappingType

This property contains the input features required by the data processor, in the form of a mapping type instance. It defines the structure and types of the inputs expected by the node.

NODE_OBJ = 'node_object'

The object associated with the node.

The value of this property is dependent on the type of node. For nodes of type NodeType.DATA_PROCESSOR, this property refers to the processor instance of the node.

NODE_TYPE = 'node_type'

Indicates the type of node.

Type: NodeType

This property indicates the type of data processor. It helps in categorizing and identifying the nature of the processor in the data flow graph.

OUT_FEATURE_TYPE = 'out_feature_type'

Represents the output features associated with the node.

Type: DType

This property contains the output feature produced by the data processor, as a DType instance. It defines the structure and types of data that is generated by this node.

OUT_PARTITION = 'out_partition'

Represents the partition the node points to.

Type: PartitionId

This property indicates the specific partition of the data flow graph that the node output belongs to.

PARTITION = 'partition'

Represents the partition to which the node belongs.

Type: PartitionId

This property indicates the specific partition of the data flow graph that the node is part of, which can be used to group nodes by different semantics.

class NodeType(value)

Bases: str, Enum

Enum representing types of nodes in the data flow graph.

CAST = 'CAST_NODE'

Represents a cast node in the data flow graph.

This type of node is responsible for type conversion within the data flow. It transforms data from one type to another, ensuring compatibility between different nodes or preparing the data for specific processing requirements.

COLLECT = 'COLLECT_NODE'

Represents a collect node in the data flow graph.

This type of node collects features from multiple upstream nodes into a single (nested) feature. This enables the combination of outputs from various sources or transformations into a unified structure, which can be further processed downstream.

CONST = 'CONST_NODE'

Represents a constant node in the data flow graph.

This type of node introduces constant values into the data flow, serving as fixed inputs to the subsequent processing stages.

DATA_AGGREGATOR = 'DATA_AGGREGATOR_NODE'

Represents a data aggregator node in the data flow graph.

This type of node is responsible for aggregating samples. Aggregator nodes typically perform dataset-wide computations.

DATA_AUGMENTOR = 'DATA_AUGMENTOR_NODE'

Represents a data augmentor node in the data flow graph.

This type of node is responsible for modifying the dataset by generating new samples from existing ones or filtering out certain samples. Data augmentor nodes are used to expand or contract the dataset.

DATA_PROCESSOR = 'DATA_PROCESSOR_NODE'

Represents a data processor node in the data flow graph.

This type of node represents a data processing component within the data flow graph. Data processors perform specific transformations on input data and produce output data based on defined processing logic.

DEBUG = 'DEBUG_NODE'

Represents a debug node for inspecting data flow.

Debug nodes do not transform data or produce integrated outputs. They facilitate monitoring and debugging through mechanisms like logging or asserting.

SOURCE = 'SOURCE_NODE'

Represents a source node in the data flow graph.

This type of node acts as the starting point of the data flow graph, typically representing raw input data sources.

TRACE = 'TRACE_NODE'

Represents a trace node in the data flow graph.

This type of node is responsible for tracing values through different partitions of the graph. It transforms data of a specific partition into the index-space of a target partition.

class Partition(value)

Bases: str, Enum

Enum representing predefined partitions in the data flow graph.

AGGREGATED = 'eb819333-b501-1c18-8c53-c786ed62c2f9'

Represents the partition containing aggregated values.

This partition includes all aggregated values in a data flow, i.e. nodes that process the output of aggregator nodes.

CONST = '8826d916-cdfb-21c6-c1ff-91a761565a70'

Represents the partition containing all constant nodes.

This partition includes nodes that hold constant values used in the data processing flow. This convers the actual constant nodes introducing constant values to the flow, as well as computations on only constant values.

DEFAULT = '2416da6e-c212-cddb-8d88-00160eb686b2'

Represents the default partition for nodes.

This partition is assigned to the source node and is inherited by its sub-graph.

add_node(node_obj: Any, node_type: NodeType, inputs: dict[str, str], output_dtype: DType, partition: str, out_partition: str, node_id: str) → str

Adds a new node to the data flow graph.

This function creates a node in the graph with specified attributes such as type, inputs, and output data type. It ensures the graph remains a Directed Acyclic Graph (DAG) after adding the node. The function also adds the dependency edges between the new node and its input nodes.

Parameters:

• node_obj (Any) – The object associated with the node.

• node_type (DataFlowGraph.NodeType) – The type of the node.

• inputs (dict[str, NodeOd]) – A dictionary mapping input names to NodeId that represent dependencies of this node on other nodes in the graph.

• partition (PartitionId) – The partition of the node.

• out_partition (PartitionId) – The output partition of the node.

• output_dtype (DType) – The output data type produced by this node.

• node_id (None | NodeId) – The unique identifier for the node.

Returns:

The node id of the added node.

Return type:

NodeId

add_source_node(dtype: DType, node_id: str) → str

Add a the source node to the graph.

This method adds a source node to the graph, which acts as the initial data provider for the data flow.

Parameters:

• dtype (DType) – The data type representing the source features.

• node_id (NodeId) – The id of the node.

Returns:

The node id of the added source node.

Return type:

NodeId

Raises:

RuntimeError – If the graph already contains a source node

dependency_graph(nodes: set[str], stop_nodes: set[str] = {}) → DataFlowGraph

Generate the dependency subgraph for a given set of nodes.

This method generates a subgraph that includes all nodes that the specified nodes depend on, either directly or indirectly, up to any defined cut-off points.

Parameters:

• nodes (set[NodeId]) – The node IDs for which to generate the dependency graph.

• stop_nodes (set[NodeId]) – A set of node IDs that serve as cut-off points in the traversal. When a dependency chain reaches any node in :node:`stop_nodes`, it stops there, excluding that node’s dependencies from the subgraph. This allows limiting the scope of the dependency graph by excluding deeper dependencies beyond these nodes.

Returns:

A subgraph representing the dependencies of the specified nodes, excluding paths beyond any nodes in stop_nodes.

Return type:

DataFlowGraph

property depth: int

Computes the total depth of the data flow graph.

The depth is defined as the maximum level of any node in the graph, where the root node has a depth of 0. This property calculates the depth by finding the maximum depth attribute among all nodes in the graph.

Returns:

The total depth of the graph.

Return type:

int

drop_partition(partition: str) → DataFlowGraph

Drop a specified partition from the graph.

This method creates a subgraph from the current graph by excluding nodes that belong to a specified partition.

Parameters:

partition (PartitionId) – The partition identifier.

Returns:

The subgraph excluding nodes of the specified partition.

Return type:

DataFlowGraph

format(format: str | Template = "[{{ node_id[:4] }}] {% if node_type == 'SOURCE_NODE' %}Source{% else %}{{ node_object }}{% endif %}") → MultiDiGraph

Creates a formatted copy with node labels generated using a Jinja2 template.

This method applies a user-specified format to the attributes of each node in the graph, producing a new graph where each node has a label attribute rendered from the provided template. The edges in the graph are copied without modification.

Attributes available in the template are specified by the NodeAttribute enum.

Parameters:

format (str | Template) – A Jinja2 template string or a precompiled Jinja2 Template object. The template can reference any of the node attributes, as well as the node_id. Defaults to DEFAULT_NODE_FORMAT.

Returns:

A new graph where nodes have a label attribute generated

from the provided template, and edges are identical to those in the original graph.

Return type:

nx.MultiDiGraph

classmethod from_dict(data: dict) → DataFlowGraph

Deserializes a dictionary into a data flow graph.

This method converts a dictionary (typically one generated by to_dict) back into a DataFlowGraph instance. It handles the deserialization of node types, partitions, feature types, and node objects. For each node, the method reconstructs its associated object based on the node’s type. Additionally, edge data is deserialized and used to re-establish links between the nodes.

Parameters:

data (dict) – A dictionary representation of a data flow graph, including node data and links between them.

Returns:

The deserialized data flow graph.

Return type:

DataFlowGraph

get_output_dtype(node_id: str) → None | DType

Helper function to get the output data type of a node.

Parameters:

node_id (NodeId) – The id of the node.

Returns:

The output data type of the node.

Return type:

None | DType

Raises:

RuntimeError – If the node id is not contained in the graph.

get_partition(partition: str) → DataFlowGraph

Extract a subgraph containing only nodes from a specific partition.

This method creates a subgraph from the current graph by selecting nodes that belong to a specified partition.

Parameters:

partition (PartitionId) – The partition identifier.

Returns:

The subgraph containing nodes of the specified partition.

Return type:

DataFlowGraph

recompute_depths() → None

Recompute the depth of all nodes in the data flow graph.

This method recalculates the depth of each node based on the topological order of the graph. The depth of a node is defined as the length of the longest path from the source node to the node.

property src_dtype: MappingType

Get the source data type.

This property retrieves the data type of the source node in the data flow graph. The source data type defines the structure and types of features expected by the source node, which serve as the initial inputs to the data flow graph.

Returns:

The data type of the source node.

Return type:

MappingType

property src_node_id: None | str

Get the source node ID of the data flow graph.

This property returns the source node ID associated with the data flow graph. The source node is the entrypoint for inputs to the data flow.

Returns:

The uuid of the source node. None if graph has no source node.

Return type:

None | NodeId

subgraph_in_edges(subgraph: DataFlowGraph) → list[tuple[int, int, str]]

Get incoming edges to a subgraph from nodes outside the subgraph.

This method returns a list of edges that point to nodes within the specified subgraph from nodes outside the subgraph.

Parameters:

subgraph (DataFlowGraph) – The subgraph of interest.

Returns:

The incoming edges to the subgraph.

Return type:

list[tuple[int, int, str]]

subgraph_out_edges(subgraph: DataFlowGraph) → list[tuple[int, int, str]]

Get outgoing edges from a subgraph to nodes outside the subgraph.

This method returns a list of edges that point from nodes within the specified subgraph to nodes outside the subgraph.

Parameters:

subgraph (DataFlowGraph) – The subgraph of interest.

Returns:

The outgoing edges from the subgraph.

Return type:

list[tuple[int, int, str]]

to_dict() → dict

Serializes the data flow graph into a dictionary representation.

This method generates a dictionary that represents the entire data flow graph, including the nodes and edges. Each node is serialized based on its type, with special handling for constants, casts, and various data processing nodes. Additionally, the feature types for each node are serialized.

Returns:

A dictionary representation of the data flow graph, including nodes

with their serialized objects and feature types, and links between them.

Return type:

dict

to_string(format: str = "[{{ node_id[:4] }}] {% if node_type == 'SOURCE_NODE' %}Source{% else %}{{ node_object }}{% endif %}", ascii_only: bool = True, vertical_chains: bool = True) → str

Generates a string representation of the graph.

This method provides a textual representation of the graph, with nodes and edges displayed in a human-readable format. Nodes are labeled according to the specified format string, and additional options allow customization of the output style.

Attributes available in the template are specified by the NodeAttribute enum.

Parameters:

• format (str) – A Jinja2 template string used to generate labels for the nodes. The template can reference any of the node attributes defined by the NodeAttribute enum, as well as the node_id. Defaults to DEFAULT_NODE_FORMAT.

• ascii_only (bool) – If True, the output will use only ASCII characters. Defaults to True.

• vertical_chains (bool) – If True, the output will display chains of nodes in a vertical layout for better readability. Defaults to True.

Returns:

A string representation of the graph, formatted according to the specified options.

Return type:

str

property width: int

Computes the width of the data flow graph.

The width is defined as the maximum number of nodes present at any single depth level in the graph. This property calculates the width by grouping nodes by their depth and finding the largest group.

Returns:

The maximum width of the graph.

Return type:

int

hyped.core.graph.logical_imply(x, y)

Returns the logical implication x -> y.