hyped.core.features.features module¶

Features Module.

This module defines the core classes and utilities for managing and validating different types of features. Features represent various data types and their structure, validated and resolved using Pydantic schemas. This allows users to define flexible and complex data structures while ensuring type safety and compatibility with the data flow system.

Features act as the high-level interface for describing data in a structured and extensible manner. Each feature is associated with a reference (BaseReference) that links it to a specific node or placeholder in the graph, providing precise context and origin information. These references can point to concrete nodes or act as forward declarations for features yet to be defined.

A core functionality of features is their ability to resolve complex type annotations, such as unions or nested structures, into concrete feature instances. Pydantic handles this resolution during validation, ensuring that users can define dynamic or multi-type features with confidence. This capability is particularly valuable when dealing with nested schemas or type unions, as it ensures the correct feature type is inferred and instantiated.

Additionally, the module includes utilities for registering and executing feature-specific methods dynamically, allowing for extensible functionality without introducing tight coupling or circular dependencies.

class hyped.core.features.features.BoolFeature(ref: BaseReference)[source]¶

Bases: PrimitiveFeature[PrimitiveType(_arrow_type=DataType(bool))]

A primitive feature representing a boolean value.

T = ~T¶

where(a: T, b: T) → T[source]¶

Selects values based on the Boolean condition.

For each element, if the corresponding value in this BoolFeature is True, the value from a is selected, otherwise, the value from b is used.

Parameters:

a (T) – The value or feature to select when the condition is True.
b (T) – The value or feature to select when the condition is False.

Returns:

A new feature with values conditionally selected from a or b.

Return type:

class hyped.core.features.features.ClassLabelFeature(ref: BaseReference)[source]¶

Bases: Int64Feature

A base class for defining strongly-typed categorical class labels.

This class provides a mechanism for defining and validating class labels, typically used to represent categorical data in datasets. Each class label corresponds to an integer ID and is mapped to an Int64Feature instance. Subclasses of ClassLabelFeature are intended to define the set of class labels in a similar way to an IntEnum.

Subclasses automatically inherit functionality for validation and schema generation, ensuring that the defined labels are consistent and complete. Missing label IDs are identified and handled during validation.

Defining Class Labels¶

Subclassing ClassLabelFeature allows users to define class labels as attributes, similar to defining members in an Enum:

Example:

class Labels(ClassLabelFeature):
    FIRST = 0
    SECOND = 1

# This defines a feature with two class labels:
# - Label `FIRST` corresponds to ID 0.
# - Label `SECOND` corresponds to ID 1.

classmethod from_names(names: list[str]) → type[ClassLabelFeature][source]¶

Dynamically creates class label type from a list of class label names.

This method dynamically defines a custom ClassLabelFeature subclass by providing a list of class label names. Each label is automatically assigned a unique integer ID starting from 0, based on its position in the list.

This is particularly useful when the class labels are dynamically generated or need to be defined programmatically, instead of being hardcoded as class-level attributes.

Parameters:: names (list[str]) – A list of class label names. Each name in the list represents a unique label, and the index of the name determines its corresponding integer ID.
Returns:: A dynamically created subclass of ClassLabelFeature, where each label name in the input list is assigned as a class-level attribute with its corresponding integer ID as the value.
Return type:: type[ClassLabelFeature]

Example:

from my_module import ClassLabelFeature

# Define class labels programmatically
label_names = ["NEGATIVE", "NEUTRAL", "POSITIVE"]
CustomLabels = ClassLabelFeature.from_names(label_names)

# Access the dynamically created class labels
print(CustomLabels.NEGATIVE)  # Output: 0
print(CustomLabels.NEUTRAL)  # Output: 1
print(CustomLabels.POSITIVE)  # Output: 2

class hyped.core.features.features.ComparableFeatureMixin[source]¶

Bases: object

Mixin providing comparison operations for primitive features.

This mixin defines methods for comparing primitive features and for obtaining minimum and maximum values. Subclasses implementing primitive features can extend their functionality by inheriting from this mixin.

max() → Self[source]¶

Returns the maximum value of the feature.

Returns:: A new feature instance representing the maximum value.
Return type:: Self

min() → Self[source]¶

Returns the minimum value of the feature.

Returns:: A new feature instance representing the minimum value.
Return type:: Self

class hyped.core.features.features.ExcludeFieldIf(condition: Callable[[BaseConfig, dict[str, Feature], ValidationSession], bool])[source]¶

Bases: AfterValidator

A Pydantic validator that conditionally excludes a field from the mapping.

This class applies a callable condition to determine whether a field should be excluded from the mapping. The condition is evaluated with the node’s configuration (BaseConfig) and the validation session (ValidationSession) as inputs, returning True to exclude the field or False to include it.

For an usage example, see _MappingFeature.

func: Callable[[Any], Any] | Callable[[Any, ValidationInfo[Any]], Any]¶

class hyped.core.features.features.Feature(ref: BaseReference)[source]¶

Bases: MethodRegistryMixin, Generic[DataType]

Base class for defining features in a data flow graph.

A feature serves as the primary user interface for defining and working with structured data in a data flow system. It wraps a reference instance (BaseReference) that describes its origin in the graph, enabling precise modeling of the flow of data between nodes. The data type (dtype) is inferred directly from the reference.

Features are responsible for:

Validation: Features use Pydantic schemas to validate the data structure and type, ensuring compatibility with the defined feature.
Resolution: Features can be resolved to concrete instances using Pydantic’s validation mechanism. This resolution process is critical when users define complex data structures, such as unions of features or nested structures. Pydantic ensures that these annotations are validated and resolved to the appropriate concrete feature type.
Extensibility: Through the dynamic method registry, features support the addition of custom methods for extended functionality. This decouples feature definitions from specific implementation details, avoiding circular dependencies.

property dtype: DataType¶: The data type of the feature inferred from the reference.

filter(condition: BoolFeature) → Self[source]¶

Filters elements based on a boolean condition.

Parameters:: condition (Bool) – A boolean array indicating which elements to retain.
Returns:: The filtered values that satisfy the given condition.
Return type:: T

ref: BaseReference¶: The reference to the feature.

class hyped.core.features.features.Float32Feature(ref: BaseReference)[source]¶

Bases: PrimitiveFeature[PrimitiveType(_arrow_type=DataType(float))], ComparableFeatureMixin, StatisticalFeatureMixin

A primitive feature representing a 32-bit floating-point number.

mean() → Float32Feature[source]¶

Calculates the mean (average) of the feature’s values.

This method computes the average of the values associated with the feature.

Returns:: A new feature representing the mean of the original feature’s values.
Return type:: Float32Feature

class hyped.core.features.features.Float64Feature(ref: BaseReference)[source]¶

Bases: PrimitiveFeature[PrimitiveType(_arrow_type=DataType(double))], ComparableFeatureMixin, StatisticalFeatureMixin

A primitive feature representing a 64-bit floating-point number.

class hyped.core.features.features.Int16Feature(ref: BaseReference)[source]¶

Bases: PrimitiveFeature[PrimitiveType(_arrow_type=DataType(int16))], ComparableFeatureMixin, StatisticalFeatureMixin

A primitive feature representing a signed 16-bit integer.

class hyped.core.features.features.Int32Feature(ref: BaseReference)[source]¶

Bases: PrimitiveFeature[PrimitiveType(_arrow_type=DataType(int32))], ComparableFeatureMixin, StatisticalFeatureMixin

A primitive feature representing a signed 32-bit integer.

class hyped.core.features.features.Int64Feature(ref: BaseReference)[source]¶

Bases: PrimitiveFeature[PrimitiveType(_arrow_type=DataType(int64))], ComparableFeatureMixin, StatisticalFeatureMixin

A primitive feature representing a signed 64-bit integer.

class hyped.core.features.features.Int8Feature(ref: BaseReference)[source]¶

Bases: PrimitiveFeature[PrimitiveType(_arrow_type=DataType(int8))], ComparableFeatureMixin, StatisticalFeatureMixin

A primitive feature representing a signed 8-bit integer.

hyped.core.features.features.MappingFeature[source]¶: alias of _MappingFeature

class hyped.core.features.features.PrimitiveFeature(ref: BaseReference)[source]¶

Bases: Feature[DataType]

Base class for primitive feature types.

The PrimitiveFeature class extends Feature to represent features with primitive data types.

class hyped.core.features.features.SequenceFeature(ref: BaseReference)[source]¶

Bases: Sequence[T], Feature[SequenceType]

A feature representing a sequence of items.

The SequenceFeature class models a feature where the data type is a sequence, supporting indexing, slicing, and length operations while maintaining type safety and feature reference consistency.

U = ~U¶

foreach(fn: Callable[[T], U]) → SequenceFeature[U][source]¶

Apply a transformation function to each element in the sequence.

This method applies the provided transformation function to each element in the sequence.

Parameters:: fn (Callable[[T], U]) – A function to apply to each element of the sequence.
Returns:: A new SequenceFeature instance containing the transformed sequence, with the same structural layout as the original.
Return type:: SequenceFeature[U]

index(val: T, default: None | T = None) → Int32Feature[source]¶

Returns a feature representing the first index of a given value in the sequence.

This method is used to dynamically resolve the index of a specific item (val) within the sequence during execution.

Parameters:

val (T) – The value to search for within the sequence.
default (None | T, optional) – The value to return if val is not found in the sequence. If not provided, a ValueError will be raised if val is not found

Returns:

A feature that resolves to the zero-based index of the first occurrence of val in the sequence.

Return type:

Int32Feature

Raises:

ValueError – If val is not found in the sequence.

length() → int | Int32Feature[source]¶

Returns the length of the sequence.

Returns:: The number of elements in the sequence. An integer is returned if the length is fixed. Otherwise a feature is returned which resolves to the sequence length during execution.
Return type:: int | Int32Feature

max() → T[source]¶

Returns the maximum value in the sequence.

Returns:: A feature representing the maximum value in the sequence.
Return type:: T

min() → T[source]¶

Returns the minimum value in the sequence.

Returns:: A feature representing the minimum value in the sequence.
Return type:: T

pad(fill_value: T, length: None | int = None) → SequenceFeature[T][source]¶

Pad the sequence to a specified length with a given fill value.

If length is provided, the sequence is padded to the specified length. If length is None, the sequence is padded to match the length of the longest sequence in the current batch.

Parameters:

fill_value (T) – The value to use for padding.
length (None | int) – The desired length to pad the sequence to. Defaults to None, in which case the longest sequence in the batch is used.

Returns:

A new SequenceFeature instance containing the padded sequence.

Return type:

SequenceFeature[T]

ref: BaseReference¶: The reference to the feature.

sum() → T[source]¶

Returns the sum of the sequence.

Returns:: A feature representing the sum of the sequence.
Return type:: T

unpack() → T[source]¶

unpack(return_index: Literal[False]) → T

unpack(return_index: Literal[True]) → tuple[T, Int32Feature]

Unpack the sequence into its values.

This method retrieves the values of the sequence, and if return_index is set to True, it also includes the index mapping that relates the values to their original structure.

Parameters:

return_index (bool, optional) – If True, the method returns both the sequence values and their indices. Defaults to False.

Returns:

The unpacked sequence values and optionally the trace: indices if return_index is set to True.

Return type:

T | tuple[T, Int32Feature]

class hyped.core.features.features.StatisticalFeatureMixin[source]¶

Bases: object

Mixin class that adds statistical operations for features.

This mixin provides common statistical operations that can be applied to primitive features, such as sum and mean. These operations calculate aggregate values for the feature, helping to analyze and summarize the data.

mean() → Float64Feature[source]¶

Calculates the mean (average) of the feature’s values.

This method computes the average of the values associated with the feature. It is intended for use with numerical data types.

Returns:: A new feature representing the mean of the original feature’s values.
Return type:: Float64Feature

sum() → Self[source]¶

Calculates the sum of the feature’s values.

This method computes the total sum of the values associated with the feature. It is intended for use with numerical data types.

Returns:: A new feature representing the sum of the original feature’s values.
Return type:: Self

class hyped.core.features.features.StringFeature(ref: BaseReference)[source]¶

Bases: PrimitiveFeature[PrimitiveType(_arrow_type=DataType(string))]

A primitive feature representing a string value.

capitalize() → StringFeature[source]¶

Capitalizes the string (first character uppercase, others lowercase).

Returns:: A new StringFeature with the string capitalized.
Return type:: StringFeature

contains(pattern: str) → BoolFeature[source]¶

Check if a pattern is contained in the string.

Parameters:

string (String) – The string column to search.
pattern (str) – The pattern to match in each string.

Returns:

A column of boolean values indicating whether the string contains the pattern.

Return type:

BoolFeature

endswith(pattern: str) → BoolFeature[source]¶

Checks if the string ends with the specified pattern.

Parameters:: pattern (str) – The suffix to check for.
Returns:: True if the string ends with the pattern, otherwise False.
Return type:: BoolFeature

find(pattern: str) → Int32Feature[source]¶

Finds the first occurrence of a pattern in the string.

Note that the resulting index presents the index of the first occurance of the pattern in the string in bytes. It give unexpected results depending on the string encoding.

Parameters:: pattern (str) – The substring to search for.
Returns:: The index of the first occurrence of the pattern, in bytes, or -1 if not found.
Return type:: Int32Feature

format(*args: Any, **kwargs: Any) → StringFeature[source]¶

Formats the string using provided positional and keyword arguments.

Parameters:

*args (Any) – Positional arguments for formatting.
**kwargs (Any) – Keyword arguments for formatting.

Returns:

The formatted string feature.

Return type:

StringFeature

lower() → StringFeature[source]¶

Converts the string to lowercase.

Returns:: A new StringFeature with all characters in lowercase.
Return type:: StringFeature

lstrip(characters: str = ' ') → StringFeature[source]¶

Strips leading characters from the string.

Parameters:: characters (str) – The characters to strip. Defaults to whitespace.
Returns:: A new StringFeature with leading characters stripped.
Return type:: StringFeature

replace(pattern: str, replacement: str) → StringFeature[source]¶

Replaces occurrences of a pattern with a replacement string.

Parameters:

pattern (str) – The substring to replace.
replacement (str) – The replacement string.

Returns:

A new StringFeature with replacements applied.

Return type:

StringFeature

rsplit(pattern: str = ' ', maxsplits: None | int = None) → SequenceFeature[StringFeature][source]¶

Splits the string by a delimiter from the right.

Parameters:

pattern (str) – The delimiter to split on. Defaults to a space.
maxsplits (None | int) – The maximum number of splits to perform. Defaults to None.

Returns:

A sequence of substrings resulting from the split.

Return type:

SequenceFeature[StringFeature]

rstrip(characters: str = ' ') → StringFeature[source]¶

Strips trailing characters from the string.

Parameters:: characters (str) – The characters to strip. Defaults to whitespace.
Returns:: A new StringFeature with trailing characters stripped.
Return type:: StringFeature

split(pattern: str = ' ', maxsplits: None | int = None) → SequenceFeature[StringFeature][source]¶

Splits the string by a delimiter.

Parameters:

pattern (str) – The delimiter to split on. Defaults to a space.
maxsplits (None | int) – The maximum number of splits to perform. Defaults to None.

Returns:

A sequence of substrings resulting from the split.

Return type:

SequenceFeature[StringFeature]

startswith(pattern: str) → BoolFeature[source]¶

Checks if the string starts with the specified pattern.

Parameters:: pattern (str) – The prefix to check for.
Returns:: True if the string starts with the pattern, otherwise False.
Return type:: BoolFeature

strip(characters: str = ' ') → StringFeature[source]¶

Strips leading and trailing characters from the string.

Parameters:: characters (str) – The characters to strip. Defaults to whitespace.
Returns:: A new StringFeature with stripped characters.
Return type:: StringFeature

swapcase() → StringFeature[source]¶

Swaps the case of all characters in the string.

Returns:: A new StringFeature with case-swapped characters.
Return type:: StringFeature

title() → StringFeature[source]¶

Converts the string to title case.

Returns:: A new StringFeature with each word capitalized.
Return type:: StringFeature

upper() → StringFeature[source]¶

Converts the string to uppercase.

Returns:: A new StringFeature with all characters in uppercase.
Return type:: StringFeature

class hyped.core.features.features.UInt16Feature(ref: BaseReference)[source]¶

Bases: PrimitiveFeature[PrimitiveType(_arrow_type=DataType(uint16))], ComparableFeatureMixin, StatisticalFeatureMixin

A primitive feature representing an unsigned 16-bit integer.

class hyped.core.features.features.UInt32Feature(ref: BaseReference)[source]¶

Bases: PrimitiveFeature[PrimitiveType(_arrow_type=DataType(uint32))], ComparableFeatureMixin, StatisticalFeatureMixin

A primitive feature representing an unsigned 32-bit integer.

class hyped.core.features.features.UInt64Feature(ref: BaseReference)[source]¶

Bases: PrimitiveFeature[PrimitiveType(_arrow_type=DataType(uint64))], ComparableFeatureMixin, StatisticalFeatureMixin

A primitive feature representing an unsigned 64-bit integer.

class hyped.core.features.features.UInt8Feature(ref: BaseReference)[source]¶

Bases: PrimitiveFeature[PrimitiveType(_arrow_type=DataType(uint8))], ComparableFeatureMixin, StatisticalFeatureMixin

A primitive feature representing an unsigned 8-bit integer.

hyped.core.features.features.build_feature_from_annotation(annotation: Any, typevar_mapping: dict[TypeVar, DType] = {}, session: ValidationSession = ValidationSession(_session_id=None, _contexts={}, _ref_count=0), context: dict[str, Any] = {'config': None}) → Feature[source]¶

Build a feature from a given annotation using a forward reference.

This function inspects the provided annotation and resolves any type parameters or type variables, then constructs a feature accordingly. If the annotation includes type parameters, it creates a generic validation model and resolves the correct feature types. This function supports features with generic annotations and resolves them to concrete feature types.

The underlying reference is a ForwardReference instance with a data type corresponding to the resolved feature type.

Parameters:

annotation (Any) – The annotation that describes the feature’s type, which can include type parameters or type variables.
typevar_mapping (dict[TypeVar, types.DType]) – A mapping that associates type variables with their corresponding types. Defaults to an empty dictionary.
session (ValidationSession) – The validation session.
context (dict[str, Any]) – A context dictionary that can provide additional information to the validation process. Defaults to an empty dictionary.

Returns:

The feature built from the annotation and reference. This feature matches: the structure defined by the annotation and resolves any type parameters.

Return type:

Feature

hyped.core.features.features.build_feature_from_reference(ref: BaseReference, fallback_dtype: None | DType = None) → Feature[source]¶

Build a feature from a given reference.

This function takes a reference and a data type, and constructs the appropriate feature based on the type of the data.

Parameters:

ref (BaseReference) – The reference to the feature being created.
fallback_dtype (None | dtypes.DType) – The fallback dtype used in case the dtype cannot be inferred from the reference.

Returns:

The corresponding feature based on the type of dtype.

Return type:

Feature

Raises:

TypeError – If the dtype is not recognized.

hyped.core.features.features.get_original_bases(cls: type) → type[source]¶: Return the class’s “original” bases prior to modification by __mro_entries__.