hyped.core.features.dtypes module¶

Data Type (DType) System Module.

This module defines a comprehensive type system for representing data types used in a structured framework. It includes abstractions for primitive types as well as nested structures including sequence and mapping types.

hyped.core.features.dtypes.BoolType = PrimitiveType(_arrow_type=DataType(bool))¶: Boolean type.

class hyped.core.features.dtypes.ClassLabelType(_arrow_type: Final[pa.DataType] = DataType(int64), names: None | tuple[str] = None)[source]¶

Bases: PrimitiveType

Represents a class label type, typically used for categorical labels in datasets.

classmethod from_dict(data: dict) → PrimitiveType[source]¶

Constructs a PrimitiveType instance from a dictionary representation.

Parameters:: data (dict) – A dictionary with the following keys: - type (str): Must be “PrimitiveType”. - arrow_type (str): The string representation of the PyArrow type.
Returns:: An instance of the PrimitiveType class.
Return type:: PrimitiveType
Raises:: ValueError – If the type field is not “PrimitiveType”.

property hf_feature: Features¶

Returns the corresponding HuggingFace Feature for this class label type.

Returns:: The Hugging Face Feature representation of the type.
Return type:: datasets.Features

names: None | tuple[str] = None¶: A tuple of class label names, which must be provided during initialization.

to_dict() → dict[source]¶

Converts the PrimitiveType instance to a dictionary representation.

Returns:

A dictionary with the following keys:

type (str): The type identifier (“PrimitiveType”).
arrow_type (str): The string representation of the PyArrow type.

Return type:

dict

class hyped.core.features.dtypes.DType[source]¶

Bases: ABC

Abstract base class for types in the system.

This class provides the foundation for defining types within the system. Subclasses must implement the arrow_type property to map their type to a corresponding PyArrow data type

abstract property arrow_type: DataType¶

Abstract property that returns the corresponding PyArrow data type.

This property must be implemented by subclasses to provide the PyArrow data type representation for the specific type.

Returns:: The PyArrow data type representation of the type.
Return type:: pa.DataType

abstractmethod classmethod from_dict(data: dict) → DType[source]¶

Constructs an object from its dictionary representation.

Subclasses must implement this method to enable deserialization of the object from a dictionary.

Parameters:: data (dict) – The dictionary representation of the object.
Returns:: An instance of the type reconstructed from the dictionary.
Return type:: DType

abstract property hf_feature: Features¶

Abstract property that returns the corresponding HuggingFace Feature.

This property must be implemented by subclasses to provide the HuggingFace Feature representation for the specific type.

Returns:: The Hugging Face Feature representation of the type.
Return type:: datasets.Features

abstractmethod to_dict() → dict[source]¶

Converts the object to a dictionary representation.

This method must be implemented by subclasses to provide a consistent dictionary serialization format.

Returns:: A dictionary representing the object.
Return type:: dict

hyped.core.features.dtypes.Float16Type = PrimitiveType(_arrow_type=DataType(halffloat))¶: 16-bit floating point type.

hyped.core.features.dtypes.Float32Type = PrimitiveType(_arrow_type=DataType(float))¶: 32-bit floating point type.

hyped.core.features.dtypes.Float64Type = PrimitiveType(_arrow_type=DataType(double))¶: 64-bit floating point type.

hyped.core.features.dtypes.Int16Type = PrimitiveType(_arrow_type=DataType(int16))¶: 16-bit signed integer type.

hyped.core.features.dtypes.Int32Type = PrimitiveType(_arrow_type=DataType(int32))¶: 32-bit signed integer type.

hyped.core.features.dtypes.Int64Type = PrimitiveType(_arrow_type=DataType(int64))¶: 64-bit signed integer type.

hyped.core.features.dtypes.Int8Type = PrimitiveType(_arrow_type=DataType(int8))¶: 8-bit signed integer type.

class hyped.core.features.dtypes.MappingType(fields: tuple[tuple[str, DType]])[source]¶

Bases: DType, Mapping[str, DType]

Represents a mapping type in the typing system.

This type associates field names with their corresponding types.

property arrow_schema: Schema¶

Returns the PyArrow schema for this mapping.

Returns:: The PyArrow schema representation.
Return type:: pa.Schema

property arrow_type: StructType¶

Returns the corresponding PyArrow data type for this mapping.

Returns:: The PyArrow struct type representation.
Return type:: pa.StructType

classmethod construct(fields: dict[str, DType]) → MappingType[source]¶

Creates a MappingType instance from a dictionary of fields.

Parameters:: fields (dict[str, DType]) – A dictionary mapping keys to their types.
Returns:: A new MappingType instance.
Return type:: MappingType

fields: tuple[tuple[str, DType]]¶: The fields in the mapping, represented as a tuple of key-type pairs.

classmethod from_dict(data: dict) → MappingType[source]¶

Deserializes a MappingType instance from a dictionary.

Parameters:: data (dict) – A dictionary containing the serialized form of a MappingType.
Returns:: A new instance of MappingType constructed from the data.
Return type:: MappingType
Raises:: ValueError – If the “type” field in the dictionary is not “MappingType”.

property hf_feature: Features¶

Returns the corresponding HuggingFace dataset features for this mapping.

This property converts the PyArrow schema associated with the mapping type into a HuggingFace Features object.

Returns:: The Hugging Face Features object representing the mapping type.
Return type:: datasets.Features

to_dict() → dict[source]¶

Serializes the MappingType instance to a dictionary.

Converts the mapping’s fields into a dictionary format, suitable for JSON serialization or storage.

Returns:: A dictionary representing the mapping type.
Return type:: dict

class hyped.core.features.dtypes.PrimitiveType(_arrow_type: DataType)[source]¶

Bases: DType

Represents a primitive type in the typing system.

A primitive type corresponds directly to a PyArrow primitive data type such as int32, float64, or bool. This class provides methods for serialization and deserialization of primitive types.

property arrow_type: DataType¶

Returns the corresponding PyArrow data type for this primitive type.

Returns:: The PyArrow data type representation.
Return type:: pa.DataType

classmethod from_dict(data: dict) → PrimitiveType[source]¶

Constructs a PrimitiveType instance from a dictionary representation.

Parameters:: data (dict) – A dictionary with the following keys: - type (str): Must be “PrimitiveType”. - arrow_type (str): The string representation of the PyArrow type.
Returns:: An instance of the PrimitiveType class.
Return type:: PrimitiveType
Raises:: ValueError – If the type field is not “PrimitiveType”.

property hf_feature: Features¶

Returns the corresponding HuggingFace Feature for this primitive type.

Returns:: The Hugging Face Feature representation of the type.
Return type:: datasets.Features

to_dict() → dict[source]¶

Converts the PrimitiveType instance to a dictionary representation.

Returns:

A dictionary with the following keys:

type (str): The type identifier (“PrimitiveType”).
arrow_type (str): The string representation of the PyArrow type.

Return type:

dict

class hyped.core.features.dtypes.SequenceType(value_type: DType, length: int = 4294967295)[source]¶

Bases: DType, Sequence

Represents a sequence type in the typing system.

property arrow_type: ListType¶

Returns the corresponding PyArrow data type for this sequence.

Returns:: The PyArrow list type representation.
Return type:: pa.ListType

classmethod from_dict(data: dict) → SequenceType[source]¶

Deserializes a dictionary representation into a SequenceType instance.

Parameters:: data (dict) – A dictionary containing serialized information of a sequence type.
Returns:: The deserialized sequence type.
Return type:: SequenceType
Raises:: ValueError – If the dictionary does not represent a SequenceType.

property hf_feature: Features¶

Returns the corresponding HuggingFace Feature for this sequence type.

Returns:: The Hugging Face Feature representation of the type.
Return type:: datasets.Features

length: int = 4294967295¶

The length of the sequence

Set to UNDEFINED_SEQUENCE_LENGTH in case of dynamic sequence lengths.

to_dict() → dict[source]¶

Serializes the sequence type to a dictionary representation.

Returns:: A dictionary containing the serialized information of the sequence type.
Return type:: dict

value_type: DType¶: The type of elements within the sequence.

hyped.core.features.dtypes.StringType = PrimitiveType(_arrow_type=DataType(string))¶: String type.

hyped.core.features.dtypes.UInt16Type = PrimitiveType(_arrow_type=DataType(uint16))¶: 16-bit unsigned integer type.

hyped.core.features.dtypes.UInt32Type = PrimitiveType(_arrow_type=DataType(uint32))¶: 32-bit unsigned integer type.

hyped.core.features.dtypes.UInt64Type = PrimitiveType(_arrow_type=DataType(uint64))¶: 64-bit unsigned integer type.

hyped.core.features.dtypes.UInt8Type = PrimitiveType(_arrow_type=DataType(uint8))¶: An 8-bit unsigned integer type.

hyped.core.features.dtypes.UNDEFINED_SEQUENCE_LENGTH = 4294967295¶

A constant representing an undefined sequence length.

This value is used when the length of a sequence is unknown or unspecified.

hyped.core.features.dtypes.build_dtype_from_arrow_type(arrow_type: DataType) → DType[source]¶

Build a data type from a given Arrow type.

Parameters:: arrow_type (pa.DataType) – The Arrow type to convert.
Returns:: The corresponding data type.
Return type:: DType
Raises:: TypeError – If the Arrow type is unsupported.

hyped.core.features.dtypes.build_dtype_from_dict(data: dict) → DType[source]¶

Constructs a DType instance from a dictionary.

This function determines the type of the serialized data and calls the appropriate from_dict method to reconstruct the corresponding DType instance.

Parameters:: data (dict) – A dictionary representing the serialized form of a DType, containing a “type” field.
Returns:: An instance of the appropriate DType subclass.
Return type:: DType
Raises:: ValueError – If the “type” field does not correspond to a recognized DType subclass.

Build a data type from a given Hugging Face feature.

Parameters:: feature (FeatureType) – The Hugging Face feature to convert.
Returns:: The corresponding data type.
Return type:: DType
Raises:: TypeError – If the feature type is unsupported.

hyped.core.features.dtypes.build_dtype_from_python_object(obj: Any) → DType[source]¶

Build a data type from a Python object.

Parameters:

obj (Any) – The object to derive the data type from.

Returns:

The corresponding data type.

Return type:

DType

Raises:

TypeError – If the object type is unsupported.
RuntimeError – If there are inconsistencies in list item types.

hyped.core.features.dtypes.cast_dtype(src_dtype: DType, tgt_dtype: DType) → DType[source]¶

Perform type casting of a source to a target data type.

This function attempts to cast a source data type (src_dtype) to a target data type (tgt_dtype). It supports casting between various data types such as sequences, mappings, and primitive types. If the casting is not feasible due to type mismatches or constraints (e.g., incompatible lengths for sequences or mismatched keys for mappings), an exception is raised.

Special Behavior for Mappings: - When both src_dtype and tgt_dtype are MappingType, the output

retains all fields from src_dtype, but fields specified in tgt_dtype are cast to their corresponding target types.

Fields in src_dtype that are not specified in tgt_dtype remain unchanged in the output.
Keys in tgt_dtype must be a subset of the keys in src_dtype. If not, a RuntimeError is raised.

Example:

src_dtype = MappingType.construct({"field": Int32Type, "other": BoolType})
tgt_dtype = MappingType.construct({"field": Int64Type})

result = cast_dtype(src_dtype, tgt_dtype)

# result:
# MappingType.construct({"field": Int64Type, "other": BoolType})

Parameters:

src_dtype (DType) – The source data type.
tgt_dtype (DType) – The target data type.

Returns:

The resulting data type after a successful cast.

Return type:

DType

Raises:

RuntimeError – If the casting operation is not feasible due to type mismatches or constraints.

hyped.core.features.dtypes.common_dtype(*dtypes: DType) → DType[source]¶

Determine the common dtype for a set of input dtypes.

This function identifies a dtype to which all input dtypes can be cast without loss of information, based on a predefined priority hierarchy. It supports primitive types, sequences, and mappings.

Parameters:

dtypes (DType) – The input dtypes to compare. These can be PrimitiveType, SequenceType, or MappingType objects.

Returns:

The common dtype that can represent all input dtypes.

Return type:

DType

Raises:

AssertionError – If no dtypes are provided or the dtypes cannot be combined.
RuntimeError – If dtypes are incompatible or cannot be resolved to a common dtype.

hyped.core.features.dtypes.is_dtype_subset(dtype_a: DType, dtype_b: DType) → bool[source]¶

Recursively checks if dtype_a is a subset of dtype_b.

Parameters:

dtype_a (DType) – The type that should be a subset.
dtype_b (DType) – The type that should be a superset.

Returns:

True if dtype_a is a subset of dtype_b, False otherwise.

Return type:

bool