DatasetsPorter

Export Chonkie’s Chunks into a Hugging Face Dataset.

The DatasetsPorter exports a list of Chunk objects into a Hugging Face Dataset object. This is particularly useful for saving your processed chunks in a standardized format for training models, sharing, or archiving.

Installation

The DatasetsPorter requires the datasets library. You can install it with:

pip install "chonkie[datasets]"

For general installation instructions, see the Installation Guide.

Initialization

To get started, simply import and initialize the porter.

from chonkie import DatasetsPorter

porter = DatasetsPorter()

Parameters

chunks · list[Chunk] · required

The list of Chunk objects to be exported.

save_to_disk · bool · default: "True"

If True, the dataset will be saved to the location specified in the path parameter.

path · str · default: "chunks"

The local directory path where the dataset should be saved. This is only used if save_to_disk is True.

**kwargs · Any

Additional keyword arguments to be passed directly to the datasets.Dataset.save_to_disk method. This allows you to control aspects like the number of shards or processes.

Usage

The DatasetsPorter can either return a Dataset object directly for in-memory use or save it to disk.

Return a Dataset Object

By default, the porter returns a Dataset object without writing any files.

from chonkie import Chunk

chunks = [
    Chunk(text="This is the first chunk.", start_index=0, end_index=25, token_count=5),
    Chunk(text="This is the second chunk.", start_index=26, end_index=52, token_count=5),
]

# Get the dataset in memory
dataset = porter.export(chunks)

print(dataset)
# Expected output:
# Dataset({
#     features: ['text', 'start_index', 'end_index', 'token_count', 'context'],
#     num_rows: 2
# })

Save a Dataset to Disk

To save the dataset, set save_to_disk=True and provide a path. The method will still return the Dataset object.

# Save the dataset to a directory named "my_exported_chunks"
dataset = porter.export(chunks, save_to_disk=True, path="my_exported_chunks")

# You can now find the dataset files in the "my_exported_chunks" directory

Using as a Callable

The porter can also be used as a callable, which is an alias for the export method.

# Get the dataset in memory
dataset = porter(chunks)

# Save the dataset to disk
porter(chunks, save_to_disk=True, path="my_exported_chunks")

Return Type

The export method (and the __call__ method) will always return a datasets.Dataset object, regardless of whether it is saved to disk. This allows you to immediately work with the dataset after exporting.