TokenChunker

Split text into fixed-size token chunks with configurable overlap

The TokenChunker splits text into chunks based on token count, ensuring each chunk stays within specified token limits.

API Reference

To use the TokenChunker via the API, check out the API reference documentation.

Installation

TokenChunker is included in the base installation of Chonkie. No additional dependencies are required.

For installation instructions, see the Installation Guide.

Initialization

from chonkie import TokenChunker

# Basic initialization with default parameters
chunker = TokenChunker(
    tokenizer="gpt2",  # Supports string identifiers
    chunk_size=512,    # Maximum tokens per chunk
    chunk_overlap=128  # Overlap between chunks
)

# Using a custom tokenizer
from tokenizers import Tokenizer
custom_tokenizer = Tokenizer.from_pretrained("your-tokenizer")
chunker = TokenChunker(
    tokenizer=custom_tokenizer,
    chunk_size=512,
    chunk_overlap=128
)

Parameters

tokenizer · Union[str, Any] · default: "gpt2"

Tokenizer to use. Can be a string identifier or a tokenizer instance

chunk_size · int · default: "512"

Maximum number of tokens per chunk

chunk_overlap · Union[int, float] · default: "0"

Number or percentage of overlapping tokens between chunks

return_type · Literal['chunks', 'texts'] · default: "chunks"

Whether to return chunks as Chunk objects or plain text strings

Usage

Single Text Chunking

text = "Some long text that needs to be chunked into smaller pieces..."
chunks = chunker.chunk(text)

for chunk in chunks:
    print(f"Chunk text: {chunk.text}")
    print(f"Token count: {chunk.token_count}")
    print(f"Start index: {chunk.start_index}")
    print(f"End index: {chunk.end_index}")

Batch Chunking

texts = [
    "First document to chunk...",
    "Second document to chunk..."
]
batch_chunks = chunker.chunk_batch(texts)

for doc_chunks in batch_chunks:
    for chunk in doc_chunks:
        print(f"Chunk: {chunk.text}")

Using as a Callable

# Single text
chunks = chunker("Text to chunk...")

# Multiple texts
batch_chunks = chunker(["Text 1...", "Text 2..."])

Supported Tokenizers

TokenChunker supports multiple tokenizer backends:

• TikToken (Recommended)

  import tiktoken
  tokenizer = tiktoken.get_encoding("gpt2")
  

• AutoTikTokenizer

  from autotiktokenizer import AutoTikTokenizer
  tokenizer = AutoTikTokenizer.from_pretrained("gpt2")
  

• Hugging Face Tokenizers

  from tokenizers import Tokenizer
  tokenizer = Tokenizer.from_pretrained("gpt2")
  

• Transformers

  from transformers import AutoTokenizer
  tokenizer = AutoTokenizer.from_pretrained("gpt2")
  

Return Type

TokenChunker returns chunks as Chunk objects. Chunks object include a custom Context class for additional metadata alongside other attributes:

@dataclass
class Context:
    text: str
    token_count: int
    start_index: Optional[int] = None
    end_index: Optional[int] = None

@dataclass
class Chunk:
    text: str           # The chunk text
    start_index: int    # Starting position in original text
    end_index: int      # Ending position in original text
    token_count: int    # Number of tokens in chunk
    context: Optional[Context]  # Contextual information (if any)