Chunkers Overview

Overview of the different chunkers available in Chonkie TS

Chonkie TS provides multiple chunking strategies to handle different text processing needs in JavaScript and TypeScript environments. Each chunker is designed to be easy to use, fast, and consistent with the core principles outlined in the concepts page.

TokenChunker

Splits text into fixed-size token chunks. Best for maintaining consistent chunk sizes and working with token-based models.

SentenceChunker

Splits text at sentence boundaries. Perfect for maintaining semantic completeness at the sentence level.

RecursiveChunker

Recursively chunks documents into smaller, semantically meaningful pieces using customizable rules. Great for long or structured documents.

Availability

The following chunkers are currently available in Chonkie TS:

Chunker	Available
TokenChunker	✅
SentenceChunker	✅
RecursiveChunker	✅

Common Interface

All chunkers in Chonkie TS share a consistent, async interface:

// Import and create a chunker instance (async)
const chunker = await TokenChunker.create({ chunkSize: 256 });

// Chunk a single string
const chunks = await chunker("Some text to chunk!");

// Or chunk a batch of texts
const batchChunks = await chunker(["Text 1", "Text 2"]);

All chunkers are directly callable as async functions after creation: await chunker(text) or await chunker([text1, text2]).

F.A.Q.

Are all the chunkers async and thread-safe?

Yes, all chunkers in Chonkie TS are async and safe to use in concurrent environments. Performance may vary depending on the chunker and workload.