Installation

Installing Chonkie and its various components

Chonkie follows a modular approach to dependencies, keeping the base installation lightweight while allowing you to add extra features as needed.

Basic Installation

For basic token and sentence chunking capabilities.

Python

pip

pip install chonkie

This installs our basic chunkers, plus the Python API SDK.

To use advanced features locally, skip ahead to Installation Options

JavaScript

Install the core package for local chunking

npm

npm install @chonkiejs/core

To use custom tokenizers, install the @chonkiejs/token package

npm

npm install @chonkiejs/token

To use the API, install the @chonkiejs/cloud package

npm

npm install @chonkiejs/cloud

Installation Options

Chonkie provides several installation options to match your specific needs:

Python

# Basic installation (TokenChunker, SentenceChunker, RecursiveChunker)
pip install chonkie

# For Hugging Face Hub support
pip install "chonkie[hub]"

# For visualization support (e.g., rich text output)
pip install "chonkie[viz]"

# For the default semantic provider support (includes Model2Vec)
pip install "chonkie[semantic]"

# For OpenAI embeddings support
pip install "chonkie[openai]"

# For Cohere embeddings support
pip install "chonkie[cohere]"

# For Jina embeddings support
pip install "chonkie[jina]"

# For SentenceTransformer embeddings support (required by LateChunker)
pip install "chonkie[st]"

# For CodeChunker support
pip install "chonkie[code]"

# For NeuralChunker support (BERT-based)
pip install "chonkie[neural]"

# For SlumberChunker support (Genie/LLM interface)
pip install "chonkie[genie]"

# For Groq Genie support (fast inference)
pip install "chonkie[groq]"

# For Cerebras Genie support (fastest inference)
pip install "chonkie[cerebras]"

# For installing multiple features together
pip install "chonkie[st, code, genie]"

# For all features
pip install "chonkie[all]"

Chunker Availability

The following table shows which chunkers are available with different installation options:

Chunker	Default	embeddings	’all’	Chonkie JS	API
TokenChunker
FastChunker
RecursiveChunker
SentenceChunker
TableChunker
SemanticChunker
LateChunker
CodeChunker
NeuralChunker
SlumberChunker

Embeddings Availability

Different embedding providers are available with different installation options:

Embeddings Provider	Default	’model2vec'	'st'	'openai'	'semantic'	'all’
Model2VecEmbeddings
SentenceTransformerEmbeddings
OpenAIEmbeddings

Dependencies

Here’s what each installation option adds:

Installation Option	Additional Dependencies
Default	tqdm, numpy, chonkie-core, tenacity
’hub’	+ huggingface-hub, jsonschema
’viz’	+ rich
’model2vec’	+ tokenizers, model2vec, numpy
’st’	+ tokenizers, sentence-transformers, accelerate
’openai’	+ openai, tiktoken, pydantic
’cohere’	+ tokenizers, cohere
’jina’	+ tokenizers
’semantic’	+ tokenizers, model2vec
’code’	+ tree-sitter, tree-sitter-language-pack, magika
’neural’	+ transformers, torch
’genie’	+ pydantic, google-genai
’groq’	+ pydantic, groq
’cerebras’	+ pydantic, cerebras-cloud-sdk
’litellm’	+ litellm, tiktoken, tokenizers
’all’	all above dependencies

Important Notes

• We provide separate semantic and all installs pre-packaged that might match other installation options breeding redundancy. This redundancy is intentional to provide users with the best experience and freedom to choose their preferred means.
• The semantic and all optional installs may change in future versions, so what you download today may not be the same for tomorrow.
• Installing either ‘semantic’ or ‘openai’ extras will enable SemanticChunker, as it can work with any embeddings provider. The difference is in which embedding providers are available for use with this chunker.

Logging

Chonkie logs warnings and errors by default. Control logging with the CHONKIE_LOG environment variable:

export CHONKIE_LOG=off     # Disable logging
export CHONKIE_LOG=warning # Warnings and errors (default)
export CHONKIE_LOG=info    # More verbose
export CHONKIE_LOG=debug   # Everything

See Logging for more details.