Overview of the different refinery available in Chonkie

Overview

Refinery Overview

Chonkie

The no-nonsense RAG chunking library that's lightweight, fast, and ready to CHONK your texts!

Introduction

🦛 Chonkie ✨

Installing Chonkie and its various components

Installation

Concepts

Chonkie Cloud

Pricing

Overview of the different chunkers available in Chonkie

Chunkers Overview

Split text into fixed-size token chunks with configurable overlap

Token Chunker

Split text into chunks while preserving sentence boundaries

Sentence Chunker

Recursively chunk documents into smaller chunks.

Recursive Chunker

Split code into chunks based on code structure

Code Chunker

Split text into chunks based on semantic similarity

Semantic Chunker

Split text using Semantic Double-Pass Merging for improved context preservation

SDPM Chunker

Split text into chunks based on a late-bound token count

Late Chunker

Split text using a fine-tuned BERT model to detect semantic shifts

Neural Chunker

Agentic chunking powered by generative models via the Genie interface

Slumber Chunker

Overview of the different embeddings available in Chonkie

Embeddings Overview

Automatically select the best embeddings handler for your use case

AutoEmbeddings

Embed text using SentenceTransformer embedding models

SentenceTransformer

SentenceTransformerEmbeddings

OpenAI

OpenAIEmbeddings

Model2Vec

Model2VecEmbeddings

Custom

Create your own custom embeddings handler

Refine chunks by adding overlapping context from adjacent chunks.

Overlap Refinery

Embeddings Refinery

Overview of the different handshakes available in Chonkie

Handshakes Overview

Export Chonkie's Chunks into a Chroma collection.

Chroma Handshake

Export Chonkie's Chunks into a Qdrant collection.

Qdrant Handshake

Export Chonkie's Chunks into a Turbopuffer database.

Turbopuffer Handshake

Export Chonkie's Chunks into a PostgreSQL database with pgvector.

Pgvector Handshake

Visualizer

Hubbie is a utility for accessing Chonkie's saved recipes.

Hubbie

Explore cutting-edge chunking capabilities with Chonkie's experimental features

Experimental Features Overview

Advanced AST-based code chunking with intelligent semantic preservation

CodeChunker

🦛 Chonkie TS ✨

Overview of the different chunkers available in Chonkie TS

Recursively chunk documents into smaller, semantically meaningful pieces using customizable rules.

Validates the provided API key. Use this endpoint to check if your API key is valid and active.

Validate API Key

Splits text into chunks based on token count, ensuring each chunk stays within specified token limits.

Splits text into chunks while preserving complete sentences, ensuring that each chunk maintains proper sentence boundaries and context.

Recursively chunks documents into smaller chunks. It is a good choice for documents that are long but well structured.

Splits text into chunks based on semantic similarity, ensuring that related content stays together in the same chunk.

Extends semantic chunking using a double-pass merging approach (Semantic Double-Pass Merging). Connects related content even if not consecutive.

Implements the late chunking strategy. Encodes the entire text, then splits recursively, deriving chunk embeddings from the full document embedding.

Splits code into chunks based on its structure, leveraging Abstract Syntax Trees (ASTs) to create contextually relevant segments.

Splits text using a fine-tuned BERT model to detect semantic shifts, creating topic-coherent chunks.

Agentic chunking powered by generative models (via Genie interface) for intelligent text splitting.

Adds embeddings to a list of provided chunks using a specified embedding model.

Adds overlapping context (prefix or suffix) to a list of provided chunks based on specified arguments.

Chonkie's Release Notes and Updates 🦛✨

Changelog

The no-nonsense RAG chunking library. Lightweight, fast, and ready to CHONK your texts

Getting Started

Chunkers

Embeddings

Refinery

Handshakes

Utils

Experimental

Refinery Overview

OverlapRefinery

EmbeddingsRefinery