Show HN: Chonkie – A Fast, Lightweight Text Chunking Library for RAG • Hacker News Board

Show HN: Chonkie – A Fast, Lightweight Text Chunking Library for RAG

#1

Post by bhavnicksm » Sun, Nov 10, 2024, 3:58 PM UTC

I built Chonkie because I was tired of rewriting chunking code for RAG applications. Existing libraries were either too bloated (80MB+) or too basic, with no middle ground.

Core features:

- 21MB default install vs 80-171MB alternatives

- 33x faster token chunking than popular alternatives

- Supports multiple chunking strategies: token, word, sentence, and semantic

- Works with all major tokenizers (transformers, tokenizers, tiktoken)

- Zero external dependencies for basic functionality

Technical optimizations:

- Uses tiktoken with multi-threading for faster tokenization

- Implements aggressive caching and precomputation

- Running mean pooling for efficient semantic chunking

- Modular dependency system (install only what you need)

Benchmarks and code: https://github.com/bhavnicksm/chonkie

Looking for feedback on the architecture and performance optimizations. What other chunking strategies would be useful for RAG applications?

Show HN: Chonkie – A Fast, Lightweight Text Chunking Library for RAG
github.com

Re: Show HN: Chonkie – A Fast, Lightweight Text Chunking Library for RAG

#2

Post by mixeden » Sun, Nov 10, 2024, 4:49 PM UTC

> Token Chunking: 33x faster than the slowest alternative

1) what

Re: Show HN: Chonkie – A Fast, Lightweight Text Chunking Library for RAG

#3

Post by samlinnfer » Sun, Nov 10, 2024, 4:55 PM UTC

How does it work for code? (Chunking code that is)

Re: Show HN: Chonkie – A Fast, Lightweight Text Chunking Library for RAG

#4

Post by rkharsan64 » Sun, Nov 10, 2024, 5:02 PM UTC

mixeden wrote: post #2

> Token Chunking: 33x faster than the slowest alternative 1) what

There's only 3 competitors in that particular benchmark, and the speedup compared to the 2nd is only 1.06x.

Edit: Also, from the same table, it seems that only this library was ran after warming up, while others were not. https://github.com/bhavnicksm/chonkie/blob/main/benchmarks/R...

Re: Show HN: Chonkie – A Fast, Lightweight Text Chunking Library for RAG

#5

Post by bravura » Sun, Nov 10, 2024, 5:15 PM UTC

One thing I've been looking for, and was a bit tricky implementing myself to be very fast, is this:

I have a particular max token length in mind, and I have a tokenizer like tiktoken. I have a string and I want to quickly find the maximum length truncation of the string that is Does chonkie handle this?

Re: Show HN: Chonkie – A Fast, Lightweight Text Chunking Library for RAG

#6

Post by spullara » Sun, Nov 10, 2024, 5:20 PM UTC

21MB? to split text? have you analyzed the footprint?

Re: Show HN: Chonkie – A Fast, Lightweight Text Chunking Library for RAG

#7

Post by ch1kkenm4ss4 » Sun, Nov 10, 2024, 5:24 PM UTC

Chonkie and lightweight? Good naming!

Re: Show HN: Chonkie – A Fast, Lightweight Text Chunking Library for RAG

#8

Post by opendang » Sun, Nov 10, 2024, 5:25 PM UTC

[flagged]

Re: Show HN: Chonkie – A Fast, Lightweight Text Chunking Library for RAG

#9

Post by nostrebored » Sun, Nov 10, 2024, 5:42 PM UTC

samlinnfer wrote: post #3

How does it work for code? (Chunking code that is)

Poorly, just like it does for text.

Chunking is easily where all of these problems die beyond PoC scale.

I’ve talked to multiple code generation companies in the past week — most are stuck with BM25 and taking in whole files.

Re: Show HN: Chonkie – A Fast, Lightweight Text Chunking Library for RAG

#10

Post by simonw » Sun, Nov 10, 2024, 5:43 PM UTC

Would it make sense for this to offer a chunking strategy that doesn't need a tokenizer at all? I love the goal to keep it small, but "tokenizers" is still a pretty huge dependency (and one that isn't currently compatible with Python 3.13).

I've been hoping to find an ultra light-weight chunking library that can do things like very simple regex-based sentence/paragraph/markdown-aware chunking with minimal additional dependencies.