Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Chonkie – A Fast, Lightweight Text Chunking Library for RAG

github.com

Post reply on HN
Print view View original

11–20 of 38 posts

Re: Show HN: Chonkie – A Fast, Lightweight Text Chunking Library for RAG

#11

Post by parhamn »

simonw wrote: post #10

Would it make sense for this to offer a chunking strategy that doesn't need a tokenizer at all? I love the goal to keep it small, but "tokenizers" is still a pretty huge dependency (and one that isn't currently compatible with Python 3.13). I've been hoping to find an ultra light-weight chunking library that can do things like very simple regex-based sentence/paragraph/markdown-aware chunking with minimal additional…

Across a broad enough dataset (char count / 4) is very close to the actual token count in english -- we verified across millions of queries. We had to switch to using an actual tokenizer for chinese and other unicode languages, as that simple formula misses the mark for context stuffing.

The more complicated stuff is the effective bin-packing problem that emerges depending on how much different contextual sources you have.

Re: Show HN: Chonkie – A Fast, Lightweight Text Chunking Library for RAG

#13

Post by bhavnicksm »

samlinnfer wrote: post #3

How does it work for code? (Chunking code that is)

Right now, we haven't worked on adding support for code -- some things like comments (#, //) have punctuations that adversely affect chunking, along with indentation and other issues.

But, it's on the roadmap, so please hold on!

Re: Show HN: Chonkie – A Fast, Lightweight Text Chunking Library for RAG

#15

Post by bhavnicksm »

spullara wrote: post #6

21MB? to split text? have you analyzed the footprint?

Just to clarify, the 21MB is the size of the package itself! Other package sizes are way larger.

Memory footprint of the chunking itself would vary widely based on the dataset, and it's not something we tested on... usually other providers don't test it either, as long as it doesn't bust up the computer/server.

If saving memory during runtime is important for your application, let me know! I'd run some benchmarks for it...

Thanks!

Re: Show HN: Chonkie – A Fast, Lightweight Text Chunking Library for RAG

#16

Post by bhavnicksm »

petesergeant wrote: post #14

> What other chunking strategies would be useful for RAG applications? I’m using o1-preview for chunking, creating summary subdocuments.

That's pretty cool! I believe a research paper called LumberChunker recently evaluated that to be pretty decent as well.

Thanks for responding, I'll try to make it easier to use something like that in Chonkie in the future!

Re: Show HN: Chonkie – A Fast, Lightweight Text Chunking Library for RAG

#17

Post by bhavnicksm »

rkharsan64 wrote: post #4
mixeden wrote: post #2

> Token Chunking: 33x faster than the slowest alternative 1) what

There's only 3 competitors in that particular benchmark, and the speedup compared to the 2nd is only 1.06x. Edit: Also, from the same table, it seems that only this library was ran after warming up, while others were not. https://github.com/bhavnicksm/chonkie/blob/main/benchmarks/R...

TokenChunking is really limited by the tokenizer and less by the Chunking algorithm. Tiktoken tokenizers seem to do better with warm-up which Chonkie defaults to -- which is also what the 2nd one is using.

Algorithmically, there's not much difference in TokenChunking between Chonkie and LangChain or any other TokenChunking algorithm you might want to use. (except Llamaindex, I don't know what mess they made for 33x slower algo)

If you only want TokenChunking (which I do not recommend completely), better than Chonkie or LangChain, just write your own for production :) At least don't install 80MiB packages for TokenChunking, Chonkie is 4x smaller than them.

That's just my honest response... And these benchmarks are just the beginning, future optimizations on SemanticChunking which would increase the speed-up from the current 2nd (2.5x right now) to even higher.

Re: Show HN: Chonkie – A Fast, Lightweight Text Chunking Library for RAG

#18

Post by bhavnicksm »

bravura wrote: post #5

One thing I've been looking for, and was a bit tricky implementing myself to be very fast, is this: I have a particular max token length in mind, and I have a tokenizer like tiktoken. I have a string and I want to quickly find the maximum length truncation of the string that is Does chonkie handle this?

I don't fully understand what you mean by "maximum length truncation of the string"; but if you're talking about splitting the sentence into 'chunks' which have token counts less than a pre-specified max_token length then, yes!

Is that what you meant?

Re: Show HN: Chonkie – A Fast, Lightweight Text Chunking Library for RAG

#19

Post by bhavnicksm »

Thank you so much for giving Chonkie a chance! Just to note Chonkie is still in beta mode (with v0.1.2 running) with a bunch of things planned for it. It's an initial working version, which seemed promising enough to present.

I hope that you will stick with Chonkie for the journey of making the 'perfect' chunking library!

Thanks again!

Re: Show HN: Chonkie – A Fast, Lightweight Text Chunking Library for RAG

#20

Post by xivusr »

opendang wrote: post #8

[flagged]

IMO comments like this go against the spirit of HN - why not offer more constructive feedback? Implying defects without suggestions on how to improve (or proof) is low effort and what I expect on a YouTube comment thread, not HN.
Post reply on HN