Out of curiosity where does the 21 MiB come from? The codebase clone is 1.2 MiB and the src folder is only 68 KiB.
Show HN: Chonkie – A Fast, Lightweight Text Chunking Library for RAG
21–30 of 38 posts
Re: Show HN: Chonkie – A Fast, Lightweight Text Chunking Library for RAG
#22> What other chunking strategies would be useful for RAG applications? I’m using o1-preview for chunking, creating summary subdocuments.
That's pretty cool! I believe a research paper called LumberChunker recently evaluated that to be pretty decent as well. Thanks for responding, I'll try to make it easier to use something like that in Chonkie in the future!
Ah, that's an interesting paper, and a slightly different approach to what I'm doing, but possibly a superior one. Thanks!
Re: Show HN: Chonkie – A Fast, Lightweight Text Chunking Library for RAG
#23Also check out https://github.com/D-Star-AI/dsRAG/ for a bit more involved chunking strategy.
Re: Show HN: Chonkie – A Fast, Lightweight Text Chunking Library for RAG
#24Also check out https://github.com/D-Star-AI/dsRAG/ for a bit more involved chunking strategy.
This looks pretty amazing. I will take it for a spin next week. I want to make a RAG that will answer questions related to my new car. The manual is huge and it is often hard to find answers in it, so I think this will be a big help to owners of the same car. I think your library can help me chunk that huge PDF easily.
Re: Show HN: Chonkie – A Fast, Lightweight Text Chunking Library for RAG
#25> Token Chunking: 33x faster than the slowest alternative 1) what
There's only 3 competitors in that particular benchmark, and the speedup compared to the 2nd is only 1.06x. Edit: Also, from the same table, it seems that only this library was ran after warming up, while others were not. https://github.com/bhavnicksm/chonkie/blob/main/benchmarks/R...
How does it compare with NLTK's chunking library? I have found that it works very well for sentence segmentation.
Re: Show HN: Chonkie – A Fast, Lightweight Text Chunking Library for RAG
#26Also check out https://github.com/D-Star-AI/dsRAG/ for a bit more involved chunking strategy.
This looks pretty amazing. I will take it for a spin next week. I want to make a RAG that will answer questions related to my new car. The manual is huge and it is often hard to find answers in it, so I think this will be a big help to owners of the same car. I think your library can help me chunk that huge PDF easily.
How many tokens is the manual?
Re: Show HN: Chonkie – A Fast, Lightweight Text Chunking Library for RAG
#27Out of curiosity where does the 21 MiB come from? The codebase clone is 1.2 MiB and the src folder is only 68 KiB.
Dependencies in the venv?
Re: Show HN: Chonkie – A Fast, Lightweight Text Chunking Library for RAG
#28Would it make sense for this to offer a chunking strategy that doesn't need a tokenizer at all? I love the goal to keep it small, but "tokenizers" is still a pretty huge dependency (and one that isn't currently compatible with Python 3.13). I've been hoping to find an ultra light-weight chunking library that can do things like very simple regex-based sentence/paragraph/markdown-aware chunking with minimal additional…
I made a rudimentary semantic chunking in just a few lines of code.
I just removed one sentence at a time from the left until there was a jump in the embedding distance. Then repeated for the right side.
Re: Show HN: Chonkie – A Fast, Lightweight Text Chunking Library for RAG
#29Earlier quoted context omitted.
This looks pretty amazing. I will take it for a spin next week. I want to make a RAG that will answer questions related to my new car. The manual is huge and it is often hard to find answers in it, so I think this will be a big help to owners of the same car. I think your library can help me chunk that huge PDF easily.
How many tokens is the manual?
if its more than the 2M that will fit in gemini context then I want to know what car it is.
Re: Show HN: Chonkie – A Fast, Lightweight Text Chunking Library for RAG
#30Review: Chonkie is an MIT license project to help with chunking your sentences. It boasts fixed length, word length, sentence and semantic methods. The instructions for installing and usage are simple.
The Benchmark numbers are massaged to look really impressive but upon scrutiny the improvements are at most The author is Bhavnick S. Minhas, an early career ML engineer with both research and industry experience and very prolific with his GitHub contributions.