31–38 of 38 posts
How does it work for code? (Chunking code that is)
Poorly, just like it does for text. Chunking is easily where all of these problems die beyond PoC scale. I’ve talked to multiple code generation companies in the past week — most are stuck with BM25 and taking in whole files.
One thing I've been looking for, and was a bit tricky implementing myself to be very fast, is this: I have a particular max token length in mind, and I have a tokenizer like tiktoken. I have a string and I want to quickly find the maximum length truncation of the string that is Does chonkie handle this?
I don't fully understand what you mean by "maximum length truncation of the string"; but if you're talking about splitting the sentence into 'chunks' which have token counts less than a pre-specified max_token length then, yes! Is that what you meant?
Given a list sentences, find the largest in order group of sentences which fit into a max token length such that the sentences contain a natural coherence.
In my case I used a fuzzy token limit and the chunker would choose a smaller group of sentences that fit into a single paragraph or a single common structure instead of cramming every possible sentence until it ran out of room. It would do the same going over the limit if it would be beneficial to do so.
A simple example would be having an alphabetized set and instead of making one chunk A items through part of B items it would end at A items with tokens to spare, or if it were only an extra 10% it would finish the B items. Most of the time it just decided to use paragraphs to end chunks instead of continuing into the middle of the next one.
edit: Get some Moo Deng jokes in the docs!
What's RAG?
Think of it as if ChatGPT (or other models) didn't just have the embedded unstructured knowledge in their weights from learning, but also an extra DB on the side with specific structured knowledge that it can lookup on the fly.
Earlier quoted context omitted.
Poorly, just like it does for text. Chunking is easily where all of these problems die beyond PoC scale. I’ve talked to multiple code generation companies in the past week — most are stuck with BM25 and taking in whole files.
What do they use BM25 for? RAG?
Would it make sense for this to offer a chunking strategy that doesn't need a tokenizer at all? I love the goal to keep it small, but "tokenizers" is still a pretty huge dependency (and one that isn't currently compatible with Python 3.13). I've been hoping to find an ultra light-weight chunking library that can do things like very simple regex-based sentence/paragraph/markdown-aware chunking with minimal additional…
[1] https://gist.github.com/LukasKriesch/e75a0132e93ca989f8870c4...