Viewing profile — bhavnicksm

Hey! Author of the blog here. This is pretty cool~ Thanks for suggesting this, I will read this in detail and add it to the next (0.5.0) release of memchunk.

thanks! this is still pretty early, please let us know if you face any issues with the library, database or anything else :)

it doesn't right now, but the fallback feature is planned for in a future release. mostly because there's no simple way to handle the classic fallbacks like aws, gcp and azure, and…

We open-sourced catsu, a Python client for embedding APIs. The problem: every embedding provider has a different SDK with different bugs. OpenAI has undocumented token limits. Voya…

Hey! Right now, some things are somewhat hard-coded to be Cloudflare compatible. If someone's willing, you can just deploy this without Cloudflare, but you'd need to dig into the c…

Yes, that's quite fair re:Cloudflare! I couldn't find the right words to describe this, in comparison to something like Github Gist. I suppose "Own-your-data" since the D1 db gener…

I'm sure folks here have seen pastebins a thousand times. There's no innovation left in this space – and that's kind of the point. When I wanted to self-host a pastebin, every opti…

Thank you so much for giving Chonkie a chance! Just to note Chonkie is still in beta mode (with v0.1.2 running) with a bunch of things planned for it. It's an initial working versi…

I don't fully understand what you mean by "maximum length truncation of the string"; but if you're talking about splitting the sentence into 'chunks' which have token counts less t…

TokenChunking is really limited by the tokenizer and less by the Chunking algorithm. Tiktoken tokenizers seem to do better with warm-up which Chonkie defaults to -- which is also w…

That's pretty cool! I believe a research paper called LumberChunker recently evaluated that to be pretty decent as well. Thanks for responding, I'll try to make it easier to use so…

Just to clarify, the 21MB is the size of the package itself! Other package sizes are way larger. Memory footprint of the chunking itself would vary widely based on the dataset, and…

Right now, we haven't worked on adding support for code -- some things like comments (#, //) have punctuations that adversely affect chunking, along with indentation and other issu…

Haha~ thanks!

I built Chonkie because I was tired of rewriting chunking code for RAG applications. Existing libraries were either too bloated (80MB+) or too basic, with no middle ground. Core fe…