Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Launch HN: Chonkie (YC X25) – Open-Source Library for Advanced Chunking

news.ycombinator.com

Post reply on HN
Print view View original

31–40 of 43 posts

Re: Launch HN: Chonkie (YC X25) – Open-Source Library for Advanced Chunking

#31

Post by zackify »

You guys should steal the ideas I had in mind and partially implemented on https://github.com/zackify/revect

Similar to you I saw a lot of bloated projects out there. Mine is 90mb container.

I want to do what your project does but in addition have extensions for every day apps that index into a db.

Your private database for all ai interactions.

I also have a cloud version using the mcp auth spec, but it’s all for fun and probably not worth releasing.

Do you have any plans to do further use cases such as this?

Re: Launch HN: Chonkie (YC X25) – Open-Source Library for Advanced Chunking

#34

Post by snyy »

elpalek wrote: post #33

Do you have a benchmark for comparing different chunking methods? Your existing benchmark is to compare different libraries.

We don't yet, but our library comes with a visualization tool that you can use to compare chunkers directly. https://docs.chonkie.ai/python-sdk/utils/visualizer

Re: Launch HN: Chonkie (YC X25) – Open-Source Library for Advanced Chunking

#35

Post by snyy »

blef wrote: post #32

> - Code Chunking: Chunks code files by creating an AST and finding ideal split points. I'd be interested to use it for SQL, did you try? Does it works well with it? I'm not familiar with the tree-sitter library

Yes :) Code chunker is fantastic for SQL

Re: Launch HN: Chonkie (YC X25) – Open-Source Library for Advanced Chunking

#36

Post by snyy »

zackify wrote: post #31

You guys should steal the ideas I had in mind and partially implemented on https://github.com/zackify/revect Similar to you I saw a lot of bloated projects out there. Mine is 90mb container. I want to do what your project does but in addition have extensions for every day apps that index into a db. Your private database for all ai interactions. I also have a cloud version using the mcp auth spec, but it’s all for fun…

We want to be the platform that connects documents to AI for all applications. Consequently, we want to cover all use cases, including the ones you mentioned :)

Re: Launch HN: Chonkie (YC X25) – Open-Source Library for Advanced Chunking

#37

Post by snyy »

hweller wrote: post #26

Congratulations on the launch! would be awesome to see support for MongoDB Atlas as one of the vector stores and Voyage AI as an embedding provider if you are interested. I can imagine quite a few customers that would prefer a lightweight interface for chunking- lmk how I can help make that happen from the Mongo side!

We're working on Mongo integrations!

Re: Launch HN: Chonkie (YC X25) – Open-Source Library for Advanced Chunking

#38

Post by mbeissinger »

mritchie712 wrote: post #22

We ( https://www.definite.app/ ) have a use case I'd imagine is common for people building agents. When a user works with our agent, they may end up with a large conversation thread (e.g. 200k+ tokens) with many SQL snippets, query results and database metadata (e.g. table and column info). For example, if they ask "show me any companies that were heavily engaged at one point, but I haven't talked to in the last 90 d…

You might want to check out the conversational chunking from this paper:

On Memory Construction and Retrieval for Personalized Conversational Agents https://arxiv.org/abs/2502.05589

Re: Launch HN: Chonkie (YC X25) – Open-Source Library for Advanced Chunking

#39

Post by ChromaticPanic »

When I was looking at your library last week, It didn't look like there was a direct way to use my own embedding model endpoints. For example, I run snowflake arctic embed in vllm and it would be good be able to use it with Chonkie's semantic chunkers.
Post reply on HN