Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Chonky – a neural approach for text semantic chunking

github.com

Post reply on HN
Print view View original

31–38 of 38 posts

Re: Show HN: Chonky – a neural approach for text semantic chunking

#31

Post by kamranjon »

Interesting! I worked previously for a company that did automatic generation of short video clips from long videos. I fine-tuned a t5 model by taking many Wikipedia articles and removing the new line characters and training it to insert them.

The idea was that paragraphs are naturally how we segment distinct thoughts in text, and would translate well to segmenting long video clips. It actually worked pretty well! It was able to predict the paragraph breaks in many texts that it wasn’t trained on at all.

The problems at the time were around context length and dialog style formatting.

I wanted to try and approach the problem in a less brute force way by maybe using sentence embedding and calculating the probability of a sentence being a “paragraph ending” sentence - which would likely result in a much smaller model.

Anyway this is really cool! I’m excited to dive in further to what you’ve done!

Re: Show HN: Chonky – a neural approach for text semantic chunking

#32

Post by vunderba »

mathis-l wrote: post #10

You might want to take a look at https://github.com/segment-any-text/wtpsplit It uses a similar approach but the focus is on sentence/paragraph segmentation generally and not specifically focused on RAG. It also has some benchmarks. Might be a good source of inspiration for where to take chonky next.

This is the library that I use, mainly around very noisy IRC chat transcripts and it works pretty well. OP I'd love to see a paragraph matching comparison benchmark against wtpsplit to see how well Chonky stacks up.

Re: Show HN: Chonky – a neural approach for text semantic chunking

#33

Post by DeveloperErrata »

serjester wrote: post #22
michaelmarkell wrote: post #19

It seems to me like chunking (or some higher order version of it like chunking into knowledge graphs) is the highest leverage thing someone can work on right now if trying to improve intelligence of AI systems like code completion, PDF understanding etc. I’m surprised more people aren’t working on this.

Chunking is less important in the long context era with most people just pulling in top 20 K. You obviously don’t want to butcher it, but you’ve got a lot of room for error.

Trueish - for orgs that can't use API models for regulatory or security reasons, or that just need really efficient high throughput models, setting up your own infra for long context models can still be pretty complicated and expensive. Careful chunking and thoughtful design of the RAG system often still matters a lot in that context.

Re: Show HN: Chonky – a neural approach for text semantic chunking

#34

Post by throwaway7783 »

legel wrote: post #29

Very cool! The training objective is clever. The 50+ filters at Ecodash.ai for 90,000 plants came from a custom RAG model on top of 800,000 raw web pages. Because LLM’s are expensive, chunking and semantic search for figuring out what to feed into the LLM for inference is a key part of the pipeline nobody talks about. I think what I did was: run all text through the cheapest OpenAI embeddings API… then, I recall that…

Have you tried the bm25 + vector search + reranking pipeline for this?

Re: Show HN: Chonky – a neural approach for text semantic chunking

#35

Post by J_Shelby_J »

michaelmarkell wrote: post #19

It seems to me like chunking (or some higher order version of it like chunking into knowledge graphs) is the highest leverage thing someone can work on right now if trying to improve intelligence of AI systems like code completion, PDF understanding etc. I’m surprised more people aren’t working on this.

That makes me feel better about spending so much time implementing this balanced text chunker last year. https://github.com/ShelbyJenkins/llm_utils

It splits an input text into equal sized chunks using DFS and parallelization (rayon) to do so relatively quickly.

However, the goal for me is to use a n LLM to split text by topic. I’m thinking I will implement it as an API saas service on top of it being OSS. Do you think it’s a viable business? You send a library of text, and receive a library of single topic context chunks as output.

Re: Show HN: Chonky – a neural approach for text semantic chunking

#36

Post by J_Shelby_J »

serjester wrote: post #22
michaelmarkell wrote: post #19

It seems to me like chunking (or some higher order version of it like chunking into knowledge graphs) is the highest leverage thing someone can work on right now if trying to improve intelligence of AI systems like code completion, PDF understanding etc. I’m surprised more people aren’t working on this.

Chunking is less important in the long context era with most people just pulling in top 20 K. You obviously don’t want to butcher it, but you’ve got a lot of room for error.

“Performance is less important in an era of multi-core CPUs.”

Re: Show HN: Chonky – a neural approach for text semantic chunking

#38

Post by searchguy »

legel wrote: post #29

Very cool! The training objective is clever. The 50+ filters at Ecodash.ai for 90,000 plants came from a custom RAG model on top of 800,000 raw web pages. Because LLM’s are expensive, chunking and semantic search for figuring out what to feed into the LLM for inference is a key part of the pipeline nobody talks about. I think what I did was: run all text through the cheapest OpenAI embeddings API… then, I recall that…

Hey, thanks for unpacking what you did at ecodash.ai.

Did you manually curate the queries that you did LLM query expansion on (generating a large number of diverse queries), or did you simply use the query log?

Post reply on HN