Just to understand: The model is trained to put paragraph breaks into text. The training dataset is books (in contrast for instance to scientific articles or advertising flyers).
It shouldn't break sentences at commas, right?
Show HN: Chonky – a neural approach for text semantic chunking
11–20 of 38 posts
Re: Show HN: Chonky – a neural approach for text semantic chunking
#12So I could use this to index i.e. a fiction book in a vector db, right?
And the semantic chunking will possibly provide better results at query time for rag, did I understand that correctly?
Re: Show HN: Chonky – a neural approach for text semantic chunking
#13So I could use this to index i.e. a fiction book in a vector db, right? And the semantic chunking will possibly provide better results at query time for rag, did I understand that correctly?
Yes and yes you are correct!
Re: Show HN: Chonky – a neural approach for text semantic chunking
#14Just to understand: The model is trained to put paragraph breaks into text. The training dataset is books (in contrast for instance to scientific articles or advertising flyers). It shouldn't break sentences at commas, right?
No it shouldn't but since it neural net there is a small chance though.
Re: Show HN: Chonky – a neural approach for text semantic chunking
#15You mention that the fine tuning time took half a day, have you ever thought to reduce that time?
Re: Show HN: Chonky – a neural approach for text semantic chunking
#16You mention that the fine tuning time took half a day, have you ever thought to reduce that time?
Actually day and a half :). I'm all for it but unfortunately I have pretty old hardware.
Re: Show HN: Chonky – a neural approach for text semantic chunking
#17Pretty cool. What use case did you have for this? Text with paragraph breaks missing seems fairly exotic.
Re: Show HN: Chonky – a neural approach for text semantic chunking
#18> I took the base distilbert model
I read "the base Dilbert model", all sorts of weird ideas going through my head, concluded I should re-read and made the same mistake again XD
Guess I better take a break and go for a walk now...
Re: Show HN: Chonky – a neural approach for text semantic chunking
#19It seems to me like chunking (or some higher order version of it like chunking into knowledge graphs) is the highest leverage thing someone can work on right now if trying to improve intelligence of AI systems like code completion, PDF understanding etc. I’m surprised more people aren’t working on this.
Re: Show HN: Chonky – a neural approach for text semantic chunking
#20Does it work on other languages?