Viewing profile — chiccomagnus

IMHO, your article is missing an important point: 90% of implementations today flatten documents to plain text before chunking them. Why not consider the visual appearance that the…

How does credits works?

If you don't want to reinvent the wheel, we have built exactly that, goggle "Preprocess"

Have you compared this solution with tools like Preprocess, Reducto, etc.. ? I'm curious about the performance gain you can achieve with your approach

It seems it tries to always extract tables even if the content is just text. Is not working and at a similar price you can get high performing solutions like preprocess.co and simi…

I'm curious to know how you handle the chunking of complex documents

If for "documents" you mean PDF, Word, Powerpoints, Excel you should try preprocess.co If that’s not what you meant, can you please clarify?

Do you use naive chunking? Have you tried something else?

By chance, have you tried preprocess.co for text extraction + chucking?

I see a huge missing point here: real world files are PDF, Word, PowerPoint and Excel, not only plain text.

There are more sophisticated chunking strategies, you will lose lots of context like that

Well-written article, missing key considerations: - Titles matter, a lot: if you add the title of the section at the start of each chunk you will get 10x better embeddings and so m…

The real point that has been completely ignored is that data for real applications come from Office and PDF files, and making them plain text throws all the visual information.

Sorry, but it is really strange how haphazard it is. Am I doing something wrong? It's missing more or less all the titles The order of the text is casual The system chunks at rando…

check out this https://preprocess.co

The datasets are composed by Documents like PDFs and/or Office files?

That's a good point, and documents too needs different chunking techniques. You don't want to split a word file the same way you split an excel...

You are both right about chunking, and i think is one of the main challenges. About more intelligent chunking approaches, i think you have to give a try to to preprocess.co It's ab…

Hi everyone, I’m delvingg into optimizing RAG for ingest pipelines involving textual documents such as PDFs, Office documents (Word, PowerPoint, etc.), HTML files, emails, and plai…

doesn't the basic chunking you are using for splitting documents have any impact on results?

It's an interesting topic i'm working on at the moment. Given the ones you shared, I figured out additional challenges. Especially for PDF and Office documents, it's really importa…

Hi, I'm Nicolò Magnanini the CEO of Pigro. I'm trying to enter with my startup in ProductHunt and i hope to receive an invite. My email is nicolo.magnanini@pigro.ai What is Pigro? …