171–178 of 178 posts
> where each chunk can go through a multi-stage process, in which the output of the first stage is passed into another prompt for the next stage
Is it made possible by your custom code or is this that now OpenAI offers off of the shelf via their API?
If the latter, that would partially replace LangChain for simple pipelines.
Earlier quoted context omitted.
How to segment the document without LLM? I prefer to do all of this in 1 step with an LLM with a good prompt and few shots. With so many passes with images, the costs/time will be high with ViT being slower.
Segmenting can likely be done on a really small resolution and with a CNN, making it real short. There are some heuristic ways of doing it but i doubt you'll be able to distinguish equations from text.
In my experience, this works well but doesn't scale to all kinds of documents. For scientific papers; it can't render formulas. meta's nougat is the best model to do that. For invoices and records; donut works better. Both these models will fail in some cases so you end up running LLM to fix the issues. Even with that LLM won't be able to do tables and charts justice, as the details were lost during OCR process (bold…
I agree that vision models that actually have access to the image are a more sound approach than using OCR and trying to fix it up. It may be more expensive though, and depending on what you're trying to do it may be good enough. What I want to do is reading handwritten documents from the 18th century, and I feel like the multistep approach hits a hard ceiling there. Transkribus is multistep, but the line detecion mo…
Earlier quoted context omitted.
Ah, yes, I've found pre-processing the PDFs to sanitize against things like that has been helpful. That's a whole other process though.
What steps does that involve?
In my case all documents to be sent to the LLM (PDFs/Images/emails/etc) are already stagged in a file repository as part of a standard storage process. This entails every document being converted into a TIFF (read: rebuilt cleanly) for storage, and then into PDF upon export. This ensures that all docs are correct and don't maintain whatever went into originally creating them. I've found any number of "PDF" documents are not PDF, while others try and enforce some "protection" that makes the LLM not like the DOCS
https://github.com/louis030195/screen-pipe
And we also add LLM layers to fix errors
Thanks for sharing the info. > where each chunk can go through a multi-stage process, in which the output of the first stage is passed into another prompt for the next stage Is it made possible by your custom code or is this that now OpenAI offers off of the shelf via their API? If the latter, that would partially replace LangChain for simple pipelines.
Earlier quoted context omitted.
What steps does that involve?
Essentially what you're already doing, with one more step :) Get PDF > convert (read: rebuild) to TIFF > convert to PDF. In my case all documents to be sent to the LLM (PDFs/Images/emails/etc) are already stagged in a file repository as part of a standard storage process. This entails every document being converted into a TIFF (read: rebuilt cleanly) for storage, and then into PDF upon export. This ensures that all d…
Thanks