I have seen no decent program that can read, OCR, and analyze, and tabulate data correctly from very large PDF files with a lot of scanned information from different sources. I run my practice with pdf files- one for each patient. It is a treasure trove of actionable data. PDF filing in this manner allows me to finish my daily tasks in 4 hrs instead of 12 hrs! For sick patients who need information at the point of care, PDF has numerous advantages over usual hospital EHR portals, etc. If any smart Engineer/s are interested in working with me, please connect with me
Ingesting PDFs and why Gemini 2.0 changes everything
341–350 of 456 posts
Re: Ingesting PDFs and why Gemini 2.0 changes everything
#342I have seen no decent program that can read, OCR, and analyze, and tabulate data correctly from very large PDF files with a lot of scanned information from different sources. I run my practice with pdf files- one for each patient. It is a treasure trove of actionable data. PDF filing in this manner allows me to finish my daily tasks in 4 hrs instead of 12 hrs! For sick patients who need information at the point of ca…
I can help as can many others. Probably a good place to start though is with some of the more recent off the shelf solutions like trellis (I have no affiliation with them).
Re: Ingesting PDFs and why Gemini 2.0 changes everything
#343(disclaimer I am CEO of llamaindex, which includes LlamaParse) Nice article! We're actively benchmarking Gemini 2.0 right now and if the results are as good as implied by this article, heck we'll adapt and improve upon it. Our goal (and in fact the reason our parser works so well) is to always use and stay on top of the latest SOTA models and tech :) - we blend LLM/VLM tech with best-in-class heuristic techniques. So…
But can it do this table?!:
Re: Ingesting PDFs and why Gemini 2.0 changes everything
#344I work in fintech and we replaced an OCR vendor with Gemini at work for ingesting some PDFs. After trial and error with different models Gemini won because it was so darn easy to use and it worked with minimal effort. I think one shouldn't underestimate that multi-modal, large context window model in terms of ease-of-use. Ironically this vendor is the best known and most successful vendor for OCR'ing this specific ty…
“LLC” to “IIC” is one thing. But wouldn’t that also make it just as easy to to mistake something like “$100” for “$700”?
Re: Ingesting PDFs and why Gemini 2.0 changes everything
#345Earlier quoted context omitted.
Just commenting here so that I can find back to this comment later. You perfectly captured the AI hype in one small paragraph.
Just commenting here to say the GP is spot on. If you already have a high optimized pipeline built yesterday , then sure keep using it. But if you start dealing with PDF today, just use Gemini. Use the most human readable formats you can find because we know AI will be optimized on understanding that. Don't even think about "stitching XML files" blahblah.
Except it's more expensive, hallucinates and you are vendor locked.
Re: Ingesting PDFs and why Gemini 2.0 changes everything
#346Earlier quoted context omitted.
Wait isn't there atleast a two step process here one is semantic segmentation followed by a method like texttract for text - to avoid hallucinations? One cannot possibly say that "Text extracted by a multimodal model cannot hallucinate"? > accuracy was like 96% of that of the vendor and price was significantly cheaper. I would like to know how this 96% was tested. If you use a human to do random sample based testing,…
For an OCR company I imagine it is unconscionable to do this because if you would say OCR for an Oral History project for a library and you made hallucination errors, well you've replaced facts with fiction. Rewriting history? What the actual F.
Normal OCR (like Tesseract) can be wrong as well (and IMO this happens frequently). It won’t hallucinate/straight make shit up like an LLM, but a human needs to review OCR results if the workload requires accuracy. Even across multiple runs of the same image an OCR can give different results (in some scenarios). No OCR system is perfectly accurate, they all use some kind of machine learning/floating point/potentially nondeterministic tech.
Re: Ingesting PDFs and why Gemini 2.0 changes everything
#347(disclaimer I am CEO of llamaindex, which includes LlamaParse) Nice article! We're actively benchmarking Gemini 2.0 right now and if the results are as good as implied by this article, heck we'll adapt and improve upon it. Our goal (and in fact the reason our parser works so well) is to always use and stay on top of the latest SOTA models and tech :) - we blend LLM/VLM tech with best-in-class heuristic techniques. So…
There's an error right on your landing page [1] with the parsed document...
It's supposed to say 234.1, not 234.4
Re: Ingesting PDFs and why Gemini 2.0 changes everything
#348Earlier quoted context omitted.
This is spot on, any legacy vendor focusing on a specific type of PDF is going to get obliterated by LLMs. The problem with using an off-the-shelf provider like this is, you get stuck with their data schema. With an LLM, you have full control over the schema meaning you can parse and extract much more unique data. The problem then shifts from "can we extract this data from the PDF" to "how do we teach an LLM to extra…
> The problem then shifts from "can we extract this data from the PDF" to "how do we teach an LLM to extract the data we need, validate its performance, and deploy it with confidence into prod?" A smart vendor will shift into that space - they'll use that LLM themselves, and figure out some combination of finetunes, multiple LLMs, classical methods and human verification of random samples, that lets them not only "va…
Software is dead, if it isn't a prompt now, it will be a prompt in 6 months.
Most of what we think software is today, will just be a UI. But UIs are also dead.
Re: Ingesting PDFs and why Gemini 2.0 changes everything
#349This is using exactly the wrong tools at every stage of the OCR pipeline, and the cost is astronomical as a result. You don't use multimodal models to extract a wall of text from an image. They hallucinate constantly the second you get past perfect 100% high-fidelity images. You use an object detection model trained on documents to find the bounding boxes of each document section as _images_; each bounding box comes…
Only thing I could find about GridFormer and tables was this: https://arxiv.org/pdf/2309.14962v1
But there is no GitHub link or details on the implementation. Only model available seems to be one for removing weather effects from images: https://github.com/TaoWangzj/GridFormer
Could you care to expand on how you would use GridFormer for extracting tables from images? Seems like it's not as trivial as using something like Excalibur or Tabula, both which seem more battle-tested.
Re: Ingesting PDFs and why Gemini 2.0 changes everything
#350Earlier quoted context omitted.
> The problem then shifts from "can we extract this data from the PDF" to "how do we teach an LLM to extract the data we need, validate its performance, and deploy it with confidence into prod?" A smart vendor will shift into that space - they'll use that LLM themselves, and figure out some combination of finetunes, multiple LLMs, classical methods and human verification of random samples, that lets them not only "va…
Software is dead, if it isn't a prompt now, it will be a prompt in 6 months. Most of what we think software is today, will just be a UI. But UIs are also dead.
Software without data moats, vender lock-in, etc sure will. All the low handing fruit saas is going to get totally obliterated by LLM built-software.