What would change "everything" is if we managed to switch to "real" digital parseable formats instead of this dead tree emulation that buries all data before the arrival of AI...
Ingesting PDFs and why Gemini 2.0 changes everything
231–240 of 456 posts
Re: Ingesting PDFs and why Gemini 2.0 changes everything
#232Earlier quoted context omitted.
What would it have taken to store the plain text in some meta field in the document. Argh, so annoying.
PDF provide that capability, but editors don't produce it, probably because printing is though OS drivers that don't support it, or PDF generators that don't support it. Or they do support it but users don't know to check that option, or turn it off because it makes PDFs too large.
Do you know what this field/type is called, and I’d any of the big names (MS/Adobe etc) support creating such PDFs?
Re: Ingesting PDFs and why Gemini 2.0 changes everything
#233I work in fintech and we replaced an OCR vendor with Gemini at work for ingesting some PDFs. After trial and error with different models Gemini won because it was so darn easy to use and it worked with minimal effort. I think one shouldn't underestimate that multi-modal, large context window model in terms of ease-of-use. Ironically this vendor is the best known and most successful vendor for OCR'ing this specific ty…
> For the 4% inaccuracies a lot of them are things like the text "LLC" handwritten would get OCR'd as "IIC" which I would say is somewhat "fair".
I'm actually somewhat surprised Gemini didn't guess from context that LLC is much more likely?
I guess the OCR subsystem is intentionally conservative? (Though I'm sure you could do a second step on your end, take the output from the conservative OCR pass, and sent it through Gemini and ask it to flag potential OCR problems? I bet that would flag most of them with very few false positives and false negatives.)
Re: Ingesting PDFs and why Gemini 2.0 changes everything
#234Earlier quoted context omitted.
Getting "bitter lesson" vibes from this post
The bitter lesson is very little of the sort. If we had unlimited memory, compute and data we'd use a rank N tensor for an input of length N and call it a day. Unfortunately N^N grows rather fast and we have to do all sorts of interesting engineering to make ML calculations complete before the heat death of the universe.
> Most AI research has been conducted as if the computation available to the agent were constant (in which case leveraging human knowledge would be one of the only ways to improve performance) but, over a slightly longer time than a typical research project, massively more computation inevitably becomes available. Seeking an improvement that makes a difference in the shorter term, researchers seek to leverage their human knowledge of the domain, but the only thing that matters in the long run is the leveraging of computation. These two need not run counter to each other, but in practice they tend to. Time spent on one is time not spent on the other. There are psychological commitments to investment in one approach or the other. And the human-knowledge approach tends to complicate methods in ways that make them less suited to taking advantage of general methods leveraging computation.
Re: Ingesting PDFs and why Gemini 2.0 changes everything
#235I'm failing to understanding the Ingesting part of the Gemini 2.0? Does Gemini provide the a process to convert PDFs to Markdown API OR the LLM APIs handle it with prompt "Extract the Attached PDF" using this API: https://ai.google.dev/gemini-api/docs/document-processing?la...
Re: Ingesting PDFs and why Gemini 2.0 changes everything
#236I work in fintech and we replaced an OCR vendor with Gemini at work for ingesting some PDFs. After trial and error with different models Gemini won because it was so darn easy to use and it worked with minimal effort. I think one shouldn't underestimate that multi-modal, large context window model in terms of ease-of-use. Ironically this vendor is the best known and most successful vendor for OCR'ing this specific ty…
Wait isn't there atleast a two step process here one is semantic segmentation followed by a method like texttract for text - to avoid hallucinations? One cannot possibly say that "Text extracted by a multimodal model cannot hallucinate"? > accuracy was like 96% of that of the vendor and price was significantly cheaper. I would like to know how this 96% was tested. If you use a human to do random sample based testing,…
Can confirm using gemini, some figure numbers were hallucinated. I had to cross-check each row to make sure data extracted is correct.
Re: Ingesting PDFs and why Gemini 2.0 changes everything
#237Earlier quoted context omitted.
It's cheap now because Google is subsidizing it, no?
Spoiler: every model is deeply, deeply subsidized. At least Google's is subsidized by a real business with revenue, not VC's staring at the clock.
It's cheap because it's a Flash model, far smaller and much less compute for inference, runs on TPUs instead of GPUs.
Re: Ingesting PDFs and why Gemini 2.0 changes everything
#238Earlier quoted context omitted.
One thing people always forget about traditional OCR providers (azure, tesseract, aws textract, etc.) is that they're ~85% accurate. They are all probabilistic. You literally get back characters + confidence intervals. So when textract gives you back incorrect characters, is that a hallucination?
I'm the founder of https://doctly.ai , also pdf extraction. The hallucination in LLM extraction is much more subtle as it will rewrite full sentences sometimes. It is much harder to spot when reading the document and sounds very plausible. We're currently working on a version where we send the document to two different LLMs, and use a 3rd if they don't match to increase confidence. That way you have the option of tra…
What if you use a different prompt to check the result, did this work? I am thinking to use this approach, but now I think maybe it is better to use two different LLM like you do.
Re: Ingesting PDFs and why Gemini 2.0 changes everything
#239This is using exactly the wrong tools at every stage of the OCR pipeline, and the cost is astronomical as a result. You don't use multimodal models to extract a wall of text from an image. They hallucinate constantly the second you get past perfect 100% high-fidelity images. You use an object detection model trained on documents to find the bounding boxes of each document section as _images_; each bounding box comes…
This is a great comment. I will mention another benefit to this approach: the same pipeline works for PDFs that are digital-native and don't require OCR. After the object detection step, you collect the text directly from within the bounding boxes, and the text is error-free. Using Gemini means that you give this up.
Re: Ingesting PDFs and why Gemini 2.0 changes everything
#240Earlier quoted context omitted.
It loads the entire PDF into context, but then it would be my job to chunk the output for RAG, and just doing arbitrary fixed-size blocks, or breaking on sentences or paragraphs is not ideal. So I can ask Gemini to return chunks of variable size, where each chunk is a one complete idea or concept, without arbitrarily chopping a logical semantic segment into multiple chunks.
Fixed size chunks is holding back a bunch of RAG projects on my backlog. Will be extremely pleased if this semantic chunking solves the issue. Currently we're getting around an 78-82% success on fixed size chunked RAG which is far too low. Users assume zero results on a RAG search equates to zero results in the source data.
This is my problem as well; do you have lots of documents?