Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ingesting PDFs and why Gemini 2.0 changes everything

sergey.fyi

Post reply on HN
Print view View original

161–170 of 456 posts

Re: Ingesting PDFs and why Gemini 2.0 changes everything

#161

Post by gioazzi »

Scoundreller wrote: post #142
themanmaran wrote: post #129

Earlier quoted context omitted.

One thing people always forget about traditional OCR providers (azure, tesseract, aws textract, etc.) is that they're ~85% accurate. They are all probabilistic. You literally get back characters + confidence intervals. So when textract gives you back incorrect characters, is that a hallucination?

> You literally get back characters + confidence intervals. Oh god, I wish speech to text engines would colour code the whole thing like a heat map to focus your attention to review where it may have over-enthusiastically guessed at what was said. You no knot.

We did this for a speech to text solution in healthcare. Doctors would always review everything that was transcribed manually (you don’t want hallucinations in your prescription), and using a heatmap it was trivial to identify e.g. drugs that were pretty much always misunderstood by STT

Re: Ingesting PDFs and why Gemini 2.0 changes everything

#162

Post by ritvikpandey21 »

ritvik here from pulse. everyone’s pretty much made the right points here, but wanted to emphasize that due to the llm architecture, they predict “the most probable text string” that corresponds to the embedding, not necessarily the exact text. this non-deterministicness is awful for customers deploying in production and a lot of our customers complained about this to us initially. the best approach is to build a sort-of “agent”-based VLM x traditional layout segmentation/reading order algos, which is what we’ve done and are continuing to do.

we have a technical blog on this exact phenomena coming out in the next couple days, will attach it here when it’s out!

check us out at https://www.runpulse.com

Re: Ingesting PDFs and why Gemini 2.0 changes everything

#163

Post by twelve40 »

Isn't it amazing how one company invented a universally spread format that takes structured data from an editor (except images obviously) and converts it into a completely fucked-up unstructured form that then requires expensive voodoo magic to convert back into structured data.

Re: Ingesting PDFs and why Gemini 2.0 changes everything

#164

Post by themanmaran »

anon373839 wrote: post #136
themanmaran wrote: post #129

Earlier quoted context omitted.

One thing people always forget about traditional OCR providers (azure, tesseract, aws textract, etc.) is that they're ~85% accurate. They are all probabilistic. You literally get back characters + confidence intervals. So when textract gives you back incorrect characters, is that a hallucination?

It’s a question of scale. When a traditional OCR system makes an error, it’s confined to a relatively small part of the overall text. (Think of “Plastics” becoming “PIastics”.) When a LLM hallucinates, there is no limit to how much text can be made up. Entire sentences can be rewritten because the model thinks they’re more plausible than the sentences that were actually printed. And because the bias is always toward…

It's a bit of a pick your poison situation. You're right that traditional OCR mistakes are usually easy to catch (except when you get $30.28 vs $80.23). Compared to LLM hallucinations that are always plausibly correct.

But on the flip side, layout is often times the biggest determinant of accuracy, and that's something LLMs do a way better job on. It doesn't matter if you have 100% accurate text from a table, but all that text is balled into one big paragraph.

Also the "pick the most plausible" approach is a blessing and a curse. A good example is the handwritten form here [1]. GPT 4o gets the all the email addresses correct because it can reasonably guess these people are all from the same company. Whereas AWS treats them all independently and returns three different emails.

[1] https://getomni.ai/ocr-demo

Re: Ingesting PDFs and why Gemini 2.0 changes everything

#165

Post by wussboy »

panarky wrote: post #33
lazypenguin wrote: post #27

I work in fintech and we replaced an OCR vendor with Gemini at work for ingesting some PDFs. After trial and error with different models Gemini won because it was so darn easy to use and it worked with minimal effort. I think one shouldn't underestimate that multi-modal, large context window model in terms of ease-of-use. Ironically this vendor is the best known and most successful vendor for OCR'ing this specific ty…

This is a big aha moment for me. If Gemini can do semantic chunking at the same time as extraction, all for so cheap and with nearly perfect accuracy, and without brittle prompting incantation magic, this is huge.

Could it do exactly the same with a web page? Would this replace something like beautiful soup?

Re: Ingesting PDFs and why Gemini 2.0 changes everything

#166

Post by throw7381 »

For data extraction from long documents (100k+ tokens) how does structured outputs via providing a json schema compare vs asking one question per field (in natural language)?

Also I've been hearing good things regarding document retrieval about Gemini 1.5 Pro, 2.0 Flash and gemini-exp-1206 (the new 2.0 Pro?), which is the best Gemini model for data extraction from 100k tokens?

How do they compare against Claude Sonnet 3.5 or the OpenAI models, has anyone done any real world tests?

Re: Ingesting PDFs and why Gemini 2.0 changes everything

#167

Post by phatfish »

itissid wrote: post #126
itissid wrote: post #124

Earlier quoted context omitted.

Wait isn't there atleast a two step process here one is semantic segmentation followed by a method like texttract for text - to avoid hallucinations? One cannot possibly say that "Text extracted by a multimodal model cannot hallucinate"? > accuracy was like 96% of that of the vendor and price was significantly cheaper. I would like to know how this 96% was tested. If you use a human to do random sample based testing,…

For an OCR company I imagine it is unconscionable to do this because if you would say OCR for an Oral History project for a library and you made hallucination errors, well you've replaced facts with fiction. Rewriting history? What the actual F.

Probaly totally fine for a "fintech" (Crypto?) though. Most of them are just burning VC money anyway. Maybe a lucky customer gets a windfall because Gemini added some zeros.

Re: Ingesting PDFs and why Gemini 2.0 changes everything

#168

Post by sconeguy »

twelve40 wrote: post #163

Isn't it amazing how one company invented a universally spread format that takes structured data from an editor (except images obviously) and converts it into a completely fucked-up unstructured form that then requires expensive voodoo magic to convert back into structured data.

What would it have taken to store the plain text in some meta field in the document. Argh, so annoying.

Re: Ingesting PDFs and why Gemini 2.0 changes everything

#169

Post by shermantanktop »

twelve40 wrote: post #163

Isn't it amazing how one company invented a universally spread format that takes structured data from an editor (except images obviously) and converts it into a completely fucked-up unstructured form that then requires expensive voodoo magic to convert back into structured data.

is "put this glyph at coordinate (x,y)" really what you'd call "structured"?

Re: Ingesting PDFs and why Gemini 2.0 changes everything

#170

Post by anirudhb99 »

yzydserd wrote: post #57
lazypenguin wrote: post #27

I work in fintech and we replaced an OCR vendor with Gemini at work for ingesting some PDFs. After trial and error with different models Gemini won because it was so darn easy to use and it worked with minimal effort. I think one shouldn't underestimate that multi-modal, large context window model in terms of ease-of-use. Ironically this vendor is the best known and most successful vendor for OCR'ing this specific ty…

How do today’s LLM’s like Gemini compare with the Document Understanding services google/aws/azure have offered for a few years, particularly when dealing with known forms? I think Google’s is Document AI.

member of the gemini team here -- personally, i'd recommend directly using gemini vs the document understanding services for OCR & general docs understanding tasks. From our internal evals gemini is now stronger than these solutions and is only going to get much better (higher precision, lower hallucination rates) from here.
Post reply on HN