Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ingesting PDFs and why Gemini 2.0 changes everything

sergey.fyi

Post reply on HN
Print view View original

171–180 of 456 posts

Re: Ingesting PDFs and why Gemini 2.0 changes everything

#171

Post by llm_trw »

ajcp wrote: post #137
llm_trw wrote: post #101

This is using exactly the wrong tools at every stage of the OCR pipeline, and the cost is astronomical as a result. You don't use multimodal models to extract a wall of text from an image. They hallucinate constantly the second you get past perfect 100% high-fidelity images. You use an object detection model trained on documents to find the bounding boxes of each document section as _images_; each bounding box comes…

Not sure what service you're basing your calculation on but with Gemmini I've processed 10,000,000+ shipping documents (PDF and PNGs) of every concievable layout in one month at under $1000 and an accuracy rate of between 80-82% (humans were at 66%). The longest part of the development timeline was establishing the accuracy rate and the ingestion pipeline, which itself is massively less complex than what your workflo…

>Not sure what service you're basing your calculation on but with Gemmini

The table of costs in the blog post. At 500,000 pages per day the hardware fixed cost overcomes the software variable cost at day 240 and from then on you're paying an extra ~$100 per day to keep it running in the cloud. The machine also had to use extremely beefy GPUs to fit all the models it needed to. Compute utilization was between 5 to 10% which means that it's future proof for the next 5 years at the rate at which the data source was growing.

    | Model                       | Pages per Dollar |
    |-----------------------------+------------------|
    | Gemini 2.0 Flash            | ≈ 6,000          |
    | Gemini 2.0 Flash Lite       | ≈ 12,000*        |
    | Gemini 1.5 Flash            | ≈ 10,000         |
    | AWS Textract                | ≈ 1,000          |
    | Gemini 1.5 Pro              | ≈ 700            |
    | OpenAI 4o-mini              | ≈ 450            |
    | LlamaParse                  | ≈ 300            |
    | OpenAI 4o                   | ≈ 200            |
    | Anthropic claude-3-5-sonnet | ≈ 100            |
    | Reducto                     | ≈ 100            |
    | Chunkr                      | ≈ 100            |
There is also the fact that it's _completely_ local. Which meant we could throw in every proprietary data source that couldn't leave the company at it.

>The longest part of the development timeline was establishing the accuracy rate and the ingestion pipeline, which itself is massively less complex than what your workflow sounds like: PDF -> Storage Bucket -> Gemini -> JSON response -> Database

Each company should build tools which match the skill level of their developers. If you're not comfortable training models locally with all that entails off the shelf solutions allow companies to punch way above their weight class in their industry.

Re: Ingesting PDFs and why Gemini 2.0 changes everything

#172

Post by makeitdouble »

lazypenguin wrote: post #27

I work in fintech and we replaced an OCR vendor with Gemini at work for ingesting some PDFs. After trial and error with different models Gemini won because it was so darn easy to use and it worked with minimal effort. I think one shouldn't underestimate that multi-modal, large context window model in terms of ease-of-use. Ironically this vendor is the best known and most successful vendor for OCR'ing this specific ty…

> After trial and error with different models

As a mere occasional customer I've been scanning 4 to 5 pages of the same document layout every week in gemini for half a year, and every single week the results were slightly different.

To note the docs are bilingual so it could affect the results, but what stroke me is the lack of consistency, and even with the same model, running it two or three times in a row gives different results.

That's fine for my usage, but that sounds like a nightmare if everytime Google tweaks their model, companies have to reajust their whole process to deal with the discrepancies.

And sticking with the same model for multiple years also sound like a captive situation where you'd have to pay premium for Google to keep it available for your use.

Re: Ingesting PDFs and why Gemini 2.0 changes everything

#173

Post by llm_trw »

dr_kiszonka wrote: post #144
llm_trw wrote: post #101

This is using exactly the wrong tools at every stage of the OCR pipeline, and the cost is astronomical as a result. You don't use multimodal models to extract a wall of text from an image. They hallucinate constantly the second you get past perfect 100% high-fidelity images. You use an object detection model trained on documents to find the bounding boxes of each document section as _images_; each bounding box comes…

Impressive. Can you share anything more about this project? 500k pages a day is massive and I can imagine why one would require that much throughput.

It was a financial company that needed a tool that would out perform Bloomberg terminal for traders and quants in markets where their coverage is spotty.

Re: Ingesting PDFs and why Gemini 2.0 changes everything

#175

Post by llm_trw »

polote wrote: post #143
llm_trw wrote: post #101

This is using exactly the wrong tools at every stage of the OCR pipeline, and the cost is astronomical as a result. You don't use multimodal models to extract a wall of text from an image. They hallucinate constantly the second you get past perfect 100% high-fidelity images. You use an object detection model trained on documents to find the bounding boxes of each document section as _images_; each bounding box comes…

Do you know another model than gridformer to detect table that has an available implementation somewhere ?

We had to roll our own from research papers unfortunately.

The number one take away we got was to use much larger images than anything that anyone else ever mentioned to get good results. A rule of thumb was that if you print the png of the image it should be easily readable from 2m away.

The actual model is proprietary and stuck in corporate land forever.

Re: Ingesting PDFs and why Gemini 2.0 changes everything

#176

Post by ALittleLight »

anon373839 wrote: post #130
kbyatnal wrote: post #95

It's clear that OCR & document parsing are going to be swallowed up by these multimodal models. The best representation of a document at the end of the day is an image. I founded a doc processing company [1] and in our experience, a lot of the difficulty w/ deploying document processing into production is when accuracy requirements are high (> 97%). This is because OCR and parsing is only one part of the problem, and…

> It's clear that OCR & document parsing are going to be swallowed up by these multimodal models. I don’t think this is clear at all. A multimodal LLM can and will hallucinate data at arbitrary scale (phrases, sentences, etc.). Since OCR is the part of the system that extracts the “ground truth” out of your source documents, this is an unacceptable risk IMO.

Seems like you could solve hallucinations by repeating the task multiple times. Non-hallucinations will be the same. Hallucinations will be different. Discard and retry hallucinated sections. This increases cost by a fixed multiple, but if cost of tokens continues to fall that's probably perfectly fine.

Re: Ingesting PDFs and why Gemini 2.0 changes everything

#177

Post by jacobr1 »

thelittleone wrote: post #99
panarky wrote: post #62

Earlier quoted context omitted.

It loads the entire PDF into context, but then it would be my job to chunk the output for RAG, and just doing arbitrary fixed-size blocks, or breaking on sentences or paragraphs is not ideal. So I can ask Gemini to return chunks of variable size, where each chunk is a one complete idea or concept, without arbitrarily chopping a logical semantic segment into multiple chunks.

Fixed size chunks is holding back a bunch of RAG projects on my backlog. Will be extremely pleased if this semantic chunking solves the issue. Currently we're getting around an 78-82% success on fixed size chunked RAG which is far too low. Users assume zero results on a RAG search equates to zero results in the source data.

Also consider methods that are using reasoning to potentially dispatch additional searches based on analysis of the returned data

Re: Ingesting PDFs and why Gemini 2.0 changes everything

#178

Post by manmal »

basch wrote: post #145
itissid wrote: post #124

Earlier quoted context omitted.

Wait isn't there atleast a two step process here one is semantic segmentation followed by a method like texttract for text - to avoid hallucinations? One cannot possibly say that "Text extracted by a multimodal model cannot hallucinate"? > accuracy was like 96% of that of the vendor and price was significantly cheaper. I would like to know how this 96% was tested. If you use a human to do random sample based testing,…

Wouldn’t the temperature on something like OCR be very low. You want the same result every time. Isn’t some part of hallucination the randomness of temperature?

I can imagine reducing temp too much will lead to garbage results in situations where glyphs are unreadable.

Re: Ingesting PDFs and why Gemini 2.0 changes everything

#179

Post by llm_trw »

vikp wrote: post #158
llm_trw wrote: post #101

This is using exactly the wrong tools at every stage of the OCR pipeline, and the cost is astronomical as a result. You don't use multimodal models to extract a wall of text from an image. They hallucinate constantly the second you get past perfect 100% high-fidelity images. You use an object detection model trained on documents to find the bounding boxes of each document section as _images_; each bounding box comes…

Marker ( https://www.github.com/VikParuchuri/marker ) works kind of like this. It uses a layout model to identify blocks and processes each one separately. The internal format is a tree of blocks, which have arbitrary fields, but can all render to html. It can write out to json, html, or markdown. I integrated gemini recently to improve accuracy in certain blocks like tables. (get initial text, then pass to gemini to…

I used sxml [0] unironically in this project extensively.

The rendering step for reports that humans got to see was a call to pandoc after the sxml was rendered to markdown - look ma we support powerpoint! - but it also allowed us to easily convert to whatever insane markup a given large (or small) language model worked best with on the fly.

[0] https://en.wikipedia.org/wiki/SXML

Re: Ingesting PDFs and why Gemini 2.0 changes everything

#180

Post by freezed8 »

(disclaimer I am CEO of llamaindex, which includes LlamaParse)

Nice article! We're actively benchmarking Gemini 2.0 right now and if the results are as good as implied by this article, heck we'll adapt and improve upon it. Our goal (and in fact the reason our parser works so well) is to always use and stay on top of the latest SOTA models and tech :) - we blend LLM/VLM tech with best-in-class heuristic techniques.

Some quick notes: 1. I'm glad that LlamaParse is mentioned in the article, but it's not mentioned in the performance benchmarks. I'm pretty confident that our most accurate modes are at the top of the table benchmark - our stuff is pretty good.

2. There's a long tail of issues beyond just tables - this includes fonts, headers/footers, ability to recognize charts/images/form fields, and as other posters said, the ability to have fine-grained bounding boxes on the source elements. We've optimized our parser to tackle all of these modes, and we need proper benchmarks for that.

3. DIY'ing your own pipeline to run a VLM at scale to parse docs is surprisingly challenging. You need to orchestrate a robust system that can screenshot a bunch of pages at the right resolution (which can be quite slow), tune the prompts, and make sure you're obeying rate limits + can retry on failure.

Post reply on HN