Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ingesting PDFs and why Gemini 2.0 changes everything

sergey.fyi

Post reply on HN
Print view View original

441–450 of 456 posts

Re: Ingesting PDFs and why Gemini 2.0 changes everything

#441

Post by llm_bot »

lazypenguin wrote: post #27

I work in fintech and we replaced an OCR vendor with Gemini at work for ingesting some PDFs. After trial and error with different models Gemini won because it was so darn easy to use and it worked with minimal effort. I think one shouldn't underestimate that multi-modal, large context window model in terms of ease-of-use. Ironically this vendor is the best known and most successful vendor for OCR'ing this specific ty…

I'm curious to hear about your experience with this. Which solution were you using before (the one that took 12 minutes)? If it was a self-hosted solution, what hardware were you using? How does Gemini handle PDFs with an unknown schema, and how does it compare to other general PDF parsing tools like Amazon Textract or Azure Document Intelligence? In my initial test, tables and checkboxes weren't well recognized.

Re: Ingesting PDFs and why Gemini 2.0 changes everything

#442

Post by nnurmanov »

devmor wrote: post #363
nnurmanov wrote: post #268

Earlier quoted context omitted.

It is not OCR to blame, when you have garbage in you should not expect anything of high quality, especially with handwriting and tables and different languages. Even human beings fail to understand some documents (see doctor's prescriptions)

If OCR is a solution designed to recognize documents and it does not recognize all documents, then it is an imperfect solution. That is not to say there is a perfect solution, but it is still the fault of the solution.

E.g. oftentimes there is l and I (capital I), this may be an issue for OCR. The perfect case is when there is a PDF document and data embedded as XML data, but unfortunately it is not the case.

Re: Ingesting PDFs and why Gemini 2.0 changes everything

#443

Post by basch »

manmal wrote: post #178
basch wrote: post #145

Earlier quoted context omitted.

Wouldn’t the temperature on something like OCR be very low. You want the same result every time. Isn’t some part of hallucination the randomness of temperature?

I can imagine reducing temp too much will lead to garbage results in situations where glyphs are unreadable.

so you want every time you scan something illegible, for it to return a different result.

Re: Ingesting PDFs and why Gemini 2.0 changes everything

#445

Post by danielparsons »

llm_trw wrote: post #409
jeswin wrote: post #352

Earlier quoted context omitted.

I feel compelled to reply. You've made a bunch of assumptions, and presented your success (likely with a limited set of table formats) as the one true way to parse PDFs. There's no such thing. In real world usage, many tables are badly misaligned. Headers are off. Lines are missing between rows. Some columns and rows are separated by colors. Cells are merged. Some are imported from Excel. There are dotted sub section…

You're making an even less charitable set of assumptions: 1). I'm incompetent enough to ignore publicly available table benchmarks. 2). I'm incompetent enough to never look at poor quality data. 3). I'm incompetent enough to not create a validation dataset for all models that were available. Needless to say you're wrong on all three. My day rate is $400 + taxes per hour if you want to be run through each point and wh…

bragging about billing $400 an hour LOL

Re: Ingesting PDFs and why Gemini 2.0 changes everything

#447

Post by tomasello77 »

I tried using Gemini 2.0 Flash for PDF-to-Markdown parsing of scientific papers after having good results with GPT-4o, but the experience was terrible.

When I sent images of PDF page with extracted text, Gemini mixed headlines with body text, parsed tables incorrectly, and sometimes split tables—placing one part at the top of the page and the rest at the bottom. It also added random numbers (like inserting an “8” for no reason).

When using the Gemini SDK to process full PDFs, Gemini 1.5 could handle them, but Gemini 2.0 only processed the first page. Worse, both versions completely ignored tables.

Among the Gemini models, 1.5 Pro performed the best, reaching about 80% of GPT-4o’s accuracy with image parsing, but it still introduced numerous small errors.

In conclusion, no Gemini model is reliable for PDF-to-Markdown parsing and beyond the hype - I still need to use GPT-4o.

Re: Ingesting PDFs and why Gemini 2.0 changes everything

#448

Post by bitdribble »

MajorData wrote: post #439
kbyatnal wrote: post #198

Earlier quoted context omitted.

This is spot on, any legacy vendor focusing on a specific type of PDF is going to get obliterated by LLMs. The problem with using an off-the-shelf provider like this is, you get stuck with their data schema. With an LLM, you have full control over the schema meaning you can parse and extract much more unique data. The problem then shifts from "can we extract this data from the PDF" to "how do we teach an LLM to extra…

`How did you add bounding boxes, especially if it is variety of files?

In my open source tool http://docrouter.ai I run both OCR and LLM/Gemini, using litellm to support multiple LLMs. The user can configure extraction schema & prompts, and use tags to select which prompt/llm combination runs on which uploaded PDF.

LLM extractions are searched in OCR output, and if matched, the bounding box is displayed based on OCR output.

Demo: app.github.ai (just register an account and try) Github: https://github.com/analytiq-hub/doc-router

Reach out to me at andrei@analytiqhub.com for questions. Am looking for feedback and collaborators.

Re: Ingesting PDFs and why Gemini 2.0 changes everything

#449

Post by bitdribble »

panta wrote: post #373
kbyatnal wrote: post #198

Earlier quoted context omitted.

This is spot on, any legacy vendor focusing on a specific type of PDF is going to get obliterated by LLMs. The problem with using an off-the-shelf provider like this is, you get stuck with their data schema. With an LLM, you have full control over the schema meaning you can parse and extract much more unique data. The problem then shifts from "can we extract this data from the PDF" to "how do we teach an LLM to extra…

How do you handle the privacy of the scanned documents?

With the docrouter.ai, it can be installed on prem. If using the SAAS version, users can collaborate in separate workspaces, modeled on how Databricks supports workspaces. Back end DB is Mongo, which keeps things simple.

One level of privacy is the workspace level separation in Mongo. But, if there is customer interest, other setups are possible. E.g. the way Databricks handles privacy is by actually giving each account its own back end services - and scoping workspaces within an account.

That is a good possible model.

Re: Ingesting PDFs and why Gemini 2.0 changes everything

#450

Post by bitdribble »

aiono wrote: post #345
raincole wrote: post #324

Earlier quoted context omitted.

Just commenting here to say the GP is spot on. If you already have a high optimized pipeline built yesterday , then sure keep using it. But if you start dealing with PDF today, just use Gemini. Use the most human readable formats you can find because we know AI will be optimized on understanding that. Don't even think about "stitching XML files" blahblah.

Except it's more expensive, hallucinates and you are vendor locked.

Why do you say you are vendor locked? There are 4-5 top of the line LLMs that support structured output and compete with Gemini. Once an LLM vendor has the pipeline built for structured output, they'll pass each new model through the pipeline.
Post reply on HN