Ingesting PDFs and why Gemini 2.0 changes everything

Post by metadat » Thu, Feb 06, 2025, 4:49 PM UTC

jeswin wrote: post #352

llm_trw wrote: post #101

This is using exactly the wrong tools at every stage of the OCR pipeline, and the cost is astronomical as a result. You don't use multimodal models to extract a wall of text from an image. They hallucinate constantly the second you get past perfect 100% high-fidelity images. You use an object detection model trained on documents to find the bounding boxes of each document section as _images_; each bounding box comes…

I feel compelled to reply. You've made a bunch of assumptions, and presented your success (likely with a limited set of table formats) as the one true way to parse PDFs. There's no such thing. In real world usage, many tables are badly misaligned. Headers are off. Lines are missing between rows. Some columns and rows are separated by colors. Cells are merged. Some are imported from Excel. There are dotted sub section…

Post by panta » Thu, Feb 06, 2025, 5:35 PM UTC

kbyatnal wrote: post #198

lazypenguin wrote: post #27

I work in fintech and we replaced an OCR vendor with Gemini at work for ingesting some PDFs. After trial and error with different models Gemini won because it was so darn easy to use and it worked with minimal effort. I think one shouldn't underestimate that multi-modal, large context window model in terms of ease-of-use. Ironically this vendor is the best known and most successful vendor for OCR'ing this specific ty…

This is spot on, any legacy vendor focusing on a specific type of PDF is going to get obliterated by LLMs. The problem with using an off-the-shelf provider like this is, you get stuck with their data schema. With an LLM, you have full control over the schema meaning you can parse and extract much more unique data. The problem then shifts from "can we extract this data from the PDF" to "how do we teach an LLM to extra…

Post by kbyatnal » Thu, Feb 06, 2025, 5:48 PM UTC

panta wrote: post #373

kbyatnal wrote: post #198

Earlier quoted context omitted.

This is spot on, any legacy vendor focusing on a specific type of PDF is going to get obliterated by LLMs. The problem with using an off-the-shelf provider like this is, you get stuck with their data schema. With an LLM, you have full control over the schema meaning you can parse and extract much more unique data. The problem then shifts from "can we extract this data from the PDF" to "how do we teach an LLM to extra…

How do you handle the privacy of the scanned documents?

Post by ajcp » Thu, Feb 06, 2025, 6:13 PM UTC

rofl123 wrote: post #357

ajcp wrote: post #154

Earlier quoted context omitted.

In this use-case the customer just wanted data not currently in the warehouse inventory management system capatured, so here we converted a JSON response to a classic table row schema (where 1 row = 1 document) and now boom, shipping data! However we do very much recommend storing the raw model responses for audit and then at least as vector embeddings to orient the expectation that the data will need to be utilized…

> Kind of like "while we're here why don't we do what you're going to want to do at some point, even if it's not your use-case now..." wow, this is so bad. why do it now and introduce complexity and debt if you can do it later when you actually need it? you are just riding the hype wave and trying to get most out of it but that's fine.

Post by tomrod » Thu, Feb 06, 2025, 6:20 PM UTC

anonu wrote: post #376

Ingesting PDFs accurately is a noble goal which will no doubt be solved as LLMs get better. However, I need to point out that the financial statement example used in the article already has a solution: iXBRL. Many financial regulators require you to publish heavily marked up statements with iXBRL. These markups reveal nuances in the numbers that OCRing a post processed table will not understand. Of course, financial…