Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ingesting PDFs and why Gemini 2.0 changes everything

sergey.fyi

Post reply on HN
Print view View original

151–160 of 456 posts

Re: Ingesting PDFs and why Gemini 2.0 changes everything

#151

Post by erulabs »

Hrm I've been using a combo of Textract (for bounding boxes) AI for understanding the contents of the document. Textract is excellent at bounding boxes and exact-text capture, but LLMs are excellent at understanding when a messy/ugly bit of a form is actually one question, or if there are duplicate questions etc.

Correlating the two (Textract AI) output is difficult, but another round of AI is usually good at that. Combined with some text-different scoring and logic, I can get pretty good full-document understanding of questions and answer locations. I've spent a pretty absurd amount of time on this and as of yet have not launched a product with it, but if anyone is interested I'd love to chat about the pipeline!

Re: Ingesting PDFs and why Gemini 2.0 changes everything

#153

Post by pbronez »

oedemis wrote: post #103

there is also https://ds4sd.github.io/docling/ from ibm research which is mit license and track bounding boxes as rich json format

Docling has worked well for me. It handles scenarios that crashed ChatGPT Pro. Only problem is it's super annoying to install. When I have a minute I might package it for homebrew.

Re: Ingesting PDFs and why Gemini 2.0 changes everything

#154

Post by ajcp »

cpursley wrote: post #140
ajcp wrote: post #137

Earlier quoted context omitted.

Not sure what service you're basing your calculation on but with Gemmini I've processed 10,000,000+ shipping documents (PDF and PNGs) of every concievable layout in one month at under $1000 and an accuracy rate of between 80-82% (humans were at 66%). The longest part of the development timeline was establishing the accuracy rate and the ingestion pipeline, which itself is massively less complex than what your workflo…

Very cool! How are you storing it to a database - vectors? What do you do with the extracted data (in terms of being able to pull it up via some query system)?

In this use-case the customer just wanted data not currently in the warehouse inventory management system capatured, so here we converted a JSON response to a classic table row schema (where 1 row = 1 document) and now boom, shipping data!

However we do very much recommend storing the raw model responses for audit and then at least as vector embeddings to orient the expectation that the data will need to be utilized for vector search and RAG. Kind of like "while we're here why don't we do what you're going to want to do at some point, even if it's not your use-case now..."

Re: Ingesting PDFs and why Gemini 2.0 changes everything

#155

Post by scotty79 »

esafak wrote: post #141
roywashere wrote: post #121

I think it is very ironic that we chose to use PDF in many fields to archive data because it is a standard and because we would be able to open our pdf documents in 50 or 100 years time. So here we are just a couple of years later facing the challenge of getting the data out of our stupid PDF documents already!

It's not ironic. PDFs are a container , which can hold scanned documents as well as text. Scanned documents need OCR and to be analyzed for their layout. This is not a failing of the PDF format, but a problem inherent to working with print scans. I don't claim PDF is a good format. It is inscrutable to me.

Pdf is a horrible format. Even if it contains plain text it has no concept of something as simple as paragraphs.

One can wonder how much wonkiness of llms comes from errors in extracting language from pdfs.

Adobe is the most harmful software development company in existence.

Re: Ingesting PDFs and why Gemini 2.0 changes everything

#156

Post by Tostino »

panarky wrote: post #62
potatoman22 wrote: post #48

Earlier quoted context omitted.

Small point but is it doing semantic chunking, or loading the entire pdf into context? I've heard mixed results on semantic chunking.

It loads the entire PDF into context, but then it would be my job to chunk the output for RAG, and just doing arbitrary fixed-size blocks, or breaking on sentences or paragraphs is not ideal. So I can ask Gemini to return chunks of variable size, where each chunk is a one complete idea or concept, without arbitrarily chopping a logical semantic segment into multiple chunks.

I wish we had a local model for semantic chunking. I've been wanting one for ages, but haven't had the time to make a dataset and finetune that task =/.

Re: Ingesting PDFs and why Gemini 2.0 changes everything

#157

Post by faxmeyourcode »

otoburb wrote: post #46
faxmeyourcode wrote: post #37

Earlier quoted context omitted.

I've been fighting trying to chunk SEC filings properly, specifically surrounding the strange and inconsistent tabular formats present in company filings. This is giving me hope that it's possible.

>> I've been fighting trying to chunk SEC filings properly, specifically surrounding the strange and inconsistent tabular formats present in company filings. For this specific use case you can also try edgartools[1] which is a library that was relatively recently released that ingests SEC submissions and filings. They don't use OCR but (from what I can tell) directly parse the XBRL documents submitted by companies an…

I'll definitely be looking into this, thanks for the recommendation! Been playing around with it this afternoon and it's very promising.

Re: Ingesting PDFs and why Gemini 2.0 changes everything

#158

Post by vikp »

llm_trw wrote: post #101

This is using exactly the wrong tools at every stage of the OCR pipeline, and the cost is astronomical as a result. You don't use multimodal models to extract a wall of text from an image. They hallucinate constantly the second you get past perfect 100% high-fidelity images. You use an object detection model trained on documents to find the bounding boxes of each document section as _images_; each bounding box comes…

Marker (https://www.github.com/VikParuchuri/marker) works kind of like this. It uses a layout model to identify blocks and processes each one separately. The internal format is a tree of blocks, which have arbitrary fields, but can all render to html. It can write out to json, html, or markdown.

I integrated gemini recently to improve accuracy in certain blocks like tables. (get initial text, then pass to gemini to refine) Marker alone works about as well as gemini alone, but together they benchmark much better.

Re: Ingesting PDFs and why Gemini 2.0 changes everything

#159

Post by ForHackernews »

lazypenguin wrote: post #27

I work in fintech and we replaced an OCR vendor with Gemini at work for ingesting some PDFs. After trial and error with different models Gemini won because it was so darn easy to use and it worked with minimal effort. I think one shouldn't underestimate that multi-modal, large context window model in terms of ease-of-use. Ironically this vendor is the best known and most successful vendor for OCR'ing this specific ty…

This is great, I just want to highlight out how nuts it is that we have spun up whole industries around extracting text that was typically printed from a computer, back into a computer.

There should be laws that mandates that financial information be provided in a sensible format: even Office Open XML would be better than this insanity. Then we can redirect all this wasted effort into digging ditches and filling them back in again.

Re: Ingesting PDFs and why Gemini 2.0 changes everything

#160

Post by serjester »

itissid wrote: post #124
lazypenguin wrote: post #27

I work in fintech and we replaced an OCR vendor with Gemini at work for ingesting some PDFs. After trial and error with different models Gemini won because it was so darn easy to use and it worked with minimal effort. I think one shouldn't underestimate that multi-modal, large context window model in terms of ease-of-use. Ironically this vendor is the best known and most successful vendor for OCR'ing this specific ty…

Wait isn't there atleast a two step process here one is semantic segmentation followed by a method like texttract for text - to avoid hallucinations? One cannot possibly say that "Text extracted by a multimodal model cannot hallucinate"? > accuracy was like 96% of that of the vendor and price was significantly cheaper. I would like to know how this 96% was tested. If you use a human to do random sample based testing,…

The LLM's are near perfect (maybe parsing I instead of 1) - if you're using the outputs in the context of RAG, your errors are likely much much higher in the other parts of your system. Spending a ton of time and money chasing 9's when 99% of your system's errors have totally different root causes seems like a bad use of time (unless they're not).
Post reply on HN