Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ingesting PDFs and why Gemini 2.0 changes everything

sergey.fyi

Post reply on HN
Print view View original

41–50 of 456 posts

Re: Ingesting PDFs and why Gemini 2.0 changes everything

#41

Post by maeil »

bt3 wrote: post #11

One major takeaway that matches my own investigation is that Gemini 2.0 still materially struggles with bounding boxes on digital content. Google has published[1] some great material on spatial understanding and bounding boxes on photography, but identifying sections of text or digital graphics like icons in a presentation is still very hit and miss. -- [1]: https://github.com/google-gemini/cookbook/blob/a916686f95f4…

Have you seen any models that perform better at this? I last looked into this a year ago but at the time they were indeed quite bad at it across the board.

Re: Ingesting PDFs and why Gemini 2.0 changes everything

#42

Post by raunakchowdhuri »

cedws wrote: post #2

90% accuracy +/- 10%? What could that be useful for, that’s awfully low.

would encourage you to take a look at some of the real data here! https://huggingface.co/spaces/reducto/rd_table_bench

you'll find that most of the errors here are structural issues with the table or inability to parse some special characters. tables can get crazy!

Re: Ingesting PDFs and why Gemini 2.0 changes everything

#43

Post by beklein »

Great article, I couldn't find any details about the prompt... only the snippets of the `CHUNKING_PROMPT` and the `GET_NODE_BOUNDING_BOXES_PROMPT`.

Is there is any code example with a full prompt available from OP, or are there any references (such as similar GitHub repos) for those looking to get started within this topic?

Your insights would be highly appreciated.

Re: Ingesting PDFs and why Gemini 2.0 changes everything

#44

Post by fngjdflmdflg »

minimaxir wrote: post #28
fngjdflmdflg wrote: post #20

>Unfortunately Gemini really seems to struggle on this, and no matter how we tried prompting it, it would generate wildly inaccurate bounding boxes This is what I have found as well. From what I've read, LLMS do not work well with images for specific details due to image encoders which are too lossy. (No idea if this is actually correct.) For now I guess you can use regular OCR to get bounding boxes.

Modern multimodal encoders for LLMs are fine/not lossy since they do not resize to a small size and can handle arbitrary sizes, although some sizes are obviously better represented in the training set. A 8.5" x 11" paper would be common. I suspect the issue is prompt engineering related. > Please provide me strict bounding boxes that encompasses the following text in the attached image? I'm trying to draw a rectangle…

Just tried this and it did not appear to work for me. Prompt:

>Please provide me strict bounding boxes that encompasses the following text in the attached image? I'm trying to draw a rectangle around the text.

> - Use the top-left coordinate system

>this input document is 1080 x 1236 px. return the bounding boxes as integers

Re: Ingesting PDFs and why Gemini 2.0 changes everything

#45

Post by cccybernetic »

scottydelta wrote: post #14

This is what I am trying to figure out how to solve. My problem statement is: - Injest PDFs, summarize, and extract important information. - Have some way to overlay the extracted information on the pdf in the UI. - User can provide feedback on the overlaid info by accepting or rejecting the highlights as useful or not. - This info goes back in to the model for reinforced learning. Hoping to find something that can m…

Most PDF parsers give you coordinate data (bounding boxes) for extracted text. Use these to draw highlights over your PDF viewer - users can then click the highlights to verify if the extraction was correct.

The tricky part is maintaining a mapping between your LLM extractions and these coordinates.

One way to do it would be with two LLM passes:

  1. First pass: Extract all important information from the PDF
  2. Second pass: "Hey LLM, find where each extraction appears in these bounded text chunks"
Not the cheapest approach since you're hitting the API twice, but it's straightforward!

Re: Ingesting PDFs and why Gemini 2.0 changes everything

#46

Post by otoburb »

faxmeyourcode wrote: post #37
lazypenguin wrote: post #27

I work in fintech and we replaced an OCR vendor with Gemini at work for ingesting some PDFs. After trial and error with different models Gemini won because it was so darn easy to use and it worked with minimal effort. I think one shouldn't underestimate that multi-modal, large context window model in terms of ease-of-use. Ironically this vendor is the best known and most successful vendor for OCR'ing this specific ty…

I've been fighting trying to chunk SEC filings properly, specifically surrounding the strange and inconsistent tabular formats present in company filings. This is giving me hope that it's possible.

>>I've been fighting trying to chunk SEC filings properly, specifically surrounding the strange and inconsistent tabular formats present in company filings.

For this specific use case you can also try edgartools[1] which is a library that was relatively recently released that ingests SEC submissions and filings. They don't use OCR but (from what I can tell) directly parse the XBRL documents submitted by companies and stored in EDGAR, if they exist.

[1] https://github.com/dgunning/edgartools

Re: Ingesting PDFs and why Gemini 2.0 changes everything

#47

Post by minimaxir »

fngjdflmdflg wrote: post #44
minimaxir wrote: post #28

Earlier quoted context omitted.

Modern multimodal encoders for LLMs are fine/not lossy since they do not resize to a small size and can handle arbitrary sizes, although some sizes are obviously better represented in the training set. A 8.5" x 11" paper would be common. I suspect the issue is prompt engineering related. > Please provide me strict bounding boxes that encompasses the following text in the attached image? I'm trying to draw a rectangle…

Just tried this and it did not appear to work for me. Prompt: >Please provide me strict bounding boxes that encompasses the following text in the attached image? I'm trying to draw a rectangle around the text. > - Use the top-left coordinate system >this input document is 1080 x 1236 px. return the bounding boxes as integers

"Might" being the operative word, particularly with models that have less prompt adherence. There's a few other prompt massaging tricks beyond the scope of a HN comment, the decimal issue is just one optimization.

Re: Ingesting PDFs and why Gemini 2.0 changes everything

#48

Post by potatoman22 »

panarky wrote: post #33
lazypenguin wrote: post #27

I work in fintech and we replaced an OCR vendor with Gemini at work for ingesting some PDFs. After trial and error with different models Gemini won because it was so darn easy to use and it worked with minimal effort. I think one shouldn't underestimate that multi-modal, large context window model in terms of ease-of-use. Ironically this vendor is the best known and most successful vendor for OCR'ing this specific ty…

This is a big aha moment for me. If Gemini can do semantic chunking at the same time as extraction, all for so cheap and with nearly perfect accuracy, and without brittle prompting incantation magic, this is huge.

Small point but is it doing semantic chunking, or loading the entire pdf into context? I've heard mixed results on semantic chunking.

Re: Ingesting PDFs and why Gemini 2.0 changes everything

#50

Post by potatoman22 »

fecal_henge wrote: post #9

Is there an AI platform where I can paste a snip of a graph and it will generate a n th order polynomial regression for me of the trace?

I've had decent luck using some of the reasoning models for this. It helps if you task them with identifying where the points on the graph are first.
Post reply on HN