Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ingesting PDFs and why Gemini 2.0 changes everything

sergey.fyi

Post reply on HN
Print view View original

361–370 of 456 posts

Re: Ingesting PDFs and why Gemini 2.0 changes everything

#361

Post by tomrod »

sschueller wrote: post #283
silverliver wrote: post #261

We are driving full speed into a xerox 2.0 moment and this time we are doing so knowingly. At least with xerox, the errors were out of place and easy to detect by a human. I wonder how many innocent people will lose their lives or be falsely incarcerated because of this. I wonder if we will adapt our systems and procedures to account for hallucinations and "85%" accuracy. And no, outlawing use the use of AI or increa…

CCC talk about Xerox copiers changing numbers when doing OCR: https://media.ccc.de/v/31c3_-_6558_-_de_-_saal_g_-_201412282...

Would be nice to get a translation for a broader audience, glad folks are reporting this out!

Re: Ingesting PDFs and why Gemini 2.0 changes everything

#362

Post by sschueller »

tomrod wrote: post #361
sschueller wrote: post #283

Earlier quoted context omitted.

CCC talk about Xerox copiers changing numbers when doing OCR: https://media.ccc.de/v/31c3_-_6558_-_de_-_saal_g_-_201412282...

Would be nice to get a translation for a broader audience, glad folks are reporting this out!

There is a translated one: https://www.youtube.com/watch?v=zXXmhxbQ-hk

Re: Ingesting PDFs and why Gemini 2.0 changes everything

#363

Post by devmor »

nnurmanov wrote: post #268
devmor wrote: post #72

I think this is one of the few functional applications of LLMs that is really undeniably useful. OCR has always been “untrustworthy” (as in you cannot expect it to be 100% correct and know you must account for that) and we have long used ML algorithms for the process.

It is not OCR to blame, when you have garbage in you should not expect anything of high quality, especially with handwriting and tables and different languages. Even human beings fail to understand some documents (see doctor's prescriptions)

If OCR is a solution designed to recognize documents and it does not recognize all documents, then it is an imperfect solution.

That is not to say there is a perfect solution, but it is still the fault of the solution.

Re: Ingesting PDFs and why Gemini 2.0 changes everything

#364

Post by shermantanktop »

twelve40 wrote: post #163

Isn't it amazing how one company invented a universally spread format that takes structured data from an editor (except images obviously) and converts it into a completely fucked-up unstructured form that then requires expensive voodoo magic to convert back into structured data.

PDFs began as just postscript commands stored in a file. It’s a genius hack in a way that has become a Frankenstein’s monster.

Re: Ingesting PDFs and why Gemini 2.0 changes everything

#365

Post by mring33621 »

lazypenguin wrote: post #27

I work in fintech and we replaced an OCR vendor with Gemini at work for ingesting some PDFs. After trial and error with different models Gemini won because it was so darn easy to use and it worked with minimal effort. I think one shouldn't underestimate that multi-modal, large context window model in terms of ease-of-use. Ironically this vendor is the best known and most successful vendor for OCR'ing this specific ty…

Hi! Any guesstimate for pages/minute from your Gemini OCR experience? Thanks!

Re: Ingesting PDFs and why Gemini 2.0 changes everything

#366

Post by eichi »

If is is a vendor work, you should probably hire person who are competitive in software engineering space. And do we actually need significant amount of processing as a solution? If this is the case, common markdowned public pdfs should be open-sourced. We shouldn't repeat other's work.

Despite that, cheaper is better.

Re: Ingesting PDFs and why Gemini 2.0 changes everything

#367

Post by Dylan16807 »

TeMPOraL wrote: post #310
Dylan16807 wrote: post #297

Earlier quoted context omitted.

Why would the same software on the same GPU architecture use different commutations from run to run? Also if you're even considering fixed point math, you can use integer accumulators to add up your parallel chunks.

Why would the same multithreaded software run on the same CPU (not just architecture - the same physical chip) have its instructions execute in different order from run to run? Performance . Want things deterministic? You have to explicitly keep them in sync yourself. GPUs sport tens of thousands of parallel processors these days, which are themselves complex, and are linked together with more complexity, both hardwa…

Each thread on a CPU will go in the same order.

Why would the reduction step of a single neuron be split across multiple threads? That sounds slower and more complex than the naive method. And if you do decide to write code doing that, then just the code that reduces across multiple blocks needs to use integers, so pretty much no extra effort is needed.

Like, is there a nondeterministic-dot-product instruction baked into the GPU at a low level?

Re: Ingesting PDFs and why Gemini 2.0 changes everything

#368

Post by vikp »

alemos wrote: post #301
vikp wrote: post #158

Earlier quoted context omitted.

Marker ( https://www.github.com/VikParuchuri/marker ) works kind of like this. It uses a layout model to identify blocks and processes each one separately. The internal format is a tree of blocks, which have arbitrary fields, but can all render to html. It can write out to json, html, or markdown. I integrated gemini recently to improve accuracy in certain blocks like tables. (get initial text, then pass to gemini to…

what does marker add on top of docling?

Docling is a great project, happy to see more people building in the space.

Marker output will be higher quality than docling output across most doc types, especially with the --use_llm flag. A few specific things we do differently:

  - We have hybrid mode with gemini that merges tables across pages, improves quality on forms, etc.
  - we run an ordering model, so ordering is better for docs where the PDF orde ris bad
  - OCR is a lot better, we train our own model, surya - https://github.com/VikParuchuri/surya
  - References and links
  - Better equation conversion (soon including inline)

Re: Ingesting PDFs and why Gemini 2.0 changes everything

#369

Post by victorbjorklund »

sitkack wrote: post #348
TeMPOraL wrote: post #304

Earlier quoted context omitted.

> The problem then shifts from "can we extract this data from the PDF" to "how do we teach an LLM to extract the data we need, validate its performance, and deploy it with confidence into prod?" A smart vendor will shift into that space - they'll use that LLM themselves, and figure out some combination of finetunes, multiple LLMs, classical methods and human verification of random samples, that lets them not only "va…

Software is dead, if it isn't a prompt now, it will be a prompt in 6 months. Most of what we think software is today, will just be a UI. But UIs are also dead.

Can you prompt a salesforce replacement for an org with 100 000 employees?

Re: Ingesting PDFs and why Gemini 2.0 changes everything

#370

Post by PaulHoule »

lazypenguin wrote: post #27

I work in fintech and we replaced an OCR vendor with Gemini at work for ingesting some PDFs. After trial and error with different models Gemini won because it was so darn easy to use and it worked with minimal effort. I think one shouldn't underestimate that multi-modal, large context window model in terms of ease-of-use. Ironically this vendor is the best known and most successful vendor for OCR'ing this specific ty…

Where I work we've had great success at using LLMs to OCR paper documents that look like

https://static.foxnews.com/foxnews.com/content/uploads/2023/...

but were often written with typewriters long ago to get nice structured tabular output. Deals with text being split across lines and across pages just fine.

Post reply on HN