Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ingesting PDFs and why Gemini 2.0 changes everything

sergey.fyi

Post reply on HN
Print view View original

281–290 of 456 posts

Re: Ingesting PDFs and why Gemini 2.0 changes everything

#281

Post by michalsustr »

brookst wrote: post #246
pigscantfly wrote: post #222

Earlier quoted context omitted.

This isn't really true unfortunately -- mixture of experts routing seems to suffer from batch non-determinism. No one has stated publicly exactly why this is, but you can easily replicate the behavior yourself or find bug reports / discussion with a bit of searching. The outcome and observed behavior of the major closed-weight LLM APIs is that a temperature of zero no longer corresponds to deterministic greedy sampli…

If temperature is zero, and weights are weights, where is the non-deterministic behavior coming from?

I recently attended a STAC conference where they claimed the GPUs themselves are not deterministic. The hand-wavy speculation is they need to temperature control the cores and the flop ops may be reordered during that process. (By temperature I mean physical temperature, not some nn sampling parameter). On such large scale of computation these small differences can show up in the actually different tokens.

Re: Ingesting PDFs and why Gemini 2.0 changes everything

#282

Post by fancyfredbot »

michalsustr wrote: post #281
brookst wrote: post #246

Earlier quoted context omitted.

If temperature is zero, and weights are weights, where is the non-deterministic behavior coming from?

I recently attended a STAC conference where they claimed the GPUs themselves are not deterministic. The hand-wavy speculation is they need to temperature control the cores and the flop ops may be reordered during that process. (By temperature I mean physical temperature, not some nn sampling parameter). On such large scale of computation these small differences can show up in the actually different tokens.

I can assure you this isn't true. Having worked with GPUs for many years in an application where consist results are important it's not only possible but actually quite easy to ensure consistent inputs produce consistent results. The temperature and clock speed do not affect the order of operations, only the speed, and this doesn't affect the results. This is the same as with any modern CPU which will also adjust clock for temperature.

Re: Ingesting PDFs and why Gemini 2.0 changes everything

#283

Post by sschueller »

silverliver wrote: post #261

We are driving full speed into a xerox 2.0 moment and this time we are doing so knowingly. At least with xerox, the errors were out of place and easy to detect by a human. I wonder how many innocent people will lose their lives or be falsely incarcerated because of this. I wonder if we will adapt our systems and procedures to account for hallucinations and "85%" accuracy. And no, outlawing use the use of AI or increa…

CCC talk about Xerox copiers changing numbers when doing OCR:

https://media.ccc.de/v/31c3_-_6558_-_de_-_saal_g_-_201412282...

Re: Ingesting PDFs and why Gemini 2.0 changes everything

#285

Post by nitwit005 »

twelve40 wrote: post #163

Isn't it amazing how one company invented a universally spread format that takes structured data from an editor (except images obviously) and converts it into a completely fucked-up unstructured form that then requires expensive voodoo magic to convert back into structured data.

People kind of dump whatever in pdf files, so I don't think a cleaner file format would do as much as you might think.

Digital fax services will generate pdf files, for example. They're just image data dumped into a pdf. Various scanners will also do so.

Re: Ingesting PDFs and why Gemini 2.0 changes everything

#286

Post by hackernewds »

vikp wrote: post #158
llm_trw wrote: post #101

This is using exactly the wrong tools at every stage of the OCR pipeline, and the cost is astronomical as a result. You don't use multimodal models to extract a wall of text from an image. They hallucinate constantly the second you get past perfect 100% high-fidelity images. You use an object detection model trained on documents to find the bounding boxes of each document section as _images_; each bounding box comes…

Marker ( https://www.github.com/VikParuchuri/marker ) works kind of like this. It uses a layout model to identify blocks and processes each one separately. The internal format is a tree of blocks, which have arbitrary fields, but can all render to html. It can write out to json, html, or markdown. I integrated gemini recently to improve accuracy in certain blocks like tables. (get initial text, then pass to gemini to…

It's funny you astroturf your own project in a thread where another is presenting tangential info about their own

Re: Ingesting PDFs and why Gemini 2.0 changes everything

#287

Post by wodenokoto »

brookst wrote: post #246
pigscantfly wrote: post #222

Earlier quoted context omitted.

This isn't really true unfortunately -- mixture of experts routing seems to suffer from batch non-determinism. No one has stated publicly exactly why this is, but you can easily replicate the behavior yourself or find bug reports / discussion with a bit of searching. The outcome and observed behavior of the major closed-weight LLM APIs is that a temperature of zero no longer corresponds to deterministic greedy sampli…

If temperature is zero, and weights are weights, where is the non-deterministic behavior coming from?

Temperature changes the distribution that is sampled, not if a distribution is sampled.

Temperature changes the softmax equation[1], not weather or not you are sampling from the softmax result or choosing the highest probability. IBM's documentation corroborates this, saying you need to set do_sample to True in order for the temperature to have any effect, e.g., T changes how we sample, not if we sample [2].

A similar discussion on openai forum also claim that the RNG might be in a different state from run to run, although I am less sure about that [3]

[1] https://pelinbalci.com/2023/10/16/Temperature_parameter.html

[2] https://www.ibm.com/think/topics/llm-temperature#:~:text=The...

[3] https://community.openai.com/t/clarifications-on-setting-tem...

Re: Ingesting PDFs and why Gemini 2.0 changes everything

#288

Post by mejutoco »

makeitdouble wrote: post #172
lazypenguin wrote: post #27

I work in fintech and we replaced an OCR vendor with Gemini at work for ingesting some PDFs. After trial and error with different models Gemini won because it was so darn easy to use and it worked with minimal effort. I think one shouldn't underestimate that multi-modal, large context window model in terms of ease-of-use. Ironically this vendor is the best known and most successful vendor for OCR'ing this specific ty…

> After trial and error with different models As a mere occasional customer I've been scanning 4 to 5 pages of the same document layout every week in gemini for half a year, and every single week the results were slightly different. To note the docs are bilingual so it could affect the results, but what stroke me is the lack of consistency, and even with the same model, running it two or three times in a row gives di…

> and every single week the results were slightly different.

This is one of the reasons why open source offline models will always be part of the solution, if not the whole solution.

Re: Ingesting PDFs and why Gemini 2.0 changes everything

#289

Post by diptanu »

We started with using LLMs for parsing at Tensorlake (https://docs.tensorlake.ai), tried Qwen, Gemini, OpenAI, pretty much everything under the sun. My thought was we could skip 5-6 years of development IDP companies have done on specialized models by going to LLMs.

On information dense pages, LLMs often hallucinate half of the times, they have trouble understanding empty cells in tables, doesn't understand checkboxes, etc.

We had to invest heavily into building a state of the art layout understanding model and finally a table structure understanding for reliability. LLMs will get there, but there are some ways to go there.

Where they do well is in VQA type use cases, ask a question, very narrowly scoped, they will work much better than OCR+Layout models, because they are much more generalizable and flexible to use.

Re: Ingesting PDFs and why Gemini 2.0 changes everything

#290

Post by cma »

vikp wrote: post #158
llm_trw wrote: post #101

This is using exactly the wrong tools at every stage of the OCR pipeline, and the cost is astronomical as a result. You don't use multimodal models to extract a wall of text from an image. They hallucinate constantly the second you get past perfect 100% high-fidelity images. You use an object detection model trained on documents to find the bounding boxes of each document section as _images_; each bounding box comes…

Marker ( https://www.github.com/VikParuchuri/marker ) works kind of like this. It uses a layout model to identify blocks and processes each one separately. The internal format is a tree of blocks, which have arbitrary fields, but can all render to html. It can write out to json, html, or markdown. I integrated gemini recently to improve accuracy in certain blocks like tables. (get initial text, then pass to gemini to…

Why process separately, if there are ink smudges, photocopier glitches, etc. wouldn't it guess some stuff better from richer context, like acronyms in rows used across the other tables?
Post reply on HN