Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Vision language models are blind

vlmsareblind.github.io

Post reply on HN
Print view View original

131–140 of 202 posts

Re: Vision language models are blind

#131

Post by _lvbh »

Spivak wrote: post #127
spookie wrote: post #120

Earlier quoted context omitted.

You don't really need a LLM for OCR. Hell, I suppose they just run a python script in its VM and rephrase the output. At least that's what I would do. Perhaps the script would be a "specialist model" in a sense.

It's not that you need an LLM for OCR but the fact that an LLM can do OCR (and handwriting recognition which is much harder) despite not being made specifically for that purpose is indicative of something. The jump from knowing "this is a picture of a paper with writing on it" like what you get with CLIP to being able to reproduce what's on the paper is, to me, close enough to seeing that the difference isn't meaning…

GPT-4v is provided with OCR

Re: Vision language models are blind

#132

Post by [deleted] »

FrenchDevRemote wrote: post #47
sweezyjeezy wrote: post #8

Entertaining, but I think the conclusion is way off. > their vision is, at best, like that of a person with myopia seeing fine details as blurry is a crazy thing to write in an abstract. Did they try to probe that hypothesis at all? I could (well actually I can't) share some examples from my job of GPT-4v doing some pretty difficult fine-grained visual tasks that invalidate this. Personally, I rate this paper [1], wh…

> their vision is, at best, like that of a person with myopia seeing fine details as blurry It's not that far from reality, most models sees images in very low resolution/limited colors, so not so far from this description

[deleted]

Re: Vision language models are blind

#133

Post by infiar »

scarface_74 wrote: post #119
jordan_bonecut wrote: post #116

This is an interesting article and goes along with how I understand how such models interpret input data. I'm not sure I would characterize the results as blurry vision, but maybe an inability to process what they see in a concrete manner. All the LLMs and multi-modal models I've seen lack concrete reasoning. For instance, ask ChatGPT to perform 2 tasks, to summarize a chunk of text and to count how many words are in…

Out of curiosity, I tried your test with ChatGPT 4o https://chatgpt.com/share/79c5c6e1-e6a9-441b-acb3-54882303a8... Of course as usual, LLMs are horrible with Math. Funny enough, the next time it verified the word count by counting it out until I specifically told it to use Python https://chatgpt.com/share/79e7b922-9b0f-4df9-98d0-2cd72d7041...

This counting words task reminded me of a youtube video: https://www.youtube.com/watch?v=-9XKiOXaHlI Maybe LLMs are somehow more like monkeys.

Re: Vision language models are blind

#134

Post by energy123 »

joelburget wrote: post #128

Vision Transformers do a shocking amount of compression in the tokenizer. In the [Chameleon paper]( https://arxiv.org/pdf/2405.09818 ) they say the tokenizer "encodes a 512 × 512 image into 1024 discrete tokens from a codebook of size 8192". That's 256 pixels per token (512 * 512 / 1024). If we assume that a pixel is 24 bits (3x 8 bit channels), this implies that they've compressed 256 * 24 = 6144 bits into 13 = (log…

GPT-4o is very good at some visual tasks like optical character recognition. So the selective blindness might just be what you say here -- all of its capacity is dedicated to minimizing loss on a few narrow tasks that had the most training data (like OCR). So it's not necessarily an inherent failure of the architecture to generalize, it could just be a capacity issue that will naturally be resolved with more scale.

Re: Vision language models are blind

#135

Post by letmevoteplease »

_lvbh wrote: post #131
Spivak wrote: post #127

Earlier quoted context omitted.

It's not that you need an LLM for OCR but the fact that an LLM can do OCR (and handwriting recognition which is much harder) despite not being made specifically for that purpose is indicative of something. The jump from knowing "this is a picture of a paper with writing on it" like what you get with CLIP to being able to reproduce what's on the paper is, to me, close enough to seeing that the difference isn't meaning…

GPT-4v is provided with OCR

No reason to believe that. Open source VLMs can do OCR.[1]

[1] https://huggingface.co/spaces/opencompass/open_vlm_leaderboa...

Re: Vision language models are blind

#136

Post by dsr_ »

benreesman wrote: post #94
SomaticPirate wrote: post #18

Earlier quoted context omitted.

There are quite a few "ai apologists" in the comments but I think the title is fair when these models are marketed towards low vision people ("Be my eyes" https://www.youtube.com/watch?v=Zq710AKC1gg ) as the equivalent to human vision. These models are implied to be human level equivalents when they are not. This paper demonstrates that there are still some major gaps where simple problems confound the models in unex…

If we're throwing "citation needed" tags on stuff, how about the first sentence? "Large language models with vision capabilities (VLMs), e.g., GPT-4o and Gemini-1.5 Pro are powering countless image-text processing applications" I don't know how many a "countless" is, but I think we've gotten really sloppy in terms of what counts for LLMs as a demonstrated, durable win in a concrete task attached to well-measured outc…

"maybe I missed the totally rigorous case study on how X company became Y percent more profitable by doing Z thing with LLMs (or similar), and if so I'd be grateful for citations, but neither Google nor any of the big models seem to know about it."

Goldman Sachs recently issued a report.

https://www.goldmansachs.com/intelligence/pages/gs-research/...

"We estimate that the AI infrastructure buildout will cost over $1tn in the next several years alone, which includes spending on data centers, utilities, and applications. So, the crucial question is: What $1tn problem will AI solve? Replacing low- wage jobs with tremendously costly technology is basically the polar opposite of the prior technology transitions I’ve witnessed in my thirty years of closely following the tech industry"

Re: Vision language models are blind

#137

Post by simonw »

_lvbh wrote: post #131
Spivak wrote: post #127

Earlier quoted context omitted.

It's not that you need an LLM for OCR but the fact that an LLM can do OCR (and handwriting recognition which is much harder) despite not being made specifically for that purpose is indicative of something. The jump from knowing "this is a picture of a paper with writing on it" like what you get with CLIP to being able to reproduce what's on the paper is, to me, close enough to seeing that the difference isn't meaning…

GPT-4v is provided with OCR

That's a common misconception.

Sometimes if you upload an image to ChatGPT and ask for OCR it will run Python code that executes Tesseract, but that's effectively a bug: GPT-4 vision works much better than that, and it will use GPT-4 vision if you tell it "don't use Python" or similar.

Re: Vision language models are blind

#139

Post by itkovian_ »

sweezyjeezy wrote: post #8

Entertaining, but I think the conclusion is way off. > their vision is, at best, like that of a person with myopia seeing fine details as blurry is a crazy thing to write in an abstract. Did they try to probe that hypothesis at all? I could (well actually I can't) share some examples from my job of GPT-4v doing some pretty difficult fine-grained visual tasks that invalidate this. Personally, I rate this paper [1], wh…

I think gpt4o is probably doing some ocr as preprocessing. It's not really controversial to say the vmls today don't pick up fine grained details - we all know this. Can just look at the output of a vae to know this is true.

Re: Vision language models are blind

#140

Post by thomasahle »

itkovian_ wrote: post #139
sweezyjeezy wrote: post #8

Entertaining, but I think the conclusion is way off. > their vision is, at best, like that of a person with myopia seeing fine details as blurry is a crazy thing to write in an abstract. Did they try to probe that hypothesis at all? I could (well actually I can't) share some examples from my job of GPT-4v doing some pretty difficult fine-grained visual tasks that invalidate this. Personally, I rate this paper [1], wh…

I think gpt4o is probably doing some ocr as preprocessing. It's not really controversial to say the vmls today don't pick up fine grained details - we all know this. Can just look at the output of a vae to know this is true.

If so, it's better than any other ocr on the market.

I think they just train it on a bunch of text.

Maybe counting squares in a grid was not probably considered important enough to train for.

Post reply on HN