Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Do coding assistants "see" attached images?

news.ycombinator.com

Post reply on HN
Print view View original

1–10 of 13 posts

Ask HN: Do coding assistants "see" attached images?

#1

Post by cryptography »

I've been using Cursor and I'm genuinely curious about something.

When you paste a screenshot of a broken UI and it immediately spots the misaligned div or padding issue—is it actually doing visual analysis, or just pattern-matching against common UI bugs from training data?

The speed feels almost too fast for real vision processing. And it seems to understand spatial relationships and layout in a way that feels different from just describing an image.

Are these tools using standard vision models or is there preprocessing? How much comes from the image vs. surrounding code context?

Anyone know the technical details of what's actually happening under the hood?

Re: Ask HN: Do coding assistants "see" attached images?

#2

Post by yawpitch »

They’re closed source black boxes, not even the people who built them really know what’s happening under the hood.

That said, one can reasonably infer that an LLM-based system isn’t doing any form of visual processing at all… it’s just looking at your HTML and CSS and flagging where it diverts from the statistical mean of all such structures in the training data (modulo some stochastic wandering and that it may, somehow, have mixed some measure of Rick Astley or Goatse into its multidimensional lookup table).

Re: Ask HN: Do coding assistants "see" attached images?

#3

Post by chistev »

yawpitch wrote: post #2

They’re closed source black boxes, not even the people who built them really know what’s happening under the hood. That said, one can reasonably infer that an LLM-based system isn’t doing any form of visual processing at all… it’s just looking at your HTML and CSS and flagging where it diverts from the statistical mean of all such structures in the training data (modulo some stochastic wandering and that it may, some…

> They’re closed source black boxes, not even the people who built them really know what’s happening under the hood.

Please explain

Re: Ask HN: Do coding assistants "see" attached images?

#7

Post by iswapna_ »

chistev wrote: post #3
yawpitch wrote: post #2

They’re closed source black boxes, not even the people who built them really know what’s happening under the hood. That said, one can reasonably infer that an LLM-based system isn’t doing any form of visual processing at all… it’s just looking at your HTML and CSS and flagging where it diverts from the statistical mean of all such structures in the training data (modulo some stochastic wandering and that it may, some…

> They’re closed source black boxes, not even the people who built them really know what’s happening under the hood. Please explain

LLMs are trained to predict a bunch of tokens (GenAI produces text) from all the previous seen tokens, based on the data it was trained on. It does not understand anything about the spatial relationships like lines, objects etc in an image. "Not even the people who built them" - We have no real understanding of how LLMs work, yet. Traditional ML theory (classification/regression/clustering) largely does not apply to LLM's emergent capabilities like coding, arithmetic and reasoning. No such theory exists today. People are trying.

Re: Ask HN: Do coding assistants "see" attached images?

#8

Post by chistev »

iswapna_ wrote: post #7
chistev wrote: post #3

Earlier quoted context omitted.

> They’re closed source black boxes, not even the people who built them really know what’s happening under the hood. Please explain

LLMs are trained to predict a bunch of tokens (GenAI produces text) from all the previous seen tokens, based on the data it was trained on. It does not understand anything about the spatial relationships like lines, objects etc in an image. "Not even the people who built them" - We have no real understanding of how LLMs work, yet. Traditional ML theory (classification/regression/clustering) largely does not apply to…

If I understand you, you're saying people how built them have no idea why they work?

Re: Ask HN: Do coding assistants "see" attached images?

#9

Post by muzani »

chistev wrote: post #8
iswapna_ wrote: post #7

Earlier quoted context omitted.

LLMs are trained to predict a bunch of tokens (GenAI produces text) from all the previous seen tokens, based on the data it was trained on. It does not understand anything about the spatial relationships like lines, objects etc in an image. "Not even the people who built them" - We have no real understanding of how LLMs work, yet. Traditional ML theory (classification/regression/clustering) largely does not apply to…

If I understand you, you're saying people how built them have no idea why they work?

Yup, it's emergent behavior. This has been going for a while in ML, I believe. To be fair, we know how brains work, but we don't understand why consciousness either.

Re: Ask HN: Do coding assistants "see" attached images?

#10

Post by yawpitch »

muzani wrote: post #9
chistev wrote: post #8

Earlier quoted context omitted.

If I understand you, you're saying people how built them have no idea why they work?

Yup, it's emergent behavior. This has been going for a while in ML, I believe. To be fair, we know how brains work, but we don't understand why consciousness either.

To be truly fair, we barely know how flatworm and fruit fly brains work… we haven’t the slightest clue how human brains work. Understanding consciousness is a long way off.
Post reply on HN