Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs understand nullability

dmodel.ai

Post reply on HN
Print view View original

71–80 of 143 posts

Re: LLMs understand nullability

#71

Post by jquery »

SJC_Hacker wrote: post #19
aSanchezStern wrote: post #17

Earlier quoted context omitted.

Many people don't think we have any good evidence that our brains aren't essentially the same thing: a stochastic statistical model that produces outputs based on inputs.

Thats probably the case 99% of the time. But that 1% is pretty important. For example, they are dismal at math problems that aren't just slight variations of problems they've seen before. Here's one by blackandredpenn where ChatGPT insisted the solution to problem that could be solved by high school / talented middle school students was correct, even after trying to convince it it was wrong. https://youtu.be/V0jhP7gi…

ChatGPT o1 pro mode solved it on the first try, after 8 minutes and 53 seconds of “thinking”:

https://chatgpt.com/share/67f40cd2-d088-8008-acd5-fe9a9784f3...

Re: LLMs understand nullability

#72

Post by gwern »

> Interestingly, for models up to 1 billion parameters, the loss actually starts to increase again after reaching a minimum. This might be because as training continues, the model develops more complex, non-linear representations that our simple linear probe can’t capture as well. Or it might be that the model starts to overfit on the training data and loses its more general concept of nullability.

Double descent?

Re: LLMs understand nullability

#73

Post by cess11 »

empath75 wrote: post #70
sega_sai wrote: post #44

One thing that is exciting in the text is an attempt to go away from describing whether LLM 'understands' which I would argue an ill posed question, but instead rephrase it in terms of something that can actually be measured. It would be good to list a few possible ways of interpreting 'understanding of code'. It could possibly include: 1) Type inference for the result 2) nullability 3) runtime asymptotics 4) What th…

Is there any way you can tell whether a human understands something other than by asking them a question and judging their answer? Nobody interrogates each other's internal states when judging whether someone understands a topic. All we can judge it based on are the words they produce or the actions they take in response to a situation. The way that systems or people arrive at a response is sort of an implementation…

No, most people absolutely use non-linguistic, involuntary cues when judging the responses of other people.

To not do that is commonly associated with things like being on the spectrum or cognitive deficiencies.

Re: LLMs understand nullability

#74

Post by lsy »

The article puts scare quotes around "understand" etc. to try to head off critiques around the lack of precision or scientific language, but I think this is a really good example of where casual use of these terms can get pretty misleading.

Because code LLMs have been trained on the syntactic form of the program and not its execution, it's not correct — even if the correlation between variable annotations and requested completions was perfect (which it's not) — to say that the model "understands nullability", because nullability means that under execution the variable in question can become null, which is not a state that it's possible for a model trained only on a million programs' syntax to "understand". You could get the same result if e.g. "Optional" means that the variable becomes poisonous and checking "> 0" is eating it, and "!= None" is an antidote. Human programmers can understand nullability because they've hopefully run programs and understand the semantics of making something null.

The paper could use precise, scientific language (e.g. "the presence of nullable annotation tokens correlates to activation of vectors corresponding to, and emission of, null-check tokens with high precision and accuracy") which would help us understand what we can rely on the LLM to do and what we can't. But it seems like there is some subconscious incentive to muddy how people see these models in the hopes that we start ascribing things to them that they aren't capable of.

Re: LLMs understand nullability

#75

Post by keybored »

aSanchezStern wrote: post #4
EncomLab wrote: post #2

This is like claiming a photorestor controlled night light "understands when it is dark" or that a bimetallic strip thermostat "understands temperature". You can say those words, and it's syntactically correct but entirely incorrect semantically.

The post includes this caveat. Depending on your philosophical position about sentience you might say that LLMs can't possibly "understand" anything, and the post isn't trying to have that argument. But to the extent that an LLM can "understand" anything, you can study its understanding of nullability.

People don’t use “understand” for machines in science because people may or may not believe in the sentience of machines. That would be a weird catering to panpsychism.

Re: LLMs understand nullability

#76

Post by empath75 »

cess11 wrote: post #73
empath75 wrote: post #70

Earlier quoted context omitted.

Is there any way you can tell whether a human understands something other than by asking them a question and judging their answer? Nobody interrogates each other's internal states when judging whether someone understands a topic. All we can judge it based on are the words they produce or the actions they take in response to a situation. The way that systems or people arrive at a response is sort of an implementation…

No, most people absolutely use non-linguistic, involuntary cues when judging the responses of other people. To not do that is commonly associated with things like being on the spectrum or cognitive deficiencies.

On a message board? Do you have theories about whether people on this thread understand or don't understand what they're talking about?

Re: LLMs understand nullability

#77

Post by root_axis »

nonameiguess wrote: post #10

As every fifth thread becomes some discussion of LLM capabilities, I think we need to shift the way we talk about this to be less like how we talk about software and more like how we talk about people. "LLM" is a valid category of thing in the world, but it's not a thing like Microsoft Outlook that has well-defined capabilities and limitations. It's frustrating reading these discussions that constantly devolve into o…

Encouraging the continued anthropomorphization of these models is a bad idea, especially in the context of discussing their capabilities.

Re: LLMs understand nullability

#78

Post by [deleted] »

lsy wrote: post #74

The article puts scare quotes around "understand" etc. to try to head off critiques around the lack of precision or scientific language, but I think this is a really good example of where casual use of these terms can get pretty misleading. Because code LLMs have been trained on the syntactic form of the program and not its execution, it's not correct — even if the correlation between variable annotations and request…

[deleted]

Re: LLMs understand nullability

#79

Post by root_axis »

aSanchezStern wrote: post #17
mubou wrote: post #14

Earlier quoted context omitted.

It's not age-old nor is it controversial. LLMs aren't intelligent by any stretch of the imagination. Each word/token is chosen as that which is statistically most likely to follow the previous. There is no capability for understanding in the design of an LLM. It's not a matter of opinion; this just isn't how an LLM works. Any comparison to the human brain is missing the point that an LLM only simulates one small part…

Many people don't think we have any good evidence that our brains aren't essentially the same thing: a stochastic statistical model that produces outputs based on inputs.

If you're willing to torture the analogy you can find a way to describe literally anything as a system of outputs based on inputs. In the case of the brain to LLM comparison, people are inclined to do it because they're eager to anthropomorophize something that produces text they can interpret as a speaker, but it's totally incorrect to suggest that our brains are "essentially the same thing" as LLMs. The comparison is specious even on a surface level. It's like saying that birds and planes are "essentially the same thing" because flight was achieved by modeling planes after birds.

Re: LLMs understand nullability

#80

Post by creatonez »

lsy wrote: post #74

The article puts scare quotes around "understand" etc. to try to head off critiques around the lack of precision or scientific language, but I think this is a really good example of where casual use of these terms can get pretty misleading. Because code LLMs have been trained on the syntactic form of the program and not its execution, it's not correct — even if the correlation between variable annotations and request…

> Because code LLMs have been trained on the syntactic form of the program and not its execution

What makes you think this? It has been trained on plenty of logs and traces, discussion of the behavior of various code, REPL sessions, etc. Code LLMs are trained on all human language and wide swaths of whatever machine-generated text is available, they are not restricted to just code.

Post reply on HN