Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Large language models do not recognize identifier swaps in Python

arxiv.org

Post reply on HN
Print view View original

71–80 of 103 posts

Re: Large language models do not recognize identifier swaps in Python

#71

Post by famouswaffles »

Al0neStar wrote: post #62
famouswaffles wrote: post #50

Earlier quoted context omitted.

Going to paste together snippets from comments on this thread. This bit is from the paper. > Typical programming languages have invariances and equivariances in their semantics that human programmers intuitively understand and exploit, such as the (near) invariance to the renaming of identifiers. This is bullshit lol. Identifier renaming may be a no-op for the code, or even for the compiler/interpreter, but it's abso…

@TeMPOraL and @og_kalu are the same people right?

I'm not lol. Which is exactly why I explicitly say I'm posting comments that have already been posted.

Re: Large language models do not recognize identifier swaps in Python

#72

Post by kenjackson »

YeGoblynQueenne wrote: post #66
TeMPOraL wrote: post #28

Earlier quoted context omitted.

Yup. The whole premise of this test is idiotic. When you give a piece of code to an LLM, it doesn't magically switch into "code focused mode". They write: > Typical programming languages have invariances and equivariances in their semantics that human programmers intuitively understand and exploit, such as the (near) invariance to the renaming of identifiers. But this is bullshit . Identifier renaming may be a no-op…

To be clear, did you have any trouble figuring out the correct answer in the motivating example in the article's introduction? I'm asking because sure, a human programmer might well get tripped up by a name swap if they had missed the swap, or forgot it, but when presented with a short test of three lines like the one in the article's motivating example, the majority of human programmers would pass the test easily. T…

I’m not sure MOST humans would pass the test. Maybe most decent SW engineers would.

Re: Large language models do not recognize identifier swaps in Python

#74

Post by TeMPOraL »

KMnO4 wrote: post #73

My pet peeve is when articles say “LLMs can’t do X” but they really mean “OpenAI’s Chat GPT”. It’s like saying “cars can’t run off gasoline” but their testing methodology is based around a Tesla.

Except right now, GPT-4 is the best LLM in terms od performance/capabilities, by a large degree; every other one is strictly inferior, and they all share the same architecture and training methodology. So it's like comparing cars from 10 years ago against a top-of-the-line Toyota from 10 years ago and top-of-the line Toyota from today (or BMW, or whoever does quality cars; I'm not a car person).

Re: Large language models do not recognize identifier swaps in Python

#75

Post by brenns10 »

famouswaffles wrote: post #50
brenns10 wrote: post #33

Earlier quoted context omitted.

I think the issue is just that, however silly or pointless the puzzle is, it's clearly understandable by any human who knows Python. You just use "len" any time you mean "print", and "print" any time you mean "len". And these large language models, despite being language models, simply cannot handle this concept.

Going to paste together snippets from comments on this thread. This bit is from the paper. > Typical programming languages have invariances and equivariances in their semantics that human programmers intuitively understand and exploit, such as the (near) invariance to the renaming of identifiers. This is bullshit lol. Identifier renaming may be a no-op for the code, or even for the compiler/interpreter, but it's abso…

> Swapping print() and len() to use in the same program? That would trip any human up, it's tailor-made to be difficult for us to process. And, as it again turns out, so it is for LLMs, because they really seem to process associations the same way we do (at the gut/intuition/first reaction level).

I would simply write the code I intend to write, then execute three replace operations in vim: %s/len/temporary/g, %s/print/len/g, %s/temporary/print/g.

It's not tripping me up at all. Maybe I don't think the same way that these models do?

Re: Large language models do not recognize identifier swaps in Python

#76

Post by TeMPOraL »

Al0neStar wrote: post #62
famouswaffles wrote: post #50

Earlier quoted context omitted.

Going to paste together snippets from comments on this thread. This bit is from the paper. > Typical programming languages have invariances and equivariances in their semantics that human programmers intuitively understand and exploit, such as the (near) invariance to the renaming of identifiers. This is bullshit lol. Identifier renaming may be a no-op for the code, or even for the compiler/interpreter, but it's abso…

@TeMPOraL and @og_kalu are the same people right?

No, we're not.

Re: Large language models do not recognize identifier swaps in Python

#77

Post by gduchgdjv »

KMnO4 wrote: post #73

My pet peeve is when articles say “LLMs can’t do X” but they really mean “OpenAI’s Chat GPT”. It’s like saying “cars can’t run off gasoline” but their testing methodology is based around a Tesla.

also it's hosted elsewhere. they may be updating it. they may redirect your calls to different buckets. some queries might perform worse as they are under load.

the whole "prompt science" is utter ridiculous to begin with because you can't even isolate what your testing. not to mention you're graduating with a paper on the 2000 equivalent of "how to find mp3s on Google" lol.

Re: Large language models do not recognize identifier swaps in Python

#78

Post by gduchgdjv »

TeMPOraL wrote: post #74
KMnO4 wrote: post #73

My pet peeve is when articles say “LLMs can’t do X” but they really mean “OpenAI’s Chat GPT”. It’s like saying “cars can’t run off gasoline” but their testing methodology is based around a Tesla.

Except right now, GPT-4 is the best LLM in terms od performance/capabilities, by a large degree; every other one is strictly inferior, and they all share the same architecture and training methodology. So it's like comparing cars from 10 years ago against a top-of-the-line Toyota from 10 years ago and top-of-the line Toyota from today (or BMW, or whoever does quality cars; I'm not a car person).

[flagged]

Re: Large language models do not recognize identifier swaps in Python

#79

Post by DiogenesKynikos »

I asked ChatGPT 3.5 to fix the code, modifying only the `print_len()` function. After some nudging, it came up with a very cheeky solution:

    def print_len(x):
        """Print the length of x"""
        original_print = __builtins__.print
        original_len = __builtins__.len
        
        __builtins__.print = original_len
        __builtins__.len = original_print
        
        print(len(x))
        
        __builtins__.print = original_print
        __builtins__.len = original_len
However, trying to run this function caused my Python to crash (as in, my interpreter crashed entirely), for reasons I don't understand.

One other funny thing is that ChatGPT lectured me about how swapping `print` and `len` is bad programming practice:

> Please note that modifying built-in function names like this is generally not recommended, as it can lead to confusion and unexpected behavior in your code. It's advisable to use different variable names to avoid such issues.

Quite right, ChatGPT.

Post reply on HN