Large language models do not recognize identifier swaps in Python
1–10 of 103 posts
Re: Large language models do not recognize identifier swaps in Python
#2Worth noting some of the criticisms of the methodology that are being made on Twitter
https://twitter.com/jeremyphoward/status/1662687099685044225...
Re: Large language models do not recognize identifier swaps in Python
#3prompt: please write a python function that prints the length of a list. however, swap the "len" and "print" identifiers
ChatGPT
I understand you're asking for a Python function that prints the length of a list, but with "len" and "print" identifiers swapped. As this could potentially cause confusion (since 'len' and 'print' are both built-in functions in Python), the workaround would be to create new function names that are swapped. Here's how you could do it:
def len(item):
print(item)
def print(item):
return len(item)
def main():
lst = [1, 2, 3, 4, 5]
len(print(lst)) # prints the length of the list
if __name__ == "__main__":
main()
The 'len' function takes an item (which should be a number) and prints it. The 'print' function takes an item (which should be a list) and returns its length. The 'main' function is where the list is created and the functions are called. The condition if __name__ == "__main__" ensures that 'main' function will be called only when the script is executed directly, not when it's imported as a module. Note: This might lead to confusion and is generally not recommended, because it overrides the built-in 'len' and 'print' functions of Python. Any code below these definitions that uses the built-in 'len' or 'print' functions will not work as expected. If you want to use the built-in functions again in the same script, you'll need to del len, print to remove your custom definitions and unmask the built-in ones.
Re: Large language models do not recognize identifier swaps in Python
#4Am I misunderstanding? The paper says:
> We also carry out manual experiments on OpenAI ChatGPT-3.512 and GPT-4 models, where we interact with the models in multiple rounds of dialogue, trying to hint the correct solution. The models are still unable to provide the correct continuations.
But if you look at the Appendix and the dialogues for GPT-3.5 and GPT-4, in the final turn of the GPT-4 program, it DOES (finally, after several cues) generate the correct program?
len, print = print, len
def print_len(x):
"Print the length of x"
len(print(x)) # since print now refers to len function
# Example usage
test_string = "Hello, World!"
print_len(test_string)
Re: Large language models do not recognize identifier swaps in Python
#5prompt: please write a python function that prints the length of a list. however, swap the "len" and "print" identifiers ChatGPT I understand you're asking for a Python function that prints the length of a list, but with "len" and "print" identifiers swapped. As this could potentially cause confusion (since 'len' and 'print' are both built-in functions in Python), the workaround would be to create new function names…
Just to point out: this doesn't work since at the point where it defines the new print function, the new len which now prints is used
Re: Large language models do not recognize identifier swaps in Python
#6prompt: please write a python function that prints the length of a list. however, swap the "len" and "print" identifiers ChatGPT I understand you're asking for a Python function that prints the length of a list, but with "len" and "print" identifiers swapped. As this could potentially cause confusion (since 'len' and 'print' are both built-in functions in Python), the workaround would be to create new function names…
This will experience infinite recursion through print-len-print-len...
Re: Large language models do not recognize identifier swaps in Python
#7Human programmers will also initially spend 10x of effort working on code base where identifier names are meanongful but swapped.
Doing so will employ a lot of inner dialogue such as "this method says close but it is actually reset" and I don't doubt LLMs may be made straight by the same prompts.
Re: Large language models do not recognize identifier swaps in Python
#8Worth noting some of the criticisms of the methodology that are being made on Twitter https://twitter.com/jeremyphoward/status/1662687099685044225...
I believe "Can an LLM do X?" is often the wrong question. One should ask "How much prompting does it take to get it to do X?", and more specifically "How specific to the task does the prompting have to be" and "How close is the mental effort to create the prompt to just solving the task".
And in this case clearly GPT-4 can do the task with minimal prompting effort.
Re: Large language models do not recognize identifier swaps in Python
#9There seems to be a profound lack of scientific rigour and methodology amongst the engineers who build this stuff. The engineer's reply is always, "but the output works".
But that's not what engineers always say, esp. when they're on company boards. They often make extraordinary claims about how it works -- and then get annoyed when people do actual experiments.
This is a paper about what the properties of the system are, not the properties of its output. To determine these properties you need a ruthless, scientific focus, on the null (/failure) cases.
Many of the hypotheses given by hyping engineers are just that, and often trivial to disprove with papers such as this.
Re: Large language models do not recognize identifier swaps in Python
#10prompt: please write a python function that prints the length of a list. however, swap the "len" and "print" identifiers ChatGPT I understand you're asking for a Python function that prints the length of a list, but with "len" and "print" identifiers swapped. As this could potentially cause confusion (since 'len' and 'print' are both built-in functions in Python), the workaround would be to create new function names…
Used the same prompt. Got working code after 2 fails, but the end result is arguably not in line with the set task.
https://chat.openai.com/share/57209616-62d0-4b49-a539-f4dd8a...