At least with AI you can ask it what it understands about the topic so you know what you can assume.
AI coding and the peanut butter and jelly problem
21–30 of 83 posts
Re: AI coding and the peanut butter and jelly problem
#22Earlier quoted context omitted.
In your analysis, do you account for the time taken to type a detailed specification with which to prompt the LLM? Or the time to review the code - whether by manual fixes, or iterating with the prompt, or both?
No, just the time spent typing the code.
I'm sure curiousity will get the better of me eventually, but as it stands I'm still unconvinced. Over the years I've ingrained a strong sense that just fixing things myself is easier than clearly explaining in text what needs to be done.
Re: AI coding and the peanut butter and jelly problem
#23Okay, but like. If you do have that skill to communicate clearly and describe the requirements of a novel problem, why is the AI still useful? Actually writing the code should be relatively trivial from there. If it isn't, that points to a problem with your tools/architecture/etc. Programmers IMX are, on average, far too tolerant of boilerplate.
Useful boilerplate: - documentation (reference, tutorials, overviews) - tools - logging and log analyzers - monitoring - configurability - unit tests - fuzzers - UIs - and not least: lots and lots of prototypes and iterating on ideas All of these are "trivial" once you have the main code, but they are incredibly valuable, and LLMs do a fantastic job.
I was referring specifically to boilerplate within the code itself. But sure, I can imagine some uses.
Re: AI coding and the peanut butter and jelly problem
#24At least with AI you can ask it what it understands about the topic so you know what you can assume.
You are assuming that generative AI has self-awareness.
Re: AI coding and the peanut butter and jelly problem
#25Anyone here see the CS50 peanut butter and jelly problem in person?
We did it in 3rd grade in public school.
Re: AI coding and the peanut butter and jelly problem
#26At least with AI you can ask it what it understands about the topic so you know what you can assume.
It turns out it is not a reliable approach. How a LLM works and how a LLM says it works can be completely different.
Think about it, a LLM is an autocompleter. It will give you the most probable next word each time. It doesn't mean it doesn't understand high level concepts, but in the end, it just writes stuff that is similar to its training dataset.
For example, ask it to multiply two numbers. If the number are small enough, you will get the right answer. Now ask it to explain how it did it, it will probably tell you the process as commonly taught in school, but it not actually how it did it. What it did is much weirder for us humans, and the only way to see how it actually works is to look at the internals of the neural network. The LLM can't describe it, it doesn't see inside itself, however, it has many textbooks in its training dataset, so it will grab an answer from these textbooks because that's how people answer.
Seeing how it correctly describes the multiplication process and how it can multiply small number correctly, you would assume it can also multiply large numbers (as we do), but nope, it can't, unless it has access to a separate math module, traditionally made (i.e. not a neural net).
Re: AI coding and the peanut butter and jelly problem
#27Earlier quoted context omitted.
>Actually writing the code should be relatively trivial For you, maybe. This statement assumes years of grueling training to become bilingual in a foreign programming language. And I can't type at 1000 tokens/s personally - sometimes I just want to press the voice dictate key and blab for five seconds and move on to something actually interesting.
>This statement assumes years of grueling training to become bilingual in a foreign programming language ...So, less experienced programmers are supposed to be happy that they can save time with the same technology that will convince their employers that a human isn't necessary for the position? (And, frankly, I've overall quite enjoyed the many years I've put into the craft.)
You're seeing this entirely from the perspective of people who do programming as their job. I'm seeing it from the perspective of the other 99% of society. It feels really good that they're no longer gatekept by the rigid and cryptic interfaces that prevented them from really communicating with their computer, just because it couldn't speak their native tongue.
Re: AI coding and the peanut butter and jelly problem
#28At least with AI you can ask it what it understands about the topic so you know what you can assume.
It turns out it is not a reliable approach. How a LLM works and how a LLM says it works can be completely different. Think about it, a LLM is an autocompleter. It will give you the most probable next word each time. It doesn't mean it doesn't understand high level concepts, but in the end, it just writes stuff that is similar to its training dataset. For example, ask it to multiply two numbers. If the number are smal…
> The LLM can't describe it, it doesn't see inside itself, however, it has many textbooks in its training dataset, so it will grab an answer from these textbooks because that's how people answer.
EDIT: I see now that you were referring to the answers it uses to justify the result, not the underlying computations. Sorry! You can disregard the actual comment. Leaving for completeness.
ORIGINAL COMMENT:
That's not how it works. Addition in LLMs is believed to function through different mechanisms depending on model size and architecture, but the single consistent finding across different models is that they generalize beyond the training data for at least those simple arithmetic operations.
For example: "Language models use trigonometry to do addition" https://arxiv.org/abs/2502.00873
For a different "emergent" algorithm, see Anthropic's observations: https://transformer-circuits.pub/2025/attribution-graphs/met...
Re: AI coding and the peanut butter and jelly problem
#29Anyone here see the CS50 peanut butter and jelly problem in person?
We had this in the 8th grade science class, and IMO it was much better than this Harvard version. Still PB&J with instructions. Teacher had a skeleton named "it". Anytime instructions referenced the word "it", the teacher used the skeleton in place.
Re: AI coding and the peanut butter and jelly problem
#30Okay, but like. If you do have that skill to communicate clearly and describe the requirements of a novel problem, why is the AI still useful? Actually writing the code should be relatively trivial from there. If it isn't, that points to a problem with your tools/architecture/etc. Programmers IMX are, on average, far too tolerant of boilerplate.
Exactly. This same point was mentioned on Accidental Tech Podcast last week during a section primarily about “vibe coding”. (May have been the paid-only segment)
If the LLM gets something wrong, you have to be more exact to get it to make the program do the thing you want. And when that isn’t perfect, you have to tell it exactly what you want to to do in THAT situation. And the next one. And the next one.
At that point you’re programming. It may not be the same as coding in a traditional language, but isn’t it effectively the same process? You’re having to lay out all the exact steps to take when different things happen.
So in the end have you replaced programmers or decreased the amount of programming needed? Or have you just changed the shape of the activity so it doesn’t look like what we’re used to calling programming today?
John Siracusa (one of the hosts) compared it to the idea of a fourth generation language.
From Wikipedia:
“The concept of 4GL was developed from the 1970s through the 1990s, overlapping most of the development of 3GL, with 4GLs identified as ‘non-procedural’ or ‘program-generating’ languages”.
Program generating language sounds an awful lot like what people are trying to use AI for. And these claims that we don’t need programmers anymore also sound a lot like the claims from when people were trying to make flowchart based languages. Or COBOL.
“You don’t need programmers! The managers can write their own reports”.
In fact “the term 4GL was first used formally by James Martin in his 1981 book Application Development Without Programmers” (Wikipedia again).
They keep trying. But it all ends up still being programming.