Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI coding and the peanut butter and jelly problem

iamcharliegraham.substack.com

Post reply on HN
Print view View original

71–80 of 83 posts

Re: AI coding and the peanut butter and jelly problem

#71

Post by Arcuru »

derefr wrote: post #41
zahlman wrote: post #6

Okay, but like. If you do have that skill to communicate clearly and describe the requirements of a novel problem, why is the AI still useful? Actually writing the code should be relatively trivial from there. If it isn't, that points to a problem with your tools/architecture/etc. Programmers IMX are, on average, far too tolerant of boilerplate.

An LLM is a very effective human-solution-description / pseudocode to "the ten programming languages we use at work, where I'm only really fluent in three of them, and have to use language references for the others each time I code in them" transpiler. It also remembers CLI tool args far better than I do. Before LLMs, I would often have to sit and just read a manpage in its entirety to see if a certain command-line t…

Before LLMs there existed quite a few tools to try to help with understanding CLI options; off the top of my head there are https://github.com/tldr-pages/tldr and explainshell.com

LLMs are both more general and more useful than those tools. They're more flexible and composable, and can replace those tools with a small wrapper script. Part of the reason why the LLMs can do that though is because it has those other tools as datasets to train off of.

Re: AI coding and the peanut butter and jelly problem

#73

Post by daxfohl »

grahac wrote: post #66
daxfohl wrote: post #39

Earlier quoted context omitted.

This is what I keep coming back to. I'm sure I'm not the only one here who frequently writes the code, or at least a PoC, then writes the design doc based on it. Because the code is the most concise and precise way to specify what you really want. And writing it gives you more clarity on things you might not have thought about when writing it in a document. Unrolling that into pseudocode/English almost always gets co…

Agreed although AIs today with simple project based rules can do things like check and account for error cases, and write the appropriate unit tests for those error cases. I personally have found I can often create equivalent code in less English than typing. Also it works very well where the scope is well defined like implementing interfaces or porting a library from one language to another.

Yeah, I guess it depends how much you care about the details. Sometimes you just want a thing to get done, and there are billions of acceptable ways to do it, so whatever GPT spits out is within the realm of good enough. Sometimes you want finer control, and in those cases trying to use AI exclusively is going to take longer than writing code.

Not much different from image generation really. Sometimes AI is fine, but there's always going to be a need to drop down into photoshop when you really care about some detail. Even if you could do the same thing thing with very detailed AI prompts and some trial and error, doing the thing in photoshop will be easier.

Re: AI coding and the peanut butter and jelly problem

#74

Post by kenjackson »

daveguy wrote: post #70
aleph_minus_one wrote: post #62

Earlier quoted context omitted.

> Put it this way — I’m going to give you a text based question to solve and you have a choice to get another human to solve it (randomly selected from adults in the US) or ChatGPT, and both will be given 30 minutes to read and solve the problem — which would you choose? You wouldn't randomly selected an arbitrary adult from the USA to do a brain surgery on you, so this argument is rabulistic.

I would chose a random person from my company that was hired to work in that domain to solve problems in that domain. Yes, regardless of the position. Accountant in the domain, yes. Office organizer in the domain, yes. Essentially anyone in the domain, yes. No offense, but by restricting the selection to the general human population you're setting a low bar for LLMs here.

If the bar is for LLMs to replace domain experts about four years after introduction then yes, they are failing miserably.

But if you were to go back to 2020 and ask if your take a random human over a the state of the art AI to answer a text question you’d take the random human every time except for arithmetic (and you’d have to write it in math notation and not plain English).

And if you were to ask AI experts when would you chose an AI they’d say at least not for a decade or two, if ever.

Re: AI coding and the peanut butter and jelly problem

#75

Post by iDon »

For decades people have been dreaming of higher-level languages, where a user can simply specify what they want and not how to do it (the name of the programming language Forth derives from '4th Generation Language', reflecting this idea).

Here we are - we've arrived at the next level.

The emphasis in my prompts is specification : clear and concise, defining terms as they are introduced, and I've had good results with that. I expect that we'll see specification/prompt languages evolve, in the same way that MCP has become a defacto standard API for connecting LLMs to other applications and servers. We could use a lot of the ideas from existing specification languages, and there has been a lot of work done on this over 40+ years, but my impression is they are largely fairly strict, because their motivation was provably-correct code. The ideas can be used in a more relaxed way, because prompting fits well with rapid application development (RAD) and prototyping - I think there is a sweet spot of high productivity in a kind of REPL (read/evaluate/print loop) with symbolic references and structure embedded in free-form text.

Other comments have mentioned the importance of specification and requirements analysis, and dahlfox menions being able to patch new elements into the structure in subsequent prompts (via BASIC line number insertion).

Re: AI coding and the peanut butter and jelly problem

#76

Post by euroderf »

MBCook wrote: post #30
zahlman wrote: post #6

Okay, but like. If you do have that skill to communicate clearly and describe the requirements of a novel problem, why is the AI still useful? Actually writing the code should be relatively trivial from there. If it isn't, that points to a problem with your tools/architecture/etc. Programmers IMX are, on average, far too tolerant of boilerplate.

Exactly. This same point was mentioned on Accidental Tech Podcast last week during a section primarily about “vibe coding”. (May have been the paid-only segment) If the LLM gets something wrong, you have to be more exact to get it to make the program do the thing you want. And when that isn’t perfect, you have to tell it exactly what you want to to do in THAT situation. And the next one. And the next one. At that poi…

So, which is it ? Do you want to end up writing extremely detailed requirements, in English ? Or do you want to DIY by filling your head with software-related abstractions - in some internal mental "language" that might often be beyond words - and then translating those mental abstractions to source code ?

Re: AI coding and the peanut butter and jelly problem

#77

Post by nyclounge »

kenjackson wrote: post #63
aleph_minus_one wrote: post #62

Earlier quoted context omitted.

> Put it this way — I’m going to give you a text based question to solve and you have a choice to get another human to solve it (randomly selected from adults in the US) or ChatGPT, and both will be given 30 minutes to read and solve the problem — which would you choose? You wouldn't randomly selected an arbitrary adult from the USA to do a brain surgery on you, so this argument is rabulistic.

Brain surgery requires a license. But I do expect an arbitrary adult to be able to follow instructions. Ok. How about you give me a text based task where you would pick the random adult over the LLM?

I think you and the parent may be talking about 2 different things.

Do I want to use an LLM to do it from business owner perspective? Yeah probably it is cheaper and more convenient. Which one I want to use, depending the problem we are solving here right?

I'm more concern about the integrity of the current digital infrastructure. In that sense I would NOT trust ANY thing really important to anything digital, much less to LLM. Can I use it for exploration then require an actually human expert approval/edit. Absolutely!

As long as the digital doesn't result in significant physical or financial damage.

Edit: and for HN ppl, of course the LLM will have have to be open weight and all and run locally in a air gaped GPU, preferably in a Faraday cage.

Re: AI coding and the peanut butter and jelly problem

#78

Post by daveguy »

kenjackson wrote: post #74
daveguy wrote: post #70

Earlier quoted context omitted.

I would chose a random person from my company that was hired to work in that domain to solve problems in that domain. Yes, regardless of the position. Accountant in the domain, yes. Office organizer in the domain, yes. Essentially anyone in the domain, yes. No offense, but by restricting the selection to the general human population you're setting a low bar for LLMs here.

If the bar is for LLMs to replace domain experts about four years after introduction then yes, they are failing miserably. But if you were to go back to 2020 and ask if your take a random human over a the state of the art AI to answer a text question you’d take the random human every time except for arithmetic (and you’d have to write it in math notation and not plain English). And if you were to ask AI experts when…

I wasn't talking about how impressive AI systems are, or how far they've come. I was talking about the fact that any random human with any experience in a specific field -- even though they are not a domain expert -- is going to do better than an LLM. Or, human common sense >>>> what LLMs are doing.

Re: AI coding and the peanut butter and jelly problem

#79

Post by kenjackson »

daveguy wrote: post #78
kenjackson wrote: post #74

Earlier quoted context omitted.

If the bar is for LLMs to replace domain experts about four years after introduction then yes, they are failing miserably. But if you were to go back to 2020 and ask if your take a random human over a the state of the art AI to answer a text question you’d take the random human every time except for arithmetic (and you’d have to write it in math notation and not plain English). And if you were to ask AI experts when…

I wasn't talking about how impressive AI systems are, or how far they've come. I was talking about the fact that any random human with any experience in a specific field -- even though they are not a domain expert -- is going to do better than an LLM. Or, human common sense >>>> what LLMs are doing.

We will have to agree to disagree about your fundamental point.

Re: AI coding and the peanut butter and jelly problem

#80

Post by kazinator »

> If your “sandwich” is a product that doesn’t have an obvious recipe—a novel app, an unfamiliar UX, or a unique set of features—LLMs struggle

Bzzt, nope!

If the sandwich does not have an obvious recipe --- an app similar to many that have been written before or familiar, conventional UX, or boring features found in countless existing apps --- LLMs struggle.

Fixed it for ya!

Post reply on HN