Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Some uncomfortable truths about AI coding agents

standupforme.app

Post reply on HN
Print view View original

111–120 of 122 posts

Re: Some uncomfortable truths about AI coding agents

#111

Post by catgary »

raw_anon_1111 wrote: post #106
catgary wrote: post #105

Earlier quoted context omitted.

Generally speaking, I try to ensure that the LLM is using core abstractions throughout the codebase in a consistent manner. This makes it easier for me to review any changes it makes.

Sort of a devils advocate question. If you write and review your tests and the functional and non functional requirements and the human tests for usability pass, why does the code matter? Non functional requirements: performance, security, reliability, logging etc?

Because the code is the actual thing, tests can only show that the code fails in certain cases, they don’t actually prove the code is correct.

Re: Some uncomfortable truths about AI coding agents

#112

Post by raw_anon_1111 »

catgary wrote: post #111
raw_anon_1111 wrote: post #106

Earlier quoted context omitted.

Sort of a devils advocate question. If you write and review your tests and the functional and non functional requirements and the human tests for usability pass, why does the code matter? Non functional requirements: performance, security, reliability, logging etc?

Because the code is the actual thing, tests can only show that the code fails in certain cases, they don’t actually prove the code is correct.

If you are writing the correct tests that mirror the requirements, why wouldn’t passing tests mean the code is correct?

Re: Some uncomfortable truths about AI coding agents

#113

Post by ambicapter »

joquarky wrote: post #109
piker wrote: post #43

Earlier quoted context omitted.

> I find myself able to work on projects of greater complexity than I would have been able to otherwise Yes. Now turn off the LLM and make an improvement to that code.

No problem. 30+ years of experience isn't going to disappear any time soon.

No one says it's going to disappear overnight, they're saying it's going to atrophy.

Re: Some uncomfortable truths about AI coding agents

#114

Post by cindyllm »

raw_anon_1111 wrote: post #112
catgary wrote: post #111

Earlier quoted context omitted.

Because the code is the actual thing, tests can only show that the code fails in certain cases, they don’t actually prove the code is correct.

If you are writing the correct tests that mirror the requirements, why wouldn’t passing tests mean the code is correct?

[dead]

Re: Some uncomfortable truths about AI coding agents

#115

Post by piker »

dwohnitmok wrote: post #110
piker wrote: post #77

Earlier quoted context omitted.

It made sense to me understanding that you can have a unit-profitable API but lose money on loss-leading campaigns like Code subscriptions. Those losses are amplified by encouraging usage. Perhaps I'm mistaken.

Again, that is a statement about inference time costs, not training costs.

> More usage compounds the problem only if inference is unprofitable.

No... only if you're charging full boat for that inference. As I said above, loss-leading caps are a in play here. Obviously encouraging people to use more of basically anything that is an all-you-can-eat subscription leads to less profitability. Not sure if we're talking past each other or what.

Re: Some uncomfortable truths about AI coding agents

#116

Post by viktorianer »

raw_anon_1111 wrote: post #112
catgary wrote: post #111

Earlier quoted context omitted.

Because the code is the actual thing, tests can only show that the code fails in certain cases, they don’t actually prove the code is correct.

If you are writing the correct tests that mirror the requirements, why wouldn’t passing tests mean the code is correct?

[flagged]

Re: Some uncomfortable truths about AI coding agents

#117

Post by dwohnitmok »

piker wrote: post #115
dwohnitmok wrote: post #110

Earlier quoted context omitted.

Again, that is a statement about inference time costs, not training costs.

> More usage compounds the problem only if inference is unprofitable. No... only if you're charging full boat for that inference. As I said above, loss-leading caps are a in play here. Obviously encouraging people to use more of basically anything that is an all-you-can-eat subscription leads to less profitability. Not sure if we're talking past each other or what.

We are kind of talking past each other. I'm saying something simpler. This all goes back to the original point I made in reference to your reply to johnfn:

>> The post is factoring in training costs, not just inference.

It is not because training costs are irrelevant here. Training costs do not cause your costs to go up as you accumulate more users.

None of these calculations we're talking about include training costs. You're saying that inference is unprofitable (at least given the subscription plans). I'm simply pointing out that we are talking about inference not training as you stated earlier. You are (very accurately) not talking at all about training costs.

Re: Some uncomfortable truths about AI coding agents

#118

Post by dybber »

maplethorpe wrote: post #65
dybber wrote: post #38

Earlier quoted context omitted.

I think it depends on what you find enjoyable. I think people who like the tinkering and the actual act of coding, debugging, etc. will find it less and less fun to be in this area, but people who like to look at the big picture, and solve problems, will see that they will now be better at both getting overview of larger and larger codebases and that technical debt that was never attainable to solve before can now be…

The best way to have a big picture view of a project is to build a mental model of that project in your head. Coding with LLMs removes that ability, and replaces it with an illusion.

Well if you have experience reviewing other people’s code, it is not that different than finding an idea, asking copilot to do it, and then review just as if you had a ton of junior engineers to write code for you, which also can go too far in one direction before asking for feedback.

So it really depends on your reviewing ability how maintainable code you will get. It is a bit of effort to review something “you have done” as thoroughly as something a colleague have done. Somehow I still feel sense of ownership even though the LLM did it.

I like reviewing using GitHub’s interface, so I often do a thorough review in that familiar interface while the PR is still draft, and before I have invited others to review. If I review my own code directly in my editor when the agent is done, my brain isn’t in the right context and can get distracted or skip over something.

Re: Some uncomfortable truths about AI coding agents

#119

Post by gitaarik »

dijksterhuis wrote: post #92
gitaarik wrote: post #87

Earlier quoted context omitted.

Yeah there's no solution to prompt injection, but prompt injection in itself is not a security risk. It's about what you give the LLM access to. You can give it access to your complete DB and APIs, or you can only allow it to operate on a very specific piece of data.

> but prompt injection in itself is not a security risk sorry, but this is wrong. the only time “prompt injection” is not a security risk is if no data is ever passed as input to a model, or the model output has no bearing on anything in the world anywhere. in which case, why bother with the model/system in the first place. you can exclude this security risk from your threat model. but what you’re saying there is you…

It's only a mitigations if you already gave AI access. If you use it to for example generate some data based on research on specific data you gave it, you can feed back the AI the data to a human for human verification and then have the human have the final say. That's I think the flow we should use AI for mainly.

Re: Some uncomfortable truths about AI coding agents

#120

Post by catgary »

raw_anon_1111 wrote: post #112
catgary wrote: post #111

Earlier quoted context omitted.

Because the code is the actual thing, tests can only show that the code fails in certain cases, they don’t actually prove the code is correct.

If you are writing the correct tests that mirror the requirements, why wouldn’t passing tests mean the code is correct?

Because it doesn’t? Thats why the field of formal methods exists.
Post reply on HN