Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Some uncomfortable truths about AI coding agents

standupforme.app

Post reply on HN
Print view View original

91–100 of 122 posts

Re: Some uncomfortable truths about AI coding agents

#91

Post by az09mugen »

palmotea wrote: post #2

> The role change has been described by some as becoming a sort of software engineering manager, where one writes little or no code oneself but instead supervises a team of AI coding agents as if they are a team of human junior software engineers.... > In reality, though, the code review load for software engineers will gradually increase as fewer and fewer of them are expected to supervise an ever-growing number of…

That's exactly the feeling in this quotation :

"I want AI to do my laundry and dishes so that I can do art and writing, not for AI to do my art and writing so that I can do my laundry and dishes."

-- Joanna Maciejewska, author and videogame enthusiast

Re: Some uncomfortable truths about AI coding agents

#92

Post by dijksterhuis »

gitaarik wrote: post #87
dijksterhuis wrote: post #10

As someone who worked on “prompt injection” before it was called “prompt injection” for an (unfinished) phd… yeah there is only one surefire 100% fix for “prompt injection”: use deterministic solutions ie not machine learning. ---- addendum in case someone tries to make this commonly made point -- i don't use deterministic here to mean "i've pinned the ML model weights after training". i use it in reference to the pr…

Yeah there's no solution to prompt injection, but prompt injection in itself is not a security risk. It's about what you give the LLM access to. You can give it access to your complete DB and APIs, or you can only allow it to operate on a very specific piece of data.

> but prompt injection in itself is not a security risk

sorry, but this is wrong. the only time “prompt injection” is not a security risk is if no data is ever passed as input to a model, or the model output has no bearing on anything in the world anywhere.

in which case, why bother with the model/system in the first place.

you can exclude this security risk from your threat model. but what you’re saying there is you don’t believe it’s high likelihood that someone would want to or could run an attack, or that doing so would have sufficiently low impact/severity. possibly because you’ve put mitigations in place …

> It's about what you give the LLM access to. You can give it access to your complete DB and APIs, or you can only allow it to operate on a very specific piece of data.

aha! mitigations!

limiting blast radius and/or limiting access to model input. neither of which remove the security risk, but they do reduce the possible impact/severity and/or likelihood.

it’s all about the threat model ;)

Re: Some uncomfortable truths about AI coding agents

#93

Post by yason »

norir wrote: post #63

> I recognize that it is reminiscent of a few decades ago when old timers complained about the proliferation of high level programming languages and insisted they would lead to a generation of programmers lacking a proper understanding of how the system behaves beneath all that syntactic sugar and automatic garbage collection. They won’t have the foundational skills necessary to design and build quality software. And…

The things is, they were and they were not.

It's absolutely necessary that there's a line of people somewhere who will understand the path from garbage collection to assembly instructions. We can't build upon abstractions only as long as we still run stuff on physical cpus.

But it's also unequivocally true that once we have enough long-bearded oldtimers and newtimers who do understand how writing a Python expression somewhere will end up with a register write elsewhere all the others just don't — have to.

In old times, all you had was hardware and to program you had to understand hardware. But those who then did program and did understand were the few smart people who had access to hardware. Everyone else was left out. Now we have high-level languages, scripting languages, AI, what else. As long as we can maintain the link to hardware by some people, the rest can build on that.

Re: Some uncomfortable truths about AI coding agents

#94

Post by yason »

doug_durham wrote: post #55

The copyright example is naive at best. Any AI work that has been meaningfully modified by a human can be copyrighted. So if LLM generated code is used a the starting point and it is then modified in the testing and review phases you are good. Code that has not been touched by humans is very rare.

I don't see how copyright would be any issue.

Currently: human output is copyrighted so companies sign a transfer agreement with employees that anything they produce at work belongs to the employer. The employer now owns the copyright eventhough the employee, depending on jurisdiction, still owns the moral rights (which matter not much more than squat).

With AI: the company uses AI to produce code that isn't copyrighted. The company can take it like any public domain piece of software and incorporate it into their product. Their product is copyrighted by the company. There are even no moral rights that are personal (no need to mention "This product is based on work by Claude").

Re: Some uncomfortable truths about AI coding agents

#95

Post by tpetrina »

dybber wrote: post #38
palmotea wrote: post #2

> The role change has been described by some as becoming a sort of software engineering manager, where one writes little or no code oneself but instead supervises a team of AI coding agents as if they are a team of human junior software engineers.... > In reality, though, the code review load for software engineers will gradually increase as fewer and fewer of them are expected to supervise an ever-growing number of…

I think it depends on what you find enjoyable. I think people who like the tinkering and the actual act of coding, debugging, etc. will find it less and less fun to be in this area, but people who like to look at the big picture, and solve problems, will see that they will now be better at both getting overview of larger and larger codebases and that technical debt that was never attainable to solve before can now be…

One way of framing this is that people that prefer to solve problems are actually bad at tinkering and writing good code. Hence the existence of terrible codebases written by devs thet “liked to solve problems for the customers”. It is not that clear cut that problem-solvers have that in addition to the tinkering part nor it is guaranteed that tinkerers don’t like to solve problems. Two independent axis!

Re: Some uncomfortable truths about AI coding agents

#96

Post by hexaga »

dijksterhuis wrote: post #92
gitaarik wrote: post #87

Earlier quoted context omitted.

Yeah there's no solution to prompt injection, but prompt injection in itself is not a security risk. It's about what you give the LLM access to. You can give it access to your complete DB and APIs, or you can only allow it to operate on a very specific piece of data.

> but prompt injection in itself is not a security risk sorry, but this is wrong. the only time “prompt injection” is not a security risk is if no data is ever passed as input to a model, or the model output has no bearing on anything in the world anywhere. in which case, why bother with the model/system in the first place. you can exclude this security risk from your threat model. but what you’re saying there is you…

Model output that has seen user input is user input. User input can be dealt with securely.

Re: Some uncomfortable truths about AI coding agents

#97

Post by gruez »

teatro wrote: post #89
gruez wrote: post #66

Earlier quoted context omitted.

>Sounds a lot like "self-driving" cars - "they are good enough 95%+ of the time, you’re not going to pay as much attention as you should". That might be an issue for supervised "self-driving" cars (eg. tesla FSD), but not really applicable to self driving cars as a whole. Waymo seems to be doing just fine for instance.

Aren’t Waymo‘s 5% covered by some people in the Philippines? [1] And aren’t they still failing occasionally? [2] [1]( https://www.techspot.com/news/111233-waymo-admits-autopilot-... ) [2] ( https://www.bloomberg.com/news/articles/2026-01-23/waymo-pro... )

>Aren’t Waymo‘s 5% covered by some people in the Philippines? [1]

If you look at waymo's prior blog posts, you'll realize that the people in Philippines aren't making split second decisions that was implied by "you’re not going to pay as much attention as you should".

https://waymo.com/blog/2024/05/fleet-response

>And aren’t they still failing occasionally? [2]

What's the failure rate of humans? For both AI drivers and coders, if they make less mistakes than humans, that's still a win.

Re: Some uncomfortable truths about AI coding agents

#98

Post by ajam1507 »

borealis-dev wrote: post #71
jp57 wrote: post #64

Honest question: Is inference actually being sold at a loss? I.e. does inference itself cost more to run than we're being charged, or are the companies operating at a loss because they're spending more on training than the margins they get from inference?

Companies like OpenAI and Anthropic are particularly sneaky about reporting numbers, but there's good reason to believe it is: https://www.wheresyoured.at/oai_docs/

Even if an AI company spends more on inference than their revenue, that does not mean that they are selling inference at a loss.

Re: Some uncomfortable truths about AI coding agents

#99

Post by stavros »

gruez wrote: post #97
teatro wrote: post #89

Earlier quoted context omitted.

Aren’t Waymo‘s 5% covered by some people in the Philippines? [1] And aren’t they still failing occasionally? [2] [1]( https://www.techspot.com/news/111233-waymo-admits-autopilot-... ) [2] ( https://www.bloomberg.com/news/articles/2026-01-23/waymo-pro... )

>Aren’t Waymo‘s 5% covered by some people in the Philippines? [1] If you look at waymo's prior blog posts, you'll realize that the people in Philippines aren't making split second decisions that was implied by "you’re not going to pay as much attention as you should". https://waymo.com/blog/2024/05/fleet-response >And aren’t they still failing occasionally? [2] What's the failure rate of humans? For both AI drivers a…

> What's the failure rate of humans?

5x more than Waymos, last I saw.

Re: Some uncomfortable truths about AI coding agents

#100

Post by stavros »

maplethorpe wrote: post #65
dybber wrote: post #38

Earlier quoted context omitted.

I think it depends on what you find enjoyable. I think people who like the tinkering and the actual act of coding, debugging, etc. will find it less and less fun to be in this area, but people who like to look at the big picture, and solve problems, will see that they will now be better at both getting overview of larger and larger codebases and that technical debt that was never attainable to solve before can now be…

The best way to have a big picture view of a project is to build a mental model of that project in your head. Coding with LLMs removes that ability, and replaces it with an illusion.

Does the thing work like I want it in the end? Is it fast, reliable, enjoyable to use, maintainable, cheap, efficient, resilient, etc?

If so, I don't care if I wrote it by hand or with an LLM. People who think that building something with an LLM somehow dooms the something to mediocrity are engaging in magical thinking. I can simply use as much or as little LLM as will allow me to meet my quality criteria.

Post reply on HN