Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Do you have any evidence that agentic coding works?

news.ycombinator.com

Post reply on HN
Print view View original

361–370 of 478 posts

Re: Ask HN: Do you have any evidence that agentic coding works?

#361

Post by halis »

I've had the opposite experience, but keep in mind that a lot of what gets worked on in large companies is glorified legacy CRUD apps. What I mean is, these applications have already been built with little thought about architecture, best practices and testing. These apps already have design flaws and bugs galore.

In these types of applications, there's already a lot of low hanging fruit to be had from working with an LLM.

If you're on a greenfield app where you get to make those decisions at the start, then I think I would still use the LLMs but I would be mindful of what you check into the code base. You would be better off setting up the project structure yourself and coding some things as examples of how you want the app to work. Once you have some examples in place, then you can use the LLMs to repeat the process for new screens/features.

Re: Ask HN: Do you have any evidence that agentic coding works?

#362

Post by andy_ppp »

theshrike79 wrote: post #305
andy_ppp wrote: post #261

Earlier quoted context omitted.

I honestly find AI quite poor at writing good well thought through tests, potentially because: 1. writing testable code is part of writing good tests 2. testing is actually poorly done in all the training data because humans are also bad at writing tests 3. tests should be more focused around business logic and describing the application than arbitrarily testing things in an uncanny valley of AI slop

When Vibe coding/engineering I don't think of tests in the same way as when testing human written code. I use unit tests to "lock down" current behavior so an agent rummaging around feature F doesn't break features A and B and will get immediate feedback if that happens. I'm not trying to match every edge case, but focus more on end to end tests where input and output are locked golden files. "If this comes in, this…

Yeah, I need to start accepting to some degree the world has changed - in the past when I want to understand a system I'd have read the tests, but with AI I can just ask cursor to explain what the code is doing and it's fairly good at explaining the functionality to me.

I'm not sure I feel truly comfortable yet with huge blocks of code that are not cleanly understood by humans but it's happening whether I like it or not.

Re: Ask HN: Do you have any evidence that agentic coding works?

#363

Post by disgruntledphd2 »

embedding-shape wrote: post #288
trashb wrote: post #269

Earlier quoted context omitted.

Those are some high profile (celebrity) developers. I wonder if they have measured their results? I believe that the perceived speed up of AI coding is often different from reality. The following paper backs this idea https://arxiv.org/abs/2507.09089 . Can you provide data that objects this view, based on these (celebrity) developers or otherwise?

Almost off-topic, but got me curious: How can I measure this myself? Say I want to put concrete numbers to this, and actually measure, how should I approach it? My naive approach would be to just implement it twice, once together with an LLM and once without, but that has obvious flaws, most obvious that the order which you do it with impacts the results too much. So how would I actually go about and be able to provi…

> My naive approach would be to just implement it twice, once together with an LLM and once without, but that has obvious flaws, most obvious that the order which you do it with impacts the results too much.

You'd get a set of 10-15 projects, and a set of 10-15 developers. Then each developer would implement the solution with LLM assistance and without such assistance. You'd ensure that half the developers did LLM first, and the others traditional first.

You'd only be able to detect large statistical effects, but that would be a good start.

If it's just you then generate a list of potential projects and then flip a coin as to whether or not to use the LLM and record how long it takes along with a bunch of other metrics that make sense to you.

Re: Ask HN: Do you have any evidence that agentic coding works?

#364

Post by embedding-shape »

disgruntledphd2 wrote: post #363
embedding-shape wrote: post #288

Earlier quoted context omitted.

Almost off-topic, but got me curious: How can I measure this myself? Say I want to put concrete numbers to this, and actually measure, how should I approach it? My naive approach would be to just implement it twice, once together with an LLM and once without, but that has obvious flaws, most obvious that the order which you do it with impacts the results too much. So how would I actually go about and be able to provi…

> My naive approach would be to just implement it twice, once together with an LLM and once without, but that has obvious flaws, most obvious that the order which you do it with impacts the results too much. You'd get a set of 10-15 projects, and a set of 10-15 developers. Then each developer would implement the solution with LLM assistance and without such assistance. You'd ensure that half the developers did LLM fi…

The initial question was:

> wonder if they have measured their results?

Which seems to indicate that there would be a suitable way for a single individual to be able to measure this by themselves, which is why I asked.

What you're talking about is a study and beyond the scope of a single person, and also doesn't give me the information I'd need about myself.

> If it's just you then generate a list of potential projects and then flip a coin as to whether or not to use the LLM and record how long it takes along with a bunch of other metrics that make sense to you.

That sounds like I can just go by "yeah, feels like I'm faster", which I thought exactly was parent wanted to avoid...

Re: Ask HN: Do you have any evidence that agentic coding works?

#366

Post by embedding-shape »

buckwheatmilk wrote: post #357
embedding-shape wrote: post #350

Earlier quoted context omitted.

Hah, after submitting my comment, I actually though about it because I knew someone would eventually ask :) I'm fortunate enough to live a very comfortable life after working myself to death, so I think for 20,000,000 USD I'd do it, happily so. 2,000,000 would be too little. So probably between those sit the real price to purchase my morals and ethics :)

It wasn't a shot at you personally but the point of this was that AI companies are flush with money and desperate to show any kind of growth and willing to spend money to do that. I'm sure they are finding people that have some social following and will happily pocket couple extra green bills to present AI products in a positive light with little to no actual proof.

> It wasn't a shot at you personally

No shots fired, as far as I'm aware, so np :)

> they are finding people that have some social following and will happily pocket couple extra green bills to present AI products in a positive light with little to no actual proof.

No doubt about it, I don't think people realize how pervasive this really is though, people still sometimes tell me they trusted something on HN/reddit just because it was the most upvoted answer, or that they chose a product based on what was mentioned the most etc.

Re: Ask HN: Do you have any evidence that agentic coding works?

#367

Post by SXX »

I worked years as backend and desktop software programmer, then in gamedev and now back to SaaS development mostly backend. I didnt have much success with agentic coding with "agents", but had a great success with LLM code generation while keeping all the code in context with Google Gemini.

For gamedev you can really build quite complex 2D game prototype in Pygame or Unity rapidly since 20-50KLOC is enough for a lot of indie games. And it allow you to iterate and try different ideas much faster.

Most of features are either one-shots doing all changes across codebase in one prompt or require few fixing prompts only.

It really helps to isolate simulation from all else with mandatory CQRS for gamestate.

It also helps to generate markdown readmes along the way for all major systems and keep feature checklists ih header of each file. This way LLM dont lose context ot what is being generated.

Basically I generated in 2-3 weeks projects that would take 2-3 months to implement in a team simply because there is much less delay between idea of feature and testing it in some form.

Yes - ocassiinally you will fail to write proper spec or LLM fail to generate working code, but then usually it means you revert everything and rewrite the specification and try again.

So LLMs of today are certainly suitable when "good enough" is sufficient. So they are good for prototyping. Then if you want better architecture you just guide LLM to refactor complete code.

LLMs also good for small self contained projects or microservices where all relevant information fits into context.

Re: Ask HN: Do you have any evidence that agentic coding works?

#368

Post by freetonik »

> with the claim that we should move from “validating architecture” to “validating behavior.” In practice, this seems to mean: don’t look at the code; if tests and CI pass, ship it.

But... tests and CI are also code. It may be buggy, it may not cover enough, etc. It is also likely written by an LLM in this scenario. So, it's more like a move from “validating architecture” to “LLM-based self-validating”

Re: Ask HN: Do you have any evidence that agentic coding works?

#369

Post by vonneumannstan »

yunwal wrote: post #134
vonneumannstan wrote: post #69

Claude Cowork was apparently completely written by Claude Code. So this appears to yet again be a skill issue.

> apparently completely written by Claude Code https://www.promptarmor.com/resources/claude-cowork-exfiltra... > Claude Cowork Exfiltrates files That explains it

OMG that's right, no human has ever written vulnerable code! Shut it down yall the AI thing is over!! This guy nailed it!
Post reply on HN