Live data from Hacker News

Ask HN: Do you have any evidence that agentic coding works?

news.ycombinator.com

301–310 of 478 posts

Re: Ask HN: Do you have any evidence that agentic coding works?

#301

Never forget that the majority of what you see online is biased towards edge framing, being the subject matter incredible or terrible. Just as people curate their online profiles to make their lives appear more "appealing" than they actually are, they do the same curating in other areas.

> edge framing

is this a term of art? I interpreted it as "people only show off the best of the best or the worst of the worst, while the averages don't post online", though I've never heard the term "edge framing" before

Re: Ask HN: Do you have any evidence that agentic coding works?

#303
Can you update the title? "Do you have any evidence that agentic coding works?" sounds like you're seeking verifiable data, but in your post you're soliciting advice for how to use these tools. I clicked hoping for empiricism and got the same hamster wheel of "you're holding it wrong" that I see in every social media comment section about AI tooling.

Re: Ask HN: Do you have any evidence that agentic coding works?

#304
Personally, I've had great gains in terms of small personal tools built on top of CLIs, and Emacs config. I'm also able to deliver PRs that demonstrate a general principle to someone I manage even if they work outside of my most comfortable stacks. But that's not what you asked about.

I don't have direct evidence of exactly what you're looking for (particularly the part about "someone responsible for the architecture to sign off"). Sticking strictly to that last caveat may prevent you from receiving some important signal.

> the claim that we should move from “validating architecture” to “validating behavior.”

I think these people are on the right track and the reason I think that is because of how I work with people right now.

I manage the work of ~10 developers pretty closely and am called on for advice to another ~10, while also juggling demanding stakeholders. For a while now, I've only been able to do spot checks on PRs myself. I don't consider that a major part of my job anymore. The management that is most valuable is:

1) Teaching developers about quality so that they start with better code, and give better reviews to each other 2) Teaching people to focus and move in small steps 3) Investing in guardrails 4) Metrics, e.g. it doesn't matter what code is merged, it doesn't matter if a "task" is "shipped", what matters is if the metrics say that we've had the result we expected.

As I acknowledge how flimsy my review process is, my impulse is to worry about architecture and security. But metrics and guardrails exist for those things too. Opinionated stacks help, for instance SQL injection opportunities look different enough from "normal" Rails to mean that there are linters that can catch many problems, and the linters are better than I am at this job.

Some of these tools are available for agents just as they are for humans. Some of them are woefully bad or lack good ergonomics for agents, but I wouldn't bet against them becoming better.

I agree that agentic coding changes code review, but I don't think that has to inevitably / long-term mean worse.

> half of my time went into fixing the subtle mistakes it made or the duplication it introduced

A cold hard evaluation of the effectiveness of agentic coding doesn't care about what percentage of time went into fixing bad code; it cares about the total time.

That said, I find that making an agent move in many small steps (just how I would advise a human) creates far less rework.

Re: Ask HN: Do you have any evidence that agentic coding works?

#305

It really depends by what you mean by "it works". A retrospective of the last 6months. I've had great success coding infra (terraform). It at least 10x the generation of easily verifiable and tedious to write code. Results were audited to death as the client was highly regulated. Professional feature dev is hit and miss for sure, although getting better and better. We're nowhere near full agentic coding. However, by…

I honestly find AI quite poor at writing good well thought through tests, potentially because: 1. writing testable code is part of writing good tests 2. testing is actually poorly done in all the training data because humans are also bad at writing tests 3. tests should be more focused around business logic and describing the application than arbitrarily testing things in an uncanny valley of AI slop

When Vibe coding/engineering I don't think of tests in the same way as when testing human written code.

I use unit tests to "lock down" current behavior so an agent rummaging around feature F doesn't break features A and B and will get immediate feedback if that happens.

I'm not trying to match every edge case, but focus more on end to end tests where input and output are locked golden files. "If this comes in, this exact thing must come out the other end." type of thing.

The AI can figure out what went wrong if the tests fail.

Re: Ask HN: Do you have any evidence that agentic coding works?

#306

Earlier quoted context omitted.

I've never had a job where writing Javascript has been the primary language (so far it's been C++/Java/Golang). The JS Chrome Extension is a fun side project. Using Augment in a work context, I'm primarily using it for Golang and Python code, languages where I'm pretty proficient but AI tools give me a decent efficiency boost. I understand the emotional satisfaction of letting loose an easy snarky comment, of course,…

[flagged]

> If you are any good with those four languages, you are leagues ahead of anyone who does Javascript full time.

That is a priggish statement, and comes across as ignorant.

I’ve been paid to program in many different languages over the years. Typescript is what I choose for most tasks these days. I haven’t noticed any real difference between my past C#, C++, C, Java, Ruby, etc programming peers and my current JavaScript ones.

Re: Ask HN: Do you have any evidence that agentic coding works?

#307

I think one fatal flaw is letting the agent build the app from scratch. I've had huge success with agents, but only on existing apps that were architected by humans and have established conventions and guardrails. Agents are really bad at architecture, but quite good at following suit. Other things that seem to contribute to success with agents are: - Static type systems (not tacked-on like Typescript) - A test suite…

I've found Go to be the most efficient language with LLMs

The language is "small", very few keywords and hasn't changed much in a decade. It also has a built in testing system with well known patterns how to use it properly.

Along with robust linters I can be pretty confident LLMs can't mess up too badly.

They do tend to overcomplicate structures a bit and require a fresh context and "see if you can simplify this" or "make those three implement a generic interface" type of prompts to tear down some of the repetition and complexity - but again it's pretty easy with a simple language.

Re: Ask HN: Do you have any evidence that agentic coding works?

#308
post #297

Just for fun, I built a first person shooter game in UE5 from scratch using agentic coding. I've only spent a couple of months on it in my free time so far, and it isn't complete yet, but it's close enough that I could definitely release an early access version with another month or so of work. The most time consuming tasks have actually been tasks that agentic coding hasn't been able to help out with, like animation…

The best way to think about Codex is an outsourced contractor.

You give it a well-defined task, it'll putter away quietly and come back with results.

I've found it to be pretty good at code reviews or large refactoring operations, not so much building new features.

Re: Ask HN: Do you have any evidence that agentic coding works?

#309
post #193

I've been programming for 20 years, and I've always been under-estimating how long things will take (no, not pressured by anyone to give firm estimates, just talking about informally when prioritizing work order together). The other day I gave an estimate to my co-worker and he said "but how long is it really going to take, because you always finish a lot quicker than you say, you say two weeks and then it takes two…

I was expecting that evidence part that OP asked for in the top comments.

[deleted]

Re: Ask HN: Do you have any evidence that agentic coding works?

#310
> The product has to work, but the code must also be high-quality.

For me this has completely changed. The code needs to work. Bonus points when it is easy for a human to follow and has comments. But I don't do a full review anymore. I skim it and if I don't see an obvious flaw, it's lgtm. I also don't look into the bytecode / assembly to check whether the compiler did a good job.

Post reply on HN