We are changing our developer productivity experiment design • Hacker News Board

Re: We are changing our developer productivity experiment design

#51

Post by camgunz » Wed, Feb 25, 2026, 9:15 AM UTC

falcor84 wrote: post #24

camgunz wrote: post #9

Unless this measures the entire SDLC longitudinally (like say, over a year) I'm not interested. I too can tell Claude Code to do things all day every day, but unless we have data on the defect rate it doesn't matter at all.

Do any of those companies collect and share data on their defect rates to give you a baseline to compare against?

That's my point. It's true codegen models generate code faster than humans do. Important remaining questions are:

* How do we scale up the other parts of the SDLC (planning, feasibility analysis, design, testing, deployment, maintenance)?

* What parts--if any--of the SDLC now take more or less time? Ex: we've seemingly cut down implementation time; does that come at the cost of maintenance, and if so is it still net worth it? Do we need to hire more designers, or do more user research?

The entire world is declaring "this is the future", but we don't even have simple data like "does this produce better code".

Re: We are changing our developer productivity experiment design

#52

Post by camgunz » Wed, Feb 25, 2026, 9:22 AM UTC

pgwhalen wrote: post #35

camgunz wrote: post #9

Unless this measures the entire SDLC longitudinally (like say, over a year) I'm not interested. I too can tell Claude Code to do things all day every day, but unless we have data on the defect rate it doesn't matter at all.

I really am quite in awe of Claude Code recently, so definitely not a naysayer, but this is a really important point. It’s so easy to create code, but am I shipping that much to prod than I used to? A bit. Obviously this highly depends on your company and your setup and risk tolerance and what not.

I mean, Brooks' Mythical Man-Month says this explicitly: adding more programmers makes projects later because of coordination costs, which we haven't figured out (coordination isn't parallelization between agents, it's "oh we discovered this problem; we need to go back to design" and so on).

Re: We are changing our developer productivity experiment design

#53

Post by ben_w » Wed, Feb 25, 2026, 10:20 AM UTC

simianwords wrote: post #43

paxys wrote: post #32

Earlier quoted context omitted.

You don't need collusion, just the VC money drying up. Economic reality will set the base price.

Why would vc money dry up?

There's only so much of it to spend before they run out.

I don't pretend to have detailed domain knowledge here, I may have seen other people's GenAI output rather than reality*, but the numbers people are throwing around for this stuff sum to trillions of USD, slightly higher than other (same caveat, perhaps also GenAI output*) claims I've seen about the total supply of money in the global venture capital markets.

* I miss the days when I could make a decent guess as to which websites were reliable and which were BS

Re: We are changing our developer productivity experiment design

#54

Post by ben_w » Wed, Feb 25, 2026, 10:35 AM UTC

SpicyLemonZest wrote: post #47

keeda wrote: post #44

Earlier quoted context omitted.

I think the issue was with incomplete context. Even before the original METR study came out, there were a number of larger-scale studies that showed a 15 - 30% boost, starting as far back as 2024. I often mention them, though they require some explanation, so this thread and linked comments may be useful: https://news.ycombinator.com/item?id=46559254 However those studies never got as much airtime as the METR study,…

I tried the "don't look too closely" thing for the first time last week. I got immediately humiliated when a reviewer asked why my commit was trying to replace the correct, elegant usage of an API the class was named after with a 4-line long franken-command using a different API with incorrect semantics. It's not like I'm not trying the new stuff, on a subjective level I think AI coding is really neat, but I just can…

It depends what you're measuring.

Don't get me wrong, my experiments with true-vibe-coding (i.e. don't even look at the code) are as yours, that the result is somewhat mediocre*.

For some cases, and I try to push beyond the limits of what LLMs can do in order to find those limits, they suck. I'd describe the output as like that of an overenthusiastic junior who reinvents the wheel badly rather than using standard approaches even when told to.

For other cases, I know that mediocre code is actually good enough: well before LLMs happened, I've seen mediocre code that still resulted in the app itself being given meaningful public accolades.

* Though, as per previous comment of mine, I can't help notice that the mediocrity is doing more and more of my previous career: https://news.ycombinator.com/item?id=46989102

Re: We are changing our developer productivity experiment design

#55

Post by ben_w » Wed, Feb 25, 2026, 11:01 AM UTC

scuff3d wrote: post #45

vessenes wrote: post #41

Earlier quoted context omitted.

For the thousandth time - they. make. a. profit. Inference margin is over 60%, today. They are spending that money training ever-larger models, so they are cashflow negative , but under almost any sane GAAP treatment that does not allow one to write down all R&D upfront (capital costs of model training), they are profitable. Should this matter to you? Only if you're making financial decisions that assume that somehow…

It always makes me laugh when people say this, because its so utterly pointless. That percentage assumes literally no other costs exist besides the direct inference cost. Even if they quit trying to make better models today, there are a mountain of recurring costs that will never go away. Retraining the models with new data, replacing/upgrading old hardware, enormous infrastructure costs related to maintaining the ac…

While true, the obvious counterpoint is that open-weight models exist, that high-end desktops can run them, that said hardware doesn't yet appear to have reached the end of the road for improvements to both purchase and operational costs, and that even if it had the moment people stop having VC money to constantly churn expensive training runs for new models it suddenly makes sense to etch the weights of whatever is SOTA at that point onto a silicon wafer and run it as a much more efficient hardware circuit without wasting the overhead that comes with software doing the same thing on general-purpose hardware.

Even if the bubble burst while I was writing this comment, even if every single current LLM provider goes the way of pets.com, AltaVista, and GeoCities, that can all happen without ending vibe coding.

Re: We are changing our developer productivity experiment design

#56

Post by dsr_ » Wed, Feb 25, 2026, 11:30 AM UTC

sgillen wrote: post #4

softwaredoug wrote: post #3

I'm a bit perplexed by the developer selection effects. I get that developers want to use AI. But are they also claiming there's not still a no/low-AI population of developers? Or that their means of selection don't find these developers? Are they worried that by splitting devs into groups of AI experience they might be measuring some confounder that causes people to choose AI / not AI in their careers?

The study was designed to have devs who are comfortable with AI perform 50% of tasks with AI and 50% without. So the problem is the population of "Developers who use AI regularly but are willing to do tasks without AI" is shrinking. >> Are they worried that by splitting devs into groups of AI experience they might be measuring some confounder that causes people to choose AI / not AI in their careers? The developer sa…

An alternative hypothesis might be "Developers who consistently use AI become unable to work without AI". It used to be well known that after a year or two away from writing code, a new manager would be a much worse dev than previously. Is a similar sort of skill shift happening? If we raise a cohort of new devs who never work without AI, do they never gain the ability?

Re: We are changing our developer productivity experiment design

#57

Post by redhale » Wed, Feb 25, 2026, 1:25 PM UTC

Wowfunhappy wrote: post #29

sjaiisba wrote: post #20

Earlier quoted context omitted.

> Anthropic claims they solely use agents to code and don't modify any code manually. Have you used CC? It shows. They did not make their fortune off this, and it’s at least lost me a customer because of how sloppy it is. The model is good, and it’s why they have to gate access to it. I’d much rather use a different harness. I do think you’re on to something though. As societal wealth further concentrates among the f…

...uh, I think Claude Code is great, actually. A lot of that is indeed just the strength of the underlying model, but the local client is great too. Plan mode, checkpoints, subagents... I've been using Claude Code for a year now, and I feel like Anthropic has steadily been eliminating pain points. It's certainly a lot better than the Gemini cli!

Allow me a momentary rant...

I love Claude Code and use it all day, every day for work. I would self identify as an unofficial Claude Code evangelist amongst my coworkers and friends.

But Claude Code is buggy as hell. Flicker is still present. Plugin/skill configuration is an absolute shitshow. The docs are (very) outdated/incomplete. The docs are also poorly organized, embarrassingly so. I know Claude Code's feature set quite well, and I still have a hard time navigating their docs to find a particular thing sometimes. Did you know Claude Code supports "rules" (similar to the original Cursor Rules)? Find where they are documented, and tell me that's intuitive and discoverable. I'm sorry, but with an unlimited token (and I assume, by now, personnel) budget, there is no excuse for the literal inventors of Claude Code to have documentation this bad.

I seriously wish they would spend some more cycles on quality rather than continuing to push so many new features. I love new features, but when I can't even install a plugin properly (without manual file system manipulation) because the configuration system is so bugged, inscrutable, and incompletely documented, I think it's obvious that a rebalancing is needed. But then again, why bother if you're winning anyway?

Side note: comparing it to Gemini CLI is simply cruel. No one should ever have to use or think about Gemini CLI.

Re: We are changing our developer productivity experiment design

#58

Post by inigyou » Wed, Feb 25, 2026, 2:24 PM UTC

mock-possum wrote: post #33

daxfohl wrote: post #15

"I don't want to do this without AI" sounds like we're already well into the brain atrophy stage of this. Now what? (I'd think about it myself but....)

I don’t want to do work around the house without a fully charged battery for my ryobi either. I don’t want to go on a groccery run without my car. Using tools is not brain atrophy

Of course, those are muscle replacement tools, not brain replacement tools. Getting groceries with your car is leg muscle atrophy and sawing with a power saw is arm muscle atrophy.

Re: We are changing our developer productivity experiment design

#59

Post by vessenes » Wed, Feb 25, 2026, 3:24 PM UTC

scuff3d wrote: post #45

vessenes wrote: post #41

Earlier quoted context omitted.

For the thousandth time - they. make. a. profit. Inference margin is over 60%, today. They are spending that money training ever-larger models, so they are cashflow negative , but under almost any sane GAAP treatment that does not allow one to write down all R&D upfront (capital costs of model training), they are profitable. Should this matter to you? Only if you're making financial decisions that assume that somehow…

It always makes me laugh when people say this, because its so utterly pointless. That percentage assumes literally no other costs exist besides the direct inference cost. Even if they quit trying to make better models today, there are a mountain of recurring costs that will never go away. Retraining the models with new data, replacing/upgrading old hardware, enormous infrastructure costs related to maintaining the ac…

Anthropic.

Listen carefully to Dario’s public statements; you could just pull his most recent Dwarkesh interview for example - worth a listen in any event.

He is guilty of an engineer’s use of the word profit when he says “we never made a profit.” But he always follows up with the real story — “every model we trained has returned 2-4x in free cashflow, counting R&D and inference”

You could say “the industry is engaged in possibly ruinous competition training ever-larger models and sucking cash to do so, and in fact if anyone stops, they’ll lose forever” and those statements might be true, but to be clear the fact that these companies are posting a loss right now is a FEATURE of how R&D works, one that lets them spend more on a race. It’s not tied to the sort of financial reality accrual accounting is designed to talk about.

Re: We are changing our developer productivity experiment design

#60

Post by keeda » Wed, Feb 25, 2026, 7:12 PM UTC

SpicyLemonZest wrote: post #47

keeda wrote: post #44

Earlier quoted context omitted.

I think the issue was with incomplete context. Even before the original METR study came out, there were a number of larger-scale studies that showed a 15 - 30% boost, starting as far back as 2024. I often mention them, though they require some explanation, so this thread and linked comments may be useful: https://news.ycombinator.com/item?id=46559254 However those studies never got as much airtime as the METR study,…

I tried the "don't look too closely" thing for the first time last week. I got immediately humiliated when a reviewer asked why my commit was trying to replace the correct, elegant usage of an API the class was named after with a 4-line long franken-command using a different API with incorrect semantics. It's not like I'm not trying the new stuff, on a subjective level I think AI coding is really neat, but I just can…

Oh yeah I can see that happening, which is why I still scan the code! However, one thing I'll add is that AI-assisted coding requires adapting your workflow. Fortunately, it largely boils down to coding best-practices on steroids: docs, tests, tooling like linters, etc.

I throw tests at everything, even minor functions, preferably integration, maybe even some E2E with Playwright in web apps, at least for the happy paths. I actually pay more attention to the tests. The amazing thing is that the AI writes all of these and uses them as feedback to fix its mistakes.

But these validation guardrails are what has been driving down the issues I encounter. Without these the AI can make mistakes, and hence will require more in-depth manual review.