Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We are changing our developer productivity experiment design

metr.org

Post reply on HN
Print view View original

41–50 of 62 posts

Re: We are changing our developer productivity experiment design

#41

Post by vessenes »

arctic-true wrote: post #7

Those developer quotes are tough to read. Rate limits are going to hit like a truck when the labs eventually need to make a profit.

For the thousandth time - they. make. a. profit. Inference margin is over 60%, today.

They are spending that money training ever-larger models, so they are cashflow negative, but under almost any sane GAAP treatment that does not allow one to write down all R&D upfront (capital costs of model training), they are profitable.

Should this matter to you? Only if you're making financial decisions that assume that somehow one day the "jig will be up" - i.e. please don't short these stocks when they float, or at least do so very judiciously.

Re: We are changing our developer productivity experiment design

#42

Post by vessenes »

Bnjoroge wrote: post #11

never been a better time to be a swe who doesnt or significantly limits the use of AI agents

I like this. I've bought a lot of CnC flatpack furniture in my day, and also employed a number of excellent cabinet makers. Room for both.

Re: We are changing our developer productivity experiment design

#43

Post by simianwords »

paxys wrote: post #32
simonw wrote: post #13

Earlier quoted context omitted.

At this point the AI labs would pretty much have to form an illegal price fixing cartel in order to jack the prices up, they've been competing to drive down prices for so long. They'd have to get the Chinese AI labs to go along with that price fixing too.

You don't need collusion, just the VC money drying up. Economic reality will set the base price.

Why would vc money dry up?

Re: We are changing our developer productivity experiment design

#44

Post by keeda »

SpicyLemonZest wrote: post #36
keeda wrote: post #21

> When surveyed, 30% to 50% of developers told us that they were choosing not to submit some tasks because they did not want to do them without AI. This implies we are systematically missing tasks which have high expected uplift from AI. In fact, one of the developers in the original study later revealed on Twitter that he had already done exactly that during the study, i.e. filtered out tasks he prefered not to do w…

As one of the naysayers who talked a lot about the original study, I enthusiastically endorse any attempt at all to actually measure AI productivity. An increase from 20% slowdown to 20% speedup over the past year seems broadly consistent with my understanding of how things have gone. I think I remain classified as a "naysayer", though, because the "booster" case has gone from "I'm multiple times more productive" to…

I think the issue was with incomplete context. Even before the original METR study came out, there were a number of larger-scale studies that showed a 15 - 30% boost, starting as far back as 2024. I often mention them, though they require some explanation, so this thread and linked comments may be useful: https://news.ycombinator.com/item?id=46559254

However those studies never got as much airtime as the METR study, and this has created an imbalanced perspective.

My take is that studies like this are extremely useful, but a lagging indicator of the true extent of AI-assisted coding. Especially since the latest tools are something else entirely.

I am not at the "never look at code again" stage, the old habits are just too ingrained... but I'm starting to look less frequently because I rarely find anything to fix. I can see a path from where I'm at to the outlandish claims people have been making.

Re: We are changing our developer productivity experiment design

#45

Post by scuff3d »

vessenes wrote: post #41
arctic-true wrote: post #7

Those developer quotes are tough to read. Rate limits are going to hit like a truck when the labs eventually need to make a profit.

For the thousandth time - they. make. a. profit. Inference margin is over 60%, today. They are spending that money training ever-larger models, so they are cashflow negative , but under almost any sane GAAP treatment that does not allow one to write down all R&D upfront (capital costs of model training), they are profitable. Should this matter to you? Only if you're making financial decisions that assume that somehow…

It always makes me laugh when people say this, because its so utterly pointless. That percentage assumes literally no other costs exist besides the direct inference cost.

Even if they quit trying to make better models today, there are a mountain of recurring costs that will never go away. Retraining the models with new data, replacing/upgrading old hardware, enormous infrastructure costs related to maintaining the actual platforms, data collection costs, payroll...

I'm not aware of a single player in the LLM space actually turning a profit, even if they're only providing inference.

Re: We are changing our developer productivity experiment design

#46

Post by scuff3d »

azan_ wrote: post #39
arctic-true wrote: post #7

Those developer quotes are tough to read. Rate limits are going to hit like a truck when the labs eventually need to make a profit.

Keep in mind that they make large profit on inference. Not enough to make up for losses on training but it won’t be a problem for Chinese labs which will just steal their weights.

Given that they built their businesses on wide spread copyright infringement and licence violations, I couldn't give less of a shit about people turning around and "stealing" from them

Re: We are changing our developer productivity experiment design

#47

Post by SpicyLemonZest »

keeda wrote: post #44
SpicyLemonZest wrote: post #36

Earlier quoted context omitted.

As one of the naysayers who talked a lot about the original study, I enthusiastically endorse any attempt at all to actually measure AI productivity. An increase from 20% slowdown to 20% speedup over the past year seems broadly consistent with my understanding of how things have gone. I think I remain classified as a "naysayer", though, because the "booster" case has gone from "I'm multiple times more productive" to…

I think the issue was with incomplete context. Even before the original METR study came out, there were a number of larger-scale studies that showed a 15 - 30% boost, starting as far back as 2024. I often mention them, though they require some explanation, so this thread and linked comments may be useful: https://news.ycombinator.com/item?id=46559254 However those studies never got as much airtime as the METR study,…

I tried the "don't look too closely" thing for the first time last week. I got immediately humiliated when a reviewer asked why my commit was trying to replace the correct, elegant usage of an API the class was named after with a 4-line long franken-command using a different API with incorrect semantics. It's not like I'm not trying the new stuff, on a subjective level I think AI coding is really neat, but I just can't ever figure out how to map what I get to the stories I hear.

Re: We are changing our developer productivity experiment design

#48

Post by pudsbuds »

SpicyLemonZest wrote: post #47
keeda wrote: post #44

Earlier quoted context omitted.

I think the issue was with incomplete context. Even before the original METR study came out, there were a number of larger-scale studies that showed a 15 - 30% boost, starting as far back as 2024. I often mention them, though they require some explanation, so this thread and linked comments may be useful: https://news.ycombinator.com/item?id=46559254 However those studies never got as much airtime as the METR study,…

I tried the "don't look too closely" thing for the first time last week. I got immediately humiliated when a reviewer asked why my commit was trying to replace the correct, elegant usage of an API the class was named after with a 4-line long franken-command using a different API with incorrect semantics. It's not like I'm not trying the new stuff, on a subjective level I think AI coding is really neat, but I just can…

You just have to give up and drink the koolaid...

But for real... My company started tracking commits per hour as a metric so I just commit as many times as I can. I don't get the luxury of even looking at my work now. They say it's faster but I've never seen so much tech debt delivered so quickly in my life.

Its going to be an interesting few years...

Re: We are changing our developer productivity experiment design

#49

Post by pudsbuds »

Philpax wrote: post #31
Wowfunhappy wrote: post #29

Earlier quoted context omitted.

...uh, I think Claude Code is great, actually. A lot of that is indeed just the strength of the underlying model, but the local client is great too. Plan mode, checkpoints, subagents... I've been using Claude Code for a year now, and I feel like Anthropic has steadily been eliminating pain points. It's certainly a lot better than the Gemini cli!

Functionality-wise, it's great, but it's a buggy mess, and it seems to be getting worse with each release.

I've been using deletated Claude agents in vscode and it crashes so much it's insane... I switched to copilot Claude local agents and it works much better.

Idk about this whole vibe coding thing though... Well see what happens

Re: We are changing our developer productivity experiment design

#50

Post by mewpmewp2 »

pudsbuds wrote: post #48
SpicyLemonZest wrote: post #47

Earlier quoted context omitted.

I tried the "don't look too closely" thing for the first time last week. I got immediately humiliated when a reviewer asked why my commit was trying to replace the correct, elegant usage of an API the class was named after with a 4-line long franken-command using a different API with incorrect semantics. It's not like I'm not trying the new stuff, on a subjective level I think AI coding is really neat, but I just can…

You just have to give up and drink the koolaid... But for real... My company started tracking commits per hour as a metric so I just commit as many times as I can. I don't get the luxury of even looking at my work now. They say it's faster but I've never seen so much tech debt delivered so quickly in my life. Its going to be an interesting few years...

Definitely need to stop squashing commits if that is the case! But no, seriously tracking git commit counts is absolutely ridiculous. Maybe you can have AI autonomously work on useless documentation that no one will read, with 1 commit per 100 lines of markdown?
Post reply on HN