Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Are LLM merge rates not getting better?

entropicthoughts.com

Post reply on HN
Print view View original

111–120 of 175 posts

Re: Are LLM merge rates not getting better?

#111

Post by jmalicki »

lich_king wrote: post #109
suttontom wrote: post #105

Earlier quoted context omitted.

This is unfair and dismissive of many roles. Coordination in a massive, technically complex company that has to adhere to laws and regulations is a critical role. I don't get why people shit on certain roles (I'm a SWE). Our PgMs reduce friction and help us be more productive and focused. Technical writers produce customer-facing content and code, and have nothing to do with supporting internal bureaucracy. There are…

You realize that the reason you need to manage this organizational complexity is largely because the organization is so huge?... The reality is that you could run LinkedIn with far, far fewer people. You probably need fewer than 100 for core engineering, and likely less than 1,000 overall if you include compliance, sales, and so on - especially since a lot of overseas compliance stuff is outsourced to consulting firm…

When doubling the number of employees can triple your revenue, you do it.

Keeping a website running with high uptime is not the goal. Maximizing revenue and profit is. The extra people aren't waste, they're what drive the incremental imperceptible changes that make these companies profitable.

Re: Are LLM merge rates not getting better?

#112

Post by _zagj »

juancn wrote: post #90

Well, on one hand they lack new data. Lot's of new code came out of an LLM, so it feeds back. On the other hand, LLMs tend to go for an average by their nature (if you squint enough). What's more common in their training data, it's more common in the output, so getting them better without fundamental changes, requires one to improve the training data on average too which is hard. What did improve a lot is the tooling…

> Well, on one hand they lack new data. Lot's of new code came out of an LLM, so it feeds back.

Supposedly model curation is a Big Deal at Big AI, and they're especially concerned about Ouroboros effects and poisoned data. Also people are still contributing to open source and open sourcing new projects, something that should have slowed to trickle by 2023, once it became clear that from now on, you're just providing the fuel for the machines that will ultimately render you unemployable (or less employable), and that these machines will completely disregard your license terms, including those of the most permissive licenses that seek only attribution, and that you're doing all of this for free.

Re: Are LLM merge rates not getting better?

#113

Post by _zagj »

BloondAndDoom wrote: post #52

I feel like anyone used AI coding tools before 11/25 and after 1/26 (with frontier models) will say there has been a massive jump in, there is a difference between whether LLM can do a specific task or pass some arguably arbitrary checks by maintainers vs. what the are capable of. We still have tons of gaps about how to build and maintain code with AI, but LLM themselves getting better at an unbelievable pace, even w…

> I feel like anyone used AI coding tools before 11/25 and after 1/26 (with frontier models) will say there has been a massive jump in, there is a difference between whether LLM can do a specific task or pass some arguably arbitrary checks by maintainers vs. what the are capable of.

How much of that is the model and how much of that is the tooling built around it? Also why is the tooling, specifically Claude Code, so buggy?

Re: Are LLM merge rates not getting better?

#114

Post by furyofantares »

jygg4 wrote: post #72
cj wrote: post #21

Earlier quoted context omitted.

I agree with your sentiment, but I think we've yet to see the full application of the current technology. (Even if LLMs themselves don't improve, there's significant opportunity for people to use it in ways not currently being done)

The issue with llm’s is trust. I don’t see that ever going away. Humans have learned to trust other humans over a large time scale with rules in place to control behaviour.

When the dust settles, for example if LLM's were to stop improving today, we would come to learn their exact capabilities, what they can do reliably and what they can't.

Once we know what they can do well and how to get them to do it well, and what they can't, you could say we "trust" them to do the first category well and just stop trying to get it to do the second category.

Re: Are LLM merge rates not getting better?

#115

Post by mountainriver »

mike_hearn wrote: post #3

That's an interesting claim, but I don't see it in my own work. They have got better but it's very hard to quantify. I just find myself editing their work much less these days (currently using GPT 5.4).

Yeah same, and all my coworkers feel the same.

Most of us have been coding for ages. I actually find it really odd people keep trying to disprove things that are relatively obvious with LLMs

Re: Are LLM merge rates not getting better?

#116

Post by ehnto »

lich_king wrote: post #100
curiouscube wrote: post #25

There is a decent case for this thesis to hold true especially if we look at the shift in training regimes and benchmarking over the last 1-2 years. Frontier labs don't seem to really push pure size/capability anymore, it's an all in focus on agentic AI which is mainly complex post-training regimes. There are good reasons why they don't or can't do simple param upscaling anymore, but still, it makes me bearish on AGI…

> In practice this still doesn't mean 50 % of white collar can't be automated though. Let me ask you this, though: if we wanted to, what percentage of white collar jobs could have been automated or eliminated prior to LLMs? Meta has nearly 80k employees to basically run two websites and three mobile apps. There were 18k people working at LinkedIn! Many big tech companies are massive job programs with some product on…

There are many reasons why we are seeing cuts economically, but the fact that it is possible to make such large cuts is because there were way too many people working at these companies. They had so much cheap money that they over-hired, now money isn't so cheap and they need to reduce headcount. AI need not enter the conversation to get to that point.

Re: Are LLM merge rates not getting better?

#117

Post by thesz »

yorwba wrote: post #69
wongarsu wrote: post #47

I don't find this very compelling. If you look at the actual graph they are referencing but never showing [1] there is a clear improvement from Sonnet 3.7 -> Opus 4.0 -> Sonnet 4.5. This is just hidden in their graph because they are only looking at the number of PRs that are mergable with no human feedback whatsoever (a high standard even for humans). And even if we were to agree that that's a reasonable standard, G…

Yes, I think this is basically an instance of the "emergent abilities mirage." https://arxiv.org/abs/2304.15004 If you measure completion rate on a task where a single mistake can cause a failure, you won't see noticeable improvements on that metric until all potential sources of error are close to being eliminated, and then if they do get eliminated it causes a sudden large jump in performance. That's fine if you ju…

  > until all potential sources of error are close to being eliminated
This is what PSP/TSP did - one has to (continually) review its' own work to identify most frequent sources of (user facing) defects.

  >  if you also want to predict future performance, you need to break it down into smaller components and track each of them individually.
This is also one of tenets of PSP/TSP. If you have a task with estimate longer that a day (8 hours), break it down.

This is fascinating. LLM community discovers PSP/TSP rules that were laid over more than twenty years ago.

What LLM community miss is that in PSP/TSP it is an individual software developer who is responsible to figure out what they need to look after.

What I see is that it is LLM users who try to harness LLMs with what they perceive as errors. It's not that LLMs are learning, it is that users of LLMs are trying to stronghold these LLMs with prompts.

Re: Are LLM merge rates not getting better?

#118

Post by orwin »

GaggiX wrote: post #87
orwin wrote: post #83

Earlier quoted context omitted.

Yes, slight increase in that kind of accuracy. And newer models still generate absurd stuff. Ask for an historical picture, like 'a London market in the 18th century', and it is still as historically wrong as it was 2 years ago. It is useful for fantasy/sci-fi though, I use them a lot. But I don't see the point of newer models since late 2024.

I have no idea how a London market in the 18th century should look like so I cannot challenge that but recent models like nano banana ones, Z-image (on a lesser extend) can generate images that are essentially indistinguishable from actual stock photos, this wasn't true for late 2024 models, with wonky backgrounds, too smooth skins and general lack of details (the classic AI-look that AI images had).

Mmm, I mostly generate scenes, not people, and when I do, it's always in a not photorealistic way. It is true that newer models have more details, I think that I put that into 'slight accuracy improvements', is it really major? Or is it mostly for close up people/animals and it is likely that I just didn't notice?

It will be wrong on a lot of details. Basically you would get a market scene that feels 18th century gb, but will use 18th century russian/french/Austrian details, or 19th century/20th century British artefacts, or a mix of both. And the further you go from western places, the higher the error rate is. Basically generating fiction scenes. That's pretty much my usecase, so that's fine, but I won't ever use it to illustrate a historical TTrpg.

Re: Are LLM merge rates not getting better?

#119

Post by sumeno »

sunaurus wrote: post #39

I am pretty convinced that for most types of day to day work, any perceived improvements from the latest Claude models for example were total placebo. In blind tests and with normal tasks, people would probably have no idea if they're using Opus 4.5 or 4.6.

This has basically been my experience since Sonnet 3.5. I've been working on a personal project on and off with various models and things since then and the biggest difference between then and now is that it will do larger chunks of work than it did before, but the quality of the code is not particularly better, I still have to do a lot of cleanup and it still goes off the rails pretty frequently. I have to do fewer individual prompts, but the time spent reviewing the code takes longer because I also have to mentally process and fix larger chunks of code too

Is it a better user experience now? Yes. Has it boosted my productivity on this project? Absolutely.

But it still needs a ton of hand holding for anything complicated and I still deal with tons of "OK, this bug is fixed now!" followed by manually confirming a bug still exists.

Re: Are LLM merge rates not getting better?

#120

Post by lukan »

FuckButtons wrote: post #107
BoumTAC wrote: post #81

Earlier quoted context omitted.

It's because they are getting so good it's impossible to recognize them. Haiku 4.5 is already so good it's ok for 80% (95%?) of dev tasks.

I must be writing very different software than you, I keep opus on a tight leash and it still comes to the strangest conclusions.

Very possible. Some things work like a charm on first try for me, others you can spell it out again and again. And then yet again. Something to do with training data, obviously.
Post reply on HN