Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Are LLM merge rates not getting better?

entropicthoughts.com

Post reply on HN
Print view View original

161–170 of 175 posts

Re: Are LLM merge rates not getting better?

#161

Post by nitwit005 »

aix1 wrote: post #141
cicko wrote: post #136

Earlier quoted context omitted.

I don't think that reviewing code is so important as reviewing results. Nobody is reviewing the IL or assembly code when they write in higher level languages. It's the end result that matters in most cases.

But we don't evolve IL or assembly code as the system evolves. We regenerate it from scratch every time. It is therefore not important whether some intermediate version of that low-level code was completely impossible to understand. It is not so with LLM-written high-level code. More often than not, it does need to be understood and maintained by someone or something. These days, I mainly focus on two things in LLM c…

We get stuck reviewing the output assembly when it's broken, and that does happen from time to time. The reason that doesn't happen often is that generation of assembly follows strict rules, which people have tried their best to test. That's not the behavior we're going to get out of a LLM.

Re: Are LLM merge rates not getting better?

#162

Post by contextfree »

aix1 wrote: post #141
cicko wrote: post #136

Earlier quoted context omitted.

I don't think that reviewing code is so important as reviewing results. Nobody is reviewing the IL or assembly code when they write in higher level languages. It's the end result that matters in most cases.

But we don't evolve IL or assembly code as the system evolves. We regenerate it from scratch every time. It is therefore not important whether some intermediate version of that low-level code was completely impossible to understand. It is not so with LLM-written high-level code. More often than not, it does need to be understood and maintained by someone or something. These days, I mainly focus on two things in LLM c…

Yes, prompts aren't analogous to higher-level code, they're analogous to wizards or something like that which were always rightly viewed with suspicion.

Re: Are LLM merge rates not getting better?

#163

Post by contextfree »

zx8080 wrote: post #149
aerhardt wrote: post #61

I feel that two things are true at the same time: 1) Something happened during 2025 that made the models (or crucially, the wrapping terminal-based apps like Claude Code or Codex) much better. I only type in the terminal anymore. 2) The quality of the code is still quite often terrible. Quadruple-nested control flow abounds. Software architecture in rather small scopes is unsound. People say AI is “good at front end”…

> People say AI is “good at front end” but I see the worst kind of atrocities there It's commonly universal to say "AI is great in X", where one is not professional in X. It's because that's how AI is designed: to output tokens according to stats, not logic, not semantic, and not meaning: stats.

Reading discussions online and comparing them to my own experience makes me feel crazy, because I've found today's LLMs and agents to be seemingly good at everything except writing code. Including everything else in software engineering around code (debugging, reviewing, reading code, brainstorming architecture, etc.) as well as discussing various questions in the humanities and sciences where I'm a dilettante. But whenever I've asked them to generate any substantial amount of code, beyond a few lines to demonstrate usage of some API I'm unfamiliar with, the results have always been terrible and I end up either throwing it out or rewriting almost all of it myself and spending more time than if I'd just written it myself from the start.

It's occurred to me that maybe this just shows that I'm better at writing code and/or worse at everything else than I'd realized.

Re: Are LLM merge rates not getting better?

#165

Post by 0x3f »

ahtihn wrote: post #159
0x3f wrote: post #144

Earlier quoted context omitted.

I think the person you're replying to is perfectly aware of the correlation, considering it was a primary feature of their comment.

Not really? The main point of their comment is that companies could be much smaller based on their experience at a much smaller company. I'm implying that big companies couldn't make as much money as they do without all the employees they have.

Their last para seems to acknowledge the correlation, but flips your assumed causal direction. I.e. they seem to be implying that the that excess money causes the complexity.

Re: Are LLM merge rates not getting better?

#166

Post by jplusequalt »

BoppreH wrote: post #147
globular-toast wrote: post #146

Earlier quoted context omitted.

On the contrary, it's the machines that have the luxury of specialisation. I'm faster than a car when scrambling up a mountain. It's easy to be fast when all you do is drive on smooth roads. I'm stronger than a hydraulic press when carrying a load over a distance. It's easy to be strong when you're limited to a few centimetres of movement. Intelligence of LLMs is a trick; they are literally trained to sound intellige…

We're probably thinking about this at very different levels. Here's what I meant: I can ask Claude for "a bilingual German-Russian poem about the side effects of the most common drugs used in anesthesia". I would bet my left shoe that if I asked people on the street, no one will do a better job than Claude. And to me, answering questions correctly is a very good metric for intelligence. We can debate whether that's r…

>I would bet my left shoe that if I asked people on the street, no one will do a better job than Claude.

Because it's unrealistic to expect such a niche population to be out and wandering around. What it sounds like you want is something that masquerades at intelligence, but is really alien.

Re: Are LLM merge rates not getting better?

#167

Post by BloondAndDoom »

_zagj wrote: post #113
BloondAndDoom wrote: post #52

I feel like anyone used AI coding tools before 11/25 and after 1/26 (with frontier models) will say there has been a massive jump in, there is a difference between whether LLM can do a specific task or pass some arguably arbitrary checks by maintainers vs. what the are capable of. We still have tons of gaps about how to build and maintain code with AI, but LLM themselves getting better at an unbelievable pace, even w…

> I feel like anyone used AI coding tools before 11/25 and after 1/26 (with frontier models) will say there has been a massive jump in, there is a difference between whether LLM can do a specific task or pass some arguably arbitrary checks by maintainers vs. what the are capable of. How much of that is the model and how much of that is the tooling built around it? Also why is the tooling, specifically Claude Code, so…

90% model if not more, look at terminal benchmark terminus tool, that mostly proves it

Re: Are LLM merge rates not getting better?

#168

Post by jmalicki »

kreyenborgi wrote: post #139
jmalicki wrote: post #110

Earlier quoted context omitted.

Meta has 80k employees to run the world's most massive engine of commerce through advertising and matching consumers to products. They build generative AI tools so people can make ads more easily. They have some of the most sophisticated tracking out there. They have shadow profiles on nearly everyone. Have you visited a website? You have a shadow profile even if you don't have a Facebook account. They know who your…

So like parent said, mostly bs jobs that would improve the product if removed

Totally fair! I think my point might be this is more malice than incompetence.

Re: Are LLM merge rates not getting better?

#169

Post by SkyPuncher »

sunaurus wrote: post #39

I am pretty convinced that for most types of day to day work, any perceived improvements from the latest Claude models for example were total placebo. In blind tests and with normal tasks, people would probably have no idea if they're using Opus 4.5 or 4.6.

4.6 has been a very, very slight regression for me, but the tradeoff is they've added better compaction - and now larger context windows. That's a reasonable tradeoff for me.

Re: Are LLM merge rates not getting better?

#170

Post by [deleted] »

utopiah wrote: post #53

I gave up on trying months ago, you can see the timeline on top of https://fabien.benetou.fr/Content/SelfHostingArtificialIntel... Truth is I'm probably wrong. I should keep on testing ... but at the same time I precisely gave up because I didn't think the trend was fast enough to keep on investing on checking it so frequently. Now I just read this kind of post, ask around (mainly arguing with comments asking for gen…

[deleted]
Post reply on HN