Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The last six months in LLMs in five minutes

simonwillison.net

Post reply on HN
Print view View original

581–590 of 631 posts

Re: The last six months in LLMs in five minutes

#581

Post by magicalhippo »

mrcsharp wrote: post #236
dawnerd wrote: post #176

Earlier quoted context omitted.

Also the least fun part of development. Maybe I’m the weird one but I like to just jump right in, planning every last detail before writing code is boring.

> planning every last detail before writing code is boring Not only that but you can't really plan everything. It is impossible. Without LLMs, with every line of code you are making a decision or discovering something new that must be dealt with or realizing how the current thing might impact something else and so on. There is no way for a programmer to consider all of these little things ahead of time and if an atte…

> Without LLMs, with every line of code you are making a decision or discovering something new that must be dealt with or realizing how the current thing might impact something else and so on.

Part of this is true, part of it the agents catch at least a non-trivial portion of. If you prompt it to do a review, especially with a specific angle like ensuring sustained write performance, or how it will work when the future extensions are implemented, they do often catch a lot of issues.

I agree you lose a fair bit of the sense of "it feels like I'm doing something wrong", or "this doesn't seem optimal" etc. I think the skill in using these tools is to determine when you need that control and where it doesn't really matter.

Re: The last six months in LLMs in five minutes

#582

Post by whateveracct »

16bitvoid wrote: post #514
freedomben wrote: post #460

Earlier quoted context omitted.

I've noticed that the bar for "quality" when people judge AI is often significantly higher than what they'd hold a human to. I'm not saying GP et al are doing this (I haven't looked myself), but it is a widespread pattern I've noticed both professionally and personally. I don't know why it is.

The bar isn't any higher. There's just no grace given. No one is judging a hobby project made by a human on quality, and the person who the hobby project belongs to will rarely say that their code is high quality. And in a professional setting, I think people are fine with "good enough" but they're not going to claim anything is high-quality. But people are so quick to label their vibe-coded codebase as high quality…

Grace shouldn't be given though. The code from vibe coding should pass the review bar as-is. If you need to iterate, you've defeated the purpose.

Because the end result is people committing bad code. For some random hobby project, sure who cares. But people are using this at work. The codebase is rotting in a new innovative way.

Either the bar has to be set at "actually good code comes out of vibe coding" or you have to accept that codebases are going to steadily become less usable by human coders who use their fingers to type in emacs.

Suddenly every dev needs an agent to even work with the slop. Seems like an outcome Anthropic would love though....

Re: The last six months in LLMs in five minutes

#583

Post by exabrial »

There's also an inflection point in Feb-April: Claude got considerably worse, and arguably has not really recovered since then. They claim it's fixed, but my experience it is not as great as it once was. 4.7 is still useless.

Waiting for the next event at this point. Hoping that "inference becomes cheap" when Groq hardware gets delivered.

Re: The last six months in LLMs in five minutes

#584

Post by th0ma5 »

prettyblocks wrote: post #516
hollowturtle wrote: post #512

Earlier quoted context omitted.

I'm curious. What makes you think that me sharing an example(which one of the many?) of what I actually tried would somehow add something to the conversation? What's the usefulness of just an anecdotal example? As I said we have a plenty of different envs, codebases, requirements. Things are complex. You're posing it like I tried just one time. It's been hundreds of hours of tries and I just found out what works best…

In my experience most people with the type of critique I'm seeing from you have only tried it one time or have not taken the time to invest in an environment/process that will work for agentic coding. My question is not so much about sharing a cherry picked example, but the question was more like "have you tried in earnest to make it work". That's the part that wasn't clear from your original post. But you say you ha…

[dead]

Re: The last six months in LLMs in five minutes

#585

Post by ivandotcodes »

Reading through the thread, a lot of the inflection point debate seems to come down to people talking past each other about what got better. My read is that the models themselves didn't really jump in capability around November, but the harnesses around them got considerably more reliable, and the RLVR work earlier in 2025 had been training the models specifically to behave well inside those harnesses, so when the two met you got a compounding effect that felt like a step change even though neither piece was that dramatic on its own.

I think that's probably why everyone in this thread has such different experiences - someone whose workflow is mostly asking a model for code and pasting it in would have seen modest improvement and would reasonably wonder what the fuss is about whereas someone who was already running agents on 20-step loops would have felt a much bigger shift, because the thing that used to kill those runs was the failure at step 12 cascading into garbage by step 20, and that got a lot better.

The local model story Simon kind of glosses over is interesting for the same reason - a 20GB model drawing a decent pelican on a laptop is a cute data point in isolation. The thing worth noticing is that a competent local model inside a good harness now gets you closer to frontier performance than running the frontier model without a harness does.

Re: The last six months in LLMs in five minutes

#586

Post by johndough »

simonw wrote: post #574
johndough wrote: post #567

Earlier quoted context omitted.

I've got a few. Pick whichever you like. Factor 135066410865995223349603216278805969938881475605667027524485143851526510604859533833940287150571909441798207282164471551373680419703964191743046496589274256239341020864383202110372958725762358509643110564073501508187510676594629205563685529475213500852879416377328533906109750544334999811150056977236890927563 in less than 24 hours. Come up with a way to sample from LLMs…

I should have said "hard, but realistic".

Fair enough. I am optimistic that 2 and 3 will be doable eventually.

Re: The last six months in LLMs in five minutes

#587

Post by myaccountonhn »

gib444 wrote: post #89

Starting from zero today, how would someone quickly get upto speed with the latest and greatest AI tooling on an extremely limited budget? Is the only choice to pay for the "max" plans? Or just read so much about it that you bs your way through an interview and then use the company's resources? Simon, I'm curious too how much you invest each month researching all the latest and great AI tech?

Opencode go + pi.dev is 10$ a month.

Re: The last six months in LLMs in five minutes

#588

Post by vessenes »

grey-area wrote: post #428
vessenes wrote: post #394

Earlier quoted context omitted.

To your point, I didn't even look at the code.. :) Okay, I looked at the codex code. it's super reasonable -- separation of concerns, operating on a state model, it's not over designed. I did not hate it. I also noted that codex put in a CRT simulator loop which is a nice touch. I think a year ago this would have taken a lot of back and forth and arguing; to me that's kind of the point of Simon's article -- a lot mor…

Sorry I meant the code a year ago - it took a bit more hand-holding at that point and it was a mishmash of different things, but I feel it’s just slightly easier now - still similar. Haven’t looked into this one just had a quick play. Thanks for trying it out! I think his article is for the last 6 months - my feeling is progress with LLMs has stalled recently and generated code still has problems with accuracy and co…

I agree with that. Right now, you choose:

Subtle bugs in understanding the spec but strong arch and coding (codex)

Or

Subtle bugs in implementation but good understanding of the spec (claude).

Re: The last six months in LLMs in five minutes

#589

Post by joe_the_user »

romaniv wrote: post #515

> there’s zero chance any AI lab would train a model for such a ridiculous task. A lot of people here stated that this is a ridiculous metric, but no one seems to remember that it was introduced in the initial GPT report ("Sparks of Artificial General Intelligence: Early experiments with GPT-4" [1]) by Microsoft about 3 years ago. Shortly after that it was parroted by a network of booster accounts and became a thing…

My own informal test when generative AI came out has been "a picture of an old man riding a bicycle over a river". I just ran it for chatgpt with the standard model I have (5.5). It shows the old man on an old bicycle with the bicycle on a slack line and the slack line extending over the river with a medieval village in the background.

The point is that the prompt has a subtle ambiguity - "how is the old man going over the river?". My sense is that most humans would quickly imagine a conventional bridge with a road on it leading over a river and with the river background being in an area developed enough to allow bridge going over it.

So the implication I draw is these things can find/generate stuff that roughly satisfies the conditions (and are getting better at this) but they still fail add the assumptions that people would draw.

So my conclusion is that LLMs are getting better and better at "what they" but there are going to be places where they fail to satisfy human common assumptions.

Re: The last six months in LLMs in five minutes

#590

Post by goatlover »

TeMPOraL wrote: post #451
HPsquared wrote: post #319

Earlier quoted context omitted.

That's a halting problem, I think.

And Claude can actually tackle it the same way as humans do - here in the real world, where we don't have time to let some nonsense like "mathematically proven to be unsolvable" to stand between us and our goals: it can eyeball the code and give a good enough guess .

It can certainly get in a loop burning up tokens before deciding to exit some time later.
Post reply on HN