Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The last six months in LLMs in five minutes

simonwillison.net

Post reply on HN
Print view View original

461–470 of 631 posts

Re: The last six months in LLMs in five minutes

#461

Post by JohnKemeny »

vessenes wrote: post #291
emil-lp wrote: post #144

Earlier quoted context omitted.

Yes, the "pelican riding a bicycle" is the ultimate test of not understanding how LLMs work. Well, a combination of that and believing that replication of test data is a good measure of progress.

Spicy — why does it show ultimate non-understanding?

because success comes from reproducing a memorized pattern rather than transferable reasoning?

At the same time failure proves little because most humans also could not manually create a correct SVG of a pelican riding a bicycle.

What is it exactly that such a test is testing?

In which situation would you measure the "competence" of a human being by asking them to write an SVG of a pelican riding a bicycle?

Re: The last six months in LLMs in five minutes

#462

Post by hollowturtle »

epolanski wrote: post #418
hollowturtle wrote: post #409

Earlier quoted context omitted.

>> But we should stop talking about 1s and 0s > I agree, but you contradicted yourself just one line above. >> > For generating production code even with a lot of steering and baby sitting? Absolutely not with this last sentence I obviously meant in my experience, it's not that hard. I don't buy your facts are highly biased towards web development, that's a common mistake here on HN to think it's the totality of the…

I've quoted you two tools (Ghostty and Redis) whose development now regularly uses AI assistance to deliver production code. I quoted those because their authors shared their experiences, the strengths and the limits of the tooling. There's many more, from Flask to Docker, from Ruby to FastAPI or Tanstack. LLVM has integrated AI-generated PRs, so did Swift and Mojo. Sasha Levin has pushed into Linux Nvidia-related ke…

You cited mostly web tech, which proves my point ;) Is antirez uses extensively agents to contribute to redis doesn't mean it's a becoming industry trend. I'd say quite the contrary, it isn't in the gaming industry for example, where novel ideas matter. And btw Antirez and Linus for example, put a lot of effort into steering agent into doing the right thing for them which is totally different than "these tools become just good"

Re: The last six months in LLMs in five minutes

#463

Post by ShinyLeftPad »

munksbeer wrote: post #392
ShinyLeftPad wrote: post #52

Earlier quoted context omitted.

If you're famous, you'll be fine. If you're in retiring age, you don't care. Otherwise, good luck! We put ourselves on the street by not protesting what is happening.

I think the general population earning median wages will have very little sympathy for first world software engineers earning vast amounts of money. What are you going to tell them? Suddenly you're earning what they're earning for sitting at a desk every day?

General population, you mean non SWEs? Because there are many SWEs around the world who earn median wage and who stand to lose it all as the avalanche of firings is ramping up.

Non SWEs (salespeople, clerks, secretaries, assistants, taxi drivers, writers, 3D modelers, artists, designers) are of course going the same way. Unless they are protected (unionized or such), why would they have sympathy for SWEs? People of our ilk are the ones causing this (to them and to ourselves). What I will tell them is to not repeat our mistake, organize and protest.

Re: The last six months in LLMs in five minutes

#464

Post by forlorn_mammoth »

Glohrischi wrote: post #402
hollowturtle wrote: post #209

> The coding agents got really good It's since november 2025, the so called "inflection point", that I'm still wondering for who coding agents become "really good". All I observe they got better at tool call and answering questions about big codebases, especially if the question has a vague pattern to search, and they're superuseful for that! For generating production code even with a lot of steering and baby sitting…

I had a really fun day yesterday because anthropics limits on their normal 20$ subscription allowed me to play around for the whole day without hitting a limit. Its 'production' code because its a small browser game which has very small to 0 requirements on security and being perfect but high requirements on 'ever even doing this' and 'fun'. The code it generated hat 0 compiletime errors. I was able to descripe 10 th…

> The code it generated hat 0 compiletime errors

And no spelling errors either!

Also,

> Really? What duplication did you actually find? I count a few small ones in buildMounts and ReadPrompt, maybe 20 lines or so, but hardly anything worthy of such an epithet

>> embedding-shape 1 hour ago | root | parent | next [–]

>>The duplication I'm seeing isn't just "same text repeated" but structural duplication. Doing a quick 5 minute look again just to give you some pointers; runtime.MountSpec construction in buildMounts, Workdir vs aux-dir mount-mode handling, repeated one-off mount append blocks, overlay detection and so on, the list goes on. Just those should account for 200+ lines.

If you don't see any errors or problems, is it because there aren't any problems to see, or because they take a trained eye to spot?

Re: The last six months in LLMs in five minutes

#465

Post by hollowturtle »

DennisP wrote: post #458
hollowturtle wrote: post #209

> The coding agents got really good It's since november 2025, the so called "inflection point", that I'm still wondering for who coding agents become "really good". All I observe they got better at tool call and answering questions about big codebases, especially if the question has a vague pattern to search, and they're superuseful for that! For generating production code even with a lot of steering and baby sitting…

Steve Yegge wrote about this in his book Vibe Coding . He says it takes about a year of experience before you're consistently getting good results. He writes about lots of different techniques for doing that, but also says a lot of it comes down to just getting a feel for when the LLM is going to go haywire.

It's been 4 years of using them for me, before writing a book I'd wait to have a decade of experience to share with others, otherwise it would have the same value as a book on a react tutorial

Re: The last six months in LLMs in five minutes

#466

Post by biophysboy »

sigmoid10 wrote: post #315
GistNoesis wrote: post #285

Last 6 months is humanity losing control of LLMs. - Memory market cornering which mitigated the adoption of local AI despite great open model being released. - Fast penetration of IP exfiltrating tools in companies world-wide. - Developers producing more code that they can read. - Autonomous agents killing Open Source by siphoning the attention economy - Autonomous agents destroyed online communities (including HN) -…

If you only read bad news (i.e. mass news these days since that sells better) this will be the picture. But I have personally seen some insane stuff happen in biotech. Like, I can't believe we're lucky enough to possibly live our life in this kind of future. We already have actual therepeutics developed using Alphafold being tested right now in real clinical trials, but the next generation of stuff that will go into…

Protein structure is not a rate-limiting step in drug discovery.

Re: The last six months in LLMs in five minutes

#467

Post by freedomben »

timr wrote: post #348
kstenerud wrote: post #347

Earlier quoted context omitted.

I haven't done any CSS/HTML/JS level work with Claude yet. I've mainly been using it for systems level stuff. LLMs have traditionally had problems with visual rendering (the good ol' pelican on the bicycle test). I wonder if this is more of the same?

In this case, the visual display was fine -- I was instructing it to fix bad code from a previous round that happened to deliver the right results. Like I said, this is just an example that happens to be CSS. I see this stuff daily, if not hourly.

Great example.

Just IME, the quality of the prompt often significantly affects whether it does bad stuff like your example. It's not easy by any stretch and I'm still getting there, but I'm up to a couple dozen or so "Agent Instructions" in my CLAUDE.md files for various projects that have to say things like: "when doing TDD, don't write tests to verify bug fixes in tests" because the agent is really good at following things literally. I am sure it will continue to improve, but until then every project needs some bandaid things like that.

Re: The last six months in LLMs in five minutes

#468

Post by vdelpuerto »

Philip-J-Fry wrote: post #248
kstenerud wrote: post #219

Earlier quoted context omitted.

The polarization comes from the very disparate coding experiences and output quality that different people find when using these tools. For example, I've had the opposite experience of yours, generating very high quality work using Claude (such as https://github.com/kstenerud/yoloai ). Just in dealing with all the bugs and idiosyncrasies in the technologies I'm using, the agent has been a godsend in discovering and c…

I don't want to offend (it's AI coded anyway :)) but that does not scream "high quality" to me. The headline gif on that repo just paints a terrible picture. It can't draw a box correctly, there's random underscores all over the screen. The UI itself is just incredibly incoherent. I don't even know what I'm looking at. Like, no it doesn't seem like very high quality work... It just seems like a vibe coded tool. Edit:…

That is the same fight the 2D animators were having with 3D aninmation 30 years ago. The resolution is likely to be the same: the tool wins but the fundamentals stay, and the line between competent and incompetent practitioners moves but does not disappear.

Re: The last six months in LLMs in five minutes

#470

Post by nijave »

hollowturtle wrote: post #209

> The coding agents got really good It's since november 2025, the so called "inflection point", that I'm still wondering for who coding agents become "really good". All I observe they got better at tool call and answering questions about big codebases, especially if the question has a vague pattern to search, and they're superuseful for that! For generating production code even with a lot of steering and baby sitting…

Have had fairly good luck with Claude Code Opus 4.7 on xhigh effort.

I think it more reliably does IaC with established patterns especially when it can do a dry run.

Python is pretty decent but usually you need good prompting and a little bit of steering to prevent slop. The slop usually works tho

Codex w/ gpt-5.5 seems faster but maybe just a bit below Opus 4.7 quality.

I gave Opus access to a repl (pyrasite-ng) in a running Python process and it managed to find an 8 year old "memory leak"--a module level cache with no eviction. It did that using GC module and exploring the heap. I was pretty happy with that outcome. It would have been quite challenging for me to find myself without at least a few weeks of deep diving into memory leak hunting docs/resources.

Post reply on HN