Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The last six months in LLMs in five minutes

simonwillison.net

Post reply on HN
Print view View original

511–520 of 631 posts

Re: The last six months in LLMs in five minutes

#511

Post by voncheese »

noisy_boy wrote: post #476
DennisP wrote: post #458

Earlier quoted context omitted.

Steve Yegge wrote about this in his book Vibe Coding . He says it takes about a year of experience before you're consistently getting good results. He writes about lots of different techniques for doing that, but also says a lot of it comes down to just getting a feel for when the LLM is going to go haywire.

> but also says a lot of it comes down to just getting a feel for when the LLM is going to go haywire That has been my experience too. The days when I'm very focused, being extra deliberate and constantly questioning/examining/challenging things, the results are much better. Autopilot days just go through in a daze and the outcome is objectively worse. This has made me much more hands-on and pushed me towards models…

+1 to all of this. The challenge can be staying focused and thinking when the AI assistant is (1) moving very fast and (2) often times doing multiple things at the same time.

I know I have struggled to keep up, and fall into the trap of approving things (either commands or recommendations) without taking the time to really process and think about them.

It's a bit like the age old problem of "it's super easy to ask questions, and can be super hard to answer many of them". So the economy of the conversation gets out of whack fast.

Re: The last six months in LLMs in five minutes

#512

Post by hollowturtle »

prettyblocks wrote: post #485
hollowturtle wrote: post #209

> The coding agents got really good It's since november 2025, the so called "inflection point", that I'm still wondering for who coding agents become "really good". All I observe they got better at tool call and answering questions about big codebases, especially if the question has a vague pattern to search, and they're superuseful for that! For generating production code even with a lot of steering and baby sitting…

I'm curious. What have you actually tried? Are you just prompting the LLM with one off tasks? For good results, you need to take the time to read the documentation for the harness you are using and configure your environment. This tuning can take dozens hours to nail down. Then there's the actual approach for working on your projects. Many people that have good results with agentic coding actually spend the bulk of t…

I'm curious. What makes you think that me sharing an example(which one of the many?) of what I actually tried would somehow add something to the conversation? What's the usefulness of just an anecdotal example?

As I said we have a plenty of different envs, codebases, requirements. Things are complex.

You're posing it like I tried just one time. It's been hundreds of hours of tries and I just found out what works best for me, like everyone should do. My original post above isn't that hard to understand.

Let me stress this out again:

> That's why the debate is so polizered imo, there isn't a shared experience

Re: The last six months in LLMs in five minutes

#513

Post by bluGill »

jiggawatts wrote: post #257
hollowturtle wrote: post #235

Earlier quoted context omitted.

Don't want to be rough, but I'd like to read experiences about novelty ideas that solve people real problems in the real world, your project it's just about selling new shovels. As I commented on another thread > If you're trying to solve a HARD problem people REALLY have, it's a novelty that agents can't help with, otherwise if it gets 97% there MAYBE it's just a signal that your idea isn't that novel!

As a random example of a "hard" problem solved by AI that I couldn't have realistically done myself, despite having decades of wide industry experience: Reverse engineering a proprietary protocol from a binary executable. I heard about people finding security vulnerabilities in compiled code with the combination of Claude Mythos wired up to a disassembler like NSA's Ghidra. Someone here mentioned that GPT 5.5 "extra…

> There is no way I could have reverse engineered this myself from compiled C++ code and/or packet captures! The format isn't self-describing and is incredibly dense (similar to NetFlow). In a hex viewer it looks like line noise!

I think you could have. However I don't think you would have - there is a big difference. It is a lot of work to to that, and people who try normally give up. However if your boss told you could have. Note that I suspect from your story this is more like give this to a dozen people and in 2 years you get results - at a cost of several million dollars.

Re: The last six months in LLMs in five minutes

#514

Post by 16bitvoid »

freedomben wrote: post #460
kstenerud wrote: post #427

Earlier quoted context omitted.

You consider adding in-place constructed items to an array to be code duplication?

I've noticed that the bar for "quality" when people judge AI is often significantly higher than what they'd hold a human to. I'm not saying GP et al are doing this (I haven't looked myself), but it is a widespread pattern I've noticed both professionally and personally. I don't know why it is.

The bar isn't any higher. There's just no grace given. No one is judging a hobby project made by a human on quality, and the person who the hobby project belongs to will rarely say that their code is high quality. And in a professional setting, I think people are fine with "good enough" but they're not going to claim anything is high-quality.

But people are so quick to label their vibe-coded codebase as high quality and no grace is going to be given to a machine.

What comments are you seeing that are calling code from humans high-quality?

Re: The last six months in LLMs in five minutes

#515

Post by romaniv »

> there’s zero chance any AI lab would train a model for such a ridiculous task.

A lot of people here stated that this is a ridiculous metric, but no one seems to remember that it was introduced in the initial GPT report ("Sparks of Artificial General Intelligence: Early experiments with GPT-4" [1]) by Microsoft about 3 years ago. Shortly after that it was parroted by a network of booster accounts and became a thing every clueless AI hype peddler does to "test" models.

100% marketing, 0% science.

[1] https://arxiv.org/pdf/2303.12712

Re: The last six months in LLMs in five minutes

#516

Post by prettyblocks »

hollowturtle wrote: post #512
prettyblocks wrote: post #485

Earlier quoted context omitted.

I'm curious. What have you actually tried? Are you just prompting the LLM with one off tasks? For good results, you need to take the time to read the documentation for the harness you are using and configure your environment. This tuning can take dozens hours to nail down. Then there's the actual approach for working on your projects. Many people that have good results with agentic coding actually spend the bulk of t…

I'm curious. What makes you think that me sharing an example(which one of the many?) of what I actually tried would somehow add something to the conversation? What's the usefulness of just an anecdotal example? As I said we have a plenty of different envs, codebases, requirements. Things are complex. You're posing it like I tried just one time. It's been hundreds of hours of tries and I just found out what works best…

In my experience most people with the type of critique I'm seeing from you have only tried it one time or have not taken the time to invest in an environment/process that will work for agentic coding.

My question is not so much about sharing a cherry picked example, but the question was more like "have you tried in earnest to make it work". That's the part that wasn't clear from your original post. But you say you have, and you weren't impressed. Fair enough. I'm not trying to convince you otherwise, but I encourage people to give the tools a fair chance before throwing up their hands and deciding it's meh.

Having said all that, you're right there isn't a shared experience.

Re: The last six months in LLMs in five minutes

#517

Post by Shocka1 »

> One of my projects was a vibe-coded implementation of JavaScript in Python—a loose port of MicroQuickJS—which I called micro-javascript. You can try it out in your browser in this playground.

I'd like to remind everyone here that people on this forum used to actually code truly remarkable and pointless stuff like this, with zero LLMs, using nothing but their brains and motivation from who the heck knows where from.

Re: The last six months in LLMs in five minutes

#518

Post by malfist »

embedding-shape wrote: post #496
malfist wrote: post #434

Earlier quoted context omitted.

There's a cowboy artist that paints with his penis and does amazing work. If I tried that it'd turn out incredibly poorly, I prefer to paint with paintbrushes. Just because the naked cowboy can paint well with just his penis, doesn't mean a penis is the right tool for painting. It doesn't matter how you hold your penis, it's not the right tool.

Hmm, ok, I think the penis in case is a bit distracting, can you de-analogize this to their real terms and tell me what this is supposed to mean and be related to developing with LLMs?

Just because you _can_ do something with a tool, doesn't mean it's the right tool for the job. Just because someone has contorted their entire process to adapt to a misshapen tool, and gotten good results, doesn't mean that's the right thing to do.

It is reasonable to both use the right tool for the right job, and demand better tools than you currently have. Success with the wrong tool in the wrong job doesn't mean it's the right tool for the right job.

Re: The last six months in LLMs in five minutes

#519

Post by JeremyNT »

hollowturtle wrote: post #209

> The coding agents got really good It's since november 2025, the so called "inflection point", that I'm still wondering for who coding agents become "really good". All I observe they got better at tool call and answering questions about big codebases, especially if the question has a vague pattern to search, and they're superuseful for that! For generating production code even with a lot of steering and baby sitting…

> It's since november 2025, the so called "inflection point", that I'm still wondering for who coding agents become "really good".

I think this may depend on the sorts of work you do. For those of us who mostly live in web using established frameworks, that's about when I came to conclude they could do everything and do it well.

I can have opencode discover third party APIs and generate fully working solutions that are well integrated into an existing long-lived codebase. I still review the MRs by hand but I only ever discover spec errors or style issues, not defects in the code itself. This was a big change from ~summer 2025.

This is a really well defined space though with strong conventions. If you're doing something more interesting YMMV.

Re: The last six months in LLMs in five minutes

#520

Post by wasabi991011 »

kstenerud wrote: post #371
andai wrote: post #364

Earlier quoted context omitted.

I think you can fix that by setting an environment variable (regarding the terminal?) but it was a while since I checked. (I was running Claude as a subprocess and had similar issues.) Also this reminds me of a principle I learned from a mentor. "People are visual buyers. If it looks good, people will think the code is good." Unfortunately it doesn't matter whose fault the janky TUI is, people will see that and assoc…

It's more along the lines of: Anyone with an axe to grind will find something to grind it on. Early stage products will have some rough edges. We've seen that in Docker, Kubernetes, AWS, Azure, LXC, KVM, etc. And people griped and raged about the sheer incompetence of the maintainers and utter lack of quality, but they still used those tools even before the rough edges were polished away and folks finally settled dow…

> Early stage products will have some rough edges. We've seen that in Docker, Kubernetes, AWS, Azure, LXC, KVM, etc.

Is your product really the same complexity as these?

Post reply on HN