Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The last six months in LLMs in five minutes

simonwillison.net

Post reply on HN
Print view View original

571–580 of 631 posts

Re: The last six months in LLMs in five minutes

#571

Post by everforward »

kstenerud wrote: post #293
pprotas wrote: post #284

Earlier quoted context omitted.

You claim "very high quality" but can't even get the basic UI working properly. You wrap tmux and a container in 2k lines of code and claim quality, I think the comment above was aimed at this claim.

The UI is working properly. Interfering with Anthropic's UI, or any of the other agent harness' UIs it supports, would be madness incarnate. I also strongly suspect that you'd only taken a cursory glance at the top of the readme prior to passing judgment.

I looked through and there's a bunch of stuff that's in poor coding practice.

E.g.

https://github.com/kstenerud/yoloai/blob/main/internal/fileu... https://github.com/kstenerud/yoloai/blob/main/internal/mcpsr... https://github.com/kstenerud/yoloai/blob/main/runtime/regist... https://github.com/kstenerud/yoloai/blob/main/runtime/exec.g... Just out of curiosity, I enabled some other linters and it looks bad. Excluding test files, there are 110 functions with a cyclomatic complexity over 10 and 7 that are _over 50_. The worst is at 86, which is mind-boggling.

Could probably find more, but you get the drift. I'm sure it runs, but stylistically this is more along the lines of what I would expect an intern to do.

This is also sort of nit-picky, but like half the stuff in https://github.com/kstenerud/yoloai/blob/main/docs/dev/backe... isn't idiosyncratic, it's just the way those things work and a lot of them aren't even tricky. The one linked is particularly blatant; that's not limited to os.Stat that's literally just how permissions work. Denying permission on inodes is a property of the folder, not the file.

Re: The last six months in LLMs in five minutes

#572

Post by habinero »

timr wrote: post #334
kstenerud wrote: post #219

Earlier quoted context omitted.

The polarization comes from the very disparate coding experiences and output quality that different people find when using these tools. For example, I've had the opposite experience of yours, generating very high quality work using Claude (such as https://github.com/kstenerud/yoloai ). Just in dealing with all the bugs and idiosyncrasies in the technologies I'm using, the agent has been a godsend in discovering and c…

While reading this thread, I literally just caught an agent putting in the following CSS selector in a rule: > .row > div > div, .alert This is fairly simple CSS, not multi-threaded systems development. A bar low enough that you could trip over it. I catch this kind of stuff all the time (literally every run), but only because I read every line. Most of it wouldn't be the end of the world for any particular task, but…

> I think the people doing the heaviest breathing around the elimination of programmers either aren't very good at programming, or they're not paying close attention.

Yeah, absolutely. People think you're picking on, like, code formatting and no, dawg, your code doesn't do what you think it does, or it only handles the happiest of happy paths.

I do find it funny when people get mad about you critiquing their AI project. You didn't even write it, dude.

Re: The last six months in LLMs in five minutes

#573

Post by dasil003 »

hollowturtle wrote: post #553
topherhunt wrote: post #546

Earlier quoted context omitted.

@hollowturtle I'm surprised - do you really find that sota models aren't good enough to generate production code with steering and babysitting? My experience (Claude Code, mostly Opus 4.6) is that it's fantastic at this. At least in JS + TS + Elixir + Ruby. It does indeed need babysitting, my mental model is that it's an exoskeleton not a junior dev, but IME it's a friggin badass exoskeleton, easily 10x-ing my speed…

It really depends on the task, but, in my experience, small to medium and bigger codebases, the amount of steering to get quality code is not worth it. I see patterns and solutions emerging from hand coding, I'm not the other way around, I can't start with a prompt, unless again I have the feeling that the task can be one-shot with minimumn effort and context. Starting with a prompt, or in plan mode, it's not how I t…

I think this depends a lot on the task, the existing codebase, and the taste of the operator.

In general I tend to agree with you if you're talking a codebase you are deeply familiar with, the value-add from have agents write the code probably ranges from very small to negative in most cases.

On the other hand if you're trying to make changes in systems you are not familiar with, LLMs are a huge speed boost to folks with enough experience to sniff out what would be a bad path essentially via socratic method to the agent.

Obviously there are no silver bullets and no substitute for judgment. I will say though, I'll tradeoff ugly local code for good data models and interfaces any day of the week, and there is definitely an archetype of engineer that is very precious about code without good judgment on where it matters and where it doesn't.

Re: The last six months in LLMs in five minutes

#574

Post by simonw »

johndough wrote: post #567
simonw wrote: post #403

Earlier quoted context omitted.

Give me a "hard problem" and I'll give you a Codex or Claude Code transcript showing how I'd use them to tackle it.

I've got a few. Pick whichever you like. Factor 135066410865995223349603216278805969938881475605667027524485143851526510604859533833940287150571909441798207282164471551373680419703964191743046496589274256239341020864383202110372958725762358509643110564073501508187510676594629205563685529475213500852879416377328533906109750544334999811150056977236890927563 in less than 24 hours. Come up with a way to sample from LLMs…

I should have said "hard, but realistic".

Re: The last six months in LLMs in five minutes

#575

Post by okamiueru »

sigmoid10 wrote: post #315
GistNoesis wrote: post #285

Last 6 months is humanity losing control of LLMs. - Memory market cornering which mitigated the adoption of local AI despite great open model being released. - Fast penetration of IP exfiltrating tools in companies world-wide. - Developers producing more code that they can read. - Autonomous agents killing Open Source by siphoning the attention economy - Autonomous agents destroyed online communities (including HN) -…

If you only read bad news (i.e. mass news these days since that sells better) this will be the picture. But I have personally seen some insane stuff happen in biotech. Like, I can't believe we're lucky enough to possibly live our life in this kind of future. We already have actual therepeutics developed using Alphafold being tested right now in real clinical trials, but the next generation of stuff that will go into…

AlphaFold is not an LLM. As such, it isn't a fitting example for "good news" related to LLMs.

Re: The last six months in LLMs in five minutes

#576

Post by travisgriggs »

hollowturtle wrote: post #553
topherhunt wrote: post #546

Earlier quoted context omitted.

@hollowturtle I'm surprised - do you really find that sota models aren't good enough to generate production code with steering and babysitting? My experience (Claude Code, mostly Opus 4.6) is that it's fantastic at this. At least in JS + TS + Elixir + Ruby. It does indeed need babysitting, my mental model is that it's an exoskeleton not a junior dev, but IME it's a friggin badass exoskeleton, easily 10x-ing my speed…

It really depends on the task, but, in my experience, small to medium and bigger codebases, the amount of steering to get quality code is not worth it. I see patterns and solutions emerging from hand coding, I'm not the other way around, I can't start with a prompt, unless again I have the feeling that the task can be one-shot with minimumn effort and context. Starting with a prompt, or in plan mode, it's not how I t…

> quality code

Probably where the mismatch is in this discussion. The measure of what is quality code is all over the place. For some, some form of "good enough" is quality. And for others, metrics like terseness, readability, vacuous amounts of comments, cleverness, various fuzzy measures of "idiomatic", etc, make "quality code" much more of a moving target.

Re: The last six months in LLMs in five minutes

#578

Post by nektro »

sigmoid10 wrote: post #315
GistNoesis wrote: post #285

Last 6 months is humanity losing control of LLMs. - Memory market cornering which mitigated the adoption of local AI despite great open model being released. - Fast penetration of IP exfiltrating tools in companies world-wide. - Developers producing more code that they can read. - Autonomous agents killing Open Source by siphoning the attention economy - Autonomous agents destroyed online communities (including HN) -…

If you only read bad news (i.e. mass news these days since that sells better) this will be the picture. But I have personally seen some insane stuff happen in biotech. Like, I can't believe we're lucky enough to possibly live our life in this kind of future. We already have actual therepeutics developed using Alphafold being tested right now in real clinical trials, but the next generation of stuff that will go into…

Alphafold isn't generative and using this as a rebuttal to OP is bad faith

Re: The last six months in LLMs in five minutes

#579

Post by lopsotronic »

LZ_Khan wrote: post #57

I'm curious how the 6 months have looked from a non-programmer's perspective. What kind of co-working tools and similar optimizations have people from other fields experienced?

I've always been a "power user", making little python programs and figuring out new ways to do things with seemingly unrelated systems. My knowledge is shallow, but very broad.

A year and a few jobs ago I was genuinely up against a wall I could not see breaking through, not if I wanted to ever sleep again. Hundreds of completely bespoke customers. Hideous archaic tooling. Two of us. It was bad times. So I started paying for Claude - desperation move, to try and vibe my way out. Honestly, it's been a little bit like having superpowers.

Not just code generation, which has been great, but gaining knowledge and understanding with incredible velocity - sort of like how RSS felt back in the day, or when Google stopped being worthless in the very end of the 20th C. When Wikipedia started.

So where am I now? Well, I ditched the hell job (I didn't really drink the koolaid of their "Enterprise Solution" anyway), and got a regular day job in my core competency. I guess I do a lot of what is called "vibe coding", all kinds of utilities, what I call my "extracurriculars". A graph view for Asciidoc in VSC to show includes, xrefs, partial includes. Graph view for everything actually - it's surprisingly insightful for PDM and config management. Analysis tools for sensor faults based on Python open source astronomy tools. All sorts of converters and aggregators and cleaners for a devil's piss bucket of enterprise systems. A bazillion new MapTools macros for gaming, making complex RPG systems nearly pushbutton. A little harvest of local LLM systems doing all sorts of things, like my "Reviewinator" for copy edit. I could type the rest of the day and wouldn't come close to the end of the list.

So, pretty amazing. Very interesting systems with what must be some N-dimensional geometry underlying, maybe a signal to an underlying principle of emergence. Who knows?

In the long term, it's going to be Enterprise Software that eats the big losses from these systems. For all sorts of reasons, but mostly because Enterprise is where software goes to die. It's all bespoke to hell, it's all ancient, no one is working there because they want to. So a domain expert, with AI assist and a little know how, is probably going to whip up a superior set of tools in a short enough time to make it really worthwhile. Watch that space: SAP, Siemens, Teamcenter, SalesForce. Watch their consulting revenue.

Re: The last six months in LLMs in five minutes

#580

Post by wolrah »

oooyay wrote: post #506
kstenerud wrote: post #389

Earlier quoted context omitted.

The issue is likely that the tmux session being generated is for some reason not propagating all term caps. Most likely it's an interop issue between tmux and docker and the image running under docker - possibly even something with the terminal client that the pipeline doesn't like somewhere. Claude Code correctly reduces its display to 7-bit ASCII in response (still functional, although less pretty). Once I get arou…

What's really interesting in this comment chain is an observation I've expressed a lot more lately. When someone knows an LLM was involved they raise their expectations. I do it too in my own work and I have to remind myself things like "this bug would've also likely occurred with a human working at this level of complexity." The real question is did the operator arbitrarily and knowingly increase the level of comple…

> The real question is did the operator arbitrarily and knowingly increase the level of complexity or is it appropriate for the task.

There's one major reason to have higher expectations for autonomous systems (of all kinds, not just LLM-powered) than for humans, at least those intended to be deployed at scale, and that's the scale. If a human makes a mistake, has biases, or even intentionally breaks the rules the impact of their actions is limited by the nature of them being a human, where something like an autonomous driving system, a coding agent, etc. is intended to be deployed by the thousands, millions, or more and any problematic behaviors happen at that scale.

There are obviously millions of bad drivers out there, but every one of the human ones is bad in different ways. If Waymo pushes a bad update there could be tens of thousands of "drivers" that suddenly become bad in identical ways.

Humans also have the ability to learn from our mistakes. The ones you'd want to have working for you usually don't make the same one twice. LLMs are pretty good at making the same mistake repeatedly, even the simplest things like basic math or counting letters.

Post reply on HN