381–390 of 525 posts
Notes and two pelicans: https://simonwillison.net/2025/Nov/24/claude-opus/
I added Opus 4.5 to my benchmark of 30 alternatives to your now-classic pelican-bicycle prompt (e.g., “Generate an SVG of a dragonfly balancing a chandelier”). Nine models are now represented: https://gally.net/temp/20251107pelican-alternatives/index.ht...
Earlier quoted context omitted.
Totally agree with this. I have seen many cases where a dumber model gets trapped in a local minima and burns a ton of tokens to escape from it (sometimes unsuccessfully). In a toy example (30 minute agentic coding session - create a markdown -> html compiler using a subset of commonmark test suite to hill climb on), dumber models would cost $18 (at retail token prices) to complete the task. Smarter models would see…
For me this is sub agent usage. If I ask Claude Code to use 1-3 subagents for a task, the 5 hour limit is gone in one or two rounds. Weekly limit shortly after. They just keep producing more and more documentation about each individual intermediate step to talk to each other no matter how I edit the sub agent definitions.
Earlier quoted context omitted.
My workflow was usually to use Gemini 2.5 Pro (now 3.0) for high-level architecture and design. Then I would take the finished "spec" and have Sonnet 4.5 perform the actual implementation.
I use plan mode in claude code, then use gpt-5 in codex to review the plan and identify gaps and feed it back to claude. Results are amazing.
Earlier quoted context omitted.
I once heard a devils advocate say, “if child porn can be fully AI generated and not imply more exploitation of real children, and it’s still banned then it’s about control not harm.” Attack away or downvote my logic.
So how exactly did you train this AI to produce CSAM?
The burying of the lede here is insane. $5/$25 per MTok is a 3x price drop from Opus 4. At that price point, Opus stops being "the model you use for important things" and becomes actually viable for production workloads. Also notable: they're claiming SOTA prompt injection resistance. The industry has largely given up on solving this problem through training alone, so if the numbers in the system card hold up under a…
Earlier quoted context omitted.
> these agents are not up to the task of writing production level code at any meaningful scale This is obviously not true, starting with the AI companies themselves. It's like the old saying "half of all advertising doesn't work; we just don't which half that is." Some organizations are having great results, while some are not. From the multiple dev podcasts I've listened to by AI skeptics have had a lightbulb moment…
Not a skeptic, I use AI for coding daily and am working on a custom agent setup because, through my experience for more than a year, they are not up to hard tasks. This is well known I thought, as even the people who build the AIs we use talk about this and acknowledge their limitations.
This is gonna be game-changing for the next 2-4 weeks before they nerf the model. Then for the next 2-3 months people complaining about the degradation will be labeled “skill issue”. Then a sacrificial Anthropic engineer will “discover” a couple obscure bugs that “in some cases” might have lead to less than optimal performance. Still largely a user skill issue though. Then a couple months later they’ll release Opus 4…
There are two possible explanations for this behavior: the model nerf is real, or there's a perceptual/psychological shift. However, benchmarks exist. And I haven't seen any empirical evidence that the performance of a given model version grows worse over time on benchmarks (in general.) Therefore, some combination of two things are true: 1. The nerf is psychologial, not actual. 2. The nerf is real but in a way that…
That's case #2 for you but I think the explanation I've proposed is pretty likely.
Earlier quoted context omitted.
This isn’t how you should be benchmarking models. You should give it the same task n times and see how often it succeeds and/or how long it takes to be successful (see also the 50% time horizon metric by METR).
I did not say that I only ran the prompt once per attempt. When I say that second time it failed it means that I spent hours to restart, clear context, giving hints, everything to help the model to produce something that works.
I've played around with Gemini 3 Pro in Cursor, and honestly: I find it to be significantly worse than Sonnet 4.5. I've also had some problems that only Claude Code has been able to really solve; Sonnet 4.5 in there consistently performs better than Sonnet 4.5 anywhere else. I think Anthropic is making the right decisions with their models. Given that software engineering is probably one of the very few domains of AI…