581–590 of 1001 posts
Earlier quoted context omitted.
Incremental gains compounds.
meta threw in the towel when it came to producing AI models since their gains couldn't keep up with China.
I generated pelicans riding bicycles on both thinking level low and thinking level high: https://gist.github.com/simonw/68560eddb0b268a8417f80ceb7304... The high one is notably better - the bicycle frame is the correct shape, unlike thinking level low. For comparison, here's Opus 4.7: https://gist.github.com/simonw/afcb19addf3f38eb1996e1ebe749c...
I bet someone shares this link every time you post about bicycles, but since I didn't see anyone share it yet in this thread, I'll take the opportunity to do so: https://www.gianlucagimini.it/portfolio-item/velocipedia/ Turns out even humans can be pretty bad at drawing bicycles :)
No guarantees is why LLM is akin to gambling. Every new context is essentially picking someone out of the crowd.
A rambling comment: I think this is the first time we've had a third minor version bump on a frontier Anthropic model. (I count the 0.5s as major here, because they've been issued non-sequentially and also corresponded to massive capability leaps, eg, Sonnet 3.5, Opus 4.5). So now the Opus 4.5 family has successors 4.6, 4.7, and 4.8, each posting fairly modest claimed gains. My own experience w/ 4.6 and 4.7 are that…
> (it's smarter than me?) I genuinely hope that you're joking with that statement. Or this is a bot. Or an ARG. Or Art. Help.
A rambling comment: I think this is the first time we've had a third minor version bump on a frontier Anthropic model. (I count the 0.5s as major here, because they've been issued non-sequentially and also corresponded to massive capability leaps, eg, Sonnet 3.5, Opus 4.5). So now the Opus 4.5 family has successors 4.6, 4.7, and 4.8, each posting fairly modest claimed gains. My own experience w/ 4.6 and 4.7 are that…
“Maybe my own tastes are saturated now” It might be saturated for smaller scopes of work, but it’s not hard to see the cracks when you scale up what you ask of SOTA models/agents. One example, to try and single shot prompt coding a ChatGPT equivalent chatbot. Sure it will spit something out, but the feature depth, UX subtitles, backend integration, and lots of pragmatic engineering decisions along the way will just n…
Why are you people saying all these things.
We'll probably see long-distance space travel long before a degree in generic problem identification and solving becomes totally useless.
Reminder the only benchmark that really matters is the one that measures the ability for the model to do real world tasks that someone would pay for on Upwork that would take ~12 hrs for a human to do. The best model has a https://labs.scale.com/leaderboard/rli Stop frying your brain with these useless tools, reducing your output to the mean. You people are betting your competency on the quality and quantity of token…
> Who is more valuable as individual, the owner of a watch factory in Vietnam or the guy who makes one watch a month in Switzerland?
With that framing, I'm not sure what the answer is. I suppose it depends on your priorities
My fav coding benchmark for frontier models is to build a simple RTS game in one file (js/html/css). Claude Code with Opus 4.8 in ultracode mode nailed it, the best result so far: https://bsky.app/profile/senko.net/post/3mmwnrkwboc2v The prompt was: Create a simple but functional real time strategy (RTS) game similar to old WarCraft, StarCraft or Command & Conquer games. The player should be able to build buildings,…
Do you have a collection of these benchmark apps saved anywhere? I'd be particularly interested in seeing the relative cost differences between different models in a use case like this.
But I just vibe-coded a handy list of all the tests I did (unfortunately without the commentary I usually leave in social media posts -- I should add those at some point): https://senko.net/vibecode-bench/
seems like a really minor upgrade?
Earlier quoted context omitted.
I like that benchmark. You should throw the results up on GitHub pages so people can try out the games.
Yeah! Host on GitHub pages, so it's easy to click a link and play!
I have a static server of my own, so here's my list (of all the tests I published so far): https://senko.net/vibecode-bench/