I would like to see completely independent test results of these companies' products. I'm skeptical because every AI company claims their new product is the best.
Claude Sonnet 4.5
211–220 of 819 posts
Re: Claude Sonnet 4.5
#212Interesting quirk on first use: "`temperature` and `top_p` cannot both be specified for this model. Please use only one."
Why might this be, does anyone know?
Re: Claude Sonnet 4.5
#213Interesting, in the new 2.0.0 claude code they got rid of the "Plan with Opus then switch to Sonnet" feature. I hope they're correct in Sonnet being good enough to Plan too, because i quite preferred Opus planning. It wasn't necessarily "better", just more predictable in my experience. Also as a Max $200 user, feels weird to be paying for an Opus tailored sub when now the standard Max $100 would be preferred since th…
I'm also a max user and I just _leave_ it on Opus 4.1 - I've never hit a rate limit.
I'm on the 25x MAX plan and if I go full hog on multiple projects I might see the yellow "Approaching Opus limits" message in Claude Code, but I have yet to have it lock me down, I usually slip right into the next 5h block and the message vanishes.
Re: Claude Sonnet 4.5
#214> Practically speaking, we’ve observed it maintaining focus for more than 30 hours on complex, multi-step tasks. Really curious about this since people keep bringing it up on Twitter. They mention it pretty much off-handedly in their press release and doesn't show up at all in their system card. It's only through an article on The Verge that we get more context. Apparently they told it to build a Slack clone and left…
This is obviously much more than just taking an LLM an letting it run for 30 hours. You have to build a whole environment together with external tool integration and context management and then tune the prompts and perhaps even set up a multi-agent system. I believe that if someone puts a ton of work into this you can have an LLM run for that long and still produce sellable outputs, but let's not pretend like this is something that average devs can do by buying some API tokens and kicking off a frontier model.
Re: Claude Sonnet 4.5
#215Earlier quoted context omitted.
Well, here's an even more insane claim: I'm infinity times more productive, as I just wouldn't even start projects without the LLM to sidestep my ADHD. Then, when the LLM invariably fucks up, I step in and finish things myself! Here are a few projects that I made these past few months that wouldn't have been possible without LLMs: * https://github.com/skorokithakis/dracula - A simple blood test viewer. * https://www.…
Did you make any money off any of that or was it all just labors of love type of stuff? I'm enjoying woodworking...
This is all my hobby, for my job I use Claude in a way that doesn't involve code, but is still very useful. It's basically what inspired Lucid, above, when I realized I find coming up with solutions very easy, but find explaining them very hard, because I assume the other person knows too much and I don't elaborate enough.
LLMs are a great rubber duck, plus they can write the document for you at the end.
Re: Claude Sonnet 4.5
#216Please y'all, when you list supportive or critical complaints based on your actual work, include some specifics of the task and prompt. Like actual prompt, actual bugs, actual feature, etc. I've had great success with both ChatGPT and Claude for years, am around 3x sustained output increase in my professional work, and kicking off and finishing new side projects / features that I used to simply not ever finish. BUT t…
I had a complete shocker with all of Claude, GitHub Copilot, and ChatGPT when trying to prototype an iOS app in Swift around 12 months ago. They would all really struggle to generate anything usable, and making any progress was incredibly slow due to all the problems I was running into. This was in stark contrast to my experience with TypeScript/NextJS, Python, and C#. Most of the time output quality for these was at…
This is more or less my experience with Go right now.
For a bunch of reasons I want to avoid the standard React, Typescript, and Node stack but the sheer velocity that might enable from the LLM side might make it worth it.
Re: Claude Sonnet 4.5
#217I just ran this through a simple change I’ve asked Sonnet 4 and Opus 4.1, and it fails too. It’s a simple substitution request where I provide a Lint error that suggests the correct change. All the models fail. I could ask someone with no development experience to do this change and they could. I worry everyone is chasing benchmarks to the detriment of general performance. Or the next token weight for the incorrect c…
At this point it would be an interesting idea, to collect examples, in a form of a community database, were LLMs miserably fail. I have examples myself...
Re: Claude Sonnet 4.5
#218Earlier quoted context omitted.
Also getting a perfect score on AIME (math) is pretty cool. Tongue in cheek: if we progress linearly from here software engineering as defined by SWE bench is solved in 23 months.
The reason they get a perfect score on AIME is because every question on AIME had lots of thought put into it, and it was made sure that everything was a possible. SWE-bench, and many other AI benchmarks, have lots of eval noise, where there is no clear right answer, and getting higher than a certain percentage means you are benchmaxxing.
> SWE-bench, and many other AI benchmarks, have lots of eval noise
SWE-bench has lots of known limitations even with its ability to reduce solution leakage and overfitting.
> where there is no clear right answer
This is both a feature and a bug. If there is no clear answer then how do you determine whether an LLM has progressed? It can't simply be judged on making "more right answers" on each release.
Re: Claude Sonnet 4.5
#219I am a paying subscriber to Gemini, Claude and OpenAI. I don't know if it's me, but over the last few weeks I've got to the conclusion ChatGPT is very strongly leading the race. Every answer it gives me is better - it's more concise and more informative. I look forward to testing this further, but out of the few runs I just did after reading about this - it isn't looking much better
What about Grok, are they catching up?
Grok has been free for over a month now and for me it has certainly proven itself competent at most tasks that you would otherwise have to pay for with Claude, ChatGPT, etc.
Re: Claude Sonnet 4.5
#220Please y'all, when you list supportive or critical complaints based on your actual work, include some specifics of the task and prompt. Like actual prompt, actual bugs, actual feature, etc. I've had great success with both ChatGPT and Claude for years, am around 3x sustained output increase in my professional work, and kicking off and finishing new side projects / features that I used to simply not ever finish. BUT t…
I had a complete shocker with all of Claude, GitHub Copilot, and ChatGPT when trying to prototype an iOS app in Swift around 12 months ago. They would all really struggle to generate anything usable, and making any progress was incredibly slow due to all the problems I was running into. This was in stark contrast to my experience with TypeScript/NextJS, Python, and C#. Most of the time output quality for these was at…
Going from past discussions, there seem to be two issues there. One is that Swift has changed massively since it came out and huge swathes of examples and articles and such online, that LLMs are trained on, are out of date and thus pollute the training set.
Another issue is that Apple developer docs are largely sequestered behind JavaScript that makes them hard for scrapers to parse.
At least, those are the two explanations I’ve seen that seem plausible.