The Claude models seem to be focused on generating code more than anything else. Are they still competitive with Open AI and Google for more general use cases, or have they sacrificed that?
Claude Haiku 4.5
221–230 of 292 posts
Re: Claude Haiku 4.5
#222Very preliminary testing is very promising, seems far more precise in code changes over GPT-5 models in not ingesting irrelevant to the task at hand code sections for changes which tends to make GPT-5 as a coding assistant take longer than sometimes expected. With that being the case, it is possible that in actual day-to-day use, Haiku 4.5 may be less expensive than the raw cost breakdown may appear initially, though…
Those numbers don’t mean anything without average token usage stats.
Fair point of course and it is still far to early to make a definitive statement, but in my still limited experience throughout the night, I have seen Haiku 4.5 be far better in using what I'd consider a justifiable amount of input tokens over e.g. GPT-5 models. Sonnets recent versions also had been better on this front over OpenAIs current best, but I try (not always succeed) to take prior experience and expectation out of the equation when evaluating models.
Additionally, the AA cost to run benchmark suite numbers are very encouraging [0] and Haiku 4.5 without reasoning is always an option too. Tested that even less, but there is some indication that reasoning may not be necessary for reasonable output performance [1][2][3].
In retrospect, I perhaps would have been served better starting with "reasoning" disabled, will have to do some self-blinded comparisons between model outputs over the coming weeks to rectify that. Am trying my best not to make a judgement yet, but compared to other recent releases, Haiku 4.5 has a very interesting, even distribution.
GPT-5 models were and continue to be encouraging for price/performance with a reliable 400k window and good adherence to prompts with multi minute (beyond 10) adherence, but from the start weren't the fastest and ingests every token there is in a code base with reckless abandon.
No Grok model ever performed for me like they seem to during the initial hype
GLM-4.6 is great value but still not solid enough for tool calls, not that fast, etc. so if you can afford something more reliable I'd go for that, but encouraging.
Recent Anthropic releases were good at code output quality, but not as reliable beyond 200k vs GPT-5, not exactly fast either when looking at token/sec, though task completion generally takes less time due to more efficient ingestion vs GPT-5 and of course rather expensive.
Haiku 4.5, if they can continue to offer it at such speeds with such low latency and at this price, cupeled with encouraging initial output quality and efficient ingestion of repos seems to be designed in a far more balanced manner, which I welcome. Course with 200k being a hard limit, that is a clear downside compared to GPT-5 (and Gemini 2.5 Pro though that has its own reliability issues in tool calling) and I have yet to test whether it can go beyond 8 min on chains of tool calls with intermittent code changes without suffering similar degradation to other recent Anthropic models, but I am seeing the potential for solid value here.
[0] https://artificialanalysis.ai/?models=gpt-5-codex%2Cgpt-5-mi...
[1] Claude 4.5 Haiku 198.72 tok/sec 2382 tokens Time-to-First: 1.0 sec https://t3.chat/share/35iusmgsw9
[2] Claude 4.5 Haiku 197.51 tok/sec 3128 tokens Time-to-First: 0.91 sec https://t3.chat/share/17mxerzlj1
[3] Claude 4.5 Haiku 154.75 tok/sec 2341 tokens Time-to-First: 0.50 sec https://t3.chat/share/96wfkxzsdk
Re: Claude Haiku 4.5
#223This is surprisingly competent. A couple of months ago I evaluated some leading models on a bunch of text adventures[1]. Typical regression coefficients would be +0.02 for top level models like Sonnet and Gemini 2.5 Pro, but notably also Gemini 2.5 Flash. (The baseline is GPT 5 Chat, i.e. the one where OpenAI routes to a thinking model only when they determine it's needed.)
When I include an attempt from Haiku 4.5 in the mix, most coefficients stay similar, but Haiku itself gets a +0.05. This must be a statistical fluke, because that would be insanely impressive – in particular for a cheaper model. I guess I'm adding samples to some of these after all...
[1]: https://entropicthoughts.com/evaluating-llms-playing-text-ad...
Edit: It was a fluke. Back to +0.01 after one more go at all games.
Re: Claude Haiku 4.5
#224What do Claude Code users do for tab auto complete, if anything? GitHub Copilot Free tier?
It's more of a conversational TUI than it is a code editor. You tell Claude to code for you. No need for autocomplete in that regime.
Re: Claude Haiku 4.5
#225Earlier quoted context omitted.
wow, grok code fast is really cheap
it writes bad code and blinding speed
From "The Psychology of Computer Programming":
After months of effort, a particular application was still not working, so a consultant was called in from another part of the company. He concluded that the existing approach could never be made to work reliably. While on his way home he realized how it could be done. After a few days work he had a demonstration program working and presented it to the original programming team.
Team leader: How long does your program take when processing?
Consultant: About 10 seconds per case.
Team leader: But our program only takes 1 second. {Team look smug at this point}
Consultant: But your program doesn't work. If the program doesn't have to work then I can make it as fast as you like.Re: Claude Haiku 4.5
#226I am really interested in the future of Opus; is it going to be an absolute monster, and continue to be wildly expensive? Or is the leap from 4 -> 4.5 for it going to be more modest.
Technically, they released Opus 4.1 a few weeks ago, so that alone hints at a smaller leap from 4.1 -> 4.5, compared to the leap from Sonnet 4 -> 4.5. That is, of course, if those version numbers represent anything but marketing, which I don't know.
Bizarrely they already call Opus 4.1 “legacy brainstorming model”.
Re: Claude Haiku 4.5
#227I've benchmarked it on the Extended NYT Connections ( https://github.com/lechmazur/nyt-connections/ ). It scores 20.0 compared to 10.0 for Haiku 3.5, 19.2 for Sonnet 3.7, 26.6 for Sonnet 4.0, and 46.1 for Sonnet 4.5.
This is such a cool benchmark idea, love it Do you have any other cool benchmarks you like? Especially any related to tools
You could try wordle on it. But from my own experience all of them are pretty bad. They're not smart enough to pick up the colours represented as letters. The only one that actually was good was Qwen surprisingly.
Re: Claude Haiku 4.5
#228Very preliminary testing is very promising, seems far more precise in code changes over GPT-5 models in not ingesting irrelevant to the task at hand code sections for changes which tends to make GPT-5 as a coding assistant take longer than sometimes expected. With that being the case, it is possible that in actual day-to-day use, Haiku 4.5 may be less expensive than the raw cost breakdown may appear initially, though…
One of the main issues I had with Claude Code (maybe it‘s the harness?) was that the agent tends to NOT read enough relevant code before it makes a change.
This leads to unnecessary helper functions instead of using existing helper functions and so on.
Not sure if it is an issue with the models or with the system prompts and so on or both.
Re: Claude Haiku 4.5
#229The Claude models seem to be focused on generating code more than anything else. Are they still competitive with Open AI and Google for more general use cases, or have they sacrificed that?
Which of the 1,000 other use cases? :)
To give one example, Opus and Sonnet IMO remain the #1 and #2 for writing informative prose. They're not entirely free of slop, but the ratio is lower than Gemini and especially GPT.
Re: Claude Haiku 4.5
#230I'm not seeing it as a model option in Claude Code for my Pro plan. Perhaps, it'll roll out eventually? Anyone else seeing it with the same plan?
Same here but you can get it using claude --model claude-haiku-4-5-20251001