201–210 of 292 posts
What is the use case for these tiny models? Is it speed? Is it to move on device somewhere? Or is it to provide some relief in pricing somewhere in the API? It seems like most use is through the Claude subscription and therefore the use case here is basically non-existent.
Earlier quoted context omitted.
I am a professional developer so I don't care about the costs. I would be willing to pay more for 4.5 Haiku vs 4.5 Sonnet because the speed is so valuable. I spend way to much time waiting for the cutting edge models to return a response. 73% on SWE Bench is plenty good enough for me.
How do you review code when the LLM can produce so much so fast?
Earlier quoted context omitted.
Those numbers don’t mean anything without average token usage stats.
Exactly, token per dollar rates are useful, but without knowing the typical input output token distribution for each model on this specific task, the numbers alone don’t give a full picture of cost.
Earlier quoted context omitted.
I vastly prefer the manual caching. There are several aspects of automatic caching that are suboptimal, with only moderately less developer burden. I don’t use Anthropic much but I wish the others had manual cache options
What's sub-optimal about the OpenAI approach, where you get 90% discount on tokens that you've previously sent within X minutes?
for example if a user sends a large number of tokens, like a file, and a question, and then they change the question.
I've benchmarked it on the Extended NYT Connections ( https://github.com/lechmazur/nyt-connections/ ). It scores 20.0 compared to 10.0 for Haiku 3.5, 19.2 for Sonnet 3.7, 26.6 for Sonnet 4.0, and 46.1 for Sonnet 4.5.
Do you have any other cool benchmarks you like? Especially any related to tools
Ain't nobody got time to pick models and compare features. It's annoying enough having to switch from one LLM ecosystem to another all the time due to vague usage restrictions. I'm paying $20/mo to Anthropic for Claude Code, to OpenAI for Codex, and previously to Cursor for...I don't even know what. I know Cursor lets you select a few different models under the covers, but I have no idea how they differ, nor do I car…
Then don't? Seems like a weird thing to complain about.
I just use whatever's available. I like Claude for coding and ChatGPT for generic tasks, that's the extent of my "pick and compare"
Earlier quoted context omitted.
What's sub-optimal about the OpenAI approach, where you get 90% discount on tokens that you've previously sent within X minutes?
because you can have multiple breakpoints with Anthropic's approach, whereas with OpenAI, you only have breakpoints for what was sent. for example if a user sends a large number of tokens, like a file, and a question, and then they change the question.
Earlier quoted context omitted.
It's not about storing data on disk, it's about keeping data resident in memory.
Deepseek pioneered automatic prefix caching and caches on SSD. SSD reads are so fast compared to LLM inference that I can't think of a reason to waste ram on it.
TtFT will get slower if you export kv cache to SSD.
Ain't nobody got time to pick models and compare features. It's annoying enough having to switch from one LLM ecosystem to another all the time due to vague usage restrictions. I'm paying $20/mo to Anthropic for Claude Code, to OpenAI for Codex, and previously to Cursor for...I don't even know what. I know Cursor lets you select a few different models under the covers, but I have no idea how they differ, nor do I car…
You either live with what you’re using or you change around and fiddle with things constantly.