261–270 of 424 posts
How is that economically possible? I’m so confused by those prices
how many times do you have to be metaphorically hit in the head with a brick before you realize inference margins at api pricing were 80%+
Seems like they're working to destroy the local LLM argument. Right now Haiku is $1/$5 in/out. You can grind out $12,000 worth of haiku (or arguably, sonnet) class tokens in about 5 months on a Blackwell RTX 6000 96GB especially if using concurrency. BUT, but, if you use a g6e.xlarge on aws it's now more expensive than buying tokens from OpenAI @ $0.20/$1.20. It also destroys "the Mac Mini argument", pushing the ROI…
The local LLM argument never really held water tbh. You can get surprisingly good performance for lightweight tasks locally, but you're just fighting economies of scale if you're going trying to beat a datacenter on cost.
How is that economically possible? I’m so confused by those prices
how many times do you have to be metaphorically hit in the head with a brick before you realize inference margins at api pricing were 80%+
Earlier quoted context omitted.
If you have agents and users, you can run evals and see how far the models go. Luna is not greatest in tool calls, but if you define your problem well and the tools well, it is comparable to Gemini 4 Flash with much lower price tag.
Luna is good as an end user model for simple tasks like classification, but not as a coding model. Also do you mean Gemini 3.6 Flash? 4 doesn't exist, and Gemma 4 exists but doesn't have a Flash option.
We run an agent company and outside coding the new Gemini 3.6 Flash and GPT 5.6 Luna are very interesting. Luna can do a bit of research and create reports. Gemini is great for computer use.
For programming it's all Kimi K3 now.
Earlier quoted context omitted.
I'd like to imagine the things that can be done with this speed and the current frontier models.
Seriously, if Fable or even Opus was this fast that would be a real game changer.
Earlier quoted context omitted.
https://taalas.com/ has done it already for a wildly obsolete model. 14000 tokens per second. https://chatjimmy.ai/ is their interactive. Tiny context, very dumb, but absurdly fast. Imagine this as a tool call for claude code for trivial changes - the tool call from the harness takes longer than the execution.
Holy crap, I was not prepared for how fast it responded. I just wrote "Just wanted to see how fast you are! Can you write me a quick story about a tiger who lives inside a block of cheese the size of a house?" I pressed Enter, and the response was instant . > Generated in 0.037s • 14,205 tok/s This is unbelievable.
This is crazy.
Earlier quoted context omitted.
this type of thing usually means you are the product
I don't see how this follows. The cost of nails has fallen by 95% over the last century. It's because the cost of manufacturing has fallen. Not because they are selling the information of nail consumers. Tokens are not normal software, because they have marginal cost, and I think people who are used to software economics really struggle with this. With token generation there really can be manufacturing cost efficienc…
No it hasn't!
A century ago, some nails cost 2.5% of disposable income, and now the same nails cost 2.3% - only a little cheaper.
The cost of nails has remained remarkably consistent for a century. The problem is that you have ignored the depreciation of money.
Let's assume California prices and income and pick a bigger retail package of nails as you might use for building a house. The numbers used to calculate percentages: in 1926 a 50lb keg of 4" nails was $2.75 and median after tax income might be $108 per month. In 2026 a 50lb carton of 4" nails is $106 and income might be $4,516. Albeit I assume nails are now more readily available and the quality of nails is likely better; and perhaps I should have compared galvinised nail prices.
Earlier quoted context omitted.
https://taalas.com/ has done it already for a wildly obsolete model. 14000 tokens per second. https://chatjimmy.ai/ is their interactive. Tiny context, very dumb, but absurdly fast. Imagine this as a tool call for claude code for trivial changes - the tool call from the harness takes longer than the execution.
Holy crap, I was not prepared for how fast it responded. I just wrote "Just wanted to see how fast you are! Can you write me a quick story about a tiger who lives inside a block of cheese the size of a house?" I pressed Enter, and the response was instant . > Generated in 0.037s • 14,205 tok/s This is unbelievable.
"LMS algorithm in bash"
Just barfed it up lol.
Amazing.
Earlier quoted context omitted.
Holy crap, I was not prepared for how fast it responded. I just wrote "Just wanted to see how fast you are! Can you write me a quick story about a tiger who lives inside a block of cheese the size of a house?" I pressed Enter, and the response was instant . > Generated in 0.037s • 14,205 tok/s This is unbelievable.
For what its worth the frontier lab models can surely be a lot faster if they wanted them to be but theyre supply constrained so theyre doing stuff like multi tenancy. Since you cant self host them no one outside the labs really knows speed as a solo tenant
Earlier quoted context omitted.
It's a feature offered in ChatGPT and other platforms, though probably gated behind paid subscriptions.
Used to be part of the $20/mo plan but it's not anymore (not sure if they removed it completely). However GPT-5.6 is pretty good at researching if you prompt it right, I've regularly had it spend 5+ minutes researching topic with lots of web searches.