291–300 of 322 posts
Earlier quoted context omitted.
What is it with AI SaaS naming themselves "openxyz" when there is 0% open about them?
It's the next co-opted buzzword after "democratize".
This isn't a good analysis, and it's because it keeps rounding everything up. He rounds up the cost of electricity by 10%. He has a range of power use, takes the high end (which is 2x the low end) and multiplies it by the inflated electricity cost. But then they talk about using a newly purchased Mac to do the inference, running at full capacity, 24/7. Why would you do that? Apple silicon is fast but the author point…
Not sure where 40 tokens per second is coming from. I’ve seen 95-100 tokens per second on M5 Max 128GB running Gemma 4 31B. I’ve done experiments where it is faster than Claude Opus 4.5 for the same prompts.
> M5 Max 128GB
Earlier quoted context omitted.
The article makes no sense. I can't use OpenRouter as a general purpose computing device. Why are we comparing a whole computer to a single purpose SaaS?
No, that’s not the point. I think this is to help people who are thinking about getting a beefier Mac so they can run their LLMs on it too. Some in particular want a dedicated Mac Mini or Studio for this purpose. The breakdown, even if slightly flawed, offers a good insight into the economics of it. For most people, they might be better off with OpenRouter models and providers supporting Zero Data Retention. On the c…
Like with OpenAI for a year?
” In June 2025, the court ordered OpenAI to retain its consumer and API customer chat logs indefinitely, including any that had been deleted, so they could be investigated […]”
https://www.techspot.com/news/109839-openai-no-longer-requir...
Earlier quoted context omitted.
You also have control over your costs. It is reasonable to assume that tokens will cost significantly more in the near to medium future as the market consolidates and subsidies decline.
Google, Microsoft, Meta, Anthropic, OpenAI, Oracle and others are going to be looking to recoup all the money that they’ve spent to date. Why would the price go down in the future?
The providers have spent a fortune on wireless, pulled a lot of fiber/cable, and it's cheaper than it was when it started.
Earlier quoted context omitted.
The hardware has multiple uses for the same cost. The pay-per-use server does not.
The author isn't pricing in the multiple uses. You either compare it apples to apples or you don't. If you're using the machine for general purpose computing on top of inference then the amortized hardware costs are pointless to measure. This is exactly what I said.
Earlier quoted context omitted.
Uber has been profitable since 2023.
And it was founded in 2009, 14 years without turning a profit
I'll invest in inference hardware whenever the economics make sense, not because of my prediction that prices will get higher (also, because smaller models keep getting better and they might just suffice for most of my use cases in a few years time).
My biggest worries in terms of cost is in regards to training. Whenever that gets too expensive for Beijing to pay, we won't be getting new SOTA open small models to run on local hardware, which, again, reinforces the decision to use providers for AI inference for now.
Earlier quoted context omitted.
Real estate is generally a "good" investment as it's considered a relatively safe way to get significant leverage. 5x leverage in the case of a 20% deposit, or even up to 20x leverage with countries that allow for 5% deposits (New Zealand). In addition, the interest payments almost always end up being near the rent the owner would have paid, so mortgage payments are higher, but that increase is generally (and quickly…
> relatively safe way to get significant leverage This only works if housing prices keep rising. This post could have been written in 2007.
This gives us a $1,995 a month payment when we purchased and a $2,142 a month payment today, due to higher assessed value for the tax.
We can see upsides and downsides in both cases. Rent would have been quite a bit cheaper in 2007, but it has very nearly caught up by now. Meanwhile, you're probably talking about renting maybe a 2 bed/1 bath apartment, whereas the median single-family house is more like 4 beds/2 baths, with a yard. Whether or not that extra space and privacy matters to you likely depends a lot on whether you're single or have or ever plan to have a family. You could have invested into something like the S&P 500, which has historically returned about 10.5% since 1957 annually in nominal returns. Let's just kind of naively split the difference here and assume you can invest $1,000 saved on rent versus mortage a month for the first 10 years and $200 a month for the next 9. That would have gotten you somewhere around $240,000 by now. Meanwhile, you're looking at about $248,000 in home equity by now for the purchase case.
Choose different parameters if you please, but I'm not really seeing the case for renting here over the long term, and that's in spite of choosing the single worst time in the last century you could have made the purchase.
Earlier quoted context omitted.
Not sure where 40 tokens per second is coming from. I’ve seen 95-100 tokens per second on M5 Max 128GB running Gemma 4 31B. I’ve done experiments where it is faster than Claude Opus 4.5 for the same prompts.
can you provide your configurations pls ?
Configuration:
Gemma 4 31B Instruct Q6K Context size 40960 LM Studio 0.4.13+1 Metal llama.cpp v2.14.0 LM Studio MLX (Apple M5) v1.6.0
Here are my results:
prompt eval time = 32545.36 ms / 5625 tokens ( 5.79 ms per token, 172.84 tokens per second) eval time = 20227.99 ms / 310 tokens ( 65.25 ms per token, 15.33 tokens per second) total time = 52773.35 ms / 5935 tokens
This was for interacting with a local MCP service, running a tool that returns a ~20KB text file to the agent to add to the chat context.
I'm seeing about the same number of tokens/second on an M2 Ultra that I have access to (also with 128GB of memory).
This is surely apples-to-oranges to the OP results (and I don't spend a great deal of time benchmarking these things, so my methodology might be lacking), but it's interesting seeing okay performance for a top open model. For most use, however, I find Gemma 4 26B A4B (Q6K) to be good enough (esp. for MCP calling) and much much faster (~1,200 tokens/second).