591–600 of 682 posts
Earlier quoted context omitted.
Friends Don't Let Friends Use Ollama https://news.ycombinator.com/item?id=47788385
What do you use instead?
The post suggests that you need an rtx 5090 use it, which is currently selling for around $5,000 USD. I wouldn't exactly call that "my device", since my device costs about 25% of that for the entire computer. For the same cost, you could run on a frontier model on a pro plan for two years. The economics dont make a lot of sense for this to me, so I would love some input on why people want to do this instead (privacy,…
Remember when we needed 200 servers for an enterprise website because Apache used one process or thread per connection - and Nginx collapsed that into a single box overnight? That moment for LLMs is near. It’s going to move us from the big iron era of AI to small portable brains. Nature has already proved it’s possible with 20 watts and very little heat generation. And I think the data center buildout will end in car…
I don’t remember that and I was there! The idea that the performance delta between Apache and nginx for any normal workload is anything like 20,000% is absurd.
The post suggests that you need an rtx 5090 use it, which is currently selling for around $5,000 USD. I wouldn't exactly call that "my device", since my device costs about 25% of that for the entire computer. For the same cost, you could run on a frontier model on a pro plan for two years. The economics dont make a lot of sense for this to me, so I would love some input on why people want to do this instead (privacy,…
It seems exceedingly unlikely that the current Pro plan costs will hold for the next two years. The subsidization train is going to end eventually.
Because I don’t think “the subsidization train is going to end” necessarily means current pricing becomes impossible.
If capital keeps pouring into frontier AI, companies still have an incentive to subsidize access while competing for users and market share. And if that subsidization starts drying up, there’s even more incentive to bring inference costs down by making smaller and cheaper models catch up to today’s frontier capabilities.
So either way, I’m not sure you can extrapolate from the cost of serving current frontier models to what equivalent capability will cost two years from now.
Earlier quoted context omitted.
FYI, llama-server can now be run in router mode so llama-swap is probably only needed for more exotic scenarios.
I'm running it in router mode, but people on Reddit were recommending people use llama-swap instead. Am I missing something by using router mode?
Earlier quoted context omitted.
Considering how all the big players are playing fast [1] and loose [2] with limits, billing [3] and adding undisclosed changes that burn your tokens on autopilot [4], it can't happen soon enough. [1]: Limits may change without notice, including due to capacity constraints. - https://support.google.com/gemini/answer/16275805?sjid=14713... . [2]: "standard limits" are never defined - https://support.google.com/gemini/a…
Not to mention all the other ways they can screw you: - Middle of the day, servers busy? Swap to Sonnet while pretending it's still Opus. Many people won't notice, and nobody can prove anything if they suspect. - Middle of the night, server load is light? Put it into extra thinky mode so it burns more tokens to ramp up the bills. Flip the switch where it gets really pedantic about writing lots of extra test cases and…
Earlier quoted context omitted.
It’s really interesting timing, Qwen over thinking is what kills it for me. I’m just glad we have more options in this size class now.
Just to play devil’s advocate: you can’t compare Qwen to a (proprietary/closed source) hosted model and deduce that Qwen is overthinking, as Qwen gives you the full reasoning/thinking trace while all the proprietary models now give you only a summary “to prevent distillation”, making it hard to properly compare apples to apples here.
Will be interesting to see how Qwen3.8 27B compares against this once it releases this week. Seems like dense 30B is back in fashion? EDIT: An open weight version of Muse Spark 1.2 is going to be released as well: https://x.com/alexandr_wang/status/2086756152034066792 https://xcancel.com/alexandr_wang/status/2086756152034066792
Based on the benchmarks, it seems that Muse Glimmer barely edges out against Qwen3.6 27B, except for tool-calling skills (MCP, etc.). I wouldn't be surprised if they released it now because they are afraid they wouldn't beat Qwen3.8 27B.
The whole class of what can reasonably fit in a single GPU is an interesting category of LLM, and based on the results I've seen from 3.6 35B A3B and 27B versus what existed a year prior, it seems there's a lot of room for advancement.