281–290 of 652 posts
Earlier quoted context omitted.
I use it with a strix halo server. 35B runs stupidly fast. 27B is about 700 TPS prefill and 30 TPS token generation. Which interestedly is about what Kimi K3 gives me depending on provider.
what hardware do you use or recommend for this? never heard of it until today.
They've also announced Qwen3.8-27B being released open-weight next week. Qwen3.6-27B is widely regarded as one of the best local models, especially since nothing else comes close to it, that isn't benchmaxxed, without being significantly larger. If 3.8 truly improves upon it that would be awesome.
Some advice I got from another HN Mac user was to run local models in energy saver mode. You'll get slightly reduced tokens, but the laptop won't overheat and the fans won't go wild.
Earlier quoted context omitted.
> $200 in rent every month into the large model providers We all know that is hugely subsidized, and I guarantee that OpenAI and Anthropic are looking to enshittify that ASAP. The enterprise users, however, are not subsidized like that. They pay per token. And some developers in those companies are chewing down a lot of tokens. Self-hosting an open weight model could be a massive savings very quickly. It also gives t…
We all ASSUME that. For all we know, inference might be dirt cheap, they might just be hiking the API prices so high for us to think subscriptions are subsidized.
Now, the one wildcard in all of this could be Google. They are on the eighth generation of their TPU and have been holding their cards extremely close to the vest. I don't think anybody has a good read on exactly how much capacity they have. Most things you can kind of figure out the overall business numbers and what's going on in Google--the TPU area is one of the exceptions. I know a couple of big customers and even they don't have any visibility on that front.
They've also announced Qwen3.8-27B being released open-weight next week. Qwen3.6-27B is widely regarded as one of the best local models, especially since nothing else comes close to it, that isn't benchmaxxed, without being significantly larger. If 3.8 truly improves upon it that would be awesome.
There seems to be quite a gap between the small ones and the enormous ones these days.
Earlier quoted context omitted.
Qwen3.6-35B is my daily driver for AI, and what convinced me to cancel my Claude subscription back in April. The Qwen3.6 line is easily the best local model I've tried, and I've tried a lot. I've got it diligently grinding away on my laptop right now, reviewing and fixing some bugs in my F# code.
On what hardware do you run the model locally, if so?
The weaker point is prompt prefill, which starts at 1,400 tokens/sec but decreases significantly at high contexts. That said, for agentic scenarios, if you're using a harness that doesn't needlessly bust the cache, it doesn't feel slow.
I really hope they release a Qwen 3.8 35B, although the lack of a mention seems ominous.
Earlier quoted context omitted.
[flagged]
Maybe you're anti-AI because you're really anti-outsourcing your thinking to some remote corporation you don't control? That's one of my main issues with AI anyways, the thought of having all my data go through some sketchy foreign (to me) entity with questionable motives and under a questionable regime. Local AI solves for all of those.
If you're learning about model inference, then it's a different and you are definitely not anti-AI in that case.
Earlier quoted context omitted.
that is not an model issue but a harness one, if you don't want it to have io to files or run command then simple don't give it access to the tools.
Yes, I don't. I have my own harness where the agent is only able to do a set of predefined things and none of them involves the internet. However, I imagine the model could be a lot smaller if it does not have the baggage of coding and programming in general. I mostly just need it to mostly be a very advanced NLP model that is able to figure the set of tools that it needs to call with what argument. Then it will just…
They've also announced Qwen3.8-27B being released open-weight next week. Qwen3.6-27B is widely regarded as one of the best local models, especially since nothing else comes close to it, that isn't benchmaxxed, without being significantly larger. If 3.8 truly improves upon it that would be awesome.
There's a lot of AI models that each 'win' one week over the next. The pure definition of a commodity market and not a 'winner takes all market' as valuations would imply