621–630 of 651 posts
After having been a happy user of Qwen3.6-27B for a few weeks, due to being away from the hardware, I'm currently forced to use Claude Sonnet 4.6 It is such a downgrade. I don't understand how that's even possible. The thing has so many strongly-held opinions I did not ever ask it for, talking just way too much and generally feeling somehow dumber. Of course, being significantly larger, it will encode more knowledge,…
This is the kind of thing that Anthropic et al should be worried about. As it becomes easier and easier to run local models, the ceiling of what they'll be able to charge will get lower and lower. Not that nobody will be willing to pay $$$$$ per month, but a lot of people are going to multiply the per-month charge by 12 or 24 and say "Could I set up a local model for less than that, and have it pay for itself within…
so long as there's no algorithm breakthrough
Earlier quoted context omitted.
> The best "free" experience I've found is using OpenCode with Big Pickle. I have absolutely zero interest in free. I honestly don't think I'm even remotely in the same demographic as people using free tiers / models. I want to pay. I don't want my data used for training. I want it to be open. I want it to be consistently up (more than Claude!). I want it to be fast. I don't want it to be subsidized as that's just an…
If you think your data isn’t being hoovered up I’d like to point out that every model is possible due to federal crimes committed to obtain the information they were trained on. Regardless of how much you are paying, your data is worth another petty civil infraction.
But I agree with your larger point. AI companies have copied Uber's aggressive posture, pushing the legal envelope with expectations of positive return. Surely they'll continue doing the same in other areas.
Earlier quoted context omitted.
Oh man! I had no idea I could do this at all! What do you usually tweak it to? I feel like 8 GB is probably still a reasonable amount to give the rest of the OS.
I've got a 32 GB MBPro, and I set it to 27700, which I haven't seen a problem with so far.
Earlier quoted context omitted.
But why would developers _step down_ to a Mac?
Are you making an honest attempt at stating your opinion that there is absolutely no reason to prefer a MacBook over your chosen device? Or are you just being glib because you’ve got a misplaced superiority complex?
They're overpriced toys. Too little CPU, too little GPU, too little RAM, and what RAM you do have has too little memory bandwidth to serve both the CPU and GPU at the same time (which still hasn't been fixed for the newest M series CPUs: the GPU can still starve the CPU out, they still do not share bandwidth correctly).
And since this is about local inference: dealing with Metal sucks, and dealing with the NPU sucks. NPUs are nightmares to infer with because you're always stuck using models quantized for them exclusively, which often means poorly quantized models (such as giving up mixed quant models, and burning your finite RAM just to keep the model from going insane).
Not only that, they can't meaningfully run Linux, so they have no life after the desktop, they just become e-waste.
Hell, even if I held one hand behind my back and _only_ looked at unified RAM machines, Apple doesn't currently sell an M5 Mac Studio, and the MBP M5 /w 128GB of RAM starts at $5.4k, and presumably, the M5 Mac Studio will cost roughly the same.
I can buy the overpriced Framework Workstation with a better CPU, a better GPU, and can run Linux or Windows and that starts at $3.5k.
Or I can just not be a dumb shit and build an inference rig with a 9800x3D, however much normal RAM I want (48 or 64GB is enough here), and then drop two 9060XTs (16GB, $500 each) in there, or two W9700 in there (9070XT /w 32GB, $1100 each), or a single 5090 ($2k, but only 32GB, the worst option of the three), and infer several times faster than all the above options while spending less money.
Inference requires two important things Apple either can't do, or can't do cheaply: RAM bandwidth and RAM amount.
I don't know about good, I use a lot of local models and they're still pretty painful to run locally You have dense models (qwen 27b, gemma 31b) who are pretty smart, but pretty slow You have MoE models (gemma 26b, qwen 35b, north mini code 30b) who are pretty fast, but make a lot of mistakes You need a lot of memory to run these well, quantization makes tool calling weaker, so most run at 4 bit quants and are wonder…
Maybe we shouldn't be running these models on laptops with their thermally constrained form factor, and we shouldn't expect quick inference on a par with a large cloud-based platform either, at least not for near-SOTA model quality. It's still worth it to avoid becoming massively reliant on centralized services.
I bought the memory at the end of the last year, and I was thinking, maybe this is excessive. No game will use that much memory, in a decade or more.
Now I realize it was one of the best purchases ever, I run qwen3-coder-next on it for just the cost of the electricity, while the coding and agents and whatever else is done in a laptop. Yeah, it is slower, I don't care. Infinite tokens is better than a few.
The cloud is another computer, but in this case it is mine =)
fav-ed this post and will check again when another one poped on front page and says "local models is perfect now"
I love running two models locally: qwen3.6 27B 8bit (dense) and qwen3.6 35B 4bit (MoE). The 27B is the smarter, more reliable one - but it is slower. The 35B is faster, still very smart but below 27B, a bit less reliable. The reason is the MoE - Mixture of Experts architecture, which only activates a subset of parameters, making the model much much faster. I run the 27B on a MacBook Pro M5 Max + 40 GPU cores + 128GB…
how are you connecting the 35B model to your mailbox, for email classification?
The beauty of it is that Hermes itself has a cron too - every 4h Hermes will wake up and check if the email ingestion is working fine. If not, it will fix it. Funnily this is one of the most robust setups I've seen in a while, it is like having your own little DevOps waking up at night, fixing the infra when needed.
Earlier quoted context omitted.
The fact OpenRouter strips https to reroute screams danger already.
What do you mean? Are you objecting that they communicate with the provider on your behalf? But how else would you design such a system? Plumbing you straight through would require nonstandard certificate juggling and they wouldn't be able to implement their core service of providing a standardized API nor could they transparently route your request to the fastest / cheapest / whatever provider on the fly nor could t…