421–430 of 651 posts
Earlier quoted context omitted.
could or will? I am pretty sure even among software engineers, much fewer than 1% are going to spend their money on that. Most software engineers know how to spend their money responsibly.
That’s not at all what you said though, was it?
ROCm stack is not for people though who aren’t willing to dig in and patch things themselves.
Earlier quoted context omitted.
I haven't spent a dime on cloud inference, so cannot make a direct comparison like you. But I can 100% attest to the fact that Qwen3.6-27B is a very capable local model for coding tasks. Over the last month and a half I've been using it almost daily, either on my M2 Ultra or on my RTX 5090 box. I use it for small mundane tasks at ggml-org [0] - nothing really impressive, but definitely a helpful tool for a maintainer…
> Currently, I have a very lightweight harness - the pi agent with everything stripped (`pi -nc --offline`) and a short system prompt [1] to align it a bit with my style This really is the secret to getting the most out of these models IMO. Pi is so damned good. I have a strongly tuned Pi for running Step 3.7 Flash (IQ4_XS) and Qwen 3.6 27B (FP8) Also, thank you for llama.cpp mate :)
Anybody used a tinybox? https://tinygrad.org/#tinybox The most "affordable" option is red v2 with 64GB GPU ram and costs $12,000. This is only ("only") 1.5x-3x the price of a beefy desktop ( https://pcpartpicker.com/builds/ ), and could crush inference work even on bigger models. It could support coding tasks for a small team of developers, or run an AI agent for every person in your household...
If you have $12K to spend, you may be better off with DGX Spark or a Mac with 128GB VRAM. That can (barely) fit DeepSeek V4 Flash.
Earlier quoted context omitted.
I wonder if it is better to have a machine somewhere running a model for you maybe shared with a few others. I could probably justify a M6 Mac Studio with hopefully 256gb RAM and have a few people all with access to one agreed upon model. I think maybe laptops are too warm and clunky for this.
The problem is that the moment you introduce shared remote hardware there's a slippery slope leading right back down to "just pay an inference host for model tokens". If you're transmitting your prompts over the internet to a trusted host you might as well just let that host be DeepInfra or together.ai or one of the many other providers already in that business.
Earlier quoted context omitted.
That’s not at all what you said though, was it?
Read it again.
> could or will? much fewer than 1% are going to spend their money on that.
It’s ok to change your point, you don’t need to get combative.
Not that it makes any difference, given their ~10% market share.
I don't know about good, I use a lot of local models and they're still pretty painful to run locally You have dense models (qwen 27b, gemma 31b) who are pretty smart, but pretty slow You have MoE models (gemma 26b, qwen 35b, north mini code 30b) who are pretty fast, but make a lot of mistakes You need a lot of memory to run these well, quantization makes tool calling weaker, so most run at 4 bit quants and are wonder…
IMO running local models "well" still requires an expensive hardware investment. You really want 96GB of VRAM on a modern Blackwell arch to run these models with decent KV cache. Trying to run them on a unified memory Mac, an AI Max AMD processor, or a DGX Spark-alike is really just asking for trouble. Prefill kills perf. If you throw the right GPUs at the problem, they become much better - but still not quite in the…
Programmers are used to paying nothing for tools. A basic laptop (SSD, multi core, 16GB of RAM) is hugely powerful if you are building in C/C++/Rust, even python. But all of a sudden it's no good, and we're back to using someone else's computer, hiring our tools every day. Worse, we get a different model every day, and maybe we aren't allowed to borrow the good tools some days because some mafioso are shaking down th…