Running AI on timesharing mainframes does seem like an odd final state for the world.
Running local models is good now
271–280 of 651 posts
Re: Running local models is good now
#272[flagged]
Re: Running local models is good now
#273Earlier quoted context omitted.
> The best "free" experience I've found is using OpenCode with Big Pickle. I have absolutely zero interest in free. I honestly don't think I'm even remotely in the same demographic as people using free tiers / models. I want to pay. I don't want my data used for training. I want it to be open. I want it to be consistently up (more than Claude!). I want it to be fast. I don't want it to be subsidized as that's just an…
These competent open models you want to use were trained on data from people like you and me. I wonder if there are competent models trained purely on permissive open-source code like MIT or Apache 2.0.
MIT and Apache 2.0 both require attribution, so it's not like limiting to those would help in license compliance.
Re: Running local models is good now
#274[deleted]
Re: Running local models is good now
#275[flagged]
Re: Running local models is good now
#276This is the kind of thing that Anthropic et al should be worried about. As it becomes easier and easier to run local models, the ceiling of what they'll be able to charge will get lower and lower. Not that nobody will be willing to pay $$$$$ per month, but a lot of people are going to multiply the per-month charge by 12 or 24 and say "Could I set up a local model for less than that, and have it pay for itself within…
The general consensus is that local models will continue to improve drastically, but hosted models will as well. There will _always_ be a pretty big gulf of capability between what you can do with a desk full of hardware at home vs a few racks of hardware in a datacenter. That seems to be the real "moat" of hosted models at this point in time: access to capital. What's interesting/exciting is that local models are _a…
There is also the thing of workflow.
We have set up something where you create a ticket, Make sure it contains enough information, and with the right tag added it will make a branch with PR for you which stays up to date based on updates to the ticket and comments on the PR.
It’s creepy in a way. But you also can’t really use local (as in workstation LLM) for that. Sure we could run something like a distributed task scheduler across all our engineer devices but just pushing it to copilot is easier.
Re: Running local models is good now
#277Earlier quoted context omitted.
Depends on which variant you pull down, but a single 5090 GPU (I know these are insanely expensive, but for context) could run either the Q8 or Q4_K_M version. It will not fit the 52GB version (BF16) on the other hand. So any modern Mac with a Pro or better processor and more than 52GB of RAM (don't forget VRAM for context window also matters!) would suffice, as someone else noted, probably a 128GB model would do the…
Note you can change the amount of shared (V)RAM reserved for the OS with: sudo sysctl iogpu.wired_limit_mb=18800 will allow you to use more, but you do need to leave a bit for the OS obviously!
Oh man! I had no idea I could do this at all! What do you usually tweak it to? I feel like 8 GB is probably still a reasonable amount to give the rest of the OS.
Re: Running local models is good now
#278Earlier quoted context omitted.
He is using a 2022 M2, which you can get that for about $2k used. That is beyond reasonable.
Global Affordability Estimate: Top 10% of global earners (~800M people) can afford a $2,000 device without major financial strain. Top 25% (~2B people) could afford it with some budget adjustments. Bottom 50% (~4B people) would find it prohibitively expensive. So for a SV top income, maybe that might look more like the weekly pet brushing budget, but for most people out there this is not that much of a no-brainer.
[deleted]
Re: Running local models is good now
#279After having been a happy user of Qwen3.6-27B for a few weeks, due to being away from the hardware, I'm currently forced to use Claude Sonnet 4.6 It is such a downgrade. I don't understand how that's even possible. The thing has so many strongly-held opinions I did not ever ask it for, talking just way too much and generally feeling somehow dumber. Of course, being significantly larger, it will encode more knowledge,…
Why would I want some half assed coding assist tool. I want something that takes in a requirement and spits out a finished product. It’s not your equal, it’s better than you.
Re: Running local models is good now
#280I think a lot of people just don't have specs like that, making it still painful.