"Just get a 64GB Mac with 1TB of storage!" LOL - some of us have a budget
Pros buy their own tools. This is why working for yourself is better than working for a corpo, you get to choose your weapon.
That's daft mate
Running local models is good now
641–650 of 651 posts
Re: Running local models is good now
#642Earlier quoted context omitted.
I don't think it is easy, otherwise this wouldn't be such a contentious and frequently discussed issue. A human who trains their brain on material they don't own then creates art or writes code based on this training, sells the product too.
Which is not illegal to do. It becomes illegal if you directly use or reference the material in your product. Consuming copyrighted material personally is fully legal, training a model on that same copy-righted material is illegal. Where is the contention? (Not a lawyer, just a nerd)
>Consuming copyrighted material personally is fully legal, training a model on that same copy-righted material is illegal. Where is the contention?
The contention is obvious in your statement. Is it illegal to train a model, and why? Why is it legal to train a human on copyright material and then sell those skills, vs training a computer and selling the skills?
This isn't black and white.
Re: Running local models is good now
#643Earlier quoted context omitted.
> thers, like coding or more advanced agentic workflows can demand much more powerful models. You can do coding and agentic fine. For coding I use qwen3.6:35b-mlx and agentic granite4.1:3b works fine. These are the models I use. - granite4.1:3b - granite4.1:30b - gpt-oss:20b - gpt-oss:120b (less so now) - mistral-small3.2 - qwen3.6:35b-mlx There will always be use cases that don't sit on your laptop, but most of what…
Why do you like gpt-oss-120b less now? What replaced it?
For me the other models are faster and results are good enough.
Re: Running local models is good now
#644Earlier quoted context omitted.
Pros buy their own tools. This is why working for yourself is better than working for a corpo, you get to choose your weapon.
That's daft mate
Is it? When you're inside the fence you get some IT cast off from the last guy that was cleaned up and re-spywared from 2-3y ago (sometimes worse!). As a hired gun (data science/engineer), I have an m3 128gb 1tb all sliders to the right MacBook Pro that works great, no spyware, rocket fast, trains/runs small models. With Apple 0%, paid in a year. I generally can name my price so it pays for itself.
If you're riding it every day, every little speed up counts. I don't wait on GitHub actions or external systems for my crunching. For that I have my local slurm cluster of mini PCs, 64 cpus and counting!
I assure you there is a better world outside of the fence, I am unlikely to return.
Re: Running local models is good now
#645Earlier quoted context omitted.
Global Affordability Estimate: Top 10% of global earners (~800M people) can afford a $2,000 device without major financial strain. Top 25% (~2B people) could afford it with some budget adjustments. Bottom 50% (~4B people) would find it prohibitively expensive. So for a SV top income, maybe that might look more like the weekly pet brushing budget, but for most people out there this is not that much of a no-brainer.
Yes, because the bottom 50%, mostly impoverished or near impoverished folks were spending money on Claude Code subscriptions instead /s
2K is 10 years of a Claude Pro subscription, which also gives you better models...
Re: Running local models is good now
#646Earlier quoted context omitted.
That's daft mate
Is it? When you're inside the fence you get some IT cast off from the last guy that was cleaned up and re-spywared from 2-3y ago (sometimes worse!). As a hired gun (data science/engineer), I have an m3 128gb 1tb all sliders to the right MacBook Pro that works great, no spyware, rocket fast, trains/runs small models. With Apple 0%, paid in a year. I generally can name my price so it pays for itself. If you're riding i…
I'm curious the last time you touched reality
Re: Running local models is good now
#647Earlier quoted context omitted.
Yep, I daily drive Qwen3.6-27B (including for work), have done pretty much since it came out. IMO it's the only (small-ish, local) model worth using, if you can run it. It might not be as good as Opus at "add X large feature" but I don't want that in a model. I want to do the thinking while it does the typing. And Qwen 3.6 27B is perfectly good at that (while in my experience models like the 35A3B and gemma are signi…
Just this morning I tweaked my single 3090 setup too: OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q8_0 OLLAMA_CONTEXT_LENGTH=180000 and that fits in 23GB. [edited for format]
Just a heads up (you may already know this) -- In the local LLM community there's a pretty strong disdain for Ollama specifically due to numerous reasons provided in this blog post: https://sleepingrobots.com/dreams/stop-using-ollama/
Re: Running local models is good now
#648Earlier quoted context omitted.
I see that OVH offers Qwen3.5-397B-A17B, which is a bit surprising to me. I thought that EU providers had to comply with the AI act where you have to provide opt-out and information about the training data once the model is sufficiently large (over 10^23 FLOPs, likely the case here), but providing information is not possible since people who train those models only give vague information at best. Does anyone know if…
OVH is acting as a "Deployer", not a "Provider", which have special meaning under the AI Act. There are much less (almost no) disclosure regulations on the deployer. https://ethicalogic.com/articles/gpai-guide-roles-public-dat...
Well, there are no disclosure requirements on Deployers because doing any kind of training or fine-tuning automatically makes you a Provider.
Re: Running local models is good now
#649[flagged]
Re: Running local models is good now
#650Earlier quoted context omitted.
There are a lot of government jobs I know of that are absolutely air-gapped. Your computer has basically no internet access, everything is stored on-prem. Hedge funds also tend to be extremely locked down, from what I saw when I interviewed. With certain data sets either having strict encryption-in-transit or a being stored in a quirky on-prem service. I can't imagine they're going to be dumping their data into Claud…
> There are a lot of government jobs I know of that are absolutely air-gapped. Your computer has basically no internet access, everything is stored on-prem. I wonder if that's because they don't know better or because of a lack of trust or costs?
It's lack of trust