31–40 of 208 posts
My biggest problem with running local LLMs on my M4 Max/128GB RAM is the prefill latency. I've since acquired two DGX Sparks, and it feels so much snappier.
Would you mind sharing your local Mac setup and which models you currently use and whether it’s GGUF or MLX? I’ve the hardware same specs.
Earlier quoted context omitted.
Would you mind sharing your local Mac setup and which models you currently use and whether it’s GGUF or MLX? I’ve the hardware same specs.
Primarily used ds4[0] by antirez [0] https://github.com/antirez/ds4
Earlier quoted context omitted.
It’s not. Do it as a hobby or for privacy but for performance just use a frontier model api. You’re paying less than cost for something that would take tens of thousands to set up locally.
That's not even remotely close to being true, even once you account for capex. You have to look at the actual usage, look at the token limits. Even if you're paying Anthropic $200k/month for scale-tier, you're going to blow through your token limits trying to run max output 24/7. Three users running Opus 4.8 at max non-stop will probably clean your monthly allowance from daddy Dario in less than a week. With an 8x MI…
Pretty expensive is an understatement. You couldn’t buy one of these if you wanted to right now. If you could it would be multiple hundreds of thousands of dollars.
> It does mean 8 multi-trillion parameter models unquantized running 24/7 without pause
You can’t even run one unquantized multi-trillion parameter (>=2T) model on 8 x MI355x with enough context for concurrent users. I don’t know how you think it’s going to run 8 of them at the same time. Did you mean 8 concurrent sessions?
Your math is way off across this post. If replacing an Anthropic subscription for a whole company was as easy as buying a box for the office and then breaking even in 2 months, it wouldn’t be some little secret that we only discover in a comment online.
Earlier quoted context omitted.
It’s not. Do it as a hobby or for privacy but for performance just use a frontier model api. You’re paying less than cost for something that would take tens of thousands to set up locally.
It does make me wonder how the hosted stuff is so cheap. For pretty much everything else, hosted/rented is more expensive but offers better convenience and flexibility. But for AI, even if you consider the total lifetime cost and are utilizing it heavily. You never break even by buying.
Earlier quoted context omitted.
It’s not. Do it as a hobby or for privacy but for performance just use a frontier model api. You’re paying less than cost for something that would take tens of thousands to set up locally.
I watched someone at a fortune 20 company get embarrassed for buying a Mac to run a 70B model in 2025. He was a lead engineer, so after he announced it wasn't going to work, everyone pretended it never happened. But we all knew.
Earlier quoted context omitted.
It’s not. Do it as a hobby or for privacy but for performance just use a frontier model api. You’re paying less than cost for something that would take tens of thousands to set up locally.
I watched someone at a fortune 20 company get embarrassed for buying a Mac to run a 70B model in 2025. He was a lead engineer, so after he announced it wasn't going to work, everyone pretended it never happened. But we all knew.
Is that supposed to be hallucination? The human or other kind. Feels like a made up URL. It's .ai, isn't it?
No mention of the performance of the models? I'm able to load a bunch of different models on my little mini-PC with 16GB RAM, but the performance is terrible. I always wonder what performance people are getting with local models that they find is acceptable?
I run a similar setup to the one he described on similar hardware. I run bifrost and llama swap though (tailscale rocks). My local model usage is for some out of band batch processing one of my personal apps uses. Basically a personalized recommender for media, it curates stuff for me based on a database i've compiled over years, so non-interactive. For that use case, I don't really care that it might take a few minu…
No mention of the performance of the models? I'm able to load a bunch of different models on my little mini-PC with 16GB RAM, but the performance is terrible. I always wonder what performance people are getting with local models that they find is acceptable?
I run a similar setup to the one he described on similar hardware. I run bifrost and llama swap though (tailscale rocks). My local model usage is for some out of band batch processing one of my personal apps uses. Basically a personalized recommender for media, it curates stuff for me based on a database i've compiled over years, so non-interactive. For that use case, I don't really care that it might take a few minu…