Just insane RAM price. Can’t wait for the bubble to pop.
AI is useful but not at the currently unsustainable price (too expensive for the provided value to corporate customers at too cheap per token vs the actual cost of creating it)
New Mac Studio with M5 Max and M5 Ultra
341–350 of 570 posts
Re: New Mac Studio with M5 Max and M5 Ultra
#342Earlier quoted context omitted.
It's more bizarre that Apple got caught with pants down. Focused on CPUs and ignored RAM. Seems like miscalculation. If they had their own fab for RAM, they could completely corner the market today.
They have no fab for CPUs, they are manufactured by Samsung and TSMC. The bottleneck is in manufacturing RAM not CPUs so there is nothing Apple can do here.
Let me rephrase. They have not booked enough capacity at contract manufacturer.
Re: New Mac Studio with M5 Max and M5 Ultra
#343Due to pricing insanity (not that Apple prices weren’t insane before the ram/ssd shortages) I’m not in the market but I do wonder if my next computer should be a Mac Studio instead of a MBP that lives its life docked. Might be better to just run a Studio and Neo for the very few times I actually need remote capabilities.
> not that Apple prices weren’t insane before the ram/ssd shortages
Funny that after the price started to increase last year (I think October/November), there was a window of a few months where Apple prices stayed the same as they were before, thus making apple prices actually good when compared to the rest of the market.
It was a unique opportunity to have acquired a 512G M3 ultra for $10k.
Re: New Mac Studio with M5 Max and M5 Ultra
#344Due to pricing insanity (not that Apple prices weren’t insane before the ram/ssd shortages) I’m not in the market but I do wonder if my next computer should be a Mac Studio instead of a MBP that lives its life docked. Might be better to just run a Studio and Neo for the very few times I actually need remote capabilities.
I have a Mac Studio M2 Max as my main driver. It's fantastic. I will get the best M5 Studio I can afford when the higher RAM options come out because I am very involved with local inference. I find that with Tailscale I can either use screen sharing if I've got a good enough internet connection or just SSH and Mosh if it's a little bit laggy. That has been good enough for me. A Mac Studio and a Neo as a companion is a really good combination. My laptop is an Apple M4 Air but it basically does nothing except for running ScreenShare and a terminal.
Re: New Mac Studio with M5 Max and M5 Ultra
#345Earlier quoted context omitted.
I’ve wanted the idea of a home server or even home “mainframe” and some terminals for me and family forever but the idea never takes off….
Same. For me, it’s one of those “solutions in search of a problem.”
It just seems somewhat more efficient to take all the computing storage, memory processes, processors, or at least all the money that go into it… the in one spot.
If you’re running a big task or small, everything scales to the appropriate size, regardless of the hardware sitting on your lap or under your desk.
At least that’s how I think of it.
Re: New Mac Studio with M5 Max and M5 Ultra
#346Earlier quoted context omitted.
you can't link onchip memory.
True, but you can link them up over thunderbolt or Ethernet. If your goal is to run local LLMs, not all weights need to live on the same computer. You can segment the workload by layers and pass the activations along the lower bandwidth interconnect with a small perf penalty. Also you get double the CPU/GPU cores allowing for better multi user/agent performance.
I was thinking the same as you as far as price per value, it does make seance to get 2x of these things IMO, but what throughput hit would you see in linking versus one machine? latency does matter, and there must be a trade off no?
Re: New Mac Studio with M5 Max and M5 Ultra
#347Earlier quoted context omitted.
1tb would likely be ~$20k - given the current >$10k price tag of 256gb. Would you still be considering it at that price?
I would probably spend up to $30k if I could get 1TB of Unified Memory, because it would allow me a guarantee to run pretty much any local model I want, including >1T parameter models with reasonable quants. I wouldn't be surprised if 512GB is close to $20k when it becomes orderable in October. The justification is less about absolute price and more about price to what it enables. 512GB really doesn't enable much ove…
I suspect we will see more companies which will burn the weights right in to silicon arise. There is already at least one company out there that showed its possible so others will follow IMO. Basically you will see the rise of disposable weights like Nintendo cartridges back in the day. Use it for a time until the better model comes out and you get a new "chip". Though there's a caveat for this business model and that requires you to pump out lots of these chips on the cheap so you are beholden to the lithography companies and what they can produce for you. If you can do this at scale and doesn't require the latest state of the art nm architecture design you are golden...
Re: New Mac Studio with M5 Max and M5 Ultra
#348> Apple’s most powerful Mac raises the bar for local AI Wow, "Local AI" mentioned in the subheading above the fold - it's really awesome to see Apple leaning into this use case and I think it will definitely pay off for them going forward. Fingers crossed Apple is able to put some engineering effort towards shipping with one of the frontier open weight models included and optimized exactly for the machine.
The local ai is used by Apple services, mostly. Though there is a Swift SDK to access it, apparently.
You can run Ollama and it’s kind with MLX models for speed.
Tend to be better than Apple AI.
Re: New Mac Studio with M5 Max and M5 Ultra
#349Re: New Mac Studio with M5 Max and M5 Ultra
#350Earlier quoted context omitted.
I would probably spend up to $30k if I could get 1TB of Unified Memory, because it would allow me a guarantee to run pretty much any local model I want, including >1T parameter models with reasonable quants. I wouldn't be surprised if 512GB is close to $20k when it becomes orderable in October. The justification is less about absolute price and more about price to what it enables. 512GB really doesn't enable much ove…
I suspect we will see more companies which will burn the weights right in to silicon arise. There is already at least one company out there that showed its possible so others will follow IMO. Basically you will see the rise of disposable weights like Nintendo cartridges back in the day. Use it for a time until the better model comes out and you get a new "chip". Though there's a caveat for this business model and tha…
> There is already at least one company out there that showed its possible so others will follow IMO.
Are your referring to Taalas / chatjimmy ?
I definitely think we'll see an ASIC-like approach in the future, especially for embedded small models where it may require minimal silicon area and can result in near-realtime performance. But at the frontier, I don't think this is a solved problem and will continue to have model weight churn that will advantage more flexible general-purpose hardware.