Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apple M3 Ultra

apple.com

Post reply on HN
Print view View original

591–600 of 1001 posts

Re: Apple M3 Ultra

#591

Post by zamalek »

alok-g wrote: post #564

Two questions for the fellow HNers: 1. What are various average joe (as opposed to researchers, etc.) use cases for running powerful AI models locally vs. just using cloud AI. Privacy of course is a benefit, but it by itself may not justify upgrades for an average user. Or are we expecting that new innovation will lead to much more proliferation of AI and use cases that will make running locally more feasible? 2. Wit…

I don't think there's a huge use-case locally, if you're happy with the subscription cost and privacy. That is, yet. Give it maybe 2 years and someone will probably invent something which local inference would seriously benefit from. I'm anticipating inference for the home appliances (something mac mini form factor that plugs into your router) but that's based on what would make logical sense for consumers, not what consumers would fall for.

Apple seems to be using LPDDR, but HBM will also likely be a key tech. SK Hynix and Samsung are the most reputable for both.

Re: Apple M3 Ultra

#592

Post by coconut08 »

epolanski wrote: post #569

Can anybody ELI5 why aren't there multi gpu builds to run LLMs locally? It feels like one should be able to build a good machine for 3/4k if not less with 6 16GB mid level gaming GPUs.

6 * 16 is still nowhere near 512gb of vram. On top of that that monster that you create requires hyper specific server grade hardware, will be huge, loud and pull down enough power to trip a circuit breaker. i'm sure most people would rather pay a 30 percent premium to get twice the ram and have a power sipping device that you can hold in the palm of your hand.

Re: Apple M3 Ultra

#593

Post by user3939382 »

christiangenco wrote: post #571
alok-g wrote: post #564

Two questions for the fellow HNers: 1. What are various average joe (as opposed to researchers, etc.) use cases for running powerful AI models locally vs. just using cloud AI. Privacy of course is a benefit, but it by itself may not justify upgrades for an average user. Or are we expecting that new innovation will lead to much more proliferation of AI and use cases that will make running locally more feasible? 2. Wit…

IMO it's all about privacy. Perhaps also availability if the main LLM providers start pulling shenanigans but it seems like that's not going to be a huge problem with how many big players are in the space. I think a great use case for this would be in a company that doesn't want all of their employees sending LLM queries about what they're working on outside the company. Buy one or two of these and give everybody a c…

Hopefully homomorphic encryption can solve this rather than building a new hardware layer everywhere.

Re: Apple M3 Ultra

#594

Post by alok-g »

archagon wrote: post #584
alok-g wrote: post #564

Two questions for the fellow HNers: 1. What are various average joe (as opposed to researchers, etc.) use cases for running powerful AI models locally vs. just using cloud AI. Privacy of course is a benefit, but it by itself may not justify upgrades for an average user. Or are we expecting that new innovation will lead to much more proliferation of AI and use cases that will make running locally more feasible? 2. Wit…

I don’t currently use AIs, but if I did, they would be local. Simply put: I can’t build my professional career around tools that I do not own.

>> ... around tools that I do not own.

That just may be dependent on how much trust you have on the providers you use. Or do you do your own electricity generation?

Re: Apple M3 Ultra

#595

Post by johnklos »

egorfine wrote: post #470
LeifCarrotson wrote: post #446

Earlier quoted context omitted.

Yep, it's apples to oranges. But sometimes you want apples, and sometimes you want oranges, so it's all good! There's a wide spectrum of potential requirements between memory capacity, memory bandwidth, compute speed, compute complexity, and compute parallelism. In the past, a few GB was adequate for tasks that we assigned to the GPU, you had enough storage bandwidth to load the relevant scene into memory and generat…

> we still work with powers of two. Please. We do. Common people don't. It's easier to write "over half a terabyte" than explain (again) to millions of people what the power of two is.

Anyone who calls 512 gigs "over half a terabyte" is bullshitting. No, thank you.

Re: Apple M3 Ultra

#596

Post by azinman2 »

hangonhn wrote: post #457
diggan wrote: post #393

Earlier quoted context omitted.

> The the question is if a llm will run with usable performance at that scale? This is the big question to have answered. Many people claim Apple can now reliably be used as a ML workstation, but from the numbers I've seen from benchmarks, the models may fit in memory, but the performance for tok/sec is so slow to not feel worth it, compared to running it on NVIDIA hardware. Although it be expensive as hell to get 51…

Do we know if is it slower because of hardware is not as well suited for the task or is it mostly a software issue -- the code hasn't been optimized to run on Apple Silicon?

Memory bandwidth is the issue

Re: Apple M3 Ultra

#597

Post by echoangle »

tomrod wrote: post #485
hedora wrote: post #349

Earlier quoted context omitted.

That’s a laptop part, so it makes different tradeoffs. Somewhere on the internet there is a tdp wattage vs performance x-y plot. There’s a pareto optimal region where all the apple and amd parts live. Apple owns low tdp, AMD owns high tdp. They duke it out in the middle. Intel is nowhere close to the line. I’d guess someone has made one that includes datacenter ARM, but I’ve never seen it.

> tdp wattage vs performance x-y plot This? https://www.videocardbenchmark.net/power_performance.html#sc...

That’s GPUs, not CPUs

Re: Apple M3 Ultra

#598

Post by Thaxll »

exabrial wrote: post #203
InTheArena wrote: post #29

Whoa. M3 instead of M4. I wonder if this was basically binning, but I thought that I had read somewhere that the interposer that enabled this for the M1 chips where not available. That Said, 512GB of unified ram with access to the NPU is absolutely a game changer. My guess is that Apple developed this chip for their internal AI efforts, and are now at the point where they are releasing it publicly for others to use.…

If Apple supported Linux (headless) natively, and we could rack m4 pros, I absolutely would use them in our Colo. The CPUs have zero competition in terms of speed, memory bandwidth. Still blown away no other company has been able to produce Arm server chips that can compete.

Apple does not make server CPUs, they make consumer low W CPUs, it's very different.

FYI Apple runs Linux in their DC, so no Apple hardware in their own servers.

Re: Apple M3 Ultra

#599

Post by archagon »

alok-g wrote: post #594
archagon wrote: post #584

Earlier quoted context omitted.

I don’t currently use AIs, but if I did, they would be local. Simply put: I can’t build my professional career around tools that I do not own.

>> ... around tools that I do not own. That just may be dependent on how much trust you have on the providers you use. Or do you do your own electricity generation?

That's quite a reductio ad absurdum. No, I don't generate my own electricity (though I could). But I don't use tools for work that can change out from under me at any moment, or that can increase 10x in price on a corporate whim.

Re: Apple M3 Ultra

#600

Post by kridsdale1 »

titzer wrote: post #466
hangonhn wrote: post #457

Earlier quoted context omitted.

Do we know if is it slower because of hardware is not as well suited for the task or is it mostly a software issue -- the code hasn't been optimized to run on Apple Silicon?

AFAICT the neural engine has accelerators for CNNs and integer math, but not the exact tensor operations in popular LLM transformer architectures that are well-supported in GPUs.

I have to assume they’re doing something like that in the lab for 4 years from now.
Post reply on HN