Lots of AI HW is focused on RAM (512GB!). I have a cost-sensitive application that needs speed (300+ TOPS), but only 1GB of RAM. Are there any HW companies focused on that space?
Isn't that just any discrete (Nvidia,AMD) GPU?
Apple M3 Ultra
171–180 of 1001 posts
Re: Apple M3 Ultra
#172Earlier quoted context omitted.
I've been buying and using MBP for 6 or 7 years now, and just assumed I could run Linux on one if I wanted to. I just spent a couple of days trying to get a 2018 MBP working with Linux and found out [edit to clarify] that my other ARM MBP basically won't work. I just want a break from MacOS, I'll be buying a Thinkpad and will probably never come back. This isn't my moaning, I understand it's their market, but if thei…
> trying to get a 2018 MBP working with Linux and found out ARM basically doesn't work. Since the M series of ARM processors didn’t come out until 2020, that would make a lot of sense.
Two separate laptops, I could have been clearer. I have an old 2018 I wanted to try it on, and my daily is M2 that would have been next.
Re: Apple M3 Ultra
#173Earlier quoted context omitted.
> This hardware is really being held back by the operating system at this point. Please elucidate.
I torrent things from two different hosts on my gigabit network. The macos stack literally cannot handle the full bandwidth I have. It fails and the machine needs to be rebooted to fix it. It’s not pretty on the way into this state, either. Other remote connections to the computer are unreliable. On Linux, running the same app in a docker container works perfectly. Transmission is the app.
>Transmission is the app.
Former Transmission user here.
I realise you didn't ask, but you might find some improvements in qBittorrent.
Re: Apple M3 Ultra
#174Whoa. M3 instead of M4. I wonder if this was basically binning, but I thought that I had read somewhere that the interposer that enabled this for the M1 chips where not available. That Said, 512GB of unified ram with access to the NPU is absolutely a game changer. My guess is that Apple developed this chip for their internal AI efforts, and are now at the point where they are releasing it publicly for others to use.…
> My guess is that Apple developed this chip for their internal AI efforts what internal AI efforts? Apple Intelligence is bunkers, and Apple MLX framework remains a hobby project for Apple
They’re taking a different and more difficult path of integrating AI with existing apps and workflows.
It’s their spin of the Google strategy of targeting providjng services to their enterprise GCP customer. I think we’ll see more out of them long term.
Re: Apple M3 Ultra
#175Earlier quoted context omitted.
Maybe .1% of tasks need this RAM, why are they charging so much?
because that's how much its worth
Its not though. For consumer computers somewhere 1k-4k there's nothing better. But for the price of 512gb of RAM you could buy that + a crazy CPU + 2x 5090s by building your own. The market fit is "needs power; needs/wants macOS; has no budget" which is so incredibly niche. But in terms of raw compute output there's absolutely no chance this is providing bang for buck
Re: Apple M3 Ultra
#176The memory amount is fantastic, memory bandwidth is half decent(~800 GB/s), and the compute capabilities are terrible(36 TOPS). For comparison, a single consumer card like the RTX 5090 is only 32 GB of memory, has 1792 GB/s memory and 3593 TOPS of compute. The use cases will be limited. While you can't run a 600B model directly like Apple says(cause you need more memory for that), you can run a quantized version, but…
You're comparing two different things. The compute level you’re talking about on the M3 Ultra is the neural engine. Not including the GPU. I expect the GPU here will be behind a 5090 for compute but not by the unrelated numbers you’re quoting. After all, the 5090 alone is multiple times the wattage of this SoC.
> After all, the 5090 alone is multiple times the wattage of this SoC.
FWIW, normalizing the wattages (or even underclocking the GPU) will still give you an Nvidia advantage most days. Apple's GPU designs are closer to AMD's designs than Nvidia's, which means they omit a lot of AI accelerators to focus on a less-LLM-relevent raster performance figure.
Yes, the GPU is faster than the NPU. But Apple's GPU designs haven't traditionally put their competitors out of a job.
Re: Apple M3 Ultra
#177Earlier quoted context omitted.
I've been buying and using MBP for 6 or 7 years now, and just assumed I could run Linux on one if I wanted to. I just spent a couple of days trying to get a 2018 MBP working with Linux and found out [edit to clarify] that my other ARM MBP basically won't work. I just want a break from MacOS, I'll be buying a Thinkpad and will probably never come back. This isn't my moaning, I understand it's their market, but if thei…
A 2018 MacBook would be an intel x86 chip. It’s incredibly easy to get Linux running on that machine.
Getting Linux running wasn't difficult. But Mint lost audio (everything else worked), the specialised Mint kernel lost both audio and wifi, and Arch lost both wifi and the onboard keyboard.
I'm sure with tinkering I could eventually get it working, but I'm well past the point of wanting to tinker with hardware and drivers to get Linux working.
Re: Apple M3 Ultra
#178Lots of AI HW is focused on RAM (512GB!). I have a cost-sensitive application that needs speed (300+ TOPS), but only 1GB of RAM. Are there any HW companies focused on that space?
Most recent GPUs will do. An older RTX 4070 is over 400 TOPS, the new RTX 5070 is around 1000 TOPS, and the RTX 5090 is around 3600 TOPS.
Re: Apple M3 Ultra
#179Ah, if we can have the hardware and the freedom of installing a good Linux repo on top of it. How is Asahi? Is it good enough? I assume, that since Asahi is focused on Apple hardware, it should have an easier time figuring out drivers and etc?
> How is Asahi? For M3 and M4 machines, hardware support is pretty derilict: https://asahilinux.org/docs/M3-Series-Feature-Support/
Thanks, looks like even M1 support has some gaps:
https://asahilinux.org/docs/M1-Series-Feature-Support/#table...
I assume anything that doesn't have "linux-asahi" is not supported -- or any WIP is not supported.
Wish I had the skills to help them. Targeting just one set of architecture, I think Asahi has more chances of success.
Re: Apple M3 Ultra
#180When would Apple silicons made natively support for OSes such as Linux? Apple seemlingly reluctant to release detailed technical reference manual for M-series SoCs, which makes running Linux natively on Apple silicon challenging.
Is it not an option to run Darwin? What would Linux offer that that would not?