Earlier quoted context omitted.
I'm curious why not. I am running a few different models on my mac studio. I'm using llama.cpp, and it performs amazingly fast for the $7k I spent.
I said in parallel.
Surely you can run smaller models together
Apple M3 Ultra
851–860 of 1001 posts
Re: Apple M3 Ultra
#852Two questions for the fellow HNers: 1. What are various average joe (as opposed to researchers, etc.) use cases for running powerful AI models locally vs. just using cloud AI. Privacy of course is a benefit, but it by itself may not justify upgrades for an average user. Or are we expecting that new innovation will lead to much more proliferation of AI and use cases that will make running locally more feasible? 2. Wit…
I don’t currently use AIs, but if I did, they would be local. Simply put: I can’t build my professional career around tools that I do not own.
Even then you're still dependent on someone training new models on sketchy data sources that they don't own.
Re: Apple M3 Ultra
#853Now with Ultra-class backdoors? https://news.ycombinator.com/item?id=43003230
It's unlikely that was a backdoor.
Re: Apple M3 Ultra
#854Re: Apple M3 Ultra
#855Who is this made for? Who needs a personal computer this powerful? Not trying to be funny - it's a genuine question. Gamers don't generally use a mac because of the lack of games and I'm guessing those who are really into LLMs use Linux for the flexibility. Video editing can be done on much cheaper hardware. Very rich LLM enthusiasts who wants to try out mac?
I don't think people into LLMs necessarily use Linux, most devs I see around use a mac, and I think I'll buy one and move out of Ubuntu if I start using them more seriously. The perfs are the key selling point as many professional use cases benefits from running LLMs locally.
I used to run Ubuntu with i3 and recently switched to a macbook air. I thought I would hate it coming from i3, but honestly when using the stage manager + tmux & nvim and betterTouchTool for keybinds it feels just as effective. The fact that fullscreen applications get their own desktop is nice too.
Re: Apple M3 Ultra
#856Earlier quoted context omitted.
Since no one specifically answered your question yet, yes, you should be able to get usable performance. A Q4_K_M GGUF of DeepSeek-R1 is 404GB. This is a 671B MoE that "only" has 37B activations per pass. You'd probably expect in the ballpark of 20-30 tok/s (depends on how much actually MBW can be utilized) for text generation. From my napkin math, the M3 Ultra TFLOPs is still relatively low (around 43 FP16 TFLOPs?),…
I actually think it’s not a coincidence and they specifically built this M3 Ultra for DeepSeek R1 4-bit. They also highlight in their press release that they tested it with 600B class LLMs (DeepSeek R1 without referring to it by name). And they specifically did not stop at 256 GB RAM to make this happen. Maybe I’m reading too much into it.
[dead]
Re: Apple M3 Ultra
#857Earlier quoted context omitted.
Asahi is a thing. For headless usage it’s pretty much ready to go already.
M3 support in Asahi is still heavily WIP. I think it doesn't even have display support, Ethernet, or Wifi yet, I think it's only serial over USB . Without any GPU or ANE support, it's not very useful for AI stuff. https://asahilinux.org/docs/M3-Series-Feature-Support/
Hmm, this page links to an out-of-tree ANE module (same as on M1/M2 I believe). No GPU support is a bummer, though.
On the other hand, you can do without display support if you’re only using it as a server. And I think USB Ethernet dongles might work for the time being?
Re: Apple M3 Ultra
#858Earlier quoted context omitted.
In fairness, the sun and dec boxes I used back then (up to about 1999) could hold their own against intel machines. Then, one day, we built a 5 machine amd athlon xp linux cluster for $2000 ($400/machine) that beat all the unix and windows server hardware by at least 10x on $/perf. It’s nice that we have more than one viable cpu vendor these days, though it seems like there’s only one viable fab company.
In 1998-1999 I had a DEC Alpha on my desktop that was really impressive, it was a 64-bit machine a few years before you could get a 64-bit Athlon.
In 1998 I somehow got my hands on a DEC OEM 21164 533mhz board for cheap. PCs were nowhere near the performance of that
at the time. It mounted in a regular PC case. A friend helped me get the power supply working (there was I think one wire to solder somewhere). Equipped with an ASUS SCSI card, an DEC Ethernet card, and an Elsa GLoria Synergy, it was a full machine. I ran Digital Unix at home on my desk on that for quite a few years. Wish I had kept it for old times sake.
One thing I remember about Alpha though was how bad the output from gcc was. Then DEC released a version of their own compilers that was command line compatible with gcc. That changed everything for open source stuff.
Re: Apple M3 Ultra
#859Earlier quoted context omitted.
The thing that people are excited about here is unified memory that the GPU can address. Mac Pro had discrete GPUs with their own memory.
Intel integrated graphics, technically also used unified memory with the standard dram
Did the Xeons in the Mac Pro even have integrated graphics?
Re: Apple M3 Ultra
#860God I wish Linux ran on Apple Silicon (with first class hardware support).
We need more mainline Linux devs to work on Apple Silicon feature gaps, instead of expecting tiny AsahiLinux to roll support up the mountain alone.