Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AMD Strix Halo RDMA Cluster Setup Guide

github.com

Post reply on HN
Print view View original

31–40 of 90 posts

Re: AMD Strix Halo RDMA Cluster Setup Guide

#31

Post by rzzzt »

pixelpoet wrote: post #23
rjzzleep wrote: post #21

Earlier quoted context omitted.

Last year you could buy a AI Max 395+ with 128G for 2.5k, now it's almost $4k. Or maybe you're right, I originally remembered 2k as well. I wanted to wait for the AI Max 395+ upgrade of my laptop, and now it makes no sense to upgrade.

I think people buying laptops for AI use are, sorry, just plain crazy. You overpay for the screen and keyboard and battery and whatever, plus you get much worse thermal performance because of basic physics (area vs volume). My Framework Desktop has a Noctua cooler which works really well. [Tangent: all my life I've been downvoted into a smoking hole in the ground, particularly on reddit r/hardware, for questioning th…

For me the smaller footprint, lower power consumption and portability (admittedly between desks only) are the three advantages of using a laptop over a desktop for these purposes.

Re: AMD Strix Halo RDMA Cluster Setup Guide

#32

Post by sspiff »

pixelpoet wrote: post #3

I have two 128gb Strix Halos and have been extremely excited about Antirez's (Redis author) work on DS4, especially with 4bit quant using two machines: https://github.com/antirez/ds4 Right now the speed isn't good for GLM 5.2, Deepseek V4 Flash speed is okay for me (actually reading the output) and quite usable. See kyuz0's great recent video here: https://www.youtube.com/watch?v=PkKXm_mKCCM With a bit more speed and…

I had a Strix Halo laptop with 128GB which unfortunately died last week. I paid 2800 euro for it. If I buy the same machine today, the sticker price is 7899.

The device was not perfect by any means, but the ability to run fairly large models is some kind of magic.

Re: AMD Strix Halo RDMA Cluster Setup Guide

#33

Post by kamranjon »

pixelpoet wrote: post #23
rjzzleep wrote: post #21

Earlier quoted context omitted.

Last year you could buy a AI Max 395+ with 128G for 2.5k, now it's almost $4k. Or maybe you're right, I originally remembered 2k as well. I wanted to wait for the AI Max 395+ upgrade of my laptop, and now it makes no sense to upgrade.

I think people buying laptops for AI use are, sorry, just plain crazy. You overpay for the screen and keyboard and battery and whatever, plus you get much worse thermal performance because of basic physics (area vs volume). My Framework Desktop has a Noctua cooler which works really well. [Tangent: all my life I've been downvoted into a smoking hole in the ground, particularly on reddit r/hardware, for questioning th…

I generally agree for everything except Macbook Pros which outperform most available desktop setups for AI tasks - but they are also now out of reach for most people after the price hikes (6.7k now for 128gb, i got mine for 4.7k just about a year ago).

Honestly I think this is just a bad time to be buying hardware - everything is marked up an insane amount that doesn't really make sense.

Re: AMD Strix Halo RDMA Cluster Setup Guide

#34

Post by kamranjon »

Tepix wrote: post #27
kamranjon wrote: post #20

Benchmarks are here: https://kyuz0.github.io/amd-strix-halo-vllm-toolboxes/ Would love to see DeepSeek V4 flash/pro and MiniMax M3 benchmarks but already these are pretty impressive, first strix Halo setup I've seen with some serious performance. EDIT: Apologies - I think I misunderstood these benchmarks - it seems this is actually very slow when compared to a M4 or M5 chip with a good amount of memory. Looking at th…

The pp speeds are really slow (50), I think there‘s room for improvement still.

Ah yea after watching one of the creators youtube videos I realize these benchmarks are combining prefill and decode which isn't super helpful - it seems this struggles with the exact same bottlenecks as all strix halo setups, memory bandwidth. It seems this is still significantly slower than equivalent memory sizing on Mac hardware.

Re: AMD Strix Halo RDMA Cluster Setup Guide

#35

Post by Hikikomori »

jmyeet wrote: post #6

So this is kind of fascinating. The main hardware costs here seem to be: - 2x Framework Desktop AI Mainboards with 128GB of RAM for $3150 each - 2x 100G Ethernet controllers for ~$500 each So the Framework board has a single PCI-e 4.0 x4 slot, which amounts to 8GB/s or 64Gbps theoretical so you're not getting 100G. Also, the 100G cards all seem to be PCI-e x16 slots for obvious reasons so you need a riser or an adapt…

What is simplex ethernet?

Re: AMD Strix Halo RDMA Cluster Setup Guide

#36

Post by pixelpoet »

rzzzt wrote: post #31
pixelpoet wrote: post #23

Earlier quoted context omitted.

I think people buying laptops for AI use are, sorry, just plain crazy. You overpay for the screen and keyboard and battery and whatever, plus you get much worse thermal performance because of basic physics (area vs volume). My Framework Desktop has a Noctua cooler which works really well. [Tangent: all my life I've been downvoted into a smoking hole in the ground, particularly on reddit r/hardware, for questioning th…

For me the smaller footprint, lower power consumption and portability (admittedly between desks only) are the three advantages of using a laptop over a desktop for these purposes.

The Strix Halo mini PCs use the exact same chip, and have a much smaller footprint than any laptop. Have you seen the size of these machines? I can and have easily popped my daily driver computer into my very small backpack to attend a demoparty for example.

With the laptop you probably won't get silent operation at the peak 100-140w, i.e. you've now massively overpaid for lower performance.

Re: AMD Strix Halo RDMA Cluster Setup Guide

#37

Post by barbacoa »

sspiff wrote: post #32
pixelpoet wrote: post #3

I have two 128gb Strix Halos and have been extremely excited about Antirez's (Redis author) work on DS4, especially with 4bit quant using two machines: https://github.com/antirez/ds4 Right now the speed isn't good for GLM 5.2, Deepseek V4 Flash speed is okay for me (actually reading the output) and quite usable. See kyuz0's great recent video here: https://www.youtube.com/watch?v=PkKXm_mKCCM With a bit more speed and…

I had a Strix Halo laptop with 128GB which unfortunately died last week. I paid 2800 euro for it. If I buy the same machine today, the sticker price is 7899. The device was not perfect by any means, but the ability to run fairly large models is some kind of magic.

>sticker price is 7899.

It's not even worth it at that point.

You can get a used enterprise grade SXM baseboard with 4-8 V100/A100 GPUs off eBay at a similar price. That will even get you actual HMB ram and NVlink. Along with 10x the AI performance, assuming you don't care about your electricity bill of course.

Re: AMD Strix Halo RDMA Cluster Setup Guide

#38

Post by throwa356262 »

pixelpoet wrote: post #36
rzzzt wrote: post #31

Earlier quoted context omitted.

For me the smaller footprint, lower power consumption and portability (admittedly between desks only) are the three advantages of using a laptop over a desktop for these purposes.

The Strix Halo mini PCs use the exact same chip, and have a much smaller footprint than any laptop. Have you seen the size of these machines? I can and have easily popped my daily driver computer into my very small backpack to attend a demoparty for example. With the laptop you probably won't get silent operation at the peak 100-140w, i.e. you've now massively overpaid for lower performance.

Can you get these from vendors like Asus and lenovo these days?

The ones I've seen on aliexpress are from unknown Chinese vendors.

Re: AMD Strix Halo RDMA Cluster Setup Guide

#39

Post by PeterStuer »

jmyeet wrote: post #6

So this is kind of fascinating. The main hardware costs here seem to be: - 2x Framework Desktop AI Mainboards with 128GB of RAM for $3150 each - 2x 100G Ethernet controllers for ~$500 each So the Framework board has a single PCI-e 4.0 x4 slot, which amounts to 8GB/s or 64Gbps theoretical so you're not getting 100G. Also, the 100G cards all seem to be PCI-e x16 slots for obvious reasons so you need a riser or an adapt…

The 512GB Mac Studio is going for around 30K used.

Re: AMD Strix Halo RDMA Cluster Setup Guide

#40

Post by pixelpoet »

throwa356262 wrote: post #38
pixelpoet wrote: post #36

Earlier quoted context omitted.

The Strix Halo mini PCs use the exact same chip, and have a much smaller footprint than any laptop. Have you seen the size of these machines? I can and have easily popped my daily driver computer into my very small backpack to attend a demoparty for example. With the laptop you probably won't get silent operation at the peak 100-140w, i.e. you've now massively overpaid for lower performance.

Can you get these from vendors like Asus and lenovo these days? The ones I've seen on aliexpress are from unknown Chinese vendors.

I have a Framework Desktop as primary PC (great cooling, beautiful case with handle) and the Bosgame M5 dedicated for AI use.

I was also a bit wary about Bosgame but TBH they've been great and the machine is rock solid, if a little noisier than and not as pretty as the FD. You can just buy from them directly and be fine, best computer deal out there by a mile.

Post reply on HN