Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AMD Strix Halo RDMA Cluster Setup Guide

github.com

Post reply on HN
Print view View original

21–30 of 90 posts

Re: AMD Strix Halo RDMA Cluster Setup Guide

#21

Post by rjzzleep »

pixelpoet wrote: post #3

I have two 128gb Strix Halos and have been extremely excited about Antirez's (Redis author) work on DS4, especially with 4bit quant using two machines: https://github.com/antirez/ds4 Right now the speed isn't good for GLM 5.2, Deepseek V4 Flash speed is okay for me (actually reading the output) and quite usable. See kyuz0's great recent video here: https://www.youtube.com/watch?v=PkKXm_mKCCM With a bit more speed and…

Last year you could buy a AI Max 395+ with 128G for 2.5k, now it's almost $4k.

Or maybe you're right, I originally remembered 2k as well. I wanted to wait for the AI Max 395+ upgrade of my laptop, and now it makes no sense to upgrade.

Re: AMD Strix Halo RDMA Cluster Setup Guide

#22

Post by francisduvivier »

rnewme wrote: post #4
pixelpoet wrote: post #3

I have two 128gb Strix Halos and have been extremely excited about Antirez's (Redis author) work on DS4, especially with 4bit quant using two machines: https://github.com/antirez/ds4 Right now the speed isn't good for GLM 5.2, Deepseek V4 Flash speed is okay for me (actually reading the output) and quite usable. See kyuz0's great recent video here: https://www.youtube.com/watch?v=PkKXm_mKCCM With a bit more speed and…

What's the advantage of ds4 over llama.cpp, esp if down the line they upstream his forked kernels?

I think mainly that he can move much faster with specific improvements targeting Deepseek on Systems with unified memory (Mac or Strix). It's a lot easier to optimize if you don't need to worry about all the other architectures. So optimize he did and it's just a lot faster than llama cpp for deepseek v4 pro and flash. Also interesting features are more doable, like SSD streaming, which makes it possible to load MOE weights for a model larger than your VRAM, I don't see that landing in llama cpp anytime soon.

Re: AMD Strix Halo RDMA Cluster Setup Guide

#23

Post by pixelpoet »

rjzzleep wrote: post #21
pixelpoet wrote: post #3

I have two 128gb Strix Halos and have been extremely excited about Antirez's (Redis author) work on DS4, especially with 4bit quant using two machines: https://github.com/antirez/ds4 Right now the speed isn't good for GLM 5.2, Deepseek V4 Flash speed is okay for me (actually reading the output) and quite usable. See kyuz0's great recent video here: https://www.youtube.com/watch?v=PkKXm_mKCCM With a bit more speed and…

Last year you could buy a AI Max 395+ with 128G for 2.5k, now it's almost $4k. Or maybe you're right, I originally remembered 2k as well. I wanted to wait for the AI Max 395+ upgrade of my laptop, and now it makes no sense to upgrade.

I think people buying laptops for AI use are, sorry, just plain crazy. You overpay for the screen and keyboard and battery and whatever, plus you get much worse thermal performance because of basic physics (area vs volume). My Framework Desktop has a Noctua cooler which works really well.

[Tangent: all my life I've been downvoted into a smoking hole in the ground, particularly on reddit r/hardware, for questioning the wisdom of laptops for high performance computing, including gaming. Everyone insists they need the mobility, and then just leave it plugged in the whole time, absolutely refusing to admit it's about aesthetic preference.]

Re: AMD Strix Halo RDMA Cluster Setup Guide

#24

Post by stymaar »

rjzzleep wrote: post #21
pixelpoet wrote: post #3

I have two 128gb Strix Halos and have been extremely excited about Antirez's (Redis author) work on DS4, especially with 4bit quant using two machines: https://github.com/antirez/ds4 Right now the speed isn't good for GLM 5.2, Deepseek V4 Flash speed is okay for me (actually reading the output) and quite usable. See kyuz0's great recent video here: https://www.youtube.com/watch?v=PkKXm_mKCCM With a bit more speed and…

Last year you could buy a AI Max 395+ with 128G for 2.5k, now it's almost $4k. Or maybe you're right, I originally remembered 2k as well. I wanted to wait for the AI Max 395+ upgrade of my laptop, and now it makes no sense to upgrade.

> Last year you could buy a AI Max 395+ with 128G for 2.5k, now it's almost $4

Only if you pay the Framework premium.

https://www.bosgamepc.com/products/bosgame-m5-ai-mini-deskto...

I don't have access to the USD price, but it's 2500€ (tax included), up from 1600€ in November when I ordered mine.

Re: AMD Strix Halo RDMA Cluster Setup Guide

#25

Post by angled »

MisterKent wrote: post #16
jmyeet wrote: post #6

So this is kind of fascinating. The main hardware costs here seem to be: - 2x Framework Desktop AI Mainboards with 128GB of RAM for $3150 each - 2x 100G Ethernet controllers for ~$500 each So the Framework board has a single PCI-e 4.0 x4 slot, which amounts to 8GB/s or 64Gbps theoretical so you're not getting 100G. Also, the 100G cards all seem to be PCI-e x16 slots for obvious reasons so you need a riser or an adapt…

I ran Ms-01s with 100GBE, copper DACs in my kubernetes cluster. Killed the NVME drives in that tiny box. I'd bet the same issue doing this with FW. And I wasn't even pushing 100GBE very hard at all, it was mostly for fun. AI + 100GBE (under load) + tiny box = unreliable and eead very quickly.

How many MS-01s did you have clustered?

And could you not use something like an N5 + iSCSI for storage?

Re: AMD Strix Halo RDMA Cluster Setup Guide

#26

Post by Tepix »

rjzzleep wrote: post #21
pixelpoet wrote: post #3

I have two 128gb Strix Halos and have been extremely excited about Antirez's (Redis author) work on DS4, especially with 4bit quant using two machines: https://github.com/antirez/ds4 Right now the speed isn't good for GLM 5.2, Deepseek V4 Flash speed is okay for me (actually reading the output) and quite usable. See kyuz0's great recent video here: https://www.youtube.com/watch?v=PkKXm_mKCCM With a bit more speed and…

Last year you could buy a AI Max 395+ with 128G for 2.5k, now it's almost $4k. Or maybe you're right, I originally remembered 2k as well. I wanted to wait for the AI Max 395+ upgrade of my laptop, and now it makes no sense to upgrade.

The cheapest ones with 128GB were 1580€/$1840 as late as mid December.

Re: AMD Strix Halo RDMA Cluster Setup Guide

#27

Post by Tepix »

kamranjon wrote: post #20

Benchmarks are here: https://kyuz0.github.io/amd-strix-halo-vllm-toolboxes/ Would love to see DeepSeek V4 flash/pro and MiniMax M3 benchmarks but already these are pretty impressive, first strix Halo setup I've seen with some serious performance. EDIT: Apologies - I think I misunderstood these benchmarks - it seems this is actually very slow when compared to a M4 or M5 chip with a good amount of memory. Looking at th…

The pp speeds are really slow (50), I think there‘s room for improvement still.

Re: AMD Strix Halo RDMA Cluster Setup Guide

#28

Post by Gareth321 »

pixelpoet wrote: post #23
rjzzleep wrote: post #21

Earlier quoted context omitted.

Last year you could buy a AI Max 395+ with 128G for 2.5k, now it's almost $4k. Or maybe you're right, I originally remembered 2k as well. I wanted to wait for the AI Max 395+ upgrade of my laptop, and now it makes no sense to upgrade.

I think people buying laptops for AI use are, sorry, just plain crazy. You overpay for the screen and keyboard and battery and whatever, plus you get much worse thermal performance because of basic physics (area vs volume). My Framework Desktop has a Noctua cooler which works really well. [Tangent: all my life I've been downvoted into a smoking hole in the ground, particularly on reddit r/hardware, for questioning th…

I’m mostly with you but there are some people who like to use one machine for both laptop and AI work, and it’s much cheaper than buying two separate devices.

Re: AMD Strix Halo RDMA Cluster Setup Guide

#29

Post by Gareth321 »

pixelpoet wrote: post #3

I have two 128gb Strix Halos and have been extremely excited about Antirez's (Redis author) work on DS4, especially with 4bit quant using two machines: https://github.com/antirez/ds4 Right now the speed isn't good for GLM 5.2, Deepseek V4 Flash speed is okay for me (actually reading the output) and quite usable. See kyuz0's great recent video here: https://www.youtube.com/watch?v=PkKXm_mKCCM With a bit more speed and…

I was hoping to buy a competent local model machine later this year but given the prices I’m shelving that for now. Especially because the frontier models are very cheap relative to the cost of building my own setup. Especially because AI specialised hardware and processors are improving very fast, meaning hardware we buy now will become obsolete for this use case much faster than for traditional computer use cases.

In 1-3 years the hardware crunch will be over, local distilled models will provide Opus 4.8 like intelligence, and the hardware will exist to provide usable performance.

Re: AMD Strix Halo RDMA Cluster Setup Guide

#30

Post by erik »

gregoryl wrote: post #14
mestadler wrote: post #10

Earlier quoted context omitted.

He did cover the Tb/USB4 ;)

Indeed, here: https://github.com/kyuz0/amd-strix-halo-vllm-toolboxes/blob/...

Looks like there is currently no RDMA support for thunderbolt, so it's a much higher latency connection. Apple has RDMA over thunderbolt working, so I wonder if it's possible on Strix Halo.
Post reply on HN