Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AMD Strix Halo RDMA Cluster Setup Guide

github.com

Post reply on HN
Print view View original

11–20 of 90 posts

Re: AMD Strix Halo RDMA Cluster Setup Guide

#11

Post by sdf4j »

gruez wrote: post #5
pixelpoet wrote: post #3

I have two 128gb Strix Halos and have been extremely excited about Antirez's (Redis author) work on DS4, especially with 4bit quant using two machines: https://github.com/antirez/ds4 Right now the speed isn't good for GLM 5.2, Deepseek V4 Flash speed is okay for me (actually reading the output) and quite usable. See kyuz0's great recent video here: https://www.youtube.com/watch?v=PkKXm_mKCCM With a bit more speed and…

>The biggest problem is all the tech companies making consumer hardware completely unaffordable, and I don't think this is accidental. Look at Micron's profits and share price lately... You realize "tech companies" isn't a monolith? Micron charging inflated prices doesn't magically benefit OpenAI. The "high prices keep out competitors" theory doesn't make much sense either. It's like saying Dennys benefits from highe…

You got it wrong. Use appliances instead of eggs. If getting an oven gets more expensive I rather keep going to Dennys.

It’s classic capex vs opex. I’d keep paying my openai subscription instead of dropping $3k to run a subpar model. If the thing costs $1k I would consider it.

Re: AMD Strix Halo RDMA Cluster Setup Guide

#12

Post by mkj »

gruez wrote: post #5
pixelpoet wrote: post #3

I have two 128gb Strix Halos and have been extremely excited about Antirez's (Redis author) work on DS4, especially with 4bit quant using two machines: https://github.com/antirez/ds4 Right now the speed isn't good for GLM 5.2, Deepseek V4 Flash speed is okay for me (actually reading the output) and quite usable. See kyuz0's great recent video here: https://www.youtube.com/watch?v=PkKXm_mKCCM With a bit more speed and…

>The biggest problem is all the tech companies making consumer hardware completely unaffordable, and I don't think this is accidental. Look at Micron's profits and share price lately... You realize "tech companies" isn't a monolith? Micron charging inflated prices doesn't magically benefit OpenAI. The "high prices keep out competitors" theory doesn't make much sense either. It's like saying Dennys benefits from highe…

openai etc are going to have a higher utilisation of the hardware so can afford it more than small companies/people. Efficient resource use matters more when they're expensive.

Re: AMD Strix Halo RDMA Cluster Setup Guide

#13

Post by [deleted] »

jmyeet wrote: post #6

So this is kind of fascinating. The main hardware costs here seem to be: - 2x Framework Desktop AI Mainboards with 128GB of RAM for $3150 each - 2x 100G Ethernet controllers for ~$500 each So the Framework board has a single PCI-e 4.0 x4 slot, which amounts to 8GB/s or 64Gbps theoretical so you're not getting 100G. Also, the 100G cards all seem to be PCI-e x16 slots for obvious reasons so you need a riser or an adapt…

[deleted]

Re: AMD Strix Halo RDMA Cluster Setup Guide

#14

Post by gregoryl »

mestadler wrote: post #10
jmyeet wrote: post #6

So this is kind of fascinating. The main hardware costs here seem to be: - 2x Framework Desktop AI Mainboards with 128GB of RAM for $3150 each - 2x 100G Ethernet controllers for ~$500 each So the Framework board has a single PCI-e 4.0 x4 slot, which amounts to 8GB/s or 64Gbps theoretical so you're not getting 100G. Also, the 100G cards all seem to be PCI-e x16 slots for obvious reasons so you need a riser or an adapt…

He did cover the Tb/USB4 ;)

Indeed, here: https://github.com/kyuz0/amd-strix-halo-vllm-toolboxes/blob/...

Re: AMD Strix Halo RDMA Cluster Setup Guide

#15

Post by layla5alive »

kcb wrote: post #7
jmyeet wrote: post #6

So this is kind of fascinating. The main hardware costs here seem to be: - 2x Framework Desktop AI Mainboards with 128GB of RAM for $3150 each - 2x 100G Ethernet controllers for ~$500 each So the Framework board has a single PCI-e 4.0 x4 slot, which amounts to 8GB/s or 64Gbps theoretical so you're not getting 100G. Also, the 100G cards all seem to be PCI-e x16 slots for obvious reasons so you need a riser or an adapt…

No reason to use fiber on short runs like that. DAC cables are cheap and better in pretty much every way over short distances. You're probably thinking of RJ-45 NICs and SFP modules which are known to run pretty hot.

+1 fiber over short distance just adds power/heat and latency compared to DAC - fiber is nice for ease of cabling and airflow, but not performance or cost when below a few meters.

Re: AMD Strix Halo RDMA Cluster Setup Guide

#16

Post by MisterKent »

jmyeet wrote: post #6

So this is kind of fascinating. The main hardware costs here seem to be: - 2x Framework Desktop AI Mainboards with 128GB of RAM for $3150 each - 2x 100G Ethernet controllers for ~$500 each So the Framework board has a single PCI-e 4.0 x4 slot, which amounts to 8GB/s or 64Gbps theoretical so you're not getting 100G. Also, the 100G cards all seem to be PCI-e x16 slots for obvious reasons so you need a riser or an adapt…

I ran Ms-01s with 100GBE, copper DACs in my kubernetes cluster. Killed the NVME drives in that tiny box. I'd bet the same issue doing this with FW. And I wasn't even pushing 100GBE very hard at all, it was mostly for fun.

AI + 100GBE (under load) + tiny box = unreliable and eead very quickly.

Re: AMD Strix Halo RDMA Cluster Setup Guide

#17

Post by sdlkj- »

This is amazing work - RDMA on these smaller unified memory boxes (somewhat) bridges the gap for consumers from the ~24GB 3090/4090/7900 card that are around to 128GB/256GB! Still not cheap, especially now, but... obtainable?

I do hope that apple opens up RDMA for their TB4 machines... ds4 using TB5 macs works great - but there are a lot of capable tb4 (M2/1) machines out there and afaik there's no hardware limitation preventing RDMA from working (at lower bandwidth, but with the latency gains!) on the older stuff.

Re: AMD Strix Halo RDMA Cluster Setup Guide

#18

Post by Tepix »

What‘s the advantage of using ConnectX-5 Ex VPI NICs instead of much cheaper ConnectX-3 VPI NICs to connect two machines directly, other than PCIe 4.0 instead of PCIe 3.0? Can they offload more tasks when doing RDMA? Solid information is hard to come by.

Re: AMD Strix Halo RDMA Cluster Setup Guide

#19

Post by pixelpoet »

rnewme wrote: post #4
pixelpoet wrote: post #3

I have two 128gb Strix Halos and have been extremely excited about Antirez's (Redis author) work on DS4, especially with 4bit quant using two machines: https://github.com/antirez/ds4 Right now the speed isn't good for GLM 5.2, Deepseek V4 Flash speed is okay for me (actually reading the output) and quite usable. See kyuz0's great recent video here: https://www.youtube.com/watch?v=PkKXm_mKCCM With a bit more speed and…

What's the advantage of ds4 over llama.cpp, esp if down the line they upstream his forked kernels?

IIRC llama.cpp doesn't implement DSv4's compressed attention mechanism, and while it does use (credited) parts of llama.cpp, it's focused on this great model for now. Much of this is covered better in the repo's readme.

Re: AMD Strix Halo RDMA Cluster Setup Guide

#20

Post by kamranjon »

Benchmarks are here: https://kyuz0.github.io/amd-strix-halo-vllm-toolboxes/

Would love to see DeepSeek V4 flash/pro and MiniMax M3 benchmarks but already these are pretty impressive, first strix Halo setup I've seen with some serious performance.

EDIT: Apologies - I think I misunderstood these benchmarks - it seems this is actually very slow when compared to a M4 or M5 chip with a good amount of memory. Looking at the creators video here: https://youtu.be/Cfl3TS7ME5s?t=734 -- it seems the performance of strix halo is much much slower than I get on my M4 MBP - which gets ~400 prefill and ~20 tok/s generation

Post reply on HN