41–50 of 90 posts
Earlier quoted context omitted.
What's the advantage of ds4 over llama.cpp, esp if down the line they upstream his forked kernels?
IIRC llama.cpp doesn't implement DSv4's compressed attention mechanism, and while it does use (credited) parts of llama.cpp, it's focused on this great model for now. Much of this is covered better in the repo's readme.
Earlier quoted context omitted.
The pp speeds are really slow (50), I think there‘s room for improvement still.
Ah yea after watching one of the creators youtube videos I realize these benchmarks are combining prefill and decode which isn't super helpful - it seems this struggles with the exact same bottlenecks as all strix halo setups, memory bandwidth. It seems this is still significantly slower than equivalent memory sizing on Mac hardware.
I have two 128gb Strix Halos and have been extremely excited about Antirez's (Redis author) work on DS4, especially with 4bit quant using two machines: https://github.com/antirez/ds4 Right now the speed isn't good for GLM 5.2, Deepseek V4 Flash speed is okay for me (actually reading the output) and quite usable. See kyuz0's great recent video here: https://www.youtube.com/watch?v=PkKXm_mKCCM With a bit more speed and…
What's the advantage of ds4 over llama.cpp, esp if down the line they upstream his forked kernels?
So this is kind of fascinating. The main hardware costs here seem to be: - 2x Framework Desktop AI Mainboards with 128GB of RAM for $3150 each - 2x 100G Ethernet controllers for ~$500 each So the Framework board has a single PCI-e 4.0 x4 slot, which amounts to 8GB/s or 64Gbps theoretical so you're not getting 100G. Also, the 100G cards all seem to be PCI-e x16 slots for obvious reasons so you need a riser or an adapt…
What is simplex ethernet?
Why do you do all this? To avoid collisions and the loss of effective bandwidth from back-offs.
It only really works with 2 computers because if you add a 3rd, now you need 12 NICs instead of 4 for unidirectional point-to-point connections.
Earlier quoted context omitted.
Last year you could buy a AI Max 395+ with 128G for 2.5k, now it's almost $4k. Or maybe you're right, I originally remembered 2k as well. I wanted to wait for the AI Max 395+ upgrade of my laptop, and now it makes no sense to upgrade.
I think people buying laptops for AI use are, sorry, just plain crazy. You overpay for the screen and keyboard and battery and whatever, plus you get much worse thermal performance because of basic physics (area vs volume). My Framework Desktop has a Noctua cooler which works really well. [Tangent: all my life I've been downvoted into a smoking hole in the ground, particularly on reddit r/hardware, for questioning th…
Earlier quoted context omitted.
What is simplex ethernet?
Imagine two computers A and B. A has two NICs, A1 and A2. B has B1 and B2. So 4 NICs total. You connect directly A1 to B1 and A2 to B2 with crossover cables. You then route all the traffic from A to B over A1 to B1 and all the traffic from B to A over B2 to A2. Why do you do all this? To avoid collisions and the loss of effective bandwidth from back-offs. It only really works with 2 computers because if you add a 3rd…
What‘s the advantage of using ConnectX-5 Ex VPI NICs instead of much cheaper ConnectX-3 VPI NICs to connect two machines directly, other than PCIe 4.0 instead of PCIe 3.0? Can they offload more tasks when doing RDMA? Solid information is hard to come by.
Thats the problem with these AMD laptop class cores, they have very little IO. They have been saying they will release in a desktop form factor, but then it probably wont have such good memory bandwidth...
The Nvidia boxes have 200Gb ethernet thats much more useful for clustering.
What‘s the advantage of using ConnectX-5 Ex VPI NICs instead of much cheaper ConnectX-3 VPI NICs to connect two machines directly, other than PCIe 4.0 instead of PCIe 3.0? Can they offload more tasks when doing RDMA? Solid information is hard to come by.
Another note: In my experience, RoCE works much better on CX4+ generation. CX3 is best with Infiniband. I think some firmwares on the CX3 generation, has a messed up config for RoCE. But running Infiniband is not a complex task, is way easier than people think, like 10x easier and faster to setup than Ethernet.
Earlier quoted context omitted.
Ah yea after watching one of the creators youtube videos I realize these benchmarks are combining prefill and decode which isn't super helpful - it seems this struggles with the exact same bottlenecks as all strix halo setups, memory bandwidth. It seems this is still significantly slower than equivalent memory sizing on Mac hardware.
How are the memory bandwidths specs of Macbooks vs this?
Benchmarks are here: https://kyuz0.github.io/amd-strix-halo-vllm-toolboxes/ Would love to see DeepSeek V4 flash/pro and MiniMax M3 benchmarks but already these are pretty impressive, first strix Halo setup I've seen with some serious performance. EDIT: Apologies - I think I misunderstood these benchmarks - it seems this is actually very slow when compared to a M4 or M5 chip with a good amount of memory. Looking at th…