The $4K is awful lot of heavy API use.
AMD Ryzen AI Halo – $4k AI Dev Kit
271–274 of 274 posts
Re: AMD Ryzen AI Halo – $4k AI Dev Kit
#272Earlier quoted context omitted.
Each memory controller interface is a not-insignificant number of PCB traces. Increasing the number of memory controllers may dramatically increase the number of PCB layers (or may not, it really depends on the CPU pinout) but it definitely will increase the number of pins on the CPU socket. This is one of the main reasons (the other is the number of PCIe lanes) why high end desktop and server CPUs have like double t…
Then what's about using 4-8 cheapest motherboards with 64Gb DDR4 and a cheap CPU, and connecting them via PCIE x16 sockets? And as for DRAM channels, typical cheap motherboard has 2 channels and 4 slots, it should not be super difficult to add 2 more channels.
To get memory mapping across systems you'd want to look into infiniband (https://en.wikipedia.org/wiki/InfiniBand) but the adapters generally aren't cheap. Common to find in super computer clusters.
If you have a desktop CPU with 2 memory channels and 2 DIMMs per channel, then the 2 DIMMs on each single channel share all of the address and data lines, there's just a chip select difference to pick which DIMM you're addressing. There is some extra loading on the lines, since you have 2 DIMMs, hence the supported speeds are usually slightly lower, but you only add 1-2 more actual PCB traces to have 2 DIMMs per channel vs 1 DIMM per channel. In contrast, adding another memory controller would add upwards of almost 100 additional PCB traces from the CPU.
Re: AMD Ryzen AI Halo – $4k AI Dev Kit
#273Earlier quoted context omitted.
Using regular DeepSeek-V4-Flash, I see 2000tok/s prompt processing and from 40 to 50 tok/s generation. Performance doesn't drop much at long contexts, DSv4 is really nice for that. With DeepSeek-V4-Flash-DSpark (Deepseek's new speculative decoding scheme), which is still barely supported anywhere, we're seeing a more steady 45-55 tok/s with bursts into the 60s.
What quant size are you using? Just picked up my second spark and a cable and wanted to try this one out for fun. I generally run a bunch of models for different purposes (easy / bulk tasks) and use APIs for harder tasks.
It's the native DeepSeek V4 Flash quant which is released in MXFP4. You have enough RAM left over with 2 sparks for ~2M tokens of working KV cache.
Re: AMD Ryzen AI Halo – $4k AI Dev Kit
#274Earlier quoted context omitted.
it's worth noting that AMD's software is universally weak and not worth any degree of reliance. and it's not just ROCm, every few months they merge a serious regression into amdgpu and sometimes even backport it into stable. they are amateurs. just a few weeks ago they backported a kernel oops amdgpu null dereference into stable, it's still not fixed.
biggest mistake was buying an amd laptop. this thing had gpu crashes from day one, and it got worse as time went on. now just playing a video and closing the lid will crash the kernel 100% of the time. amd - never again
eh, i gotta system76 pangolin with all amd and it's been solid