Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AMD Ryzen AI Halo – $4k AI Dev Kit

lttlabs.com

Post reply on HN
Print view View original

261–270 of 274 posts

Re: AMD Ryzen AI Halo – $4k AI Dev Kit

#261

Post by SwellJoe »

cmrdporcupine wrote: post #257
verdverm wrote: post #253

Earlier quoted context omitted.

Nvidia didn't make a great out-of-box experience for sure, one of the first things I do on my Spark is set init 3 and disable bluetooth. Docker was working on one, not the second. Many typical local tools used in Ai were not installed (ollama,vllm,llama.cpp... all MIA)

I mean, I never expected to have pre-packaged versions of any of those things that wouldn't just fall out of sync with upstream immediately anyways? I basically treated it as a stock Ubuntu machine but on Aarch64, and ignored the stuff they installed. And then went looking for docker images for vllm that made sense and were up to date and hopefully tuned for NVFP4 on the hardware and... ... that was more the disappoi…

Yeah, the stuff they preinstall is crazy. It's a ~16GB OS image, which is a huge pain in the ass (in addition to all the other things that are a pain in the ass about Nvidia's OS situation); I can't ship their 16GB pile of crap plus our custom 3GB of stuff on my robots, there's no reasonable way to do OTA upgrades with a pile of shit like that. It's just an unpleasant experience all around. What I always want is a nice clean basic install, and then my custom packages of everything I need goes in. Especially for embedded cases, like the Jetson line is allegedly intended for, their whole approach is just stupid, and they don't provide any documentation for how to not use their stupid thing.

Still, I'd rather have the blazing prefill speed of the Nvidia over my Strix Halo, but I wasn't willing to spend nearly twice as much for it (when I bought, the Strix Halo was $2100 and the GX10 was going for $3700, I think). Now that the difference between the two is much smaller, there's no reason to get the AMD.

Re: AMD Ryzen AI Halo – $4k AI Dev Kit

#262

Post by nightski »

verdverm wrote: post #256
nightski wrote: post #108

Earlier quoted context omitted.

They had it on sale last week for $3999, it will likely happen again. Also if you are willing to buy ASUS/Acer/MSI you can get them cheaper, in the same range as well. Those units are identical (mainboard/ram/chipset/connectivity), they only tend to differ in SSD being offered.

The more beneficial difference between DGX and OEM Spark is in cooling. DGX has had cooling issue based on user reports.

Only Asus from what I have seen. Acer/Dell units seem to run cooler than the Nvidia unit, by a noticeable margin. That said I bought the OEM units for the Gen 5 4TB SSD and the fact that it was on sale.

Re: AMD Ryzen AI Halo – $4k AI Dev Kit

#263

Post by verdverm »

nightski wrote: post #262
verdverm wrote: post #256

Earlier quoted context omitted.

The more beneficial difference between DGX and OEM Spark is in cooling. DGX has had cooling issue based on user reports.

Only Asus from what I have seen. Acer/Dell units seem to run cooler than the Nvidia unit, by a noticeable margin. That said I bought the OEM units for the Gen 5 4TB SSD and the fact that it was on sale.

I have ASUS and no thermal issues, not sure if that is what you meant, ambiguity

Re: AMD Ryzen AI Halo – $4k AI Dev Kit

#265

Post by SR2Z »

hparadiz wrote: post #194
SR2Z wrote: post #184

Earlier quoted context omitted.

I also think this is forward looking and helps reanchor expectations for RAM and memory bandwidth, which historically have been areas where companies do value engineering. The memory shortages won't last forever; when companies start adding capacity I wouldn't be surprised to see massive sticks of RAM being sold for consumers.

Before AI I kinda thought more ram and faster ram was kind of useless because I had written off increased speeds as sort of a luxury whose only application is bloated unoptimized software. Boy was I wrong about that. But then why stop there?

I mean, there's a difference between "my IDE opens 0.1s faster" and "my desktop can now write my code for me" and only one of those justifies increasing RAM production.

Re: AMD Ryzen AI Halo – $4k AI Dev Kit

#266

Post by cmrdporcupine »

SwellJoe wrote: post #261
cmrdporcupine wrote: post #257

Earlier quoted context omitted.

I mean, I never expected to have pre-packaged versions of any of those things that wouldn't just fall out of sync with upstream immediately anyways? I basically treated it as a stock Ubuntu machine but on Aarch64, and ignored the stuff they installed. And then went looking for docker images for vllm that made sense and were up to date and hopefully tuned for NVFP4 on the hardware and... ... that was more the disappoi…

Yeah, the stuff they preinstall is crazy. It's a ~16GB OS image, which is a huge pain in the ass (in addition to all the other things that are a pain in the ass about Nvidia's OS situation); I can't ship their 16GB pile of crap plus our custom 3GB of stuff on my robots, there's no reasonable way to do OTA upgrades with a pile of shit like that. It's just an unpleasant experience all around. What I always want is a ni…

The $2100 vs $3700 price difference really came down to the ConnectX hardware in the thing, I guess. But the price gap narrowed over time, now they both suck.

I am not sure why NVIDIA couldn't have released a cheaper version of the thing with just standard Ethernet on it and leave it at that. Hardly anybody can afford two or more of them to cluster via ConnectX anyways.

Re: AMD Ryzen AI Halo – $4k AI Dev Kit

#267

Post by cmrdporcupine »

verdverm wrote: post #263
nightski wrote: post #262

Earlier quoted context omitted.

Only Asus from what I have seen. Acer/Dell units seem to run cooler than the Nvidia unit, by a noticeable margin. That said I bought the OEM units for the Gen 5 4TB SSD and the fact that it was on sale.

I have ASUS and no thermal issues, not sure if that is what you meant, ambiguity

Same here, no issue at all with my ASUS Spark-thing. And back when I was selecting this, I watched a whole video that broke them all down by thermal mgmt and there wasn't anything wrong with the ASUS.

I do wish didn't buy the one that only had the 1TB nVME in it though. It's one of those expensive half-sized dealies, which makes upgrading expensive. Esp now.

Re: AMD Ryzen AI Halo – $4k AI Dev Kit

#268

Post by verdverm »

cmrdporcupine wrote: post #267
verdverm wrote: post #263

Earlier quoted context omitted.

I have ASUS and no thermal issues, not sure if that is what you meant, ambiguity

Same here, no issue at all with my ASUS Spark-thing. And back when I was selecting this, I watched a whole video that broke them all down by thermal mgmt and there wasn't anything wrong with the ASUS. I do wish didn't buy the one that only had the 1TB nVME in it though. It's one of those expensive half-sized dealies, which makes upgrading expensive. Esp now.

I bought the 1T and then did it again. I don't see the need for more sad on a vllm server, I have dozens of models and quant in that space. If I need more for data, I intend to plug in an old 4T HDD

Re: AMD Ryzen AI Halo – $4k AI Dev Kit

#269

Post by lhl »

icedchai wrote: post #217
teravor wrote: post #213

Earlier quoted context omitted.

it's worth noting that AMD's software is universally weak and not worth any degree of reliance. and it's not just ROCm, every few months they merge a serious regression into amdgpu and sometimes even backport it into stable. they are amateurs. just a few weeks ago they backported a kernel oops amdgpu null dereference into stable, it's still not fixed.

The "ROCm" situation with Strix Halo was pretty bad for a while. I think it finally stabilized late last year. You needed the right combo of ROCm, Linux kernel, and kernel firmware for it to work reliably. Whenever I rebuild llama.cpp, I wind up using the Vulkan build anyway.

I started a project w/ the assumption that fully tuned for HIP kernels could get closer to roofline and beat llama.cpp's performance (and vs their ROCm engine it does!) but after thousands of iterations/experiments my suspicion is that RADV/ACO is actually just much better than LLVM/HIP when it comes to a lower-level scheduling etc for RDNA3.

I haven't finished comprehensive tests but I found:

- Vulkan is up to 2.25X faster for most coalesced, strided and interleave variants for memory-side scheduling/access shapes

- 3.3X faster on specific dot-path sweeps, including for scalar-dequant

- For matched LDS, Vulkan can be 8-14X+ faster (!!!) than matched HIP LDS

HIP doesn't always win against RADV/ACO, but on dispatch/runtime, it does appear to be quite a bit faster than HIP/LLVM on gfx1151 (Strix Halo). I'll be publishing sharing full data once I also run vs gfx1100...

Re: AMD Ryzen AI Halo – $4k AI Dev Kit

#270

Post by dabinat »

Tenoke wrote: post #63
dabinat wrote: post #62

Earlier quoted context omitted.

A Mac Studio is a much better buy in terms of memory bandwidth, but impossible to buy in a 128 GB configuration. Honestly there aren’t great options right now and it’s probably better to wait for the market to be less insane.

I looked for one and it's impossible to find, let alone at a reasonable price + it does suffer from being harder to train/use less common models and workflows (e.g. arbitrary comfyui ones). Spark at least doesnt have that drawback, while AMD has both drawbacks. Waiting for the market to be less insane is somewhat akin to waiting for the s&p500 to drop a decent amount so you can buy in.

> Waiting for the market to be less insane is somewhat akin to waiting for the s&p500 to drop a decent amount so you can buy in.

I said less insane, not sane. If prices go down 20% but are still up 150% over a few years ago, that would still be an improvement from now.

Or it could work the other way: if new hardware comes out in 2027 such that the tokens/$ ratio works out better, that would also be less insane.

Post reply on HN