Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AMD Ryzen AI Halo – $4k AI Dev Kit

lttlabs.com

Post reply on HN
Print view View original

61–70 of 274 posts

Re: AMD Ryzen AI Halo – $4k AI Dev Kit

#61

Post by roadside_picnic »

wolttam wrote: post #43
aunty_helen wrote: post #35

256gbs memory bandwidth is about 1/4 that of a 3090. It would be a better buy with half the memory at 4x the speed.

Are you sure about that? High memory speed is great for dense models, or when serving at high concurrency. However for local single-user setups, it's often better to have access to more capable/bigger MoE models at reasonable speeds and lower concurrences, which is enabled by these platforms.

If you're using a MoE model, then why do you care about the larger RAM offered by these devices? That's the main problem with low bandwidth devices: they limit the effective ram you can make use.

I do (and have historically done) quite a work with both local LLMs and local diffusion models. I have an M3 Max MBP at 400 GB/s and also a desktop with a RTX 4090 with 1,008 GB/s

While the M3 Max MBP can serve up MoE reasonably fast (~60 token/sec)the RTX 4090 is an entirely different experience (~170 token/sec). I also do a fair bit of experimentation and am currently running a custom decoder that requires expensive look-ahead, but I'm still able to get a usable 25 token/s on the RTX.

The raison d'etre for the DGX spark is not practical home inference, but rather offering the same fundamental architecture as data center cards for a affordable CUDA prototyping. If you want to build software to run on H100s, you probably can't justify buying (and running) a single card. The DGX spark solves this by having the same fundamental setup as what those cards have.

That makes these non-NVIDIA DGX-like devices confusing to me. The entire benefit of the DGX series is the NVIDIA architecture itself.

Anyone interested in home LLMs should decide whether a Mac or a dedicated GPU is the more sensible path based on their budget and other computer use. Each has their own benefits.

Re: AMD Ryzen AI Halo – $4k AI Dev Kit

#62

Post by dabinat »

Tenoke wrote: post #51

I really want a 128gb+ machine but it's brutal to be at only 256 GB/s for $4k (especially with the drawbacks of both ARM and AMD). I fear that by the time the RTX Spark comes out it'd have to be $6k, and by the time a 128gb or more machine with 700+ GB/s comes out it'd be at $10k, way out of most consumers' hands. Edit: capitalized gb/s to GB/s.

A Mac Studio is a much better buy in terms of memory bandwidth, but impossible to buy in a 128 GB configuration. Honestly there aren’t great options right now and it’s probably better to wait for the market to be less insane.

Re: AMD Ryzen AI Halo – $4k AI Dev Kit

#63

Post by Tenoke »

dabinat wrote: post #62
Tenoke wrote: post #51

I really want a 128gb+ machine but it's brutal to be at only 256 GB/s for $4k (especially with the drawbacks of both ARM and AMD). I fear that by the time the RTX Spark comes out it'd have to be $6k, and by the time a 128gb or more machine with 700+ GB/s comes out it'd be at $10k, way out of most consumers' hands. Edit: capitalized gb/s to GB/s.

A Mac Studio is a much better buy in terms of memory bandwidth, but impossible to buy in a 128 GB configuration. Honestly there aren’t great options right now and it’s probably better to wait for the market to be less insane.

I looked for one and it's impossible to find, let alone at a reasonable price + it does suffer from being harder to train/use less common models and workflows (e.g. arbitrary comfyui ones). Spark at least doesnt have that drawback, while AMD has both drawbacks.

Waiting for the market to be less insane is somewhat akin to waiting for the s&p500 to drop a decent amount so you can buy in.

Re: AMD Ryzen AI Halo – $4k AI Dev Kit

#64

Post by bigyabai »

kcb wrote: post #46
Catloafdev wrote: post #23

These devices were great when they were cheaper than the DGX Spark. But when they cost the same price (unless the Spark has shot up too), there's no reason to buy this over a Spark. The Spark is literally a faster version of this, with better software support. Edit: And I say that as an owner of a Ryzen AI Max 395 device.

Ability to run any OS is a pretty nice benefit versus the spark.

As far as I know, you can use other OSes once the Spark's firmware is updated with LVFS.

You'll need a custom-built distro image, but that goes for like 90% of ARM hardware on Linux.

Re: AMD Ryzen AI Halo – $4k AI Dev Kit

#66

Post by wmf »

jauntywundrkind wrote: post #54
croes wrote: post #44

Earlier quoted context omitted.

Because it’s a limit of the platform https://community.frame.work/t/was-there-no-possible-way-to-...

From the replies, > A shame, really, as the Ryzen 7640U, 7840U, 7840HS, and 7940HS all support 256GB of RAM. To be fair, those platforms support dual dimms per channel, which Strix Halo would not, at least not at it's high speeds. But reciprocally Gorgon Halo 400 just launched and it supports... 192GB. And is the exact same APU. Memory chips did finally have their first big doubling per chip semi recently (available…

I assume they validated certain DRAM chips when the 395 first came out and they're just not going to validate any more. So newer DRAM is validated for the 495. We can't compare DDR5 and LPDDR5 since they are completely different; if 256 GB DDR5 is possible that doesn't mean anything.

Re: AMD Ryzen AI Halo – $4k AI Dev Kit

#67

Post by codedokode »

32 Gb DDR4 RAM module has a bandwidth of 25 Gb/s and costs $160. If you buy 8 of these, you get 256 Gb RAM with 200 Gb/s bandwidth at $1280. And if you buy 16 x 16 Gb modules (each at $60) then you can get 400 Gb/s of bandwidth for $960.

The only problem, you need 8 or 16 memory controllers. Memory controllers are not that expensive: Intel Core i3-14100F has 2 channel controller and costs $110, so we can estimate that 16-channel controller should cost not more than $880, and 8-channel controller should cost $440.

So isn't it better to make a cheap CPU with 16 DRAM controllers instead of this $4K gear having only 128 Gb? Or maybe 2 CPUs each having 8 RAM channels?

DDR5 costs 2 times more ($360 for 32 Gb) while not even having 2 times the bandwidth so it is not worth buying. It is more reasonable to make more RAM channels and stuff them with DDR4.

Re: AMD Ryzen AI Halo – $4k AI Dev Kit

#68

Post by StableAlkyne »

Tenoke wrote: post #51

I really want a 128gb+ machine but it's brutal to be at only 256 GB/s for $4k (especially with the drawbacks of both ARM and AMD). I fear that by the time the RTX Spark comes out it'd have to be $6k, and by the time a 128gb or more machine with 700+ GB/s comes out it'd be at $10k, way out of most consumers' hands. Edit: capitalized gb/s to GB/s.

[dead]

Re: AMD Ryzen AI Halo – $4k AI Dev Kit

#69

Post by wmf »

tracker1 wrote: post #59
moelf wrote: post #15

Earlier quoted context omitted.

frontier to laptop runnable open weight so far seems to be ~2 years latency, so maybe there's some hope

For that matter, just getting Chinese DRAM into the market could cool pricing down a lot to oppose the cartel.

No it won't because Chinese DRAM manufacturers have relatively low capacity and it's already being used. And in an auction, prices from different suppliers converge.
Post reply on HN