Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DeepSeek V4 Flash on a Single AMD MI300X

github.com

Post reply on HN
Print view View original

41–50 of 114 posts

Re: DeepSeek V4 Flash on a Single AMD MI300X

#41

Post by touisteur »

Lwerewolf wrote: post #8
majke wrote: post #2

I don't think you can buy a single "MI300X" unit, right? Only the box with x8 of these at a cost of ~250K EUR.

The MI350p exists and should run a decent quant (say, the ~96GB antirez mix) well, but you can get two rtx pro 6000s for one of these, or 8x (actually more) r9700 + probably the gear to run them, etc. Otherwise, you can probably buy one of these second hand from somewhere (SXM A100s are available that way) and run it in an adapter board.

I thought MI350P wasn't available yet, curious where to source it right now.

Re: DeepSeek V4 Flash on a Single AMD MI300X

#42

Post by wren6991 »

sylware wrote: post #38

Is their hardware programming interface reasonable for implementing inference of frontier models: no quantization, several tera params? BTW, how many many params open weight frontier models have? A few teras, 100s of teras?

Kimi-K3: 2.8T

Qwen3.8-Max: 2.4T

DeepSeek V4 Pro: 1.6T

DeepSeek V4 Flash: 284B

(all are total parameter counts, not active parameters)

Re: DeepSeek V4 Flash on a Single AMD MI300X

#43

Post by simlevesque »

krisknez wrote: post #11
WASDx wrote: post #7

Earlier quoted context omitted.

At 830tok/s * 1 hour that's almost 3M tokens which is just $0.54 worth of tokens at Deepseeks current output price.

How is that economically viable? They are selling at a loss?

They get all our invaluable data which they'll use to train the next model, to get more data, to train the model after.

Re: DeepSeek V4 Flash on a Single AMD MI300X

#44

Post by drob518 »

krisknez wrote: post #11
WASDx wrote: post #7

Earlier quoted context omitted.

At 830tok/s * 1 hour that's almost 3M tokens which is just $0.54 worth of tokens at Deepseeks current output price.

How is that economically viable? They are selling at a loss?

I think Deepseek is selling roughly at cost (perhaps a slight premium). They don’t guarantee that they don’t train on the submitted prompts, so I suspect they are mining the data. Mining for what? Well, who knows. Best case, mining to make Deepseek better. That said, I use Deepseek all the time. It has done a whole lot of ‘ls’ commands on my system, though.

Re: DeepSeek V4 Flash on a Single AMD MI300X

#45

Post by throwawayffffas »

majke wrote: post #2

I don't think you can buy a single "MI300X" unit, right? Only the box with x8 of these at a cost of ~250K EUR.

You can get one on ebay for like 20k, but it comes without the backplane and i dont think there is a pcie card adaptor from china like the ones for h200.

Re: DeepSeek V4 Flash on a Single AMD MI300X

#46

Post by WhitneyLand »

Tepix wrote: post #35

Unfortunately, the MI300X is an OAM module. The MI350P is the one you want: It's a PCIe card, but it has less memory: 144GB. Luckily, DeepSeek V4 Flash will run in 144GB too because it's 256 MoE exports are native MXFP4 quantized.

How do you figure that?

When they just loaded the weights alone, it was taking 156GB in vLLM. After warm-up and adding a KV cache pool, it took over 200GB.

And this implementation is already cutting down the 1M token context window you would normally get.

Re: DeepSeek V4 Flash on a Single AMD MI300X

#47

Post by dghlsakjg »

aurareturn wrote: post #10
baalimago wrote: post #3

Earlier quoted context omitted.

Give it an AI-bubble pop and these will be flooding the market.

When is it popping? Is the AI bubble in the room with us now?

Nvidia has ever so slightly underperformed the SP500 YTD (at the exact time this comment is being typed), so its basically the apocalypse already.

Re: DeepSeek V4 Flash on a Single AMD MI300X

#48

Post by WhitneyLand »

Another headline of “model runs on x”, which usually means “let’s list how much you give up to run on x”.

Dumbed down quantization?

No. Full intended inference weights preserved, so far so good.

Slow performance?

No again. Looks like you could get over 150 tokens/second.

Give up context window size?

Yes. Original model is trained for and served at 1M, this is 256k. A very practical tradeoff though. Codex is in this range, and quality does start to drop off toward the full size.

Re: DeepSeek V4 Flash on a Single AMD MI300X

#49

Post by throw10920 »

gpugreg wrote: post #16
krisknez wrote: post #11

Earlier quoted context omitted.

How is that economically viable? They are selling at a loss?

Agentic workloads are somewhere around 1%/0.5%/98.5% input/output/cached tokens. Cached tokens are pretty much free for inference providers (if they implement sparse and compressed attention properly) and throughput for input tokens is much higher. Lets assume that you've got 2 million input tokens, 1 million output tokens and 98.5 million cached tokens to process. That would cost 2 * $0.14 + 1 * $0.28 + 98.5 * $0.00…

> should not discount that DeepSeek also gets paid in data, which is probably more valuable to them

That's agentic feedback loops for training, right? Any more detail on this, such as how they actually tell whether that data is good or not? That seems like a very hard problem, and like the value of that data is low compared to just building their own, controlled RL gyms.

Re: DeepSeek V4 Flash on a Single AMD MI300X

#50

Post by fergusfinn »

nice! i think the higher HBM on Mi300x is really useful for this kind of thing

we did some work on this for 2xMi300x (kindly referenced in the readme) https://blog.doubleword.ai/deepseek-v4-flash-mi300x. https://hotaisle.xyz/quick-start hotaisle is great for getting Mi300x to experiment with

Post reply on HN