Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DeepSeek V4 Flash on a Single AMD MI300X

github.com

Post reply on HN
Print view View original

81–90 of 114 posts

Re: DeepSeek V4 Flash on a Single AMD MI300X

#81

Post by Tepix »

bwfan123 wrote: post #55
WhitneyLand wrote: post #48

Another headline of “model runs on x”, which usually means “let’s list how much you give up to run on x”. Dumbed down quantization? No. Full intended inference weights preserved, so far so good. Slow performance? No again. Looks like you could get over 150 tokens/second. Give up context window size? Yes. Original model is trained for and served at 1M, this is 256k. A very practical tradeoff though. Codex is in this r…

I am curious if there has been work to remove experts from an open-weights model. The goal would be to reduce the size to be able to run on desktop GPUs without compromising quality. For a focused usecase - say coding, you dont need a model that knows world history. And, I am not talking about quantization. If it is possible to determine which experts are active for some usecases, and surgically remove the others.

Yes. It's called REAP and from what I've seen, results aren't stellar.

Re: DeepSeek V4 Flash on a Single AMD MI300X

#82

Post by sylware »

wren6991 wrote: post #42
sylware wrote: post #38

Is their hardware programming interface reasonable for implementing inference of frontier models: no quantization, several tera params? BTW, how many many params open weight frontier models have? A few teras, 100s of teras?

Kimi-K3: 2.8T Qwen3.8-Max: 2.4T DeepSeek V4 Pro: 1.6T DeepSeek V4 Flash: 284B (all are total parameter counts, not active parameters)

Rumors say chatgpt/claude/gemini/etc are in the 100s of teras. True?

Re: DeepSeek V4 Flash on a Single AMD MI300X

#83

Post by wren6991 »

sylware wrote: post #82
wren6991 wrote: post #42

Earlier quoted context omitted.

Kimi-K3: 2.8T Qwen3.8-Max: 2.4T DeepSeek V4 Pro: 1.6T DeepSeek V4 Flash: 284B (all are total parameter counts, not active parameters)

Rumors say chatgpt/claude/gemini/etc are in the 100s of teras. True?

I'll ask my uncle (he works for Nintendo) and get back to you on that one

Re: DeepSeek V4 Flash on a Single AMD MI300X

#84

Post by sylware »

wmf wrote: post #79
sylware wrote: post #38

Is their hardware programming interface reasonable for implementing inference of frontier models: no quantization, several tera params? BTW, how many many params open weight frontier models have? A few teras, 100s of teras?

Yes, ROCm can be used to run frontier models and is being used by OpenAI, Anthropic, and Meta.

I would prefer direct hardware kernel interface.

Like linux DMABUFs with userland hardware command ring buffers (I guess this hardware ring buffer instance would be specific to a VMID and a PASID).

Re: DeepSeek V4 Flash on a Single AMD MI300X

#86

Post by qwytw »

cheema33 wrote: post #78
Aurornis wrote: post #52

Earlier quoted context omitted.

> They are selling at a loss? Definitely not. Inference is not as expensive to operate as many people seem to assume. The frontier labs are probably making a lot of money from selling tokens. It’s covering all of the R&D costs like salaries, collecting training material, and running the large training operations that costs a lot of money.

> It’s covering all of the R&D costs like salaries, collecting training material, and running the large training operations that costs a lot of money. Are you claiming that the frontier labs like OpenAI and Anthropic are actually making a profit contrary to all the claims?

They claimed that OpenAI and Anthropic have positive gross margins. I don't think there are many credible claims saying that's not the case (at least for API usage)?

Re: DeepSeek V4 Flash on a Single AMD MI300X

#87

Post by qwytw »

vehemenz wrote: post #63
atwrk wrote: post #51

Earlier quoted context omitted.

The big question is whether the demand will stay if the subsidized pricing ends . That's what the bubble talk is about. Right now all the players compete for market share and don't care about the losses (hence the debt). But what happens if no one wants to lend them anymore?

Good distinction. The demand is partially driven by the low costs, which are only low because the major providers are losing money.

There is no evidence they are losing money on inference, though?

Also if they are keeping the price low because they want to gain market share and reduce the competitiveness of Chinese models they won't be able to raise prices without providers serving open models (at cost + low margin) severely undercutting them.

Re: DeepSeek V4 Flash on a Single AMD MI300X

#88

Post by wren6991 »

sylware wrote: post #85
wren6991 wrote: post #83

Earlier quoted context omitted.

I'll ask my uncle (he works for Nintendo) and get back to you on that one

My question is that wrong?

Sorry if the joke didn't land; I have heard a lot of different numbers for the size of US labs' models, but never seen any of them substantiated, so I think you're likely to just get more rumours in answer to this question.

My personal take, with no sources: 100T sounds excessively high given they need to be able to actually serve these things on commercially available hardware. I would guess they are in the same order of magnitude as the Chinese frontier models. It's possible their edge is in RL training methods, training-time compute, and access to data (e.g. from customers' CC/Codex sessions), not in model size.

Re: DeepSeek V4 Flash on a Single AMD MI300X

#89

Post by qwytw »

aurareturn wrote: post #36
ekidd wrote: post #31

Earlier quoted context omitted.

> Well, if no body can say when it will pop, then can we really say it's a bubble and it's overvalued? Well, given the literal trillions being spent, the only ways this pays off are: 1. AI replaces a non-trivial fraction of human employees. 2. Someone builds a Culture Mind, and humans become (hopefully) pampered pets of AIs we don't understand. Seems unlikely, but it would arguably count as a payoff even if it made m…

Can we see some actual numbers, projections, models instead of vibes?

That's what you should ask the companies spending massive amounts of money on AI though?

Re: DeepSeek V4 Flash on a Single AMD MI300X

#90

Post by ak_t »

thrownaway561 wrote: post #13
WASDx wrote: post #7

Earlier quoted context omitted.

At 830tok/s * 1 hour that's almost 3M tokens which is just $0.54 worth of tokens at Deepseeks current output price.

This is exactly what I came to say. The price of Flash is so cheap that trying to run it locally or with your own hardware is pointless. I was using it about a month ago to program some stuff and ran it for 4 days non-stop and it cost me about $2.

It's expensive to run it locally at full quality, but at least on my setup, its about 5 times faster than any API, and is completely private.
Post reply on HN