Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DeepSeek V4 Flash on a Single AMD MI300X

github.com

Post reply on HN
Print view View original

71–80 of 114 posts

Re: DeepSeek V4 Flash on a Single AMD MI300X

#71

Post by monster_truck »

bwfan123 wrote: post #55
WhitneyLand wrote: post #48

Another headline of “model runs on x”, which usually means “let’s list how much you give up to run on x”. Dumbed down quantization? No. Full intended inference weights preserved, so far so good. Slow performance? No again. Looks like you could get over 150 tokens/second. Give up context window size? Yes. Original model is trained for and served at 1M, this is 256k. A very practical tradeoff though. Codex is in this r…

I am curious if there has been work to remove experts from an open-weights model. The goal would be to reduce the size to be able to run on desktop GPUs without compromising quality. For a focused usecase - say coding, you dont need a model that knows world history. And, I am not talking about quantization. If it is possible to determine which experts are active for some usecases, and surgically remove the others.

Glazing over a lot, that's how they work already, just not in the way you think. A relatively small fraction of the model is active at any given time

Re: DeepSeek V4 Flash on a Single AMD MI300X

#72

Post by monster_truck »

cyberax wrote: post #68
craftkiller wrote: post #65

Earlier quoted context omitted.

Just want to add that while the MI350P is a PCIe card, it is designed for servers. It just has a heatsink (with no fan) which the powerful full-case fans of a rackmount server are supposed to cool. So while the MI350P is certainly more attainable for us regular folk due to its formfactor, we won't be able to just drop it into our gaming PCs like a regular graphics card. That being said, if you're dropping tens of tho…

Just put a fan on it. It's just 600W, so nothing super-special is needed. Or add a water cooler.

You're going to need at least a couple loud ass IPPC 3000s to usefully move that kind of heat if you don't want it to throttle. And then another normal sized fan for the doorway of the room it's in.

Not exactly super special, but a ~constant 600W+ of heat tends to be a learning experience. It's worse than a high end gaming rig, much closer to a literal space heater. I don't work during the summer because it sucks fighting both this and the sun with AC. I do have a fan that slots into the window and can push or pull, but kicking the waste heat outside doesn't help when its humid.

Re: DeepSeek V4 Flash on a Single AMD MI300X

#73

Post by YKutsev »

NitpickLawyer wrote: post #26
thrownaway561 wrote: post #13

Earlier quoted context omitted.

This is exactly what I came to say. The price of Flash is so cheap that trying to run it locally or with your own hardware is pointless. I was using it about a month ago to program some stuff and ran it for 4 days non-stop and it cost me about $2.

> trying to run it locally or with your own hardware is pointless. Serving local models has advantages other than price. If you work in restricted industries, or have a strong need to protect your IP, or if you just value privacy more than cost, you now have options.

[flagged]

Re: DeepSeek V4 Flash on a Single AMD MI300X

#74

Post by FeepingCreature »

jack_pp wrote: post #61
atwrk wrote: post #51

Earlier quoted context omitted.

The big question is whether the demand will stay if the subsidized pricing ends . That's what the bubble talk is about. Right now all the players compete for market share and don't care about the losses (hence the debt). But what happens if no one wants to lend them anymore?

I don't think inference is subsidized, it's the training. So what happens is, there's no new models anymore or are released slower.

API inference is probably not subsidized. Coding plans absolutely are.

Re: DeepSeek V4 Flash on a Single AMD MI300X

#75

Post by wmf »

MaKey wrote: post #56
GTP wrote: post #39

Strange that in the prior art they didn't list DwarfStar, as it is able to run the same model (probably quantized differently though) in less memory. Maybe the author isn't aware of it?

The focus of this repo is the MI300X and DwarfStar doesn't include any optimizations/fixes for it.

https://github.com/antirez/ds4/pull/484

I assume this is parallel work.

Re: DeepSeek V4 Flash on a Single AMD MI300X

#76

Post by FuriouslyAdrift »

touisteur wrote: post #41
Lwerewolf wrote: post #8

Earlier quoted context omitted.

The MI350p exists and should run a decent quant (say, the ~96GB antirez mix) well, but you can get two rtx pro 6000s for one of these, or 8x (actually more) r9700 + probably the gear to run them, etc. Otherwise, you can probably buy one of these second hand from somewhere (SXM A100s are available that way) and run it in an adapter board.

I thought MI350P wasn't available yet, curious where to source it right now.

There's at least one systems integrator selling a rack server with 2x MI350Ps.

I haven't seen the cards all by themselves yet.

Re: DeepSeek V4 Flash on a Single AMD MI300X

#77

Post by FuriouslyAdrift »

_joel wrote: post #18
majke wrote: post #2

I don't think you can buy a single "MI300X" unit, right? Only the box with x8 of these at a cost of ~250K EUR.

I thought it was a consumer grade GPU until I saw the 192GB of HBM and 256GB or RAM.

It's a chopped down MI350X (roughly half the performance)

Re: DeepSeek V4 Flash on a Single AMD MI300X

#78

Post by cheema33 »

Aurornis wrote: post #52
krisknez wrote: post #11

Earlier quoted context omitted.

How is that economically viable? They are selling at a loss?

> They are selling at a loss? Definitely not. Inference is not as expensive to operate as many people seem to assume. The frontier labs are probably making a lot of money from selling tokens. It’s covering all of the R&D costs like salaries, collecting training material, and running the large training operations that costs a lot of money.

> It’s covering all of the R&D costs like salaries, collecting training material, and running the large training operations that costs a lot of money.

Are you claiming that the frontier labs like OpenAI and Anthropic are actually making a profit contrary to all the claims?

Re: DeepSeek V4 Flash on a Single AMD MI300X

#79

Post by wmf »

sylware wrote: post #38

Is their hardware programming interface reasonable for implementing inference of frontier models: no quantization, several tera params? BTW, how many many params open weight frontier models have? A few teras, 100s of teras?

Yes, ROCm can be used to run frontier models and is being used by OpenAI, Anthropic, and Meta.

Re: DeepSeek V4 Flash on a Single AMD MI300X

#80

Post by Tepix »

WhitneyLand wrote: post #46
Tepix wrote: post #35

Unfortunately, the MI300X is an OAM module. The MI350P is the one you want: It's a PCIe card, but it has less memory: 144GB. Luckily, DeepSeek V4 Flash will run in 144GB too because it's 256 MoE exports are native MXFP4 quantized.

How do you figure that? When they just loaded the weights alone, it was taking 156GB in vLLM. After warm-up and adding a KV cache pool, it took over 200GB. And this implementation is already cutting down the 1M token context window you would normally get.

For sure if you want to properly utilize the model with several users in parallel and large context you'll want two MI350P.
Post reply on HN