Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DeepSeek V4 Flash on a Single AMD MI300X

github.com

Post reply on HN
Print view View original

61–70 of 114 posts

Re: DeepSeek V4 Flash on a Single AMD MI300X

#61

Post by jack_pp »

atwrk wrote: post #51
segmondy wrote: post #33

Earlier quoted context omitted.

no they won't , the bubble is a financial thing. the demand is real and not going away.

The big question is whether the demand will stay if the subsidized pricing ends . That's what the bubble talk is about. Right now all the players compete for market share and don't care about the losses (hence the debt). But what happens if no one wants to lend them anymore?

I don't think inference is subsidized, it's the training. So what happens is, there's no new models anymore or are released slower.

Re: DeepSeek V4 Flash on a Single AMD MI300X

#62

Post by K0IN »

Tepix wrote: post #37
zhoutong wrote: post #4

Earlier quoted context omitted.

It’s available on demand from a few cloud providers. Seems like the cheapest is AMD Developer Cloud ( https://www.amd.com/en/developer/resources/cloud-access/amd-... ) powered by Digital Ocean at $1.99/hour. Edit: Now I think about it, this might be the cheapest way to run the DeepSeek V4 Flash 0731 on a dedicated inference server at original weights. I haven’t run mixed load benchmarks but I guess it’s possible to g…

If you have 2x DGX Spark it will run quite nicely. They cost only $8000 or so and use less power so you may be able to rent them cheaper than the MI300X. I found an offer to rent two at $1.65 per hour https://spark.enverge.ai/#pricing The MI300X will vastly outperform it for only a slightly higher price.

As other calculated, even with a Mi300 you could not saturae it enough with one stream to break even with the DS API, so I think renting sparks would make it even harder cause they are considerably slower.

(274gb/s vs 5.3tb/s)

Re: DeepSeek V4 Flash on a Single AMD MI300X

#63

Post by vehemenz »

atwrk wrote: post #51
segmondy wrote: post #33

Earlier quoted context omitted.

no they won't , the bubble is a financial thing. the demand is real and not going away.

The big question is whether the demand will stay if the subsidized pricing ends . That's what the bubble talk is about. Right now all the players compete for market share and don't care about the losses (hence the debt). But what happens if no one wants to lend them anymore?

Good distinction. The demand is partially driven by the low costs, which are only low because the major providers are losing money.

Re: DeepSeek V4 Flash on a Single AMD MI300X

#64

Post by somnial »

xorfish wrote: post #28

This is still quite a bit away from the performance that deepseek gets on their H800. In their DSpark paper they report a throughput of 15k tokens/s/gpu. The MI300 should be able to compete with the H800 so there are probably still quite a few optimizations that can be made.

throughput scales superlinearly with number of GPUs when networked well and deployed with wideEP, so 1x won't compare.

also it would be interesting to figure from the DSpark paper whether their numbers are consistent with the GPUs still being H800s, since they never actually say...

Re: DeepSeek V4 Flash on a Single AMD MI300X

#65

Post by craftkiller »

Tepix wrote: post #35

Unfortunately, the MI300X is an OAM module. The MI350P is the one you want: It's a PCIe card, but it has less memory: 144GB. Luckily, DeepSeek V4 Flash will run in 144GB too because it's 256 MoE exports are native MXFP4 quantized.

Just want to add that while the MI350P is a PCIe card, it is designed for servers. It just has a heatsink (with no fan) which the powerful full-case fans of a rackmount server are supposed to cool. So while the MI350P is certainly more attainable for us regular folk due to its formfactor, we won't be able to just drop it into our gaming PCs like a regular graphics card.

That being said, if you're dropping tens of thousands of dollars on graphics cards then picking up a rackmount case to go around the card is pretty insignificant.

Re: DeepSeek V4 Flash on a Single AMD MI300X

#66

Post by smallerize »

bwfan123 wrote: post #55
WhitneyLand wrote: post #48

Another headline of “model runs on x”, which usually means “let’s list how much you give up to run on x”. Dumbed down quantization? No. Full intended inference weights preserved, so far so good. Slow performance? No again. Looks like you could get over 150 tokens/second. Give up context window size? Yes. Original model is trained for and served at 1M, this is 256k. A very practical tradeoff though. Codex is in this r…

I am curious if there has been work to remove experts from an open-weights model. The goal would be to reduce the size to be able to run on desktop GPUs without compromising quality. For a focused usecase - say coding, you dont need a model that knows world history. And, I am not talking about quantization. If it is possible to determine which experts are active for some usecases, and surgically remove the others.

Experts aren't trained on separate tasks. More recent routers are designed to spread out requests even more evenly, and they were already pretty even.

Re: DeepSeek V4 Flash on a Single AMD MI300X

#68

Post by cyberax »

craftkiller wrote: post #65
Tepix wrote: post #35

Unfortunately, the MI300X is an OAM module. The MI350P is the one you want: It's a PCIe card, but it has less memory: 144GB. Luckily, DeepSeek V4 Flash will run in 144GB too because it's 256 MoE exports are native MXFP4 quantized.

Just want to add that while the MI350P is a PCIe card, it is designed for servers. It just has a heatsink (with no fan) which the powerful full-case fans of a rackmount server are supposed to cool. So while the MI350P is certainly more attainable for us regular folk due to its formfactor, we won't be able to just drop it into our gaming PCs like a regular graphics card. That being said, if you're dropping tens of tho…

Just put a fan on it. It's just 600W, so nothing super-special is needed. Or add a water cooler.

Re: DeepSeek V4 Flash on a Single AMD MI300X

#70

Post by monster_truck »

WhitneyLand wrote: post #48

Another headline of “model runs on x”, which usually means “let’s list how much you give up to run on x”. Dumbed down quantization? No. Full intended inference weights preserved, so far so good. Slow performance? No again. Looks like you could get over 150 tokens/second. Give up context window size? Yes. Original model is trained for and served at 1M, this is 256k. A very practical tradeoff though. Codex is in this r…

In my experience the 1M context is genuinely too much. The first time I swapped from OAI to DSv4P, I checked and double checked that the harness/etc was working correctly over the course of hours and hours of work thinking that I had set something up wrong because it simply never had to compact! The drop in quality is arguably less than that of what you get from compact to compact on Codex, which is good for what it is or was.

Was also surprised to learn just how much of Codex's window was being burnt on shit I didn't want or use. Sure I can pass this and that flag to eliminate most of it, but for a $200/mo product aimed at professionals, that isn't something anyone should have to janitor (also totally ignoring the bandaid of banked resets they've slapped over their repeated mistakes).

It's wild just how far $20 will get you with Deepseek, even at their new rates. Buyers Remorse is my very least favorite feeling, I felt sick thinking about what the $1200 I had given OAI this year would have gotten me had I only tried sooner.

Post reply on HN