Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DeepSeek V4 Flash on a Single AMD MI300X

github.com

Post reply on HN
Print view View original

31–40 of 114 posts

Re: DeepSeek V4 Flash on a Single AMD MI300X

#31

Post by ekidd »

aurareturn wrote: post #24
amrit3128 wrote: post #21

Earlier quoted context omitted.

Tomorrow? Next year? In 5 years? Nobody can say. But we do know that AI is overvalued, so it WILL pop.

Well, if no body can say when it will pop, then can we really say it's a bubble and it's overvalued? I can tell you it will pop in 10 years and when it pops, it will still be 20x bigger than in 2026. Does that even make any sense? People said AI bubble will pop soon in 2024 and that it was overvalued. Turns out, many AI stocks 10x, 20x since 2024. Actual usage has gone exponential as well. Anthropic revenue went from…

> Well, if no body can say when it will pop, then can we really say it's a bubble and it's overvalued?

Well, given the literal trillions being spent, the only ways this pays off are:

1. AI replaces a non-trivial fraction of human employees.

2. Someone builds a Culture Mind, and humans become (hopefully) pampered pets of AIs we don't understand. Seems unlikely, but it would arguably count as a payoff even if it made money meaningless.

Or maybe the AIs don't want pets, and you get SkyNet. Which definitely doesn't care about paying off anyone's investments.

When you look at various news articles about investors, yeah, there are definitely a lot of rich people who think that they're going to automate all human labor or just bring about the Singularity. Possibly with them in charge of the rest of us. If you don't make these kinds of wild assumptions, then yeah, this is looking like one of the biggest bubbles ever.

Re: DeepSeek V4 Flash on a Single AMD MI300X

#32

Post by tamimio »

baalimago wrote: post #3
majke wrote: post #2

I don't think you can buy a single "MI300X" unit, right? Only the box with x8 of these at a cost of ~250K EUR.

Give it an AI-bubble pop and these will be flooding the market.

Thing is, GPUs will always be on demand, look at their history, initially for gaming, then for hash cracking, then 3D rendering, then for crypto mining, and now AI training and fine tuning. When AI bubble bursts, there will be another bubble taking over.

The only solution is more companies making high end units, only competition will make it better for consumers.

Re: DeepSeek V4 Flash on a Single AMD MI300X

#33

Post by segmondy »

baalimago wrote: post #3
majke wrote: post #2

I don't think you can buy a single "MI300X" unit, right? Only the box with x8 of these at a cost of ~250K EUR.

Give it an AI-bubble pop and these will be flooding the market.

no they won't , the bubble is a financial thing. the demand is real and not going away.

Re: DeepSeek V4 Flash on a Single AMD MI300X

#34

Post by xyzzy_plugh »

gpugreg wrote: post #16
krisknez wrote: post #11

Earlier quoted context omitted.

How is that economically viable? They are selling at a loss?

Agentic workloads are somewhere around 1%/0.5%/98.5% input/output/cached tokens. Cached tokens are pretty much free for inference providers (if they implement sparse and compressed attention properly) and throughput for input tokens is much higher. Lets assume that you've got 2 million input tokens, 1 million output tokens and 98.5 million cached tokens to process. That would cost 2 * $0.14 + 1 * $0.28 + 98.5 * $0.00…

Your math is a bit funny if you're assuming the 1/0.5/98.5 ratios: you doubled input and output tokens but not cached. If you double cached tokens to match your original ratio it works out to around $1.11, and if you 10x the cached token cost it's around $6.08.

Based on your $0.83 estimate, the margin isn't great. This is within shooting distance of "at cost" which is probably pretty close to what DeepSeek is operating with, ignoring the value of the data they're collecting of course.

> And I believe that this framework still has some room for optimization for generation with high batch sizes.

If that optimization can bring this scenario closer to $0.50 then it gets pretty compelling, otherwise I'm not confident.

Re: DeepSeek V4 Flash on a Single AMD MI300X

#35

Post by Tepix »

Unfortunately, the MI300X is an OAM module. The MI350P is the one you want: It's a PCIe card, but it has less memory: 144GB.

Luckily, DeepSeek V4 Flash will run in 144GB too because it's 256 MoE exports are native MXFP4 quantized.

Re: DeepSeek V4 Flash on a Single AMD MI300X

#36

Post by aurareturn »

ekidd wrote: post #31
aurareturn wrote: post #24

Earlier quoted context omitted.

Well, if no body can say when it will pop, then can we really say it's a bubble and it's overvalued? I can tell you it will pop in 10 years and when it pops, it will still be 20x bigger than in 2026. Does that even make any sense? People said AI bubble will pop soon in 2024 and that it was overvalued. Turns out, many AI stocks 10x, 20x since 2024. Actual usage has gone exponential as well. Anthropic revenue went from…

> Well, if no body can say when it will pop, then can we really say it's a bubble and it's overvalued? Well, given the literal trillions being spent, the only ways this pays off are: 1. AI replaces a non-trivial fraction of human employees. 2. Someone builds a Culture Mind, and humans become (hopefully) pampered pets of AIs we don't understand. Seems unlikely, but it would arguably count as a payoff even if it made m…

Can we see some actual numbers, projections, models instead of vibes?

Re: DeepSeek V4 Flash on a Single AMD MI300X

#37

Post by Tepix »

zhoutong wrote: post #4
majke wrote: post #2

I don't think you can buy a single "MI300X" unit, right? Only the box with x8 of these at a cost of ~250K EUR.

It’s available on demand from a few cloud providers. Seems like the cheapest is AMD Developer Cloud ( https://www.amd.com/en/developer/resources/cloud-access/amd-... ) powered by Digital Ocean at $1.99/hour. Edit: Now I think about it, this might be the cheapest way to run the DeepSeek V4 Flash 0731 on a dedicated inference server at original weights. I haven’t run mixed load benchmarks but I guess it’s possible to g…

If you have 2x DGX Spark it will run quite nicely. They cost only $8000 or so and use less power so you may be able to rent them cheaper than the MI300X.

I found an offer to rent two at $1.65 per hour https://spark.enverge.ai/#pricing

The MI300X will vastly outperform it for only a slightly higher price.

Re: DeepSeek V4 Flash on a Single AMD MI300X

#40

Post by ux266478 »

thrownaway561 wrote: post #13
WASDx wrote: post #7

Earlier quoted context omitted.

At 830tok/s * 1 hour that's almost 3M tokens which is just $0.54 worth of tokens at Deepseeks current output price.

This is exactly what I came to say. The price of Flash is so cheap that trying to run it locally or with your own hardware is pointless. I was using it about a month ago to program some stuff and ran it for 4 days non-stop and it cost me about $2.

If you don't do any attention steering, custom decoding or meddle with the weights maybe. Services are worthless unless all you do is write positive prompts.

As others have mentioned, there's the privacy factor as well.

Post reply on HN