Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DeepSeek V4 Flash on a Single AMD MI300X

github.com

Post reply on HN
Print view View original

11–20 of 114 posts

Re: DeepSeek V4 Flash on a Single AMD MI300X

#11

Post by krisknez »

WASDx wrote: post #7
zhoutong wrote: post #4

Earlier quoted context omitted.

It’s available on demand from a few cloud providers. Seems like the cheapest is AMD Developer Cloud ( https://www.amd.com/en/developer/resources/cloud-access/amd-... ) powered by Digital Ocean at $1.99/hour. Edit: Now I think about it, this might be the cheapest way to run the DeepSeek V4 Flash 0731 on a dedicated inference server at original weights. I haven’t run mixed load benchmarks but I guess it’s possible to g…

At 830tok/s * 1 hour that's almost 3M tokens which is just $0.54 worth of tokens at Deepseeks current output price.

How is that economically viable? They are selling at a loss?

Re: DeepSeek V4 Flash on a Single AMD MI300X

#12

Post by dietr1ch »

krisknez wrote: post #11
WASDx wrote: post #7

Earlier quoted context omitted.

At 830tok/s * 1 hour that's almost 3M tokens which is just $0.54 worth of tokens at Deepseeks current output price.

How is that economically viable? They are selling at a loss?

They claim their advantage is knowing how to serve their models efficiently, which is quite possible since they design for it.

Re: DeepSeek V4 Flash on a Single AMD MI300X

#13

Post by thrownaway561 »

WASDx wrote: post #7
zhoutong wrote: post #4

Earlier quoted context omitted.

It’s available on demand from a few cloud providers. Seems like the cheapest is AMD Developer Cloud ( https://www.amd.com/en/developer/resources/cloud-access/amd-... ) powered by Digital Ocean at $1.99/hour. Edit: Now I think about it, this might be the cheapest way to run the DeepSeek V4 Flash 0731 on a dedicated inference server at original weights. I haven’t run mixed load benchmarks but I guess it’s possible to g…

At 830tok/s * 1 hour that's almost 3M tokens which is just $0.54 worth of tokens at Deepseeks current output price.

This is exactly what I came to say. The price of Flash is so cheap that trying to run it locally or with your own hardware is pointless. I was using it about a month ago to program some stuff and ran it for 4 days non-stop and it cost me about $2.

Re: DeepSeek V4 Flash on a Single AMD MI300X

#15

Post by lnenad »

WASDx wrote: post #7
zhoutong wrote: post #4

Earlier quoted context omitted.

It’s available on demand from a few cloud providers. Seems like the cheapest is AMD Developer Cloud ( https://www.amd.com/en/developer/resources/cloud-access/amd-... ) powered by Digital Ocean at $1.99/hour. Edit: Now I think about it, this might be the cheapest way to run the DeepSeek V4 Flash 0731 on a dedicated inference server at original weights. I haven’t run mixed load benchmarks but I guess it’s possible to g…

At 830tok/s * 1 hour that's almost 3M tokens which is just $0.54 worth of tokens at Deepseeks current output price.

830t/s is burst aggregate. ~500 is sustained and it's for 8 concurrent users. Meaning for $1.99/hour if you serve 8 users it's 8*$0.54, not just $0.54.

You shouldn't rent one out if you're just serving it for yourself, but from a financial standpoint if you sell to users you can take a 100% margin.

Re: DeepSeek V4 Flash on a Single AMD MI300X

#16

Post by gpugreg »

krisknez wrote: post #11
WASDx wrote: post #7

Earlier quoted context omitted.

At 830tok/s * 1 hour that's almost 3M tokens which is just $0.54 worth of tokens at Deepseeks current output price.

How is that economically viable? They are selling at a loss?

Agentic workloads are somewhere around 1%/0.5%/98.5% input/output/cached tokens. Cached tokens are pretty much free for inference providers (if they implement sparse and compressed attention properly) and throughput for input tokens is much higher.

Lets assume that you've got 2 million input tokens, 1 million output tokens and 98.5 million cached tokens to process. That would cost 2 * $0.14 + 1 * $0.28 + 98.5 * $0.0028 = $0.8358 with DeepSeek API pricing.

For comparison, it would take 2M / 8000 + 1M / 800 = 1500 seconds to process this amount of tokens with the linked framework, which is about $0.83 when we assume $2/hr for one MI300X.

However, other inference providers have 10 times higher prices for cached tokens, which results in a comfortable margin.

And we should not discount that DeepSeek also gets paid in data, which is probably more valuable to them.

And I believe that this framework still has some room for optimization for generation with high batch sizes.

Re: DeepSeek V4 Flash on a Single AMD MI300X

#17

Post by Almondsetat »

WASDx wrote: post #7
zhoutong wrote: post #4

Earlier quoted context omitted.

It’s available on demand from a few cloud providers. Seems like the cheapest is AMD Developer Cloud ( https://www.amd.com/en/developer/resources/cloud-access/amd-... ) powered by Digital Ocean at $1.99/hour. Edit: Now I think about it, this might be the cheapest way to run the DeepSeek V4 Flash 0731 on a dedicated inference server at original weights. I haven’t run mixed load benchmarks but I guess it’s possible to g…

At 830tok/s * 1 hour that's almost 3M tokens which is just $0.54 worth of tokens at Deepseeks current output price.

You get privacy for 4 times the cost

Re: DeepSeek V4 Flash on a Single AMD MI300X

#19

Post by pama »

krisknez wrote: post #11
WASDx wrote: post #7

Earlier quoted context omitted.

At 830tok/s * 1 hour that's almost 3M tokens which is just $0.54 worth of tokens at Deepseeks current output price.

How is that economically viable? They are selling at a loss?

Use nvidia hardware instead and use a larger cluster serving many more users concurrently. Easily 10x–20x higher token rate per GPU with public solutions like dynamo and sglang.

Re: DeepSeek V4 Flash on a Single AMD MI300X

#20

Post by varispeed »

_joel wrote: post #18
majke wrote: post #2

I don't think you can buy a single "MI300X" unit, right? Only the box with x8 of these at a cost of ~250K EUR.

I thought it was a consumer grade GPU until I saw the 192GB of HBM and 256GB or RAM.

To be fair the development of GPUs have stalled over the years. If they kept up with the progress instead of focusing on enterprise market, likely 256GB consumer GPU would be a norm today.
Post reply on HN