Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DeepSeek V4 Flash on a Single AMD MI300X

github.com

Post reply on HN
Print view View original

101–110 of 114 posts

Re: DeepSeek V4 Flash on a Single AMD MI300X

#101

Post by throwawayffffas »

WhitneyLand wrote: post #48

Another headline of “model runs on x”, which usually means “let’s list how much you give up to run on x”. Dumbed down quantization? No. Full intended inference weights preserved, so far so good. Slow performance? No again. Looks like you could get over 150 tokens/second. Give up context window size? Yes. Original model is trained for and served at 1M, this is 256k. A very practical tradeoff though. Codex is in this r…

To be fair, in most of the headlines, x is a Xeon from 2017 or a macbook pro or a gpu with 4 gb of vram. In this case, it's a datacenter gpu, and one that actually has more VRAM than the NVIDIA equivalent.

Re: DeepSeek V4 Flash on a Single AMD MI300X

#102

Post by sylware »

wren6991 wrote: post #88
sylware wrote: post #85

Earlier quoted context omitted.

My question is that wrong?

Sorry if the joke didn't land; I have heard a lot of different numbers for the size of US labs' models, but never seen any of them substantiated, so I think you're likely to just get more rumours in answer to this question. My personal take, with no sources: 100T sounds excessively high given they need to be able to actually serve these things on commercially available hardware. I would guess they are in the same ord…

Allright! Well, the rumors I got are around 100T for those frontier models: reading stuff here and there on internet, on discussion forums with guys pretending running AI models, etc.

This is so hard to sort the true from the false nowadays.

If LLM becomes that good at coding, I'll have to run an open weight one locally.

Re: DeepSeek V4 Flash on a Single AMD MI300X

#105

Post by sylware »

wmf wrote: post #91
sylware wrote: post #84

Earlier quoted context omitted.

I would prefer direct hardware kernel interface. Like linux DMABUFs with userland hardware command ring buffers (I guess this hardware ring buffer instance would be specific to a VMID and a PASID).

You can do that (tinygrad style) if you want.

Namely AMD would have to publish the hardware documentation (for instance command packets) of anything specific to inference (unless, this is just their GPU doing only compute).

Re: DeepSeek V4 Flash on a Single AMD MI300X

#106

Post by gpugreg »

throw10920 wrote: post #97
gpugreg wrote: post #57

Earlier quoted context omitted.

Agents usually start with ingesting the existing code base, and DeepSeek can use those code bases for pretraining. And they will have filters on top of that to throw out garbage. I am not sure how they are using the data for post-training, but there probably are ways to get signal out of it, e.g. sentiment analysis when the user begins cursing at the agent, or checking whether the user continued another session with…

> e.g. sentiment analysis when the user begins cursing at the agent, or checking whether the user continued another session with the generated code, or started a new session with the same starting point as before, i.e. they git-stashed. Thank you for elaborating, that's already useful. Anywhere I can learn more about this? I'm very interested in it!

To learn about sentiment analysis, I'd look for related datasets and then look at recent code, e.g. here: https://www.kaggle.com/datasets?search=sentiment+analysis

For more LLM-specific stuff, you can pick some agent trace dataset on https://huggingface.co/datasets?format=format%3Aagent-traces and check out what people are doing with it (usually linked on the right when you click on a dataset).

And of course https://scholar.google.com/ for research papers.

Re: DeepSeek V4 Flash on a Single AMD MI300X

#107

Post by Tepix »

K0IN wrote: post #62
Tepix wrote: post #37

Earlier quoted context omitted.

If you have 2x DGX Spark it will run quite nicely. They cost only $8000 or so and use less power so you may be able to rent them cheaper than the MI300X. I found an offer to rent two at $1.65 per hour https://spark.enverge.ai/#pricing The MI300X will vastly outperform it for only a slightly higher price.

As other calculated, even with a Mi300 you could not saturae it enough with one stream to break even with the DS API, so I think renting sparks would make it even harder cause they are considerably slower. (274gb/s vs 5.3tb/s)

Privacy is often a requirement.

Re: DeepSeek V4 Flash on a Single AMD MI300X

#108

Post by vehemenz »

qwytw wrote: post #87
vehemenz wrote: post #63

Earlier quoted context omitted.

Good distinction. The demand is partially driven by the low costs, which are only low because the major providers are losing money.

There is no evidence they are losing money on inference, though? Also if they are keeping the price low because they want to gain market share and reduce the competitiveness of Chinese models they won't be able to raise prices without providers serving open models (at cost + low margin) severely undercutting them.

There's no evidence they're making money, and we already know from the projected datacenter capacity in a few years that there will be for more supply than demand, so the major providers will have to repay that debt. Even if they are making money on inference, it's nowhere near enough to cover the bill. It's a losing proposition either way, especially with Chinese models now in play.

Re: DeepSeek V4 Flash on a Single AMD MI300X

#109

Post by [deleted] »

vehemenz wrote: post #108
qwytw wrote: post #87

Earlier quoted context omitted.

There is no evidence they are losing money on inference, though? Also if they are keeping the price low because they want to gain market share and reduce the competitiveness of Chinese models they won't be able to raise prices without providers serving open models (at cost + low margin) severely undercutting them.

There's no evidence they're making money, and we already know from the projected datacenter capacity in a few years that there will be for more supply than demand, so the major providers will have to repay that debt. Even if they are making money on inference, it's nowhere near enough to cover the bill. It's a losing proposition either way, especially with Chinese models now in play.

[deleted]

Re: DeepSeek V4 Flash on a Single AMD MI300X

#110

Post by qwytw »

vehemenz wrote: post #108
qwytw wrote: post #87

Earlier quoted context omitted.

There is no evidence they are losing money on inference, though? Also if they are keeping the price low because they want to gain market share and reduce the competitiveness of Chinese models they won't be able to raise prices without providers serving open models (at cost + low margin) severely undercutting them.

There's no evidence they're making money, and we already know from the projected datacenter capacity in a few years that there will be for more supply than demand, so the major providers will have to repay that debt. Even if they are making money on inference, it's nowhere near enough to cover the bill. It's a losing proposition either way, especially with Chinese models now in play.

There is. Specifically the pricing for open models from third party providers on OpenRouter since inference is a "commodity" at this point. Unless we think that Opus/GPT-5.6 are many times less efficient than GLM 5.2 or Kimi Openai and Anthropic are making money from inference. > it's nowhere near enough to cover the bill

Obviously it does not cover R&D, marketing and other spending but nobody has ever claimed that here.

Post reply on HN