Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An analysis of DeepSeek's R1-Zero and R1

arcprize.org

Post reply on HN
Print view View original

141–150 of 280 posts

Re: An analysis of DeepSeek's R1-Zero and R1

#141

Post by littlestymaar »

philipkiely wrote: post #36
littlestymaar wrote: post #30

Earlier quoted context omitted.

Earlier today I read a reddit comment[1] about a guy who tried running the quantized version from unsloth[2] on 4xH100 and the results was underwhelming (it ended up costing $137 per 1 million tokens). Any idea of what they're doing wrong? [1]: https://www.reddit.com/r/LocalLLaMA/comments/1icphqa/how_to_... [2]: https://unsloth.ai/blog/deepseekr1-dynamic

They're using Llama.cpp which is an amazing tool for local inference but doesn't match fast inference frameworks like TensorRT-LLM/SGLang for production speeds and throughputs on Hopper GPUs. The Unsloth quantizations are really cool, but if you want to experiment with the R1 models in a smaller form factor the R1 Distills like Llama 70B are great and should run a lot faster as they take advantage of existing optimiz…

> They're using Llama.cpp which is an amazing tool for local inference but doesn't match fast inference frameworks like TensorRT-LLM/SGLang for production speeds and throughputs on Hopper GPUs.

That's something I thought about, but it wouldn't explain much, as they are roughly two orders of magnitude off in terms of cost, only a small fraction of which could be explain by performance of the inference engine.

> The Unsloth quantizations are really cool, but if you want to experiment with the R1 models in a smaller form factor the R1 Distills like Llama 70B are great and should run a lot faster as they take advantage of existing optimizations around inferencing llama-architecture models.

What kind of optimization do you have in mind? Because Deepseek having only 37B active parameters, which means ~12GB at this level of quantization, means inference ought to be much faster that a dense 70B model, especially unquantized, no? The Llama 70B distill would benefit from speculative decoding though, but it shouldn't be enough to compensate. So I'm really curious about what kind of llama-specific optimizations, and how much speed up you think they'd bring.

Re: An analysis of DeepSeek's R1-Zero and R1

#142

Post by mrandish »

theendisney wrote: post #130
mrandish wrote: post #73

Earlier quoted context omitted.

I'm not in the space either but I think the answer is an emphatic yes. Three categories come to mind: 1. Online trolls and pranksters (who already taught several different AIs to be racist in a matter of hours - just for the LOLs). 2. Nation states like China who already require models to conform to state narratives. 3. More broadly, when training on "the internet" as a whole there is a huge amount of wrong, confused…

The part where people disagree seems fun. Im looking forwards to protoscience/unconventional science and perhaps even that what is worthy of the fringe or pseudoscience labels. The debunking there usually fails to adress the topic as it is incredibly hard to spend even a single day reading about something you "know" to be nonsense. Who has time for that? If you take a hundred thousand such topics the odds they should…

> The part where people disagree seems fun.

Apparently, you haven't been on that Internet thingie in the last five years or so... :-)

But I do agree with your point. What's interesting is the increasing number of people who act like there's some clearly objective and knowable truth about a much a larger percentage of topics than there actually is. Outside of mathematics, logic, physics and other hard sciences, the range of topics on which informed, reasonable people can disagree, at least on certain significant aspects, is vast.

That's why even the concept of having some army of "Fact Checkers" always struck me as bizarre and doomed at best, and at worst, a transparent attempt to censor and control public discourse. That more people didn't see even the idea of it as being obviously brittle is concerning.

Re: An analysis of DeepSeek's R1-Zero and R1

#143

Post by littlestymaar »

coder543 wrote: post #97
littlestymaar wrote: post #30

Earlier quoted context omitted.

Earlier today I read a reddit comment[1] about a guy who tried running the quantized version from unsloth[2] on 4xH100 and the results was underwhelming (it ended up costing $137 per 1 million tokens). Any idea of what they're doing wrong? [1]: https://www.reddit.com/r/LocalLLaMA/comments/1icphqa/how_to_... [2]: https://unsloth.ai/blog/deepseekr1-dynamic

I’m not an expert on at-scale inference, but they surely can’t have been running at a batch size of more than 1 if they were getting performance that bad on 4xH100… and I’m not even sure how they were getting performance that low even at batch size 1. Batching is essential to serving large token volumes at scale. As the comments on reddit said, those numbers don’t make sense.

> I’m not an expert on at-scale inference, but they surely can’t have been running at a batch size of more than 1 if they were getting performance that bad on 4xH100… and I’m not even sure how they were getting performance that low even at batch size 1. Batching is essential to serving large token volumes at scale.

That was my first though as well, but from a quick search it looks like Llama.cpp has a default batch size that's quite high (like 256 or 512 I don't remember exactly, which I find surprising for something that's mostly used by local users) so it shouldn't be the issue.

> As the comments on reddit said, those numbers don’t make sense.

Absolutely, hence my question!

Re: An analysis of DeepSeek's R1-Zero and R1

#144

Post by coder543 »

littlestymaar wrote: post #143
coder543 wrote: post #97

Earlier quoted context omitted.

I’m not an expert on at-scale inference, but they surely can’t have been running at a batch size of more than 1 if they were getting performance that bad on 4xH100… and I’m not even sure how they were getting performance that low even at batch size 1. Batching is essential to serving large token volumes at scale. As the comments on reddit said, those numbers don’t make sense.

> I’m not an expert on at-scale inference, but they surely can’t have been running at a batch size of more than 1 if they were getting performance that bad on 4xH100… and I’m not even sure how they were getting performance that low even at batch size 1. Batching is essential to serving large token volumes at scale. That was my first though as well, but from a quick search it looks like Llama.cpp has a default batch s…

Sure, but that default batch size would only matter if the person in question was actually generating and measuring parallel requests, not just measuring the straight line performance of sequential requests... and I have no confidence they were.

Re: An analysis of DeepSeek's R1-Zero and R1

#145

Post by nv35 »

philipkiely wrote: post #58
WhitneyLand wrote: post #52

Earlier quoted context omitted.

Can you share at a high level how you run this model? We know it’s 671B params with each MOE node at 37B… If the GPUs have say, 140GB for an H200, then do you just load up as many nodes as will fit into a GPU? How much do interconnects hurt performance vs being able to load the model into a single GPU?

Yeah so MoE doesn't really come into play for production serving -- once you are batching your requests you hit every expert at a large enough batch size so you have to think about running the models as a whole. There are two ways we can run it: - 8xH200 GPU == 8x141GB == 1128 GB VRAM - 16xH100 GPU == 8x80GB == 1280 GB VRAM Within a single node (up to 8 GPUs) you don't see any meaningful hit from GPU-to-GPU communica…

> Yeah so MoE doesn't really come into play for production serving -- once you are batching your requests you hit every expert at a large enough batch size

In their V3 paper DeepSeek talk about having redundant copies of some "experts" when deploying with expert parallelism in order to account for the different amounts of load they get. I imagine it only makes a difference at very high loads, but I thought it was a pretty interesting technique.

Re: An analysis of DeepSeek's R1-Zero and R1

#146

Post by Vampiero »

amluto wrote: post #116
mohsen1 wrote: post #31

Earlier quoted context omitted.

every time you respond to an AI model "no, you got that wrong, do it this way" you provide a very valuable piece of data to train on. With reasoning tokens there is just a lot more of that data to train on now

Does it? If I say "no, you hallucinated basically the entire content of the response", then maybe a newer training set derived from that could train on the specific fact that that specific hallucinated response is hallucinated. This seems to be of dubious value in a training set.

Nah I just insult it and tell it that it costs me 20 dollars a month and it's a huge disappointment

Re: An analysis of DeepSeek's R1-Zero and R1

#147

Post by theendisney »

mrandish wrote: post #142
theendisney wrote: post #130

Earlier quoted context omitted.

The part where people disagree seems fun. Im looking forwards to protoscience/unconventional science and perhaps even that what is worthy of the fringe or pseudoscience labels. The debunking there usually fails to adress the topic as it is incredibly hard to spend even a single day reading about something you "know" to be nonsense. Who has time for that? If you take a hundred thousand such topics the odds they should…

> The part where people disagree seems fun. Apparently, you haven't been on that Internet thingie in the last five years or so... :-) But I do agree with your point. What's interesting is the increasing number of people who act like there's some clearly objective and knowable truth about a much a larger percentage of topics than there actually is. Outside of mathematics, logic, physics and other hard sciences, the ra…

On Wikipedia you are suppose to quote the different perspectives. No one has ever accomplished this.

We can trust altman and elon to weed out the "fakenews". Finally we will get the answer which is the greatest linux distro.

> Outside of mathematics, logic, physics

No need to go outside. There are plenty of Grigori Perelmans with various levels of credibility.

Re: An analysis of DeepSeek's R1-Zero and R1

#148

Post by Uehreka »

whilenot-dev wrote: post #114
Uehreka wrote: post #101

Earlier quoted context omitted.

Not necessarily, not all tactics can be used symmetrically like that. Many of the sites they scrape feel the need to support search engine crawlers and RSS crawlers, but OpenAI feels no such need to grant automated anonymous access to ChatGPT users. And at the end of the daty, they can always look at the responses coming in and make decisions like “95% of users said these responses were wrong, 5% said these responses…

> As long as the vast majority of their data is good (and it will be) So expert answers are out of scope? Nice, looking forward to those quality data!

If you want to pick apart my hastily concocted examples, well, have fun I guess. My overall point is that ensuring data quality is something OpenAI is probably very good at. They likely have many clever techniques, some of which we could guess at, some of which would surprise us, all of which they’ve validated through extensive testing including with adversarial data.

If people want to keep playing pretend that their data poisoning efforts are causing real pain to OpenAI, they’re free to do so. I suppose it makes people feel good, and no one’s getting hurt here.

Re: An analysis of DeepSeek's R1-Zero and R1

#150

Post by Vampiero »

johnfn wrote: post #65
optimalsolver wrote: post #25

Earlier quoted context omitted.

But can o3 write a symphony? Seriously though, I'd like to hear suggestions on how to automatically evaluate an AI model's creativity, no humans in the loop.

Have you tried suno.ai?

Have _you_? It lost its novelty after a couple of days.
Post reply on HN