Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DeepSeekMath 7B achieved 51.7% on MATH benchmark

github.com

Post reply on HN
Print view View original

31–40 of 41 posts

Re: DeepSeekMath 7B achieved 51.7% on MATH benchmark

#31

Post by godelski »

rgbrgb wrote: post #11
deepseekfake wrote: post #6

I have spoken to team members, and they all say the results of this and coder are very, very much leakage (no suprisse given the result!!)

yea, that's my first thought seeing the result too. we need a reputable proprietary eval.

> reputable proprietary eval

I think this is self-conflicting. If the evaluation is proprietary then it is most certainly not reputable. We'd want open metrics where we can analyze the limitations. Of course, we'd need open data too, but that's exceptionally rare these days. Plus, a metric isn't going to really tell us if we have have spoilage or not. You can get some evidence for spoilage through a trained model, but it is less direct, fuzzier, and more tells us about what information it was able to memorize rather than if the data was spoiled.

Re: DeepSeekMath 7B achieved 51.7% on MATH benchmark

#32

Post by godelski »

CuriouslyC wrote: post #25
pclmulqdq wrote: post #10

Earlier quoted context omitted.

There's a good chance that's also true for GPT-4 given how they train. Without known completely new evals, it's hard to say that any LLM benchmark results aren't leakage.

If you're trying to prove the model has reasoning abilities, ask it the question in a language other than English, even better give it multiple sentences in different languages and tell it to answer the question without first translating the sentences.

That's not a great metric and is going to be incredibly language dependent. For example, the European languages all have a lot of similarities and so it should be unsurprising that a model trained on English can do pretty well on French and German. But then if you are to ask it a language that is fairly disjoint (say Chinese) then you are held back by the lack of language data from that dataset (or you run into the exact same issue as previously).

It's definitely a metric worth trying, but we also must recognize the extreme limits of it too. Evaluation is quite difficult and the better our models perform the more difficult evaluation actually becomes. Anyone saying otherwise is likely trying to sell you something.

Re: DeepSeekMath 7B achieved 51.7% on MATH benchmark

#33

Post by godelski »

pclmulqdq wrote: post #10
deepseekfake wrote: post #6

I have spoken to team members, and they all say the results of this and coder are very, very much leakage (no suprisse given the result!!)

There's a good chance that's also true for GPT-4 given how they train. Without known completely new evals, it's hard to say that any LLM benchmark results aren't leakage.

This is most certainly true. If you look back to my comment and the discussion from the main thread I have two quotes from the GPT 4 technical paper

> We measure cross-contamination between our evaluation dataset and the pre-training data using substring match. Both evaluation and training data are processed by removing all spaces and symbols keeping only characters (including numbers). For each evaluation example, we randomly select three substrings of 50 characters (or use the entire example if it’s less than 50 characters). A match is identified if any of the three sampled evaluation substrings is a substring of the processed training example. This yields a list of contaminated examples. We discard these and rerun to get uncontaminated scores.

> The RLHF post-training dataset is vastly smaller than the pretraining set and unlikely to have any particular question contaminated. However we did not check explicitly.

These are not great at building confidence that OpenAI does not have spoilage. Given what we know about the dedupe process (even from early 2023) this is not enough to purge contamination. Exact string matching has been the de facto method for quite some time and for quite some time we've known that this has issues. Just that 5 years ago these issues weren't as critical as they are today because performance was much lower back then.

Re: DeepSeekMath 7B achieved 51.7% on MATH benchmark

#34

Post by ianbutler »

godelski wrote: post #30
ianbutler wrote: post #28

Earlier quoted context omitted.

As my sibling comment notes their decontamination process was outlined in the paper and you can reproduce it, though it may not be sufficient. That was my initial point, I wasn’t giving you handwavy trust claims, I was saying you’d likely find it in the paper.

Sure, but I'm your sibling is essentially in agreement with what I said about the difficulties of deduplication and this is in direct contention with your initial response to me > many serious submissions from serious groups for benchmarks like this check for contamination to specifically avoid the problem you’re suggesting Despite explicitly linking a serious work (HumanEval is still a common dataset to use[0]) and…

It isn't in direct contention, I'm not certain you're reading my original message correctly. I never made any claim about difficulty though you're welcome to show that claim to me, that is your insertion. I said that serious groups do it and make their methodology available. You're welcome to reproduce these things yourself to find fault, which you clearly do.

I will say I didn't intend to debate this point. People try to solve the problem It may not be satisfactory or meet your standards, and I can't do much for your there. Sorry and good luck.

Re: DeepSeekMath 7B achieved 51.7% on MATH benchmark

#36

Post by godelski »

ianbutler wrote: post #34
godelski wrote: post #30

Earlier quoted context omitted.

Sure, but I'm your sibling is essentially in agreement with what I said about the difficulties of deduplication and this is in direct contention with your initial response to me > many serious submissions from serious groups for benchmarks like this check for contamination to specifically avoid the problem you’re suggesting Despite explicitly linking a serious work (HumanEval is still a common dataset to use[0]) and…

It isn't in direct contention, I'm not certain you're reading my original message correctly. I never made any claim about difficulty though you're welcome to show that claim to me, that is your insertion. I said that serious groups do it and make their methodology available. You're welcome to reproduce these things yourself to find fault, which you clearly do. I will say I didn't intend to debate this point. People t…

Yeah I'm not going to claim you intended to say anything or not. Miscommunication happens. Sorry about that.

How I read your original message is that the I do not need to worry about contamination because the big labs already account for this and their methods are specified in the papers. I was dissatisfied by this response because I thought I had demonstrated working knowledge of these methods through my mention of contents of the training data as well as provided links to two relevant papers, one I was criticizing about how the decontamination method was far more naive than one might expect and the other was a work showing a significant amount of contamination.

Re: DeepSeekMath 7B achieved 51.7% on MATH benchmark

#37

Post by 3abiton »

godelski wrote: post #33
pclmulqdq wrote: post #10

Earlier quoted context omitted.

There's a good chance that's also true for GPT-4 given how they train. Without known completely new evals, it's hard to say that any LLM benchmark results aren't leakage.

This is most certainly true. If you look back to my comment and the discussion from the main thread I have two quotes from the GPT 4 technical paper > We measure cross-contamination between our evaluation dataset and the pre-training data using substring match. Both evaluation and training data are processed by removing all spaces and symbols keeping only characters (including numbers). For each evaluation example, w…

I am not that verses on this topic but am curious what would be the biggest impact of leakage/spoilage on LLM perfermance? Is it similar to overfitting?

Re: DeepSeekMath 7B achieved 51.7% on MATH benchmark

#38

Post by rgbrgb »

godelski wrote: post #31
rgbrgb wrote: post #11

Earlier quoted context omitted.

yea, that's my first thought seeing the result too. we need a reputable proprietary eval.

> reputable proprietary eval I think this is self-conflicting. If the evaluation is proprietary then it is most certainly not reputable. We'd want open metrics where we can analyze the limitations. Of course, we'd need open data too, but that's exceptionally rare these days. Plus, a metric isn't going to really tell us if we have have spoilage or not. You can get some evidence for spoilage through a trained model, bu…

i don’t buy that premise. in practice we’re seeing a lot of evidence that you can’t trust the open evals because of contamination (maybe accidental, though there’s definitely incentive to cheat and move up the leaderboards).

closed/subjective ranking and evaluation has been around since there were critics. yes it’s hard to bootstrap trust, but i can’t see a way around it because the open evals can’t really be trusted either.

Re: DeepSeekMath 7B achieved 51.7% on MATH benchmark

#39

Post by godelski »

3abiton wrote: post #37
godelski wrote: post #33

Earlier quoted context omitted.

This is most certainly true. If you look back to my comment and the discussion from the main thread I have two quotes from the GPT 4 technical paper > We measure cross-contamination between our evaluation dataset and the pre-training data using substring match. Both evaluation and training data are processed by removing all spaces and symbols keeping only characters (including numbers). For each evaluation example, w…

I am not that verses on this topic but am curious what would be the biggest impact of leakage/spoilage on LLM perfermance? Is it similar to overfitting?

Yes, it'll generally lead to overfitting. This will look a lot like memorization btw. And just an fyi, you can still not diverge on the train/test split (as is common) and still overfit. That's an obvious signal but there are many ways to have a model overfit. As far as I'm aware, all giant LLMs and image generators show signs of overfitting. But note that sometimes this can be helpful. Obviously these tools are useful still so it is more about where they break than anything.

Re: DeepSeekMath 7B achieved 51.7% on MATH benchmark

#40

Post by godelski »

rgbrgb wrote: post #38
godelski wrote: post #31

Earlier quoted context omitted.

> reputable proprietary eval I think this is self-conflicting. If the evaluation is proprietary then it is most certainly not reputable. We'd want open metrics where we can analyze the limitations. Of course, we'd need open data too, but that's exceptionally rare these days. Plus, a metric isn't going to really tell us if we have have spoilage or not. You can get some evidence for spoilage through a trained model, bu…

i don’t buy that premise. in practice we’re seeing a lot of evidence that you can’t trust the open evals because of contamination (maybe accidental, though there’s definitely incentive to cheat and move up the leaderboards). closed/subjective ranking and evaluation has been around since there were critics. yes it’s hard to bootstrap trust, but i can’t see a way around it because the open evals can’t really be trusted…

I find this argument weird. I'm not saying you can trust the open evals, I'm just saying you can know their limits. Closed evals you're a lot more blind.
Post reply on HN