DeepSeekMath 7B achieved 51.7% on MATH benchmark
1–10 of 41 posts
Re: DeepSeekMath 7B achieved 51.7% on MATH benchmark
#2Related paper - https://arxiv.org/pdf/2402.03300.pdf
Re: DeepSeekMath 7B achieved 51.7% on MATH benchmark
#3Supports commercial use!
Interesting what's unsupported:
- In any way that violates any applicable national or international law or regulation or infringes upon the lawful rights and interests of any third party;
- For military use in any way;
- For the purpose of exploiting, harming or attempting to exploit or harm minors in any way;
- To generate or disseminate verifiably false information and/or content with the purpose of harming others;
- To generate or disseminate inappropriate content subject to applicable regulatory requirements;
- To generate or disseminate personal identifiable information without due authorization or for unreasonable use;
- To defame, disparage or otherwise harass others;
- For fully automated decision making that adversely impacts an individual’s legal rights or otherwise creates or modifies a binding, enforceable obligation;
- For any use intended to or which has the effect of discriminating against or harming individuals or groups based on online or offline social behavior or known or predicted personal or personality characteristics;
- To exploit any of the vulnerabilities of a specific group of persons based on their age, social, physical or mental characteristics, in order to materially distort the behavior of a person pertaining to that group in a manner that causes or is likely to cause that person or another person physical or psychological harm;
- For any use intended to or which has the effect of discriminating against individuals or groups based on legally protected characteristics or categories.
Re: DeepSeekMath 7B achieved 51.7% on MATH benchmark
#4Supports commercial use! Interesting what's unsupported: - In any way that violates any applicable national or international law or regulation or infringes upon the lawful rights and interests of any third party; - For military use in any way; - For the purpose of exploiting, harming or attempting to exploit or harm minors in any way; - To generate or disseminate verifiably false information and/or content with the p…
The irony is that anyone who was going to do those things isn't going to care about a license anyway.
Re: DeepSeekMath 7B achieved 51.7% on MATH benchmark
#5Does anyone know how much spoilage are in these datasets? Common crawl has a lot of websites in it, including Reddit and Stack*. I'm certain there are lots of questions in those datasets and we want to differentiate recall from problem solving (often confused). I have a deep distrust when using large datasets like this given a common one with 60 authors assumed writing leet code style programs by hand would mean they wouldn't appear in the training data (github) and didn't even bother to check. It's really hard to sanitize datasets of this size and deduplication is a much harder task than many realize.
Re: DeepSeekMath 7B achieved 51.7% on MATH benchmark
#6I have spoken to team members, and they all say the results of this and coder are very, very much leakage (no suprisse given the result!!)
Re: DeepSeekMath 7B achieved 51.7% on MATH benchmark
#7I have spoken to team members, and they all say the results of this and coder are very, very much leakage (no suprisse given the result!!)
That's good to know, and better to admit. Earns a lot of respect, at least for me. Recall is still a pretty useful task. I just wish more would be less afraid to admit spoilage.
Re: DeepSeekMath 7B achieved 51.7% on MATH benchmark
#8Supports commercial use! Interesting what's unsupported: - In any way that violates any applicable national or international law or regulation or infringes upon the lawful rights and interests of any third party; - For military use in any way; - For the purpose of exploiting, harming or attempting to exploit or harm minors in any way; - To generate or disseminate verifiably false information and/or content with the p…
The irony is that anyone who was going to do those things isn't going to care about a license anyway.
True, but at least the author wouldn't be liable.
Re: DeepSeekMath 7B achieved 51.7% on MATH benchmark
#9Re: DeepSeekMath 7B achieved 51.7% on MATH benchmark
#10I have spoken to team members, and they all say the results of this and coder are very, very much leakage (no suprisse given the result!!)
There's a good chance that's also true for GPT-4 given how they train. Without known completely new evals, it's hard to say that any LLM benchmark results aren't leakage.