I have spoken to team members, and they all say the results of this and coder are very, very much leakage (no suprisse given the result!!)
yea, that's my first thought seeing the result too. we need a reputable proprietary eval.
DeepSeekMath 7B achieved 51.7% on MATH benchmark
11–20 of 41 posts
Re: DeepSeekMath 7B achieved 51.7% on MATH benchmark
#12I have spoken to team members, and they all say the results of this and coder are very, very much leakage (no suprisse given the result!!)
It's actually interesting results in a sense we see the limitation of LLM to memorize complicated information correctly. Gemini ultra also reported around 50% accuracy
Re: DeepSeekMath 7B achieved 51.7% on MATH benchmark
#13Supports commercial use! Interesting what's unsupported: - In any way that violates any applicable national or international law or regulation or infringes upon the lawful rights and interests of any third party; - For military use in any way; - For the purpose of exploiting, harming or attempting to exploit or harm minors in any way; - To generate or disseminate verifiably false information and/or content with the p…
Who would qualify as a user that seeks to violate applicable laws and yet is somehow identified as an official part of some legally recognized military? Furthermore, how would anybody know?
As a dumb Army guy if I were doing military research I would just keep it on my private military internet that does not exist for non-military users.
Re: DeepSeekMath 7B achieved 51.7% on MATH benchmark
#14Supports commercial use! Interesting what's unsupported: - In any way that violates any applicable national or international law or regulation or infringes upon the lawful rights and interests of any third party; - For military use in any way; - For the purpose of exploiting, harming or attempting to exploit or harm minors in any way; - To generate or disseminate verifiably false information and/or content with the p…
Who would qualify as a user that seeks to violate applicable laws and yet is somehow identified as an official part of some legally recognized military? Furthermore, how would anybody know? As a dumb Army guy if I were doing military research I would just keep it on my private military internet that does not exist for non-military users.
Its virtue signaling. I know its over used, but seriously, who is intentionally harming minors BUT unwilling to break a ToS contract?
Re: DeepSeekMath 7B achieved 51.7% on MATH benchmark
#15Earlier quoted context omitted.
The irony is that anyone who was going to do those things isn't going to care about a license anyway.
True, but at least the author wouldn't be liable.
You either wouldn't be liable anyway (not responsible for what people use it for) or will still be held liable (took no measures to prevent malicious use).
Re: DeepSeekMath 7B achieved 51.7% on MATH benchmark
#16Earlier quoted context omitted.
Who would qualify as a user that seeks to violate applicable laws and yet is somehow identified as an official part of some legally recognized military? Furthermore, how would anybody know? As a dumb Army guy if I were doing military research I would just keep it on my private military internet that does not exist for non-military users.
Its virtue signaling. I know its over used, but seriously, who is intentionally harming minors BUT unwilling to break a ToS contract?
The Devil?
Re: DeepSeekMath 7B achieved 51.7% on MATH benchmark
#17Earlier quoted context omitted.
Who would qualify as a user that seeks to violate applicable laws and yet is somehow identified as an official part of some legally recognized military? Furthermore, how would anybody know? As a dumb Army guy if I were doing military research I would just keep it on my private military internet that does not exist for non-military users.
Its virtue signaling. I know its over used, but seriously, who is intentionally harming minors BUT unwilling to break a ToS contract?
Meta/Instagram probably
Re: DeepSeekMath 7B achieved 51.7% on MATH benchmark
#18Earlier quoted context omitted.
Who would qualify as a user that seeks to violate applicable laws and yet is somehow identified as an official part of some legally recognized military? Furthermore, how would anybody know? As a dumb Army guy if I were doing military research I would just keep it on my private military internet that does not exist for non-military users.
Its virtue signaling. I know its over used, but seriously, who is intentionally harming minors BUT unwilling to break a ToS contract?
Processed food vendors come to mind, but I get your point.
Re: DeepSeekMath 7B achieved 51.7% on MATH benchmark
#19I have spoken to team members, and they all say the results of this and coder are very, very much leakage (no suprisse given the result!!)
Why haven't they updated the github page?
Re: DeepSeekMath 7B achieved 51.7% on MATH benchmark
#20I have spoken to team members, and they all say the results of this and coder are very, very much leakage (no suprisse given the result!!)
It's actually interesting results in a sense we see the limitation of LLM to memorize complicated information correctly. Gemini ultra also reported around 50% accuracy
I think the SOTA is GPT4+tool use? I heard near 80%