I wonder if more companies should open source their eval model outputs alongside the eval results
We tried doing that here at Skyvern (eval.skyvern.com)
FrontierMath was funded by OpenAI
61–70 of 212 posts
Re: FrontierMath was funded by OpenAI
#62Earlier quoted context omitted.
OpenAI doesn't respect copyright so why would they let a verbal agreement get in the way of billion$
Can somehow explain to me how they can simply not respect copyright and get away with it? Also is this a uniquely open-ai problem, or also true of the other llm makers?
They're a rich company, they are immune from consequences
Re: FrontierMath was funded by OpenAI
#63Earlier quoted context omitted.
OpenAI doesn't respect copyright so why would they let a verbal agreement get in the way of billion$
Can somehow explain to me how they can simply not respect copyright and get away with it? Also is this a uniquely open-ai problem, or also true of the other llm makers?
Simply put, if the model isn’t producing an actual copy, they aren’t violating copyright (in the US) under any current definition.
As much as people bandy the term around, copyright has never applied to input, and the output of a tool is the responsibility of the end user.
If I use a copy machine to reproduce your copyrighted work, I am responsible for that infringement not Xerox.
If I coax your copyrighted work out of my phones keyboard suggestion engine letter by letter, and publish it, it’s still me infringing on your copyright, not Apple.
If I make a copy of your clip art in Illustratator, is Adobe responsible? Etc.
Even if (as I’ve seen argued ad nauseaum) a model was trained on copyrighted works on a piracy website, the copyright holder’s tort would be with the source of the infringing distribution, not the people who read the material.
Not to mention, I can walk into any public library and learn something from any book there, would I then owe the authors of the books I learned from a fee to apply that knowledge?
Re: FrontierMath was funded by OpenAI
#64Earlier quoted context omitted.
A lot of people have problem with selective enforcement of copyright law. Yes, changing them because it is captured by greedy cooperations would be something many would welcome. But currently the problem is that for normal folks doing what openai is doing they would be crushed (metaphorically) under the current copyright law. So it is not like all people who problems with openAI is big cudgel. Also openAI is making m…
Can you give an example of a copyright lawsuit lost by a 'normal person' that's doing the same thing OpenAI is?
Re: FrontierMath was funded by OpenAI
#65People on here were mocking me openly when I pointed out that you can't be sure LLMs (or any AIs) are actually smart unless you CAN PROVE that the question you're asking isn't in the training set (or adjacent like in this case). So with this in mind now, let me repeat: Unless you know that the question AND/OR answer are not in the training set or adjacent, do not claim that the AI or similar black box is smart.
I ran a test yesterday on ChatGPT and co-pilot asking first if it knew of a specific paper which it confirmed and then to derive simple results from which it was completely incapable of. I know this paper is not widely referenced (ie few known results in the public domain) but has been available for over 15 years with publicly accessible code written by humans. The training set was so sparse it had no ability to "understand" or even regurgitate past the summary text which it listed almost verbatim.
Re: FrontierMath was funded by OpenAI
#66Earlier quoted context omitted.
Can somehow explain to me how they can simply not respect copyright and get away with it? Also is this a uniquely open-ai problem, or also true of the other llm makers?
The FSF funded some white papers a while ago on CoPilot: https://www.fsf.org/news/publication-of-the-fsf-funded-white... . Take a look at the analysis by two academics versed in law at https://www.fsf.org/licensing/copilot/copyright-implications... starting with §II.B that explains why it might be legal. Bradley Kuhn also has a differing opinion in another whitepaper there ( https://www.fsf.org/licensing/copilot/if-s…
All of the most capable models I use have been clearly trained on the entirety of libgen/z-lib. You know it is the first thing they did, it is like 100TB.
Some of the models are even coy about it.
Re: FrontierMath was funded by OpenAI
#67Earlier quoted context omitted.
Can somehow explain to me how they can simply not respect copyright and get away with it? Also is this a uniquely open-ai problem, or also true of the other llm makers?
Simply put, if the model isn’t producing an actual copy, they aren’t violating copyright (in the US) under any current definition. As much as people bandy the term around, copyright has never applied to input, and the output of a tool is the responsibility of the end user. If I use a copy machine to reproduce your copyrighted work, I am responsible for that infringement not Xerox. If I coax your copyrighted work out…
> As much as people bandy the term around, copyright has never applied to input, and the output of a tool is the responsibility of the end user.
Where this breaks down though is that contributory infringement is a still a thing if you offer a service aids in copyright infringement and you don't do "enough" to stop it.
Ie, it would all be on the end user for folks that self host or rent hardware and run an LLM or Gen Art AI model themselves. But folks that offer a consumer level end to end service like ChatGPT or MidJourney could be on the hook.
Re: FrontierMath was funded by OpenAI
#68People on here were mocking me openly when I pointed out that you can't be sure LLMs (or any AIs) are actually smart unless you CAN PROVE that the question you're asking isn't in the training set (or adjacent like in this case). So with this in mind now, let me repeat: Unless you know that the question AND/OR answer are not in the training set or adjacent, do not claim that the AI or similar black box is smart.
I ran a test yesterday on ChatGPT and co-pilot asking first if it knew of a specific paper which it confirmed and then to derive simple results from which it was completely incapable of. I know this paper is not widely referenced (ie few known results in the public domain) but has been available for over 15 years with publicly accessible code written by humans. The training set was so sparse it had no ability to "und…
It is known that current models have terrible sample efficiency. I've been told that it's better than I thought it was, but it still isn't good.
Re: FrontierMath was funded by OpenAI
#69People on here were mocking me openly when I pointed out that you can't be sure LLMs (or any AIs) are actually smart unless you CAN PROVE that the question you're asking isn't in the training set (or adjacent like in this case). So with this in mind now, let me repeat: Unless you know that the question AND/OR answer are not in the training set or adjacent, do not claim that the AI or similar black box is smart.
This all smells like the OpenAI CEO's MO. Stupid drama for stupid reasons.
Re: FrontierMath was funded by OpenAI
#70[dead]