Live data from Hacker News

FrontierMath was funded by OpenAI

lesswrong.com

41–50 of 212 posts

Re: FrontierMath was funded by OpenAI

#41

Earlier quoted context omitted.

OpenAI doesn't respect copyright so why would they let a verbal agreement get in the way of billion$

Can somehow explain to me how they can simply not respect copyright and get away with it? Also is this a uniquely open-ai problem, or also true of the other llm makers?

“There must be in-groups whom the law protects but does not bind, alongside out-groups whom the law binds but does not protect.”

Re: FrontierMath was funded by OpenAI

#42

Earlier quoted context omitted.

OpenAI doesn't respect copyright so why would they let a verbal agreement get in the way of billion$

Can somehow explain to me how they can simply not respect copyright and get away with it? Also is this a uniquely open-ai problem, or also true of the other llm makers?

You'll find people on this forum especially using the false analogy with a human. Like these things are like or analogous to human minds, and human minds have fair use access, so why shouldn't a these?

Magical thinking that just so happens to make lots of $$. And after all why would you want to get in the way of profit^H^H^Hgress?

Re: FrontierMath was funded by OpenAI

#44
post #36

“… we have a verbal agreement that these materials will not be used in model training” Ha ha ha. Even written agreements are routinely violated as long as the potential upside > downside, and all you have is verbal agreement? And you didn’t disclose this? At the time o3 was released I wrote “this is so impressive that it brings out the pessimist in me”[0], thinking perhaps they were routing API calls to human workers…

This has me curious about ARC-AGI. Would it have been possible for OpenAI to have gamed ARC-AGI by seeing the first few examples and then quickly mechanical turking a training set, fine tuning their model, then proceeding with the rest of the evaluation? Are there other tricks they could have pulled? It feels like unless a model is being deployed to an impartial evaluator's completely air gapped machine, there's a to…

In their benchmark, they have a tag "tuned" attached to their o3 result. I guess we need they to inform us of the exact meaning of it to gauge.

Re: FrontierMath was funded by OpenAI

#45

Earlier quoted context omitted.

OpenAI doesn't respect copyright so why would they let a verbal agreement get in the way of billion$

Can somehow explain to me how they can simply not respect copyright and get away with it? Also is this a uniquely open-ai problem, or also true of the other llm makers?

The FSF funded some white papers a while ago on CoPilot: https://www.fsf.org/news/publication-of-the-fsf-funded-white.... Take a look at the analysis by two academics versed in law at https://www.fsf.org/licensing/copilot/copyright-implications... starting with §II.B that explains why it might be legal.

Bradley Kuhn also has a differing opinion in another whitepaper there (https://www.fsf.org/licensing/copilot/if-software-is-my-copi...) but then again he studied CS, not law. Nor has the FSF attempted AFAIK to file any suits even though they likely would have if it were an open and shut case.

Re: FrontierMath was funded by OpenAI

#46
People on here were mocking me openly when I pointed out that you can't be sure LLMs (or any AIs) are actually smart unless you CAN PROVE that the question you're asking isn't in the training set (or adjacent like in this case).

So with this in mind now, let me repeat: Unless you know that the question AND/OR answer are not in the training set or adjacent, do not claim that the AI or similar black box is smart.

Re: FrontierMath was funded by OpenAI

#47

Earlier quoted context omitted.

OpenAI doesn't respect copyright so why would they let a verbal agreement get in the way of billion$

Can somehow explain to me how they can simply not respect copyright and get away with it? Also is this a uniquely open-ai problem, or also true of the other llm makers?

It's because the copyright is fake and the only thing supporting it were million dollar business. It naturally crumbles while facing billion dollar business.

Re: FrontierMath was funded by OpenAI

#48
A lot of the comments express some type of deliberate cheating the benchmark. However, even without intentionally trying to game it, if anybody can repeatedly take the same test, then they'll be nudged to overfit/p-hack.

For instance, suppose they conduct an experiment and find that changing some hyper-parameter yields a 2% boost. That could just be noise, it could be a genuine small improvement, or it may be a mix of a genuine boost along with some fortunate noise. An effect may be small enough that researchers would need to rely on their gut to interpret it. Researchers may jump on noise while believing they have discovered true optimizations. Enough of these types of nudges, and some serious benchmark gains can materialize.

(Hopefully my comment isn't entirely misguided, I don't know how they actually do testing or how often they probe their test set)

Re: FrontierMath was funded by OpenAI

#49

There's something gross about OpenAI constantly misleading the public. This maneuver by their CEO will destroy FrontierMath and Epoch AI's reputation

Reminds me of the following proverb:

"The integrity of the upright guides them, but the unfaithful are destroyed by their duplicity."

(Proverbs 11:3)

Re: FrontierMath was funded by OpenAI

#50

Earlier quoted context omitted.

OpenAI doesn't respect copyright so why would they let a verbal agreement get in the way of billion$

Can somehow explain to me how they can simply not respect copyright and get away with it? Also is this a uniquely open-ai problem, or also true of the other llm makers?

> Can somehow explain to me how they can simply not respect copyright and get away with it? Also is this a uniquely open-ai problem, or also true of the other llm makers?

"Move fast and break things."[0]

Another way to phrase this is:

  Move fast enough while breaking things and regulations
  can never catch up.
0 - https://quotes.guide/mark-zuckerberg/quote/move-fast-and-bre...
Post reply on HN