Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI researcher announced GPT-5 math breakthrough that never happened

the-decoder.com

Post reply on HN
Print view View original

151–160 of 258 posts

Re: OpenAI researcher announced GPT-5 math breakthrough that never happened

#151

Post by scruple »

scosman wrote: post #136
Timsky wrote: post #86

> GPT-5 is proving useful as a literature review assistant No, it does not. It only produces a highly convincing counterfeit. I am honestly happy for people who are satisfied with its output: life is way easier for them than for me. Obviously, the machine discriminates me personally. When I spend hours in the library looking for some engineering-related math made in the 70s-80s, as a last resort measure, I can try to…

Saying it isn't useful is a bit of an overstatement. It can search, churn through 500k words in a few minutes, and come back with summaries, answers, and sources for each point. Should you blindly trust the summary? No. Should you verify key claims by clicking through to the source? Yes. Is it still incredibly useful as a search tool and productivity booster? Absolutely.

I gave it a PDF recently and asked it to help me generate some tables based on the information there in. I thought I'd be saving myself time. I spent easily twice as long as I would have if it I had done it myself. It kept making trivial mistakes, misunderstanding what was in the PDF, hallucinating, etc.

Re: OpenAI researcher announced GPT-5 math breakthrough that never happened

#152

Post by strangescript »

lukev wrote: post #47
strangescript wrote: post #18

This entire thing has been pretty disingenuous on both sides of the fence. All the anti-AI (or anti OpenAI) people are doing victory laps, but what GPT-5 Pro did is still very valuable. 1) What good is your open problem set if really its a trivial "google search" away from being solved. Why are they not catching any blame here? 2) These answers still weren't perfectly laid out for the most part. GPT-5 was still doing…

Framing this question as "AI good" OR "AI bad" is culture-war thinking. The real problem here is that there's clearly a strong incentive for the big labs to deceive the public (and/or themselves) about the actual scientific and technical capabilities of LLMs. As Karpathy pointed out on the recent Dwarkesh podcast, LLMs are quite terrible at novel problems, but this has become sort of an "Emperor's new clothes" situat…

Except they weren't intentionally trying to deceive anyone. They made the faulty assumption that these problems were non-trivial to solve and didn't think it was simply GPT-5 aggregating solutions in the wild.

Re: OpenAI researcher announced GPT-5 math breakthrough that never happened

#153

Post by lukev »

andai wrote: post #145
Timsky wrote: post #86

> GPT-5 is proving useful as a literature review assistant No, it does not. It only produces a highly convincing counterfeit. I am honestly happy for people who are satisfied with its output: life is way easier for them than for me. Obviously, the machine discriminates me personally. When I spend hours in the library looking for some engineering-related math made in the 70s-80s, as a last resort measure, I can try to…

There's this principle, I forget the name, but how everyone when reading the newspaper, when they read on a subject they're familiar with, will instantly spot all the holes, all the errors. And they will ask themselves, how was this even published in the first place? But then they flip to the next page and they read a story on a subject they're not an expert on and they just accept all of it without question. I think…

The Gell-Mann Amnesia effect. And you're absolutely right, it's extremely pronounced in LLM users.

Re: OpenAI researcher announced GPT-5 math breakthrough that never happened

#154

Post by strangescript »

puttycat wrote: post #25
strangescript wrote: post #18

This entire thing has been pretty disingenuous on both sides of the fence. All the anti-AI (or anti OpenAI) people are doing victory laps, but what GPT-5 Pro did is still very valuable. 1) What good is your open problem set if really its a trivial "google search" away from being solved. Why are they not catching any blame here? 2) These answers still weren't perfectly laid out for the most part. GPT-5 was still doing…

This is a strawman argument. No anti-AI sentiment was involved here. Simply the fact that finding and matching text on the Internet is several orders of magnitude easier than finding novel solutions to hard math problems.

You didn't read the X replies if you believe that

Re: OpenAI researcher announced GPT-5 math breakthrough that never happened

#155

Post by happy_dog1 »

Timsky wrote: post #86

> GPT-5 is proving useful as a literature review assistant No, it does not. It only produces a highly convincing counterfeit. I am honestly happy for people who are satisfied with its output: life is way easier for them than for me. Obviously, the machine discriminates me personally. When I spend hours in the library looking for some engineering-related math made in the 70s-80s, as a last resort measure, I can try to…

I wonder whether for a lot of the search & literature review-type use-cases where people are trying to use GPT-5 and similar we'd honestly be much better off with a really powerful semantic search engine? Any time you ask a chatbot to summarize the literature for you or answer your question, there's a risk it will hallucinate and give you an unreliable answer. Using LLM-generated embeddings for documents to retrieve the nearest match, by contrast, doesn't run any risk of hallucination and might be a powerful way to retrieve things that Google / Bing etc. wouldn't be able to find using their current algorithms.

I don't know if something like this already exists and I'm just not aware of it to be fair.

Re: OpenAI researcher announced GPT-5 math breakthrough that never happened

#156

Post by lukev »

strangescript wrote: post #152
lukev wrote: post #47

Earlier quoted context omitted.

Framing this question as "AI good" OR "AI bad" is culture-war thinking. The real problem here is that there's clearly a strong incentive for the big labs to deceive the public (and/or themselves) about the actual scientific and technical capabilities of LLMs. As Karpathy pointed out on the recent Dwarkesh podcast, LLMs are quite terrible at novel problems, but this has become sort of an "Emperor's new clothes" situat…

Except they weren't intentionally trying to deceive anyone. They made the faulty assumption that these problems were non-trivial to solve and didn't think it was simply GPT-5 aggregating solutions in the wild.

Knowing what I know about LLMs, from their internal architecture and from extensive experience working with them daily, I would find this kind of result highly surprising and in a clear violation of my mental model of how these things work. And I'm very far from an expert.

If a purported expert in the field can is willing to credulously publish this kind of result, it's not unreasonable to assume that either they're acting in bad faith, or (at best) are high on their own supply regarding what these things can actually do.

Re: OpenAI researcher announced GPT-5 math breakthrough that never happened

#157

Post by andrewflnr »

georgemcbay wrote: post #143
andrewflnr wrote: post #130

Earlier quoted context omitted.

So, the exact stuff Google used to be good at.

Pretty much, though Google got bad at these things well before LLMs really came on to the scene, and we can all debate which project manager was responsible and the month and year things took a downward turn, but the IMO obvious catalyst was that "Barely Good Enough" search creates more ad impressions, especially when virtually all of the bad results you are serving are links to sites that also serve Google managed a…

Oh, sure, Google was starting to take a dive almost a decade before LLMs came on the scene.

Re: OpenAI researcher announced GPT-5 math breakthrough that never happened

#158

Post by NitpickLawyer »

badsectoracula wrote: post #150
NitpickLawyer wrote: post #128

Earlier quoted context omitted.

> with pure LLMs and without a planning framwork. Sure, but isn't that moving the goalposts? Why shouldn't we use LLMs + tools if it works? If anything it shows that the early detractors weren't even considering this could work. Yann in particular was skeptical that long-context things can happen in LLMs at all. We now have "agents" that can work a problem for hours, with self context trimming, planning to md files,…

> Sure, but isn't that moving the goalposts? Why shouldn't we use LLMs + tools if it works? Personally i do not see it like that at all as one is referring to LLMs specifically while the other is referring to LLMs plus a bunch of other stuff around them. It is like person A claiming that GIF files can be used to play Doom deathmatches, person B responding that, no, a GIF file cannot start a Doom deathmatch, it is fun…

At the end of the day LLM + tools is asking the LLM to create a story with very specific points where "tool calls" are parts of the story, and "tool results" are like characters that provide context. The fact that they can output stories like that, with enough accuracy to make it worthwhile is, IMO, proof that they can "do" whatever we say they can do. They can "do" math by creating a story where a character takes NL and invokes a calculator, and another character provides the actual computation. Cool. It's still the LLM driving the interaction. It's still the LLM creating the story.

Re: OpenAI researcher announced GPT-5 math breakthrough that never happened

#159

Post by Timsky »

scosman wrote: post #136
Timsky wrote: post #86

> GPT-5 is proving useful as a literature review assistant No, it does not. It only produces a highly convincing counterfeit. I am honestly happy for people who are satisfied with its output: life is way easier for them than for me. Obviously, the machine discriminates me personally. When I spend hours in the library looking for some engineering-related math made in the 70s-80s, as a last resort measure, I can try to…

Saying it isn't useful is a bit of an overstatement. It can search, churn through 500k words in a few minutes, and come back with summaries, answers, and sources for each point. Should you blindly trust the summary? No. Should you verify key claims by clicking through to the source? Yes. Is it still incredibly useful as a search tool and productivity booster? Absolutely.

It is excellent when just finding something is enough. Most often in my practice, I am dealing with questions that have no written-down answers, meaning the probability of finding a book/article that provides one is negligible. Instead, I am looking for indirect answers or proofs before I make a final engineering decision. Yet another problem is that the language itself changes over time. For instance, at the beginning of the 20th century, the integers were called integral numbers. IMHO, LLMs poorly handle such cases when considered as a substitute for search engines. For full-text vector search, I am using https://www.recoll.org/ a real time saver for me, especially for desktop search.

Re: OpenAI researcher announced GPT-5 math breakthrough that never happened

#160

Post by scosman »

Timsky wrote: post #159
scosman wrote: post #136

Earlier quoted context omitted.

Saying it isn't useful is a bit of an overstatement. It can search, churn through 500k words in a few minutes, and come back with summaries, answers, and sources for each point. Should you blindly trust the summary? No. Should you verify key claims by clicking through to the source? Yes. Is it still incredibly useful as a search tool and productivity booster? Absolutely.

It is excellent when just finding something is enough. Most often in my practice, I am dealing with questions that have no written-down answers, meaning the probability of finding a book/article that provides one is negligible. Instead, I am looking for indirect answers or proofs before I make a final engineering decision. Yet another problem is that the language itself changes over time. For instance, at the beginni…

> GPT-5 is proving useful as a literature review assistant

> No, it does not.

> It is excellent when just finding something is enough.

Post reply on HN