Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-5 is behind schedule

wsj.com

Post reply on HN
Print view View original

61–70 of 1001 posts

Re: GPT-5 is behind schedule

#61

Post by ynniv »

anonzzzies wrote: post #20
rubymamis wrote: post #18

Earlier quoted context omitted.

I find that o1 and Sonnet 3.5 are good and bad quite equally on different things. That's why I keep asking both the same coding questions.

We do the same (all requests go to o1, sonnet and gemini and we store the results for later to compare) automatically for our research: Claude always wins. Even with specific prompting on both platforms. Especially frontend it seems o1 really is terrible.

Claude is trained on principles. GPT is trained on billions of edge cases. Which student do you prefer?

Re: GPT-5 is behind schedule

#62

Post by aimanbenbaha »

coreyh14444 wrote: post #52
PittleyDunkin wrote: post #21

Everyone's comparing o1 and claude, but neither really work well enough to justify paying for them in my experience for coding. What I really want is a mode where they ask clarifying questions , ideally many of them, before spitting out an answer. This would greatly improve utility of producing something with more value than an auto-complete.

Just tell it to do that and it will. Whenever I ask an AI for something and I'm pretty sure it doesn't have all the context I literally just say "ask me clarifying questions until you have enough information to do a great job on this."

And this chain of prompts cumulated with the improved CoT reasoner would accrue a lot more enhanced results. More in line with what the coming agentic era promises.

Re: GPT-5 is behind schedule

#63

Post by dartos »

ynniv wrote: post #60
dartos wrote: post #50

Earlier quoted context omitted.

I’m curious how, if at all, the plan to get around compounding bias in synthetic data generated by models trained in synthetic data.

Everyone's obsessed with new training tokens... It doesn't need to be more knowledgeable, it just needs to practice more. Ask any student: practice is synthetic data.

That leads to overfitting in ML land, which hurts overall performance.

We know that unique data improves performance.

These LLM systems are not students…

Also, which students graduate and are immediately experts in their fields? Almost none.

It takes years of practice in unique, often one-off, situations after graduation for most people to develop the intuition needed for a given field.

Re: GPT-5 is behind schedule

#64

Post by vintermann »

PittleyDunkin wrote: post #21

Everyone's comparing o1 and claude, but neither really work well enough to justify paying for them in my experience for coding. What I really want is a mode where they ask clarifying questions , ideally many of them, before spitting out an answer. This would greatly improve utility of producing something with more value than an auto-complete.

Yes. You can only do so much with the information you get in. The ability to ask good questions, not just of itself in internal monologue style, but actually of the user, would fundamentally make it better since it can get more information in.

As it is now, it has a bad habit of, if it can't answer the question you asked, instead answering a similar-looking question which it thinks you may have meant. That is of course a great strategy for benchmarks, where you don't earn any points for saying you don't know. But it's extremely frustrating for real users, who didn't read their question from a test suite.

Re: GPT-5 is behind schedule

#65

Post by master_crab »

h_tbob wrote: post #35
anonzzzies wrote: post #9

Earlier quoted context omitted.

Not sure what you are using it for, but it is terrible for me for coding; claude beats it always and hands down. o1 just thinks forever to come up with stuff it already tried the previous time. People say that's just prompting without pointing to real million line+ repositories or realistic apps to show how that can be improved. So I say they are making todo and hello world apps and yes, there it works really well. C…

That so weird, it’s seems like everybody here prefers Claude. I’ve been using Claude and openai in copilot and I find even 4o seems to understand the problem better. O1 definitely seems to get it right more for me.

Claude also has a better workflow UI. It’ll maintain conversation context while opening up new windows to present code suggestions.

When I was still subscribing to OpenAI (about 4 months ago) this didn’t exist.

Re: GPT-5 is behind schedule

#67

Post by radioactivist »

Xcelerate wrote: post #58
A_D_E_P_T wrote: post #2

Counterpoint: o1-Pro is insanely good -- subjectively, it's as far above GPT4 as GPT4 was above 3. It's almost too good. Use it properly for an extended period of time, and one begins to worry about the future of one's children and the utility of their schooling. o3, by all accounts, is better still. Seems to me that things are progressing quickly enough.

I had a 30 min argument with o1-pro where it was convinced it had solved the halting problem. Tried to gaslight me into thinking I just didn’t understand the subtlety of the argument. But it’s susceptible to appeal to authority and when I started quoting snippets of textbooks and mathoverflow it finally relented and claimed there had been a “misunderstanding”. It really does argue like a human though now...

I had a similar experience with regular o1 about integral that was divergent. It was adamant that it wasn't and would respond to any attempt at persuasion with variants of "its a standard integral" with a "subtle cancellation". When I asked for any source for this standard integral it produced references to support its argument that existed but didn't actually contain the integral. When I told it the references didn't have the result and backpedalled (gaslighting!) to "I never told you they were in there". When I pointed out that in fact it did it insisted this was just a "misunderstanding". It only relented when I told it Mathematica agreed the integral was divergent. It still insisted it never said that the books it pointed to contained this (false, non-sensical) result.

This was new behaviour for me to see in an LLM. Usually the problem is these things would just fold when you pushed back. I don't know which is better, but being this confidently wrong (and "lying" when confronted with it) is troubling.

Re: GPT-5 is behind schedule

#68

Post by A_D_E_P_T »

h_tbob wrote: post #35
anonzzzies wrote: post #9

Earlier quoted context omitted.

Not sure what you are using it for, but it is terrible for me for coding; claude beats it always and hands down. o1 just thinks forever to come up with stuff it already tried the previous time. People say that's just prompting without pointing to real million line+ repositories or realistic apps to show how that can be improved. So I say they are making todo and hello world apps and yes, there it works really well. C…

That so weird, it’s seems like everybody here prefers Claude. I’ve been using Claude and openai in copilot and I find even 4o seems to understand the problem better. O1 definitely seems to get it right more for me.

They're both okay for coding, though for my use cases (which are niche and involve quite a lot of mathematics and formal logic) o1/o1-Pro is better. It seems to have a better native grasp of mathematical concepts, and it can even answer very difficult questions from vague inputs, e.g.: https://chatgpt.com/share/676020cb-8574-8005-8b83-4bed5b13e1...

Re: GPT-5 is behind schedule

#69

Post by noman-land »

ericskiff wrote: post #25

What we can reasonably assume from statements made by insiders: They want a 10x improvement from scaling and a 10x improvement from data and algorithmic changes The sources of public data are essentially tapped Algorithmic changes will be an unknown to us until they release, but from published research this remains a steady source of improvement Scaling seems to stall if data is limited So with all of that taken toge…

I completely don't understand the use for synthetic data. What good it's it to train a model basically on itself?

Re: GPT-5 is behind schedule

#70

Post by jncfhnb »

qup wrote: post #38
jncfhnb wrote: post #29

Earlier quoted context omitted.

I’m a consultant. Having looked at several enterprises, there’s a lot of work being done to make a lot of things that don’t really work. The bigger the ambition, the harder they’re failing. Some well designed isolated use cases are ok. Mostly things about listening and summarizing text to aid humans. I have yet to see a successful application that is generating good content. IMO replacing the first draft of content c…

> Having looked at several enterprises, there’s a lot of work being done to make a lot of things that don’t really work. Is this a new phenomenon that started post-LLM?

I mean, no, not generally. but the success rate of other tools is much higher.

A lot of companies are trying to build these general purpose bots that just magically know everything about the company and have these but knowledge bases, but they just don’t work.

Post reply on HN