Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-5 is behind schedule

wsj.com

Post reply on HN
Print view View original

91–100 of 1001 posts

Re: GPT-5 is behind schedule

#91

Post by weird_fox »

simondotau wrote: post #44
PittleyDunkin wrote: post #21

Everyone's comparing o1 and claude, but neither really work well enough to justify paying for them in my experience for coding. What I really want is a mode where they ask clarifying questions , ideally many of them, before spitting out an answer. This would greatly improve utility of producing something with more value than an auto-complete.

Just today I got Claude to convert a company’s PDF protocol specification into an actual working python implementation of that protocol. It would have been uncreative drudge work for a human, but I would have absolutely paid a week of junior dev time for it. Instead I wrote it alongside AI and it took me barely more than an hour. The best part is, I’ve never written any (substantial) python code before.

I have to agree. It's still a bit hit or miss, but the hits are a huge time and money saver especially in refactoring. And unlike what most of the rather demeaning comments in those HN threads state, I am not some 'grunt' doing 'boilerplate work'. I mostly do geometry/math stuff, and the AIs really do know what they're talking about there sometimes. I don't have many peers I can talk to most of the time, and Claude is really helping me gather my thoughts.

That being said, I definitely believe it's only useful for isolated problems. Even with Copilot, I feel like the AIs just lack a bigger context of the projects.

Another thing that helped me was designing an initial prompt that really works for me. I think most people just expect to throw in their issue and get a tailored solution, but that's just not how it works in my experience.

Re: GPT-5 is behind schedule

#92

Post by ben_w »

ogogmad wrote: post #22
delusional wrote: post #6

Earlier quoted context omitted.

I'd say the same. I've tried a bunch of different AI tools, and none of them really seem all that helpful.

One use-case: They help with learning things quickly by having a chat and asking questions. And they never get tired or emotional. Tutoring 24/7. They also generate small code or scripts, as well as automate small things, when you're not sure how, but you know there's a way. You need to ensure you have a way to verify the results. They do language tasks like grammar-fixing, perfect translation, etc. They're 100 times…

> perfect translation

Hah, no. They're good, but they definitely make stuff up when the context gets too long. Always check their output, just the same as you already note they need for small code and scripts.

Re: GPT-5 is behind schedule

#93

Post by noman-land »

psb217 wrote: post #73
noman-land wrote: post #69

Earlier quoted context omitted.

I completely don't understand the use for synthetic data. What good it's it to train a model basically on itself?

The value of synthetic data relies on having non-zero signal about which generated data is "better" or "worse". In a sense, this what reinforcement learning is about. Ie, generate some data, have that data scored by some evaluator, and then feed the data back into the model with higher weight on the better stuff and lower weight on the worse stuff. The basic loop is: (i) generate synthetic data, (ii) rate synthetic d…

Thanks, that makes a lot more sense.

Re: GPT-5 is behind schedule

#94

Post by josh-sematic »

ADeerAppeared wrote: post #47
beacon294 wrote: post #19

Earlier quoted context omitted.

There's someone with this comment in every thread. Meanwhile, no one answers this because they are getting value. Please take the time to learn, it will give you value.

> Meanwhile, no one answers this because they are getting value. You're literally doing the same thing you're accusing of. Every HN thread is full of AI boosters claiming AI to be the future with no backing evidence. Riddle me this. If all these people are "getting value", why are all these companies losing horrendous amounts of money? Why has nobody figured out how to be profitable? > Please take the time to learn,…

There’s clearly some value. People are paying for something.

> AI start-ups generate money faster than past hyped tech companies

https://www.ft.com/content/a9a192e3-bfbc-461e-a4f3-112e63d0b...

Re: GPT-5 is behind schedule

#95

Post by CapcomGo »

anonzzzies wrote: post #20
rubymamis wrote: post #18

Earlier quoted context omitted.

I find that o1 and Sonnet 3.5 are good and bad quite equally on different things. That's why I keep asking both the same coding questions.

We do the same (all requests go to o1, sonnet and gemini and we store the results for later to compare) automatically for our research: Claude always wins. Even with specific prompting on both platforms. Especially frontend it seems o1 really is terrible.

Wins? What does this mean? Do you have any results? I see the claims that Claude is better for coding a lot but using it and using Gemini 2.0 flash and o1 and it sure doesn't seem like it.

Re: GPT-5 is behind schedule

#96

Post by ynniv »

dartos wrote: post #86
ynniv wrote: post #76

Earlier quoted context omitted.

It's overfitting when you train too large a model on too many details. Rote memorization isn't rewarding. The more concepts the model manages to grok, the more nonlinear its capabilities will be: we don't have a data problem, we have an educational one. Claude 3.5 was safety trained by Claude 3.0, and it's more coherent for it. https://www.anthropic.com/news/claudes-constitution

Overfitting can be caused by a lot of different things. Having an over abundance of one kind of data in a training set is one of those causes. It’s why many pre-processing steps for image training pipelines will add copies of images at weird rotations, amounts of blur, and different cropping. > The more concepts the model manages to grok, the more nonlinear its capabilities will be These kind of hand wavey statements…

These kind of hand wavey statements like “practice,” “grok,” and “nonlinear its capabilities will be” are not very constructive as they don’t have solid meaning wrt language models.

So, here's my hypothesis, as someone who is adjacent ML but haven't trained DNNs directly:

We don't understand how they work, because we didn't build them. They built themselves.

At face value this can be seen as an almost spiritual position, but I am not a religious person and I don't think there's any magic involved. Unlike traditional models, the behavior of DNNs is based on random changes that failed up. We can reason about their structure, but only loosely about their functionality. When they get better at drawing, it isn't because we taught them to draw. When they get better at reasoning, it isn't because the engineers were better philosophers. Given this, there will not be a direct correlation between inputs and capabilities, but some arrangements do work better than others.

If this is the case, high order capabilities should continue to increase with training cycles, as long as they are performed in ways that don't interfere with what has been successfully learned. People lamented the loss of capability that GPT 4 suffered as they increased safety. I think Anthropic has avoided this by choosing a less damaging way to tune a well performing model.

I think these ideas are supported by Wolfram's reduction of the problem at https://writings.stephenwolfram.com/2024/08/whats-really-goi...

Re: GPT-5 is behind schedule

#97

Post by nialv7 »

ericskiff wrote: post #25

What we can reasonably assume from statements made by insiders: They want a 10x improvement from scaling and a 10x improvement from data and algorithmic changes The sources of public data are essentially tapped Algorithmic changes will be an unknown to us until they release, but from published research this remains a steady source of improvement Scaling seems to stall if data is limited So with all of that taken toge…

> OpenAI’s next moat

I don't think oai has any moat at all. If you look around, QwQ from Alibaba is already pushing o1-preview performances. I think oai is only ahead by 3~6 months at most.

Re: GPT-5 is behind schedule

#98

Post by rrrrrrrrrrrryan »

master_crab wrote: post #65
h_tbob wrote: post #35

Earlier quoted context omitted.

That so weird, it’s seems like everybody here prefers Claude. I’ve been using Claude and openai in copilot and I find even 4o seems to understand the problem better. O1 definitely seems to get it right more for me.

Claude also has a better workflow UI. It’ll maintain conversation context while opening up new windows to present code suggestions. When I was still subscribing to OpenAI (about 4 months ago) this didn’t exist.

It exists as of last week with Canvas.

Re: GPT-5 is behind schedule

#99

Post by bookaway »

Stevvo wrote: post #31
ericskiff wrote: post #25

What we can reasonably assume from statements made by insiders: They want a 10x improvement from scaling and a 10x improvement from data and algorithmic changes The sources of public data are essentially tapped Algorithmic changes will be an unknown to us until they release, but from published research this remains a steady source of improvement Scaling seems to stall if data is limited So with all of that taken toge…

"With o3 now public knowledge, imagine how long it’s been churning out new thinking at expert level across every field." I highly doubt that. o3 is many orders of magnitude more expensive than paying subject matter experts to create new data. It just doesn't make sense to pay six figures in compute to get o3 to make data a human could make for a few hundred dollars.

Yes, I think they had to push this reveal forward because their investors were getting antsy with the lack of visible progress to justify continuing rising valuations. There is no other reason a confident company making continuous rapid progress would feel the need to reveal a product that 99% of companies worldwide couldn't use at the time of the reveal.

That being said, if OpenAI is burning cash at lightspeed and doesn't have to publicly reveal the revenue they receive from certain government entities, it wouldn't come as a surprise if they let the government play with it early on in exchange for some much needed cash to set on fire.

EDIT: The fact that multiple sites seem to be publishing GPT-5 stories similar to this one leads one to conclude that the o3 benchmark story was meant to counter the negativity from this and other similar articles that are just coming out.

Re: GPT-5 is behind schedule

#100

Post by rrrrrrrrrrrryan »

captainbland wrote: post #81

In my intuition it makes sense that there is going to be some significant friction in LLM development going forward. We're talking about models that will cost upwards of $1bn to train. Save for a technological breakthrough, GPT-6/7 will probably have to wait for hardware to catch up.

I think the main bottleneck right now is training data - they've basically exhausted all public sources of data, so they have to either pay humans to generate new data from scratch or pay for the reasoning models to generate (less useful) synthetic training data. The next bottleneck is hardware, and the least important bottleneck is money.
Post reply on HN