Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-5 is behind schedule

wsj.com

Post reply on HN
Print view View original

571–580 of 1001 posts

Re: GPT-5 is behind schedule

#571

Post by concerndc1tizen »

ciconia wrote: post #551
t_serpico wrote: post #208

One fundamental challenge to me is that if each training run because more and more expensive, the time it takes it to learn what works/doesn't work widens. Half a billion dollars for training a model is already nuts, but if it takes 100 iterations to perfect it, you've cumulatively spent 50 billion dollars... Smaller models may actually be where rapid innovation continues simply because of tighter feedback loops. O3…

When you think about it it's astounding how much energy this technology consumes versus a human brain which runs at ~20W [1]. [1] https://hypertextbook.com/facts/2001/JacquelineLing.shtml

20w for 20 years to answer questions slowly and error-prone at the level of a 30B model. An additional 10 years with highly trained supervision and the brain might start contributing original work.

Re: GPT-5 is behind schedule

#572

Post by TeMPOraL »

Fargren wrote: post #555
jstummbillig wrote: post #552

Earlier quoted context omitted.

You just state this as if it was obviously true, but I don't see how. Why is using LLM like reading a pop sci book and not like reading a history book? Or even less like either, because you have to continually ask questions to get anything?

A history book is written by someone who knows the topic, and then reviewed by more people who also know the topic, and then it's out there where people can read it and criticize it if it's wrong about the topic. A question asked to an AI is not reviewed by anyone, and it's ephemeral. The AI can answer "yes" today, and "no" tomorrow, so it's not possible to build a consensus on whether it answers specific questions c…

> A question asked to an AI is not reviewed by anyone, and it's ephemeral. The AI can answer "yes" today, and "no" tomorrow, so it's not possible to build a consensus on whether it answers specific questions correctly.

It's even more so with humans! Most of our conversations are, and has always been, ephemeral and unverifiable (and there's plenty of people who want to undo the little of permanence and verifiability we still have on the Internet...). Along the dimension of permanence and verifiability, asking an LLM is actually much better than asking a human - there's always a log of the conversation you had with the AI produced and stored somewhere for at least a while (even if only until you clear your temp folder), and if you can get ahold of that log, you can not just verify the answers, you can actually debug the AI. You can rerun the conversation with different parameters, different prompting, perhaps even inspect the inference process itself. You can do that ten times, hundred times, a million times, and won't be asked to come to Hague and explain yourself. Now try that with a human :).

Re: GPT-5 is behind schedule

#573

Post by jacobolus »

jstummbillig wrote: post #562
Fargren wrote: post #555

Earlier quoted context omitted.

A history book is written by someone who knows the topic, and then reviewed by more people who also know the topic, and then it's out there where people can read it and criticize it if it's wrong about the topic. A question asked to an AI is not reviewed by anyone, and it's ephemeral. The AI can answer "yes" today, and "no" tomorrow, so it's not possible to build a consensus on whether it answers specific questions c…

A pop sci fi book can be written by someone who knows the topic and reviewed by people who know the topic — and a history book can also not. LLM generated answers are more comparable to ad-hoc human expert's answers and not to written books. But it's much simpler to statistically evaluate and correct them. That is how we can know that, on average, LLMs are improving and are outperforming human experts on an increasin…

In my experience LLM generated answers are more comparable to an ad-hoc answer by a human with no special expertise, moderate google skills, but good bullshitting skills spending a few minutes searching the web, reading what they find and synthesizing it, waiting long enough for the details to get kind of hazy, and then writing up an answer off the top of their head based on that, filling in any missing material by just making something up. They can do this significantly faster than a human undergraduate student might be able to, so if you need someone to do this task very quickly / prolifically this can be beneficial (e.g. this could be effective for generating banter for video game non-player characters, for astroturfing social media, or for cheating on student essays read by an overworked grader). It's not a good way to get expert answers about anything though.

More specifically: I've never gotten an answer from an LLM to a tricky or obscure question about a subject I already know anything about that seemed remotely competent. The answers to basic and obvious questions are sometimes okay, but also sometimes completely wrong (but confidently stated). When asked follow-up questions the LLM will repeatedly directly contradict itself with additional answers each as wrong as the first, all just as confidently stated.

Re: GPT-5 is behind schedule

#574

Post by az226 »

Stevvo wrote: post #31
ericskiff wrote: post #25

What we can reasonably assume from statements made by insiders: They want a 10x improvement from scaling and a 10x improvement from data and algorithmic changes The sources of public data are essentially tapped Algorithmic changes will be an unknown to us until they release, but from published research this remains a steady source of improvement Scaling seems to stall if data is limited So with all of that taken toge…

"With o3 now public knowledge, imagine how long it’s been churning out new thinking at expert level across every field." I highly doubt that. o3 is many orders of magnitude more expensive than paying subject matter experts to create new data. It just doesn't make sense to pay six figures in compute to get o3 to make data a human could make for a few hundred dollars.

Only a matter of time. The costs are aggressively going down. And with specialized inference hardware it will go further down.

Cost of coordination is also large. Immediate answers are an advantage/selling point.

Re: GPT-5 is behind schedule

#575

Post by _w1tm »

scott_w wrote: post #560
wilg wrote: post #548

Earlier quoted context omitted.

Its good to not trust it but that's not the same as it having no idea . There is a lot of value in being close for many tasks!

I think it’s a very dangerous place to be in an area you’re not familiar with. I can read Python code and figure out if it’s what I want or not. I couldn’t read an article about physics and tell you what’s accurate and what’s not. Legal Eagle has a great video on how ChatGPT was used to present a legal argument, including made up case references! Stuff like this is why I’m wary to rely on it in areas outside of my ex…

There’s a world of difference between blindly trusting an LLM and using it to generate clues for further research.

You wouldn’t write a legal argument based on what some random stranger told you, would you?

Re: GPT-5 is behind schedule

#576

Post by OtherShrezzing »

rchaves wrote: post #553

Nah it's just a marketing problem, "GPT" and "ChatGPT" names is the biggest asset OpenAI has, people have expectations so high for GPT-5 that they cannot burn this name unless it's something truly majestic, bordering AGI at the very least. Until they are confident enough that people will be blown off by it, it's better to continue building up the hype

The Half Life 3 of the SaaS/zirp era.

Re: GPT-5 is behind schedule

#577

Post by energy123 »

captainbland wrote: post #81

In my intuition it makes sense that there is going to be some significant friction in LLM development going forward. We're talking about models that will cost upwards of $1bn to train. Save for a technological breakthrough, GPT-6/7 will probably have to wait for hardware to catch up.

It's more that you have to allocate the compute the right way.

Noam Brown's analogy is, you could train a massive one shot foundation model to predict the next best Go move, but that would be stupid. Better to use some test time search. You get better results for less money.

Same is happening in LLMs.

Re: GPT-5 is behind schedule

#578

Post by eru »

NBJack wrote: post #563
eru wrote: post #423

Earlier quoted context omitted.

> LLMs have no real sense of truth or hard evidence of logical thinking. Most humans don't have that either, most of the time.

Then we already have access to a cheaper, scalable, abundant, and (in most cases) renewable resource, at least compared to how much a few H100s cost. Take good care of them, and they'll probably outlast most a GPU's average lifespans (~10 years). We're also biodegradable.

Humans are a lot more expensive to run than inference on LLMs.

No human, especially no human whose time you can afford, comes close to the breadth of book knowledge ChatGPT has, and the number of languages is speaks reasonably well.

Re: GPT-5 is behind schedule

#579

Post by raducu »

wilg wrote: post #548
scott_w wrote: post #542

Earlier quoted context omitted.

A Scientific American issue where the authors have no idea that they don’t know a topic so just completely make up the content, including the sources. At least magazine authors are reading the sources before misunderstanding the content (or asking the authors what the research means). I don’t even trust the summaries after watching LLMs think we have meetings about my boss’s cat just because I mentioned it once as sh…

Its good to not trust it but that's not the same as it having no idea . There is a lot of value in being close for many tasks!

> Its good to not trust it but that's not the same as it having no idea. There is a lot of value in being close for many tasks!

The task is to replace hazelcast with infinispan in a stand-alone IMDG setup. You're interested in Locks and EntryProcessors.

Ghat GPT 4, o1 tell you with their enthusiastic style Infinispan has all those features.

You test it locally and it does....

But the thing is infinispan doesn't have explicit locks in client-server mode, just in embedded mode, but that's something you find out from another human who has tied doing the same thing.

Are you better off using Chat GPT in this case?

I could go on and on and on, on times Chat GPT has bullshitted me and wasted days of my time, but hey, it helps with one-liners and Copilot occasionally has spectacular method auto-complete and learns on the fly some stuff and it makes my cry when it remembers random tidbits about me that not even family members do

Re: GPT-5 is behind schedule

#580

Post by rhubarbtree »

exhaze wrote: post #160
simonw wrote: post #140

Earlier quoted context omitted.

The new idea is inference-time scaling, as seen in o1 (and o3 and Qwen's QwQ and DeepSeek's DeepSeek-R1-Lite-Preview and Google's gemini-2.0-flash-thinking-exp). I suggest reading these two pieces about that: - https://www.aisnakeoil.com/p/is-ai-progress-slowing-down - best explanation I've seen of inference scaling anywhere - https://arcprize.org/blog/oai-o3-pub-breakthrough - François Chollet's deep dive into o3 I'…

I think the wildest thing is actually Meta’s latest paper where they show a method for LLMs reasoning not in English, but in latent space https://arxiv.org/pdf/2412.06769 I’ve done research myself adjacent to this (mapping parts of a latent space onto a manifold), but this is a bit eerie, even to me.

Seems a standard approach of AI research is to “move X into the latent space” where X is some useful function (eg diffusion) previously done in the “data” or “artefact” space. So seems very pedestrian not wild to make that step.
Post reply on HN