Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI Progress

progress.openai.com

Post reply on HN
Print view View original

251–260 of 372 posts

Re: OpenAI Progress

#251

Post by atoav »

simianwords wrote: post #7

My interpretation of the progress. 3.5 to 4 was the most major leap. It went from being a party trick to legitimately useful sometimes. It did hallucinate a lot but I was still able to get some use out of it. I wouldn't count on it for most things however. It could answer simple questions and get it right mostly but never one or two levels deep. I clearly remember 4o was also a decent leap - the accuracy increased su…

To me 4 to 5 got much faster, but also worse. It is much more often ignoring explicit instructions like: "generate 10 song-titles with varying length" and it generates 10 song titles that are nearly identical length. This worked somewhat well with version 3 already..

Re: OpenAI Progress

#252

Post by rorads »

I would have liked to have seen the underlying token count, and the full responses as an optional filter. My understanding is that under the hood the GPT-5 response (as well as omitted O model responses) would end with the presented output, but would have had many paragraphs of “The user wants X, so I should try to Y. Okay this z needs to be considered” etc.

It doesn’t detract from the progress, but I think it would change how you interpret it. In some ways 4 / 4o were more impressive because they were going straight to output with a lower number of tokens produced to get a good response.

Re: OpenAI Progress

#253

Post by RugnirViking »

energy123 wrote: post #186
gordon_freeman wrote: post #122

It seems like the progress from GPT-4 to GPT-5 has plateaued: for most prompts, I actually find GPT-4 more understandable than GPT-5 [1]. [1] Read the answers from GPT-4 and 5 for this math question: "Ugh I hate math, integration by parts doesn't make any sense"

Basic prose is a saturated bench. You can't go above 100% so by definition progress will stall on such benchmarks.

You say that, but I can imagine a good maths textbook and a bad one, both technically correct and well written prose, but one is better at taking the student on a journey and understanding where people fall off and common misunderstandings without odiously re-explaining everything

Re: OpenAI Progress

#254

Post by gpt5 »

wasabi991011 wrote: post #223
raincole wrote: post #19

Earlier quoted context omitted.

I really wonder which one of us is the minority. Because I find text-davinci-001 answer is the only one that reads like a story. All the others don't even resemble my idea of "story" so to me they're 0/100.

text-davinci-001 feels more like a story, but it is also clearly incomplete, in that it is cut-off before the story arc is finished. imo GPT-5 is objectively better at following the prompt because it has a complete story arc, but this feels less satisfying since a 50 word story is just way too short to do anything interesting (and to your point, barely even feels like a story).

FWIW, I found the way it ended interesting. It realized it is being replaced, so it burned the toast out of anger/despair, but also just to hear its owner voice one last time.

Re: OpenAI Progress

#255

Post by vczf »

furyofantares wrote: post #97
simianwords wrote: post #7

My interpretation of the progress. 3.5 to 4 was the most major leap. It went from being a party trick to legitimately useful sometimes. It did hallucinate a lot but I was still able to get some use out of it. I wouldn't count on it for most things however. It could answer simple questions and get it right mostly but never one or two levels deep. I clearly remember 4o was also a decent leap - the accuracy increased su…

I have a theory about why it's so easy to underestimate long-term progress and overestimate short-term progress. Before a technology hits a threshold of "becoming useful", it may have a long history of progress behind it. But that progress is only visible and felt to researchers. In practical terms, there is no progress being made as long as the thing is going from not-useful to still not-useful. So then it goes from…

The more general pattern is “slowly at first, then all at once.”

It almost universally describes complex systems.

Re: OpenAI Progress

#256

Post by chippiewill »

andy_ppp wrote: post #173
actuallyalys wrote: post #169

One thing that appears to have been lost between GPT-4 and GPT-5 is that it no longer reminds the user that it's an AI and not a human, let alone a human expert. Maybe those genuinely annoyed people, but it seems like they were potentially useful measure to prevent users from being overly credulous GPT-5 also goes out of its way to suggest new prompts. This seems potentially useful, although potentially dangerous if…

People seem to miss the humanity of previous GPTs from my understanding. GPT5 seems colder and more precise and better at holding itself together with larger contexts. People should know it’s AI, it does not need to explain this constantly for me, but I’m sure you can add that back in with some memory options if you prefer that?

I agree. I think it's a classic UX progression thing to be removing the "I'm an AI" aspect, because it's not actually useful anymore because it's no longer a novel tool. Same as how GUIs all removed their skeuomorphs because they were no longer required.

Re: OpenAI Progress

#257

Post by TrackerFF »

NoahZuniga wrote: post #167
starchild3001 wrote: post #158

A few data points that highlight the scale of progress in a year: 1. LM Sys (Human Preference Benchmark): GPT-5 High currently scores 1463, compared to GPT-4 Turbo (04/03/2024) at 1323 -- a 140 ELO point gap. That translates into GPT-5 winning about two-thirds of head-to-head comparisons, with GPT-4 Turbo only winning one-third. In practice, people clearly prefer GPT-5’s answers ( https://lmarena.ai/leaderboard ). 2.…

The 135 iq result is on Mensa Norway, while the offline test is 120. It seems probable that similar questions to the one in Mensa are in the training data, so it probably overestimates "general intelligence".

Some iq / aptitude test sections are trivial for machines, like working memory. Wonder if those parts are just excluded? As the could really pull up the test scores.

Re: OpenAI Progress

#258

Post by elictronic »

yojo wrote: post #201
mrandish wrote: post #83

Earlier quoted context omitted.

I agree. Since I'm recently retired and no longer code much, I don't have much need for LLMs but refining a complex, niche web search is the one thing where they're uniquely useful to me. It's usually when targeting the specific topic involves several keywords which have multiple plain English meanings that return a flood of erroneous results. Because LLMs abstract keywords to tokens based on underlying meaning, you…

I’ll give you another use: LLMs are really good at unearthing the “unknown unknowns.” If I’m learning a new topic (coding or not) summarizing my own knowledge to an LLM and then asking “what important things am I missing” almost always turns up something I hadn’t considered. You’ll still want to fact check it, and there’s no guarantee it’s comprehensive, but I can’t think of another tool that provides anything close…

Coworkers and experts in a field. I can trust them much more but the better they are the less access you have.

Re: OpenAI Progress

#259

Post by vineyardmike »

muzani wrote: post #198
ralusek wrote: post #64

Earlier quoted context omitted.

The real jump was 3 to 3.5. 3.5 was the first “chatgpt.” I had tried gpt 3 and it was certainly interesting, but when they released 3.5 as ChatGPT, it was a monumental leap. 3.5 to 4 was also huge compared to what we see now, but 3.5 was really the first shock.

ChatGPT was a proper product, but as an engine, GPT-3 (davinci-001) has been my favorite all the way until 4.1 or so. It's absolutely raw and they didn't even guardrail it. 3.5 was like Jenny from customer service. davinci-001 was like Jenny the dreamer trying to make ends meet by scriptwriting, who was constantly flagged for racist opinions. Both of these had an IQ of around 70 or so, so the customer service trainin…

> 3.5 was like Jenny from customer service. davinci-001 was like Jenny the dreamer trying to make ends meet by scriptwriting, who was constantly flagged for racist opinions.

How do you use the product to get this experience? All my questions warrant answers with no personality.

Re: OpenAI Progress

#260

Post by OnlineGladiator »

ninetyninenine wrote: post #241
OnlineGladiator wrote: post #228

Earlier quoted context omitted.

Every time I use ChatGPT I become incredibly frustrated with how fucking awful it is. I've used it more than enough, time and time again (just try the new model, bro!), to know that I fucking hate it. If it works for you, cool. I think it's dogshit.

Objectively he didn't cherry pick. He responded to the person and it got it right when he used the "thinking" model WHICH he did specify in his original comment. Why don't you stick to the topic rather than just declaring it's utter dog shit. Nobody cares about your "opinion" and everyone is trying to converge on a general ground truth no matter how fuzzy it is.

All anybody is doing here is sharing their opinion unless you're quoting benchmarks. My opinion is just as useless as yours, it's just some find mine more interesting and some find yours more interesting.

How do you expect to find a ground truth from a non-deterministic system using anecdata?

Post reply on HN