Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-5.2

openai.com

Post reply on HN
Print view View original

981–990 of 1001 posts

Re: GPT-5.2

#981

Post by dolmen »

actionfromafar wrote: post #822
stacktrace wrote: post #785

Earlier quoted context omitted.

> It's still a big issue that the models will make up plausible sounding but wrong or misleading explanations for things, and verifying their claims ends up taking time. And if it's a topic you don't care about enough, you might just end up misinformed. Exactly! One important thing LLMs have made me realise deeply is "No information" is better than false information. The way LLMs pull out completely incorrect explana…

I wonder if the only way to fix this with current LLMs, would be to generate a lot synthetic data for a select number topics you really don't want it "go off the rails" with. That synthetic data would be lots of variations on that "I don't know how to do X with Y".

I would not bet on synthetic data.

LLMs are very good at detecting patterns.

Re: GPT-5.2

#982

Post by fellowniusmonk »

agentifysh wrote: post #436

Looks like they've begun censoring posts at r/Codex and not allowing complaint threads so here is my honest take: - It is faster which is appreciated but not as fast as Opus 4.5 - I see no changes, very little noticeable improvements over 5.1 - I do not see any value in exchange for +40% in token costs All in all I can't help but feel that OpenAI is facing an existential crisis. Gemini 3 even when its used from AI St…

5.2 is performing worse in technical reading comprehension for information and logic dense puzzles. It's way more confidently wrong and stubborn about understanding definitions of words.

Re: GPT-5.2

#983

Post by fellowniusmonk »

zone411 wrote: post #222

I've benchmarked it on the Extended NYT Connections benchmark ( https://github.com/lechmazur/nyt-connections/ ): The high-reasoning version of GPT-5.2 improves on GPT-5.1: 69.9 → 77.9. The medium-reasoning version also improves: 62.7 → 72.1. The no-reasoning version also improves: 22.1 → 27.5. Gemini 3 Pro and Grok 4.1 Fast Reasoning still score higher.

I think they are overfitting more, I'm seeing it perform worse on esoteric logic puzzles.

Re: GPT-5.2

#984

Post by fellowniusmonk »

nbardy wrote: post #398

Those arc agi 2 improvements are insane. Thats especially encouraging to me because those are all about generalization. 5 and 5.1 both felt overfit and would break down and be stubborn when you got them outside their lane. As opposed to Opus 4.5 which is lovely at self correcting. It’s one of those things you really feel in the model rather than whether it can tackle a harder problem or not, but rather can I go back…

5.2 seems worse on overfitting for esoteric logic puzzles in my testing. Tests using precise language where attention has to be paid to use the correct definition among many for a given word. It charges ahead with wrong definitions in a far lower accuracy and worse way now.

Re: GPT-5.2

#985

Post by deaux »

baq wrote: post #737
deaux wrote: post #598

Earlier quoted context omitted.

> 5 and 5.1 both felt overfit and would break down and be stubborn when you got them outside their lane. As opposed to Opus 4.5 which is lovely at self correcting. This is simply the "openness vs directive-following" spectrum, which as a side-effect results in the sycophancy spectrum, which still none of them have found an answer to. Recent GPT models follow directives more closely than Claude models, and are less sy…

Opus 4.5 answers most of my non-question comments with ‘you’re right.’ as the first thing in the output. At least I’m not absolutely right, I’ll take this as an improvement.

Hah, maybe 5th gen Claude will change to "you may be right".

The positive thing is that it seems to be more performative than anything. Claude models will say "you're [absolutely] right" and then immediately do something that contradicts it (because you weren't right).

Gemini 3 Pro seems to have struck a decent balance between stubbornness and you're-right-ness, though I still need to test it more.

Re: GPT-5.2

#986

Post by ric2b »

FergusArgyll wrote: post #105
elgatolopez wrote: post #29

Earlier quoted context omitted.

Where did you get that from? Cutoff date says august 2025. Looks like a newly pretrained model

> This stands in sharp contrast to rivals: OpenAI’s leading researchers have not completed a successful full-scale pre-training run that was broadly deployed for a new frontier model since GPT-4o in May 2024, highlighting the significant technical hurdle that Google’s TPU fleet has managed to overcome. - https://newsletter.semianalysis.com/p/tpuv7-google-takes-a-s... It's also plainly obvious from using it. The "Broa…

How is that a technical hurdle if they obviously were able to do it before?

It's probably just a question of cost/benefit analysis, it's very expensive to do, so the benefits need to be significant.

Re: GPT-5.2

#987

Post by dominotw »

conception wrote: post #877
BatteryMountain wrote: post #860

Earlier quoted context omitted.

My biggest problem with LLM's at this point is that they produce different and inconsistent results or behave differently, given the same prompt. The better grounding would be amazing at this point. I want to give an LLM the same prompt on different days and I want to be able to trust that it will do the same thing as yesterday. Currently they misbehave multiple times a week and I have to manually steer it a bit whic…

You need to change the temperature to 0 and tune your prompts for automated workflows.

have you tried this? this doesnt work because the way inference runs at big companies. its not just running your query in isolation.

maybe it can work if you are running your own inference.

Re: GPT-5.2

#988

Post by dominotw »

neuah wrote: post #882
AstroBen wrote: post #515

Earlier quoted context omitted.

Did you notice much improvement going from Gemini 2.5 to 3? I didn't I just think they're all struggling to provide real world improvements

Using it in a specialized subfield of neuroscience, Gemini 3 w/ thinking is a huge leap forward in terms of knowledge and intelligence (with minimal hallucinations). I take it that the majority of people on here are software engineers. If you're evaluating it on writing boilerplate code, you probably have to squint to see differences between the (excellent) raw model performances. whereas in more niche edge cases the…

what specalized usecases did you use it on and what were the outcomes.

can you share your experience and data for "leap forward" ?

Re: GPT-5.2

#989

Post by kromokromo »

biofox wrote: post #830
stacktrace wrote: post #785

Earlier quoted context omitted.

> It's still a big issue that the models will make up plausible sounding but wrong or misleading explanations for things, and verifying their claims ends up taking time. And if it's a topic you don't care about enough, you might just end up misinformed. Exactly! One important thing LLMs have made me realise deeply is "No information" is better than false information. The way LLMs pull out completely incorrect explana…

I ask for confidence scores in my custom instructions / prompts, and LLMs do surprisingly well at estimating their own knowledge most of the time.

Asking an LLM to give itself a «confidence score» is like asking a teenager to grade his own exam. I LLMs doesn’t «feel» uncertainty and confidence like we do.

Re: GPT-5.2

#990

Post by jacquesm »

svara wrote: post #765

In my experience, the best models are already nearly as good as you can be for a large fraction of what I personally use them for, which is basically as a more efficient search engine. The thing that would now make the biggest difference isn't "more intelligence", whatever that might mean, but better grounding. It's still a big issue that the models will make up plausible sounding but wrong or misleading explanations…

There are four words that would make the output of any LLM instantly 1000x more useful and I haven't seen them yet: "I do not know.".
Post reply on HN