Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-5.2

openai.com

Post reply on HN
Print view View original

951–960 of 1001 posts

Re: GPT-5.2

#951

Post by olliepro »

Wowfunhappy wrote: post #282
olliepro wrote: post #54

Earlier quoted context omitted.

Do you have a better way to measure LLMs? Measurement implies quantitative evaluation... which is the same as benchmarks.

I don’t have a good way to measure them, but I think they should be evaluated more like how we evaluate movies, or restaurants. Namely, experienced critics try them and write reviews.

It feels like this should work, but the breadth of knowledge in these models is so vast. Everyone knows how to taste, but not everyone knows physics, biology, math, every language… poetry, etc. Enumerating the breadth of valuable human tasks is hard, so both approaches suffer from the scale of the models’ surface area.

An interesting problem since the creators of OLMO have mentioned that throughout training, they use 1/3 or their compute just doing evaluations.

Edit:

One nice thing about the “critic” approach is that the restaurant (or model provider) doesn’t have access to the benchmark to quasi-directly optimize against.

Re: GPT-5.2

#952

Post by faxmeyourcode »

onraglanroad wrote: post #333

I suppose this is as good a place as any to mention this. I've now met two different devs who complained about the weird responses from their LLM of choice, and it turned out they were using a single session for everything. From recipes for the night, presents for the wife and then into programming issues the next day. Don't do that. The whole context is sent on queries to the LLM, so start a new chat for each topic.…

My boss (great engineer) had been complaining about this with his internal github copilot quality no matter the model or task. Turns out he never cleared the context. It was just the same conversation spread thin across nearly a dozen completely separate repositories because they were all in his massive vscode workspace at once.

This was earlier this year... So I started giving internal presentations on basic context management, best practices, etc after that for our engineering team.

Re: GPT-5.2

#953

Post by loa_observer »

does the model really improve? i tried several tasks today, and most of them failed, which are super easy ones.

maybe it's just because the gpt5.2 in cursor is super stupid?

Re: GPT-5.2

#954

Post by atheljcarlton »

It's dog-doo-doo. I put in my algebraic geometry final review (100's of thousands of tokens) and Gemini instantly found all the propositions, theorems, and problems that I needed in a neat list (in about 5 seconds), meanwhile ChatGPT 5.2 Thinking took 10mins before timing out and not even completing the request.

Re: GPT-5.2

#955

Post by atheljcarlton »

atheljcarlton wrote: post #954

It's dog-doo-doo. I put in my algebraic geometry final review (100's of thousands of tokens) and Gemini instantly found all the propositions, theorems, and problems that I needed in a neat list (in about 5 seconds), meanwhile ChatGPT 5.2 Thinking took 10mins before timing out and not even completing the request.

However, the model card for GPT 5.2 looks amazing, wish I could actually see that performance in action!

Re: GPT-5.2

#956

Post by XCSme »

andrepd wrote: post #838
XCSme wrote: post #828

Earlier quoted context omitted.

But most benchmarks are not about that... Are there even any "hallucination" public benchmarks?

"Benchmarks" for LLMs are a total hoax, since you can train them on the benchmarks themselves.

I would assume a good benchmark has hidden tests, or something randomly generated that is harder to game

Re: GPT-5.2

#957

Post by credit_guy »

deaux wrote: post #606
credit_guy wrote: post #185

Earlier quoted context omitted.

Not the OP, but I think "slight" here is in relation to Anthropic and Google. Claude Opus 4.5 comes at $25/MT (million tokens), Sonnet 4.5 at $22.5/MT, and Gemini 3 at $18/MT. GPT 5.2 at $14/MT is still the cheapest.

Your numbers are very off. $25 - Opus 4.5 $15 - Sonnet 4.5 $14 - GPT 5.2 $12 - Gemini 3 Pro Even if you're including input, your numbers are still off.

I used the pricing for long context (>200k) in all cases. I personally use AI as coding assistants, like lots of other people, and as such, hitting and exceeding 200k is quite the norm. The numbers you are showing are for <200k context length.

Re: GPT-5.2

#958

Post by carsoon »

ifwinterco wrote: post #766
goobatrooba wrote: post #359

I feel there is a point when all these benchmarks are meaningless. What I care about beyond decent performance is the user experience. There I have grudges with every single platform and the one thing keeping me as a paid ChatGPT subscriber is the ability to sort chats in "projects" with associated files (hello Google, please wake up to basic user-friendly organisation!) But all of them * Lie far too often with confi…

I'm not an expert but my understanding is transformers based models simply can't do some of those things, it isn't really how they work. Especially something like expressing a certainty %, you might be able to get it to output one but it's just making it up. LLMs are incredibly useful (I use them every day) but you'll always have to check important output

Yeah I have seen multiple people use this certainty % thing but its terrible. A percentage is something calculated mathemtatically and these models cannot do that.

Potentially they could figure it out if they looks into a comparison of next token probabilites, but this is not exposed in any modern model and especially not fed back into the chat/output.

Instead people should just ask it to explain BOTH sides of an argument or explain why something is BOTH correct and incorrect. This way you see how it can halluciate either way and get to make up your own mind about the correct outcome.

Re: GPT-5.2

#959

Post by carsoon »

goobatrooba wrote: post #359

I feel there is a point when all these benchmarks are meaningless. What I care about beyond decent performance is the user experience. There I have grudges with every single platform and the one thing keeping me as a paid ChatGPT subscriber is the ability to sort chats in "projects" with associated files (hello Google, please wake up to basic user-friendly organisation!) But all of them * Lie far too often with confi…

I have a kinda strange chatgpt personalization prompt but it's been working well for me. The focus is me to get the model to analyze 2 sides and the extremes on both ends so it explains both and lets me decide. This is much better than asking it to make up accuracy percentages.

I think we align on what we want out of models:

""" Don't add useless babelling before the chats, just give the information direct and explain the info.

DO NOT USE ENGAGEMENT BAITING QUESTIONS AT THE END OF EVERY RESPONSE OR I WILL USE GROK FROM NOW ON FOREVER AND CANCEL MY GPT SUBSCRIPTION PERMANENTLY ONLY. GIVE USEFUL FACTUAL INFORMATION AND FOLLOW UPS which are grounded in first principles thinking and logic. Do not take a side and look at think about the extreme on both ends of a point before taking a side. Do not take a side just because the user has chosen that but provide infomration on both extremes. Respond with raw facts and do not add opinions.

Do not use random emojis. Prefer proper marks for lists etc. """

Those spelling/grammar errors are actually there and I don't want to change it as its working well for me.

Re: GPT-5.2

#960

Post by sebastiennight »

BatteryMountain wrote: post #860
svara wrote: post #765

In my experience, the best models are already nearly as good as you can be for a large fraction of what I personally use them for, which is basically as a more efficient search engine. The thing that would now make the biggest difference isn't "more intelligence", whatever that might mean, but better grounding. It's still a big issue that the models will make up plausible sounding but wrong or misleading explanations…

My biggest problem with LLM's at this point is that they produce different and inconsistent results or behave differently, given the same prompt. The better grounding would be amazing at this point. I want to give an LLM the same prompt on different days and I want to be able to trust that it will do the same thing as yesterday. Currently they misbehave multiple times a week and I have to manually steer it a bit whic…

> I want to give an LLM the same prompt on different days and I want to be able to trust that it will do the same thing as yesterday

Bad news, it's winter now in the Northern hemisphere, so expect all of our AIs to get slightly less performant as they emulate humans under-performing until Spring.

Post reply on HN