Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-5.2

openai.com

Post reply on HN
Print view View original

841–850 of 1001 posts

Re: GPT-5.2

#841

Post by intended »

svara wrote: post #829
intended wrote: post #817

Earlier quoted context omitted.

Doubt it. I suspect it’s fundamentally not possible in the spirit you intend it. Reality is perfectly fine with deception and inaccuracy. For language to magically be self constraining enough to only make verified statements is… impossible.

Take a look at the new experimental AI mode in Google scholar, it's going in the right direction. It might be true that a fundamental solution to this issue is not possible without a major breakthrough, but I'm sure you can get pretty far with better tooling that surfaces relevant sources, and that would make a huge difference.

So lets run it through the rubric test -

What’s your level of expertise in this domain or subject? How did you use it? What were your results?

It’s basically gauging expertise vs usage to pin down the variance that seems endemic to LLM utility anecdotes/examples. For code examples I also ask which language was used, the submitters familiarity with the language, their seniority/experience and familiarity with the domain.

Re: GPT-5.2

#842

Post by BoredPositron »

Tepix wrote: post #834
arw0n wrote: post #789

Earlier quoted context omitted.

I think the better word is confabulation; fabricating plausible but false narratives based on wrong memory. Fundamentally, these models try to produce plausible text. With language models getting large, they start creating internal world models, and some research shows they actually have truth dimensions. [0] I'm not an expert on the topic, but to me it sounds plausible that a good part of the problem of confabulatio…

> false narratives based on wrong memory I don't think "wrong memory" is accurate, it's missing information and doesn't know it or is trained not to admit it. Checkout the Dwarkesh Podcast episode https://www.dwarkesh.com/p/sholto-trenton-2 starting at 1:45:38 Here is the relevant quote by Trenton Bricken from the transcript: One example I didn't talk about before with how the model retrieves facts: So you say, "What…

Architecture wise the "admit" part is impossible.

Re: GPT-5.2

#843

Post by throw310822 »

jacobedawson wrote: post #714
Workaccount2 wrote: post #666

Earlier quoted context omitted.

Using my parents as a reference, they just thought it was neat when I showed them GPT-4 years ago. My jaw was on the floor for weeks, but most regular folks I showed had a pretty "oh thats kinda neat" response. Technology is already so insane and advanced that most people just take it as magic inside boxes, so nothing is surprising anymore. It's all equally incomprehensible already.

This mirrors my experience, the non-technical people in my life either shrugged and said 'oh yeah that's cool' or started pointing out gnarly edge cases where it didn't work perfectly. Meanwhile as a techie my mind was (and still is) spinning with the shock and joy of using natural human language to converse with a super-humanly adept machine.

I don't think the divide is between technical and non-technical people. HN is full of people that are weirdly, obstinately dismissive of LLMs (stochastic parrots, glorified autocompletes, AI slop, etc.). Personal anecdote: my father (85yo, humanistic culture) was astounded by the perfectly spot-on analysis Claude provided of a poetic text he had written. He was doubly astounded when, showing Claude's analysis to a close friend, he reacted with complete indifference as if it were normal for computers to competently discuss poetry.

Re: GPT-5.2

#844

Post by energy123 »

sigmoid10 wrote: post #836
jbkkd wrote: post #769

A new model doesn't address the fundamental reliability issues with OpenAI's enterprise tier. As an enterprise customer, the experience has been disappointing. The platform is unstable, support is slow to respond even when escalated to account managers, and the UI is painfully slow to use. There are also baffling feature gaps, like the lack of connectors for custom GPTs. None of the major providers have a perfect ent…

Which tier are you? We are on the highest enterprise tier and I've found that OpenAI is a much more stable platform for high-usage than other providers. Can't say much about the UI though since I almost exclusively work with the API. I feel like UIs generally suck everywhere unless you want to do really generic stuff.

ChatGPT UI is leagues above Gemini and AI Studio in responsiveness and latency which is what I care about.

Re: GPT-5.2

#845

Post by drclau »

biofox wrote: post #830
stacktrace wrote: post #785

Earlier quoted context omitted.

> It's still a big issue that the models will make up plausible sounding but wrong or misleading explanations for things, and verifying their claims ends up taking time. And if it's a topic you don't care about enough, you might just end up misinformed. Exactly! One important thing LLMs have made me realise deeply is "No information" is better than false information. The way LLMs pull out completely incorrect explana…

I ask for confidence scores in my custom instructions / prompts, and LLMs do surprisingly well at estimating their own knowledge most of the time.

How do you know the confidence scores are not hallucinated as well?

Re: GPT-5.2

#846

Post by wooger »

capitainenemo wrote: post #251
Donald wrote: post #243

Earlier quoted context omitted.

Gemini 3 Pro Preview gets 96.8% on the same benchmark? That's impressive

And performs very well on the latest 100 puzzles too, so isn't just learning the data set (unless I guess they routinely index this repo). I wonder how well AIs would do at bracket city. I tried gemini on it and was underwhelmed. It made a lot of terrible connections and often bled data from one level into the next.

> unless I guess they routinely index this repo

This sounds like exactly the kind of thing any tech company would do when confronted with a competitive benchmark.

Re: GPT-5.2

#847

Post by Libidinalecon »

rimeice wrote: post #793
kyletns wrote: post #781

Earlier quoted context omitted.

For the record, brains are also not free of hallucinations.

I still don’t really get this argument/excuse for why it’s acceptable that LLMs hallucinate. These tools are meant to support us, but we end up with two parties who are, as you say, prone to “hallucination” and it becomes a situation of the blind leading the blind. Ideally in these scenarios there’s at least one party with a definitive or deterministic view so the other party (i.e. us) at least has some trust in the…

"The airplane wing broke and fell off during flight"

"Well humans break their leg too!"

It is just a mindlessly stupid response and a giant category error.

The way an airplane wing and a human limb is not at all the same category.

There is even another layer to this that comparing LLMs to the brain might be wrong because the mereological fallacy is attributing the brain "thinks" vs the person/system as a whole thinks.

Re: GPT-5.2

#848

Post by dfsegoat »

drclau wrote: post #845
biofox wrote: post #830

Earlier quoted context omitted.

I ask for confidence scores in my custom instructions / prompts, and LLMs do surprisingly well at estimating their own knowledge most of the time.

How do you know the confidence scores are not hallucinated as well?

they 100% are unless you provide a RUBRIC / basically make it ordinal.

"Return a score of 0.0 if ...., Return a score of 0.5 if .... , Return a score of 1.0 if ..."

Re: GPT-5.2

#849

Post by kiliankoe »

drclau wrote: post #845
biofox wrote: post #830

Earlier quoted context omitted.

I ask for confidence scores in my custom instructions / prompts, and LLMs do surprisingly well at estimating their own knowledge most of the time.

How do you know the confidence scores are not hallucinated as well?

They are, the model has no inherent knowledge about its confidence levels, it just adds plausible-sounding numbers. Obviously they _can_ be plausible, but trusting these is just another level up from trusting the original output.

I read a comment here a few weeks back that LLMs always hallucinate, but we sometimes get lucky when the hallucinations match up with reality. I've been thinking about that a lot lately.

Re: GPT-5.2

#850

Post by dmd »

austhrow743 wrote: post #633
nullbound wrote: post #360

Earlier quoted context omitted.

I will say that it is wild, if not somewhat problematic that two users have such disparate views of seemingly the same product. I say that, but then I remember my own experience just from few days ago. I don't pay for gemini, but I have paid chatgpt sub. I tested both for the same product with seemingly same prompt and subbed chatgpt subjectively beat gemini in terms of scope, options and links with current decent de…

We've been having trouble telling if people are using the same product ever since Chat GPT first got popular. The had a free model and a paid model, that was it, no other competitors or naming schemes to worry about, and discussions were still full of people talking about current capabilities without saying what model they were using. For me, "gemini" currently means using this model in the llm.datasette.io cli tool.…

FWIW i’m always using 5.1 Thinking.
Post reply on HN