Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-5.2

openai.com

Post reply on HN
Print view View original

871–880 of 1001 posts

Re: GPT-5.2

#871

Post by kznewman »

EastLondonCoder wrote: post #856
biofox wrote: post #830

Earlier quoted context omitted.

I ask for confidence scores in my custom instructions / prompts, and LLMs do surprisingly well at estimating their own knowledge most of the time.

I’m with the people pushing back on the “confidence scores” framing, but I think the deeper issue is that we’re still stuck in the wrong mental model. It’s tempting to think of a language model as a shallow search engine that happens to output text, but that metaphor doesn’t actually match what’s happening under the hood. A model doesn’t “know” facts or measure uncertainty in a Bayesian sense. All it really does is t…

Solid agree. Hallucination for me IS the LLM use case. What I am looking for are ideas that may or may not be true that I have not considered and then I go try to find out which I can use and why.

Re: GPT-5.2

#872

Post by pietz »

saejox wrote: post #752
az226 wrote: post #750

Earlier quoted context omitted.

And here is Gemini 3: https://media.licdn.com/dms/image/v2/D5610AQH7v9MtrZxxug/ima...

This is very impressive. Google really is ahead

They are definitely ahead in multi modality and I'd argue they have been for a long time. Their image understanding was already great, when their core LLM was still terrible.

Re: GPT-5.2

#873

Post by jstummbillig »

jpkw wrote: post #820
flkiwi wrote: post #426

Earlier quoted context omitted.

Someone didn't think so, lol. I debated not saying anything because the AI partisans are just so awful.

I think the above comment was a joke (Claude frequently says that whenever you challenge it, whether you are right or wrong)

At least this once the AI-ism was not spotted.

Re: GPT-5.2

#874

Post by joshribakoff »

striking wrote: post #726
razster wrote: post #697

Earlier quoted context omitted.

The latest of the big three... OpenAI, Claude, and Google, none of their models are good. I've spent too much time monitoring them than just enjoying them. I've found it easier to run my own local LLM. The latest Gemini release, I gave it another go but only for it to misspell words and drift off into a fantasy world after a few chats with help restructuring guides. ChatGPT has become lazy for some reason and changes…

What's to stop you from using the APIs the way you'd like?

The API is a way to access a model, he is criticizing the model not the access the method (at least until the last sentence where he incorrectly implied you can only script a local model, but I don’t think thats a silver bullet, in my experience that is even more challenging than starting with a working agent)

Re: GPT-5.2

#875

Post by rbranson »

BoredPositron wrote: post #842
Tepix wrote: post #834

Earlier quoted context omitted.

> false narratives based on wrong memory I don't think "wrong memory" is accurate, it's missing information and doesn't know it or is trained not to admit it. Checkout the Dwarkesh Podcast episode https://www.dwarkesh.com/p/sholto-trenton-2 starting at 1:45:38 Here is the relevant quote by Trenton Bricken from the transcript: One example I didn't talk about before with how the model retrieves facts: So you say, "What…

Architecture wise the "admit" part is impossible.

Bricken isn’t just making this up. He’s one of the leading researchers in model interpretability. See: https://arxiv.org/abs/2411.14257

Re: GPT-5.2

#876

Post by ben_w »

whalesalad wrote: post #256
breakingcups wrote: post #43

Is it me, or did it still get at least three placements of components (RAM and PCIe slots, plus it's DisplayPort and not HDMI) in the motherboard image[0] completely wrong? Why would they use that as a promotional image? 0: https://images.ctfassets.net/kftzwdyauwt9/6lyujQxhZDnOMruN3f...

to be fair that image has the resolution of a flip phone from 2003

That shouldn't be what causes this problems; if we can see it's wrong despite the low resolution, the AI isn't going to fully replace humans for all tasks involving this kind of thing.

That said, even with this kind of error rate an AI can speed *some* things up, because having a human whose sole job is to ask "is this AI correct?" is easier and cheaper than having one human for "do all these things by hand" followed by someone else whose sole job is to check "was this human output correct?" because a human who has been on a production line for 4 hours and is about ready for a break also makes a certain number of mistakes.

But at the same time, why use a really expensive general-purpose AI like this, instead of a dedicated image model for your domain? Special purpose AI are something you can train on a decent laptop, and once trained will run on a phone at perhaps 10fps give or take what the performance threshold is and how general you need it to be.

If you're in a factory and you're making a lot of some small widget or other (so, not a whole motherboard), having answers faster than the ping time to the LLM may be important all by itself.

And at this point, you can just ask the LLM to write the training setup for the image-to-bounding-box AI, and then you "just" need to feed in the example images.

Re: GPT-5.2

#877

Post by conception »

BatteryMountain wrote: post #860
svara wrote: post #765

In my experience, the best models are already nearly as good as you can be for a large fraction of what I personally use them for, which is basically as a more efficient search engine. The thing that would now make the biggest difference isn't "more intelligence", whatever that might mean, but better grounding. It's still a big issue that the models will make up plausible sounding but wrong or misleading explanations…

My biggest problem with LLM's at this point is that they produce different and inconsistent results or behave differently, given the same prompt. The better grounding would be amazing at this point. I want to give an LLM the same prompt on different days and I want to be able to trust that it will do the same thing as yesterday. Currently they misbehave multiple times a week and I have to manually steer it a bit whic…

You need to change the temperature to 0 and tune your prompts for automated workflows.

Re: GPT-5.2

#878

Post by rafaelmn »

svara wrote: post #765

In my experience, the best models are already nearly as good as you can be for a large fraction of what I personally use them for, which is basically as a more efficient search engine. The thing that would now make the biggest difference isn't "more intelligence", whatever that might mean, but better grounding. It's still a big issue that the models will make up plausible sounding but wrong or misleading explanations…

I constantly see top models (opus 4.5, gemini 3) get a stroke mid task - they will solve the problem correctly in one place, or have a correct solution that needs to be reapplied in context - and then completely miss the mark in another place. "Lack of intelligence" is very much a limiting factor. Gemini especially will get into random reasoning loops - reading thinking traces - it gets unhinged pretty fast.

Not to mention it's super easy to gaslight these models, just asserting something wrong with vaguely plausible explanation and you get no pushback or reasoning validation.

So I know you qualified your post with "for your use case", but personally I would very much like more intelligence from LLMs.

Re: GPT-5.2

#879

Post by Tepix »

BoredPositron wrote: post #842
Tepix wrote: post #834

Earlier quoted context omitted.

> false narratives based on wrong memory I don't think "wrong memory" is accurate, it's missing information and doesn't know it or is trained not to admit it. Checkout the Dwarkesh Podcast episode https://www.dwarkesh.com/p/sholto-trenton-2 starting at 1:45:38 Here is the relevant quote by Trenton Bricken from the transcript: One example I didn't talk about before with how the model retrieves facts: So you say, "What…

Architecture wise the "admit" part is impossible.

Why do you think it's impossible? I just quoted him saying 'by default, it will actually say, "I don't know the answer to this question"'

We already see that ­­- given the right prompting - we can get LLMs to say more often that they don't know things.

Re: GPT-5.2

#880

Post by rsanek »

wooger wrote: post #846
capitainenemo wrote: post #251

Earlier quoted context omitted.

And performs very well on the latest 100 puzzles too, so isn't just learning the data set (unless I guess they routinely index this repo). I wonder how well AIs would do at bracket city. I tried gemini on it and was underwhelmed. It made a lot of terrible connections and often bled data from one level into the next.

> unless I guess they routinely index this repo This sounds like exactly the kind of thing any tech company would do when confronted with a competitive benchmark.

I mean, the repo has <200 stars, it's not like it's so mainstream that you'd expect LLM makers to be watching it actively. If they wanted to game it, they could more easily do that in RL with synthetic data anyway.
Post reply on HN