Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-5.2

openai.com

Post reply on HN
Print view View original

781–790 of 1001 posts

Re: GPT-5.2

#781

Post by kyletns »

phorkyas82 wrote: post #768
svara wrote: post #765

In my experience, the best models are already nearly as good as you can be for a large fraction of what I personally use them for, which is basically as a more efficient search engine. The thing that would now make the biggest difference isn't "more intelligence", whatever that might mean, but better grounding. It's still a big issue that the models will make up plausible sounding but wrong or misleading explanations…

Isn't that what no LLM can provide: being free of hallucinations?

For the record, brains are also not free of hallucinations.

Re: GPT-5.2

#782

Post by bob1029 »

I've been looking really hard at combining Roslyn (.NET compiler platform SDK) with one of these high end tool calling models. The ability to have the LLM create custom analyzers and then verify them with a human in the loop can provide stable, compile-time guarantees of business rules that accumulate without paying for context tokens.

I feel like there is a small chance I could actually make this work in some areas of the business now. 400k is a really big context window. The last time I made any serious attempt I only had 32k tokens to work with. I still don't think these things can build the whole product for you, but if you have a structured configuration abstraction in an existing product, I think there is definitely uplift possible.

Re: GPT-5.2

#783

Post by chuckSu »

svara wrote: post #765

In my experience, the best models are already nearly as good as you can be for a large fraction of what I personally use them for, which is basically as a more efficient search engine. The thing that would now make the biggest difference isn't "more intelligence", whatever that might mean, but better grounding. It's still a big issue that the models will make up plausible sounding but wrong or misleading explanations…

[dead]

Re: GPT-5.2

#784

Post by schmuhblaster »

bob1029 wrote: post #782

I've been looking really hard at combining Roslyn (.NET compiler platform SDK) with one of these high end tool calling models. The ability to have the LLM create custom analyzers and then verify them with a human in the loop can provide stable, compile-time guarantees of business rules that accumulate without paying for context tokens. I feel like there is a small chance I could actually make this work in some areas…

Sounds interesting, could you elaborate a bit on this? (I am experimenting in a similar direction)

Re: GPT-5.2

#785

Post by stacktrace »

svara wrote: post #765

In my experience, the best models are already nearly as good as you can be for a large fraction of what I personally use them for, which is basically as a more efficient search engine. The thing that would now make the biggest difference isn't "more intelligence", whatever that might mean, but better grounding. It's still a big issue that the models will make up plausible sounding but wrong or misleading explanations…

> It's still a big issue that the models will make up plausible sounding but wrong or misleading explanations for things, and verifying their claims ends up taking time. And if it's a topic you don't care about enough, you might just end up misinformed.

Exactly! One important thing LLMs have made me realise deeply is "No information" is better than false information. The way LLMs pull out completely incorrect explanations baffles me - I suppose that's expected since in the end it's generating tokens based on its training and it's reasonable it might hallucinate some stuff, but knowing this doesn't ease any of my frustration.

IMO if LLMs need to focus on anything right now, they should focus on better grounding. Maybe even something like a probability/confidence score, might end up experience so much better for so many users like me.

Re: GPT-5.2

#786

Post by matusp »

goobatrooba wrote: post #359

I feel there is a point when all these benchmarks are meaningless. What I care about beyond decent performance is the user experience. There I have grudges with every single platform and the one thing keeping me as a paid ChatGPT subscriber is the ability to sort chats in "projects" with associated files (hello Google, please wake up to basic user-friendly organisation!) But all of them * Lie far too often with confi…

Consider using structured output. You can define a JSON with specific fields, and LLMs are only used to fill in the values.

https://ai.google.dev/gemini-api/docs/structured-output

Re: GPT-5.2

#787

Post by marcosscriven »

razster wrote: post #697
goobatrooba wrote: post #359

I feel there is a point when all these benchmarks are meaningless. What I care about beyond decent performance is the user experience. There I have grudges with every single platform and the one thing keeping me as a paid ChatGPT subscriber is the ability to sort chats in "projects" with associated files (hello Google, please wake up to basic user-friendly organisation!) But all of them * Lie far too often with confi…

The latest of the big three... OpenAI, Claude, and Google, none of their models are good. I've spent too much time monitoring them than just enjoying them. I've found it easier to run my own local LLM. The latest Gemini release, I gave it another go but only for it to misspell words and drift off into a fantasy world after a few chats with help restructuring guides. ChatGPT has become lazy for some reason and changes…

What are your best local models, and what hardware do you run them on?

Re: GPT-5.2

#788

Post by timmg »

nextworddev wrote: post #687
blitz_skull wrote: post #575

Again I just tap the sign. All of your benchmarks mean nothing to me until you include Claude Sonnet on them. In my experience, GPT hasn’t been able to compete with Claude in years for the daily “economically valuable” tasks I work on.

Claude is pretty trash for anything besides coding

That hasn't been my experience at all. I always wondered if we just get used to how to prompt a given model and that it hard to transition to another.

Re: GPT-5.2

#789

Post by arw0n »

phorkyas82 wrote: post #768
svara wrote: post #765

In my experience, the best models are already nearly as good as you can be for a large fraction of what I personally use them for, which is basically as a more efficient search engine. The thing that would now make the biggest difference isn't "more intelligence", whatever that might mean, but better grounding. It's still a big issue that the models will make up plausible sounding but wrong or misleading explanations…

Isn't that what no LLM can provide: being free of hallucinations?

I think the better word is confabulation; fabricating plausible but false narratives based on wrong memory. Fundamentally, these models try to produce plausible text. With language models getting large, they start creating internal world models, and some research shows they actually have truth dimensions. [0]

I'm not an expert on the topic, but to me it sounds plausible that a good part of the problem of confabulation comes down to misaligned incentives. These models are trained hard to be a 'helpful assistant', and this might conflict with telling the truth.

Being free of hallucinations is a bit too high a bar to set anyway. Humans are extremely prone to confabulations as well, as can be seen by how unreliable eye witness reports tend to be. We usually get by through efficient tool calling (looking shit up), and some of us through expressing doubt about our own capabilities (critical thinking).

[0] https://arxiv.org/abs/2407.12831

Re: GPT-5.2

#790

Post by andai »

svara wrote: post #765

In my experience, the best models are already nearly as good as you can be for a large fraction of what I personally use them for, which is basically as a more efficient search engine. The thing that would now make the biggest difference isn't "more intelligence", whatever that might mean, but better grounding. It's still a big issue that the models will make up plausible sounding but wrong or misleading explanations…

So there's two levels to this problem.

Retrieval.

And then hallucination even in the face of perfect context.

Both are currently unsolved.

(Retrieval's doing pretty good but it's a Rube Goldberg machine of workarounds. I think the second problem is a much bigger issue.)

Post reply on HN