Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Has degradation in the quality of ChatGPT and Claude been proven?

news.ycombinator.com

Post reply on HN
Print view View original

41–44 of 44 posts

Re: Ask HN: Has degradation in the quality of ChatGPT and Claude been proven?

#41

Post by muzani »

I've screenshot an amazing response by GPT-4 on a team Slack. Basically, I took a screenshot of a frustrating error, gave it some code, and it would find the error from an opaque message. 6 months later, it was impossible to replicate. Seems foolproof evidence to me.

We also have a side-by-side UAT comparison of Claude Sonnet 3 and Sonnet 3.5 where 3.5 tends to make wrong assumptions and yet more likely to flag itself as unsure and asking more questions. It could be a problem with our instructions more than the model itself.

There's been a lot of gaslighting from the OpenAI community though. The Claude community at least acknowledge them and encourages people to report them.

Some of the overactive rejections on Claude is related to the different prompts used in Artifacts. 3.5 is also a lot stricter with instructions.

If you want something that doesn't change, use open source.

Re: Ask HN: Has degradation in the quality of ChatGPT and Claude been proven?

#42

Post by afpx »

kolinko wrote: post #7
supriyo-biswas wrote: post #6

One of the things I've personally observed is that ChatGPT has become very verbose these days. Previously, it used to return the right amount of information in most contexts, and I can't get that behavior back with prompts asking it to be concise, because then it'll just omit important parts, prioritizing providing a extremely high-level summary that elucidates very little. No opinion on Claude because I've not had a…

+1 on verbosity - it happened when switching from 4t to 4o I think, and personally I don’t like it. Should be fizable with system prompt though.

I don't know if this is still the case, but there was a point where they were trucating and summarizing system prompts. For instance, you could ask the chat session what the current system prompt was, and it would respond with something somewhat similar to yours but much more general and vague.

Re: Ask HN: Has degradation in the quality of ChatGPT and Claude been proven?

#43

Post by afpx »

nobodywillobsrv wrote: post #23
kolinko wrote: post #7

Earlier quoted context omitted.

+1 on verbosity - it happened when switching from 4t to 4o I think, and personally I don’t like it. Should be fizable with system prompt though.

Can't we all just go test the responses with old chats?

I've tested old chats with the latest 4 and 4o models, and what had been zero-shot now sometimes can't even be done (or at least not without carefully guiding it to the answer).

My old chats say they have been migrated to 4o. But, I swear (can't confirm) that they perform better than a new 4o session. I haven't had time yet, but I wanted to side-by-side compare the responses from those old chats with the current 4o model.

Re: Ask HN: Has degradation in the quality of ChatGPT and Claude been proven?

#44

Post by h_tbob »

supriyo-biswas wrote: post #6

One of the things I've personally observed is that ChatGPT has become very verbose these days. Previously, it used to return the right amount of information in most contexts, and I can't get that behavior back with prompts asking it to be concise, because then it'll just omit important parts, prioritizing providing a extremely high-level summary that elucidates very little. No opinion on Claude because I've not had a…

If you use their developer portal/playground you can save a preset with model and system prompt like “you don’t say more than you need to”

Then you bookmark the url and get shorter replies.

Post reply on HN