Ask HN: Is GPT 4's quality lately worst than GPT 3.5?

#1

Post by agonz253 » Tue, Aug 01, 2023, 2:59 PM UTC

Has anyone else encountered this phenomenon lately? I've found myself prompting GPT 3.5 with simple questions that GPT 4 provided an incorrect answer for, and lo and behold I get a much better answer.

For ex this is GPT 4: https://chat.openai.com/share/e24501ad-8f1c-4b5a-a6d0-d933f5d1d209

And this is GPT 3.5: https://chat.openai.com/share/b9372bdc-ffff-4655-bee4-2b3f3c3b8285

In the latter case I didn't even need to ask for the order by clause as it anticipates it and provides an answer for it. GPT 4's first answer was wrong.

In the past two days I've seen at least 2 other cases where GPT 4's answer was plain wrong and GPT 3.5's was not only correct but of very high quality, reminding me of what I first felt when using GPT 4 for the first time.

Re: Ask HN: Is GPT 4's quality lately worst than GPT 3.5?

#2

Post by briga » Tue, Aug 01, 2023, 5:10 PM UTC

One thing to note when making comparisons like this is that LLM output is not deterministic, in the sense that if you ask it the same question 10 times you will get 10 different answers. So the question to ask is not, “is GPT4 better on this one specific question?”, but rather “does GPT4 produce better results on average?”. I would bet that it does, for no other reason than that it is much larger, and LLM performance seems to just scale with size. Also worth noting is that the more detailed your prompt is the better the response will be. Sometimes you have to encourage GPT to get the best results. GPT4 should be able yo handle much more complex and detailed prompts than 3.5

Re: Ask HN: Is GPT 4's quality lately worst than GPT 3.5?

#3

Post by lumost » Tue, Aug 01, 2023, 5:14 PM UTC

briga wrote: post #2

One thing to note when making comparisons like this is that LLM output is not deterministic, in the sense that if you ask it the same question 10 times you will get 10 different answers. So the question to ask is not, “is GPT4 better on this one specific question?”, but rather “does GPT4 produce better results on average?”. I would bet that it does, for no other reason than that it is much larger, and LLM performance…

GPT-4 has a capped number of responses, it also costs $20/month. If it's only marginally better than GPT-3.5, why would I pay for it?

Re: Ask HN: Is GPT 4's quality lately worst than GPT 3.5?

#4

Post by seanhunter » Tue, Aug 01, 2023, 5:17 PM UTC

You will get a much more objective comparison if you use the openai api and set the temperature parameter to zero. You may need to use the azure version of gpt-4. At least for me only gpt-3.5 is available on the “real” openAI vs gpt-4 is definitely available for me on azure.

Re: Ask HN: Is GPT 4's quality lately worst than GPT 3.5?

#5

Post by ezedv » Tue, Aug 01, 2023, 5:18 PM UTC

[flagged]

Re: Ask HN: Is GPT 4's quality lately worst than GPT 3.5?

#6

Post by briga » Tue, Aug 01, 2023, 5:21 PM UTC

lumost wrote: post #3

briga wrote: post #2

One thing to note when making comparisons like this is that LLM output is not deterministic, in the sense that if you ask it the same question 10 times you will get 10 different answers. So the question to ask is not, “is GPT4 better on this one specific question?”, but rather “does GPT4 produce better results on average?”. I would bet that it does, for no other reason than that it is much larger, and LLM performance…

GPT-4 has a capped number of responses, it also costs $20/month. If it's only marginally better than GPT-3.5, why would I pay for it?

That is entirely up to your discretion, if 3.5 fits your use case just use that

Re: Ask HN: Is GPT 4's quality lately worst than GPT 3.5?

#7

Post by tutfbhuf » Tue, Aug 01, 2023, 5:28 PM UTC

For all that have the "feeling" that GPT4 has become worse. Please try Bing Chat in creative mode, it seems that this is using some old GPT4 March Version + Bing Search. In my opinion this is much better than the current Chat GPT4 version from OpenAI.

Re: Ask HN: Is GPT 4's quality lately worst than GPT 3.5?

#8

Post by tmpz22 » Tue, Aug 01, 2023, 5:49 PM UTC

Its almost as if OpenAI used VC resources to artificially boost their products efficacy during early hype and over time we're seeing the product degrade as the company optimizes for efficiency and profitability and removes its subsidies.

Like AirBnB. Or Uber. Etc.

Re: Ask HN: Is GPT 4's quality lately worst than GPT 3.5?

#9

Post by justinlloyd » Tue, Aug 01, 2023, 6:14 PM UTC

tmpz22 wrote: post #8

Its almost as if OpenAI used VC resources to artificially boost their products efficacy during early hype and over time we're seeing the product degrade as the company optimizes for efficiency and profitability and removes its subsidies. Like AirBnB. Or Uber. Etc.

You may be right, but I think it is too early in the season for that to be true at this time, we're not even out of act 1 of the first episode of the season with regard to OpenAI. AI tech may be accelerating, but business still tends to move at glacial speed.

Re: Ask HN: Is GPT 4's quality lately worst than GPT 3.5?

#10

Post by ideamotor » Tue, Aug 01, 2023, 6:16 PM UTC

tmpz22 wrote: post #8

Its almost as if OpenAI used VC resources to artificially boost their products efficacy during early hype and over time we're seeing the product degrade as the company optimizes for efficiency and profitability and removes its subsidies. Like AirBnB. Or Uber. Etc.

Also like Walmart when they open a store next to a competitor and temporarily drop their price. Or any company with capital ever in existence.