Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Experiencing decreased performance with ChatGPT-4

community.openai.com

Post reply on HN
Print view View original

81–90 of 200 posts

Re: Experiencing decreased performance with ChatGPT-4

#81

Post by throwaway121abc »

wincy wrote: post #62
cheeko1234 wrote: post #38

I use the following test to ensure I'm on GPT4 and not 3.5. (I noticed that it did fail at this test temporarily and then got it. Not sure why. Maybe it reverts back to 3.5 when under load?) I have a 12 liter jug and a 6 liter jug. I want to measure 6 liters. How do I do it? GPT4: You actually don't need to do anything because one of your jugs is already a 6-liter jug. If you fill it up to the top, you'll have exactl…

It’s funny you say this as I just asked ChatGPT 4 and got this response. Here is a simple solution to your problem: 1. Fill the 12-liter jug completely. 2. Use the water in the 12-liter jug to fill the 6-liter jug. Now you have 6 liters remaining in the 12-liter jug, which is exactly what you need. So, you have successfully measured 6 liters.

Is GPT 3.5 is nerfed too? ..hehe XD

To measure exactly 6 liters using a 12-liter jug and a 6-liter jug, you can follow these steps:

Start with both jugs empty. Fill the 12-liter jug completely. Pour the contents of the 12-liter jug into the 6-liter jug. This leaves you with 6 liters in the 12-liter jug and a full 6-liter jug. Empty the 6-liter jug. Pour the 6 liters from the 12-liter jug into the empty 6-liter jug. Refill the 12-liter jug. Fill the 6-liter jug using the 12-liter jug until the 6-liter jug is full. This requires pouring 4 liters from the 12-liter jug into the 6-liter jug, which leaves 8 liters in the 12-liter jug. At this point, you have 6 liters in the 6-liter jug and 8 liters in the 12-liter jug. Empty the 6-liter jug. Pour the remaining 2 liters from the 12-liter jug into the empty 6-liter jug. Finally, refill the 12-liter jug and pour 6 liters from the 12-liter jug into the 6-liter jug. The 6-liter jug will now be full, and you will have successfully measured 6 liters using the given jugs.

By following these steps, you can accurately measure 6 liters using a 12-liter jug and a 6-liter jug.

Re: Experiencing decreased performance with ChatGPT-4

#82

Post by lumost »

sillysaurusx wrote: post #2

I’m convinced this is group hallucination. It must be so interesting to work at OpenAI, knowing you didn’t change a thing, and seeing that because of random chance, some small fraction of 100M users have all tricked each other that suddenly, something is different.

My suspicion is that we're collectively becoming accustomed to ChatGPT failures. These failures cause problems, and become more annoying with time. The same thing happened with voice assistants.

That being said, the safety filters have definitively changed in OpenAI. ChatGPT is definitely more prone to reminding me that it is an LLM, and it refuses to participate in pretend play which it perceives as violating its safety filters. As a trivial example, ChatGPT is less willing to generate test cases for security vulnerabilities now - or engage in speculative mathematical discussions. Instead it will simply state that it is an advanced LLM blah blah blah.

Re: Experiencing decreased performance with ChatGPT-4

#83

Post by beepbooptheory »

sillysaurusx wrote: post #2

I’m convinced this is group hallucination. It must be so interesting to work at OpenAI, knowing you didn’t change a thing, and seeing that because of random chance, some small fraction of 100M users have all tricked each other that suddenly, something is different.

At least for some people, it seems to be a (unconscious) way to save face after being so ridiculous with the hype and predictions and "these will replace doctors and lawyers" when this was all first trending.

Re: Experiencing decreased performance with ChatGPT-4

#84

Post by logicprog »

Timpy wrote: post #21
sillysaurusx wrote: post #2

I’m convinced this is group hallucination. It must be so interesting to work at OpenAI, knowing you didn’t change a thing, and seeing that because of random chance, some small fraction of 100M users have all tricked each other that suddenly, something is different.

I remember the first time I played Minecraft and I was in awe at how expansive the play world felt. Without thinking too much about it, I had the feeling that if I set off in any direction I would discover infinitely new things. After enough playtime I saw the repeating patterns and eventually it felt so small again.

This is an excellent theory IMO. It isn't that the AI has actually gotten much worse, it's that the novelty has worn off and they are finally starting to notice all of the repetitious patterns it has and mistakes it makes — the stuff people like me who never bought into the AI hype to begin with noticed from the start — but instead of realizing that maybe their initial impressions of the capabilities of large language models were wrong or based on partial information they are taking the Mandela effect route and just insisting that something outside them has fundamentally changed.

Re: Experiencing decreased performance with ChatGPT-4

#85

Post by zer0c00ler »

sillysaurusx wrote: post #2

I’m convinced this is group hallucination. It must be so interesting to work at OpenAI, knowing you didn’t change a thing, and seeing that because of random chance, some small fraction of 100M users have all tricked each other that suddenly, something is different.

Most definitely not. I doubt there is a single person at OpenAI that knows and controls the whole stack.

Changes happen at many layers, ChatGPT UI, API Gateways, moderation API, backend server hosting model, the model file itself, etc.

Each of these components is changing pretty regularly it seem.

The end result of combined changes for users is being the observed degraded performance of ChatGPT.

Re: Experiencing decreased performance with ChatGPT-4

#86

Post by clarge1120 »

acumenical wrote: post #56
sillysaurusx wrote: post #2

I’m convinced this is group hallucination. It must be so interesting to work at OpenAI, knowing you didn’t change a thing, and seeing that because of random chance, some small fraction of 100M users have all tricked each other that suddenly, something is different.

I don't agree. As someone who has written many jailbreak prompts, the very fact that earlier jailbreak prompts no longer work indicates to me that the integration has changed. The model might be the same, but filtering the input extensively might cause undefined behavior.

Great example!

Re: Experiencing decreased performance with ChatGPT-4

#87

Post by gregw134 »

cheeko1234 wrote: post #38

I use the following test to ensure I'm on GPT4 and not 3.5. (I noticed that it did fail at this test temporarily and then got it. Not sure why. Maybe it reverts back to 3.5 when under load?) I have a 12 liter jug and a 6 liter jug. I want to measure 6 liters. How do I do it? GPT4: You actually don't need to do anything because one of your jugs is already a 6-liter jug. If you fill it up to the top, you'll have exactl…

It figures it out once you let it reflect on its answer: Consider the following situation: You have a 12 liter jug and a 6 liter jug, and you want to measure out exactly 6 liters of water. First, generate an initial solution for this problem. Then, think about the solution you've generated, considering if there might be a simpler or more straightforward way to achieve the goal. If there is, please provide the more accurate or simpler solution.

Re: Experiencing decreased performance with ChatGPT-4

#88

Post by JohnMakin »

logicprog wrote: post #84
Timpy wrote: post #21

Earlier quoted context omitted.

I remember the first time I played Minecraft and I was in awe at how expansive the play world felt. Without thinking too much about it, I had the feeling that if I set off in any direction I would discover infinitely new things. After enough playtime I saw the repeating patterns and eventually it felt so small again.

This is an excellent theory IMO. It isn't that the AI has actually gotten much worse, it's that the novelty has worn off and they are finally starting to notice all of the repetitious patterns it has and mistakes it makes — the stuff people like me who never bought into the AI hype to begin with noticed from the start — but instead of realizing that maybe their initial impressions of the capabilities of large languag…

Pretty sure this is going on to some degree. It seems like there should be some kind of regression testing possible on these systems to definitively prove these claims, rather than these anecdotal stories that seem to rarely ever come with concrete examples.

Re: Experiencing decreased performance with ChatGPT-4

#89

Post by carabiner »

cheeko1234 wrote: post #38

I use the following test to ensure I'm on GPT4 and not 3.5. (I noticed that it did fail at this test temporarily and then got it. Not sure why. Maybe it reverts back to 3.5 when under load?) I have a 12 liter jug and a 6 liter jug. I want to measure 6 liters. How do I do it? GPT4: You actually don't need to do anything because one of your jugs is already a 6-liter jug. If you fill it up to the top, you'll have exactl…

You can't evaluate them with a single prompt, single execution! Any given output is just a sample from a range of possible outputs, but all of them (ALL) are considered plausible returns. You have to think of it as returning a distribution as a random variable (in high dimensional space) but unfortunately there's only 1 sample from it at a time. Maybe some day they'll let us control a random seed.

Re: Experiencing decreased performance with ChatGPT-4

#90

Post by Implicated »

throwaway121abc wrote: post #81
wincy wrote: post #62

Earlier quoted context omitted.

It’s funny you say this as I just asked ChatGPT 4 and got this response. Here is a simple solution to your problem: 1. Fill the 12-liter jug completely. 2. Use the water in the 12-liter jug to fill the 6-liter jug. Now you have 6 liters remaining in the 12-liter jug, which is exactly what you need. So, you have successfully measured 6 liters.

Is GPT 3.5 is nerfed too? ..hehe XD To measure exactly 6 liters using a 12-liter jug and a 6-liter jug, you can follow these steps: Start with both jugs empty. Fill the 12-liter jug completely. Pour the contents of the 12-liter jug into the 6-liter jug. This leaves you with 6 liters in the 12-liter jug and a full 6-liter jug. Empty the 6-liter jug. Pour the 6 liters from the 12-liter jug into the empty 6-liter jug. R…

Ummm... lmfao, this thing is on drugs.

(ChatGPT 4)

You can measure exactly 6 liters using a 12 liter jug and a 6 liter jug by following these steps:

Fill the 6-liter jug completely.

Pour the water from the 6-liter jug into the 12-liter jug. Now you have 6 liters of water in the 12-liter jug.

Fill the 6-liter jug again.

Carefully pour water from the 6-liter jug into the 12-liter jug until the 12-liter jug is full. Since the 12-liter jug already contains 6 liters, you will be able to pour in only another 6 liters to make it full. This leaves you with exactly 6 liters of water in the 6-liter jug.

Congratulations, you now have measured exactly 6 liters of water using a 12-liter jug and a 6-liter jug!

> https://chat.openai.com/share/929e68a3-9c67-44c8-8fbc-b555c1...

Post reply on HN