Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-4 is getting worse over time, not better

twitter.com

Post reply on HN
Print view View original

11–20 of 315 posts

Re: GPT-4 is getting worse over time, not better

#11

Post by airstrike »

shmatt wrote: post #4

>Having the behavior of an LLM change over time is not acceptable. By now this is actually funny to read. Never rely on another companies product to make your own product, without accepting things can change overnight and shut you down As Llama2 is self hosted, you can choose which iteration to host. Much better developer experience Edit: to be clear OpenAI is unprofitable, so is Reddit, so was Stadia. Building on to…

That's a pretty defeatist take. Surely if you pay for a service you should expect the provider to be making good faith efforts to provide the same quality of service over time? Natural degradation would be fine, but purposefully sandbagging the service so it gets worse because cheaper is unacceptable.

That we have become numb to the point that we collectively accept such poor behavior on the part of vendors in concrete cases does not make the behavior acceptable in general.

Re: GPT-4 is getting worse over time, not better

#12

Post by 79a6ed87 »

I've been paying for GPT-4 since 3 hours after its release. The decrease in quality was noticeable just one week later (on top of the cap changes from 50 messages every 4 hours to 25 messages every 3 hours)

I originally assumed that this was due to the increase in demand. It never went back to being as sharp as it was during those first hours of usage

Re: GPT-4 is getting worse over time, not better

#13

Post by brandall10 »

andai wrote: post #6
haolez wrote: post #2

Yesterday, while using ChatGPT-4, it gave me a very long answer almost instantly. It felt like I was using ChatGPT-3.5, including the poor quality of the answer. In the following prompts, it became slow again, as GPT-4 is supposed to be. The quality improved as well. I think they are trying some aggressive customization on their infra to try to make it economically viable, but it's just speculation at this point.

I heard GPT-4 described as "eight GPT-3's in a trenchcoat" but I'm not sure how accurate that is.

Several people hinted/remarked (starting w/ George Hotz, then others more closely linked to OpenAI) that it's a Mixture of Experts* approach comprised of 8 220B parameter models.

* https://arxiv.org/pdf/2101.03961.pdf

Re: GPT-4 is getting worse over time, not better

#14

Post by AbrahamParangi »

shmatt wrote: post #4

>Having the behavior of an LLM change over time is not acceptable. By now this is actually funny to read. Never rely on another companies product to make your own product, without accepting things can change overnight and shut you down As Llama2 is self hosted, you can choose which iteration to host. Much better developer experience Edit: to be clear OpenAI is unprofitable, so is Reddit, so was Stadia. Building on to…

I assume you run your own servers and operating system too?

Re: GPT-4 is getting worse over time, not better

#15

Post by hermannj314 »

It is worse in determining if a number is prime and on leetcode? Who cares?

Sounds like a "in rats" study. Not sure how those use cases relate at all to how most users use gpt.

I am happy with GPT4. It is doing absolutely wonderful for my use cases. When it comes to where I spend my money, n=1 is a valid sample size.

Re: GPT-4 is getting worse over time, not better

#16

Post by komodo »

There is a Chatbot AI product called character.ai that has suffered a marked decline in quality since its launch as they battle their users to maintain the AI’s safety protocols (similar to chatGPT “jailbreaks”). I wonder if something similar could be happening here.

Re: GPT-4 is getting worse over time, not better

#17

Post by kaba0 »

My favorites misstep from GPT-4 was when my friend asked it about the difference between vet bulb temperatures and dry bulb. You see that typo correctly (he was dictating):

> The main difference is in what they're measuring. Temperature measurement at a vet is usually taken to determine an animal's body temperature, often done rectally or via the ear. It is direct and generally provides an absolute temperature value.

> A dry bulb temperature, on the other hand, is a meteorological term that refers to the temperature of the air as measured by a standard thermometer exposed to the air but shielded from radiation and moisture. It does not take into account the effects of humidity or other factors and it is used for weather forecasting and climate studies.

Ridiculous. Like the first version of Google would have realized it’s likely a typo.. I ain’t fearing for my job anytime soon.

Post reply on HN