Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-4 is getting worse over time, not better

twitter.com

Post reply on HN
Print view View original

81–90 of 315 posts

Re: GPT-4 is getting worse over time, not better

#81

Post by insanitybit »

lhl wrote: post #35

I have not read the paper yet (in my backlog, here's the paper: https://arxiv.org/pdf/2307.09009.pdf ), but it's important note that the paper is entitled "How Is ChatGPT’s Behavior Changing over Time?" not that it's necessarily "getting worse." Here's a more nuanced (not an AI clout chasing account) discussion by Arvind Narayanan (Princeton CS prof) about the results: https://twitter.com/random_walker/status/1681489…

> One thing that I have confirmed is while the abstract and intro talk about evaluating "code generation" as if GPT-4 code generation is getting worse, In is 3.3/Figure 4 it says they judge correctness only if it's passing raw code: "We call it directly executable if the online judge accepts the answer" not whether the code snippet is actually correct (!). The latest model outputs code as triple ticked in Markdown: "In June, however, they added extra triple quotes before and after the code snippet, rendering the code not executable." I mean, this is important if you're passing code directly into an API I suppose, but I don't think this should be properly extracted to judge code generation capability.

OK yeah I mean that's kind of critical information, wtf. I've upvoted this because it needs to be the top post. That's massively important - obviously you could go from ~100% to ~0% if your judge can't handle formatting changes.

Re: GPT-4 is getting worse over time, not better

#82

Post by majkinetor »

gwd wrote: post #72
majkinetor wrote: post #63

Earlier quoted context omitted.

So the original paper from Stanford and Berkley is also linked to this AI influencer? I am really amazed by this kind of dismissal. Its totally irrelevant who posted the info and how framed it is, as long as you have access to the source.

This comment has an interesting take on it, haven't read the paper to verify the take: https://news.ycombinator.com/item?id=36781968 EDIT: FWIW I haven't noticed any such regression. I don't generally use it to find prime numbers, but I do use it for coding, and have been really impressed with what it's able to do. 8 This paper is being misinterpreted. The degradations reported are somewhat peculiar to the authors' t…

Fine tuning or not, its definitelly a proof one should not rely on it apart from very specific use cases (like lorem ipsum generator or something).

Re: GPT-4 is getting worse over time, not better

#83

Post by j45 »

brandall10 wrote: post #13
andai wrote: post #6

Earlier quoted context omitted.

I heard GPT-4 described as "eight GPT-3's in a trenchcoat" but I'm not sure how accurate that is.

Several people hinted/remarked (starting w/ George Hotz, then others more closely linked to OpenAI) that it's a Mixture of Experts* approach comprised of 8 220B parameter models. * https://arxiv.org/pdf/2101.03961.pdf

I wonder if additional layers of factories of factories approach can continue to improve it.

I'm not familiar enough with the technology but could it be possible to create a prompt, or multiple prompts to stitch together 8 similtaneous calls to GPT3.5 pulled together and see if the quality is similar to GPT4?

Re: GPT-4 is getting worse over time, not better

#84

Post by chaxor »

kaba0 wrote: post #17

My favorites misstep from GPT-4 was when my friend asked it about the difference between vet bulb temperatures and dry bulb. You see that typo correctly (he was dictating): > The main difference is in what they're measuring. Temperature measurement at a vet is usually taken to determine an animal's body temperature, often done rectally or via the ear. It is direct and generally provides an absolute temperature value.…

This seems more like a response from Bard or Llama. There is always some probability of any of these models responding in a nonsensical way like this however

Re: GPT-4 is getting worse over time, not better

#85

Post by [deleted] »

majkinetor wrote: post #63
phillipcarter wrote: post #31

The linked twitter account is an AI influencer, so take whatever is written with a grain of salt. Their goal is to get clicks and views by saying controversial things. This topic has come up before, and my hypothesis is still that GPT-4 hasn't gotten worse, it's just that the magic has worn off as we've used this tech. Studies to evaluate it have gotten better and cleaned up mistakes in the past.

So the original paper from Stanford and Berkley is also linked to this AI influencer? I am really amazed by this kind of dismissal. Its totally irrelevant who posted the info and how framed it is, as long as you have access to the source.

[deleted]

Re: GPT-4 is getting worse over time, not better

#86

Post by anononaut »

jmkni wrote: post #78
phillipcarter wrote: post #31

The linked twitter account is an AI influencer, so take whatever is written with a grain of salt. Their goal is to get clicks and views by saying controversial things. This topic has come up before, and my hypothesis is still that GPT-4 hasn't gotten worse, it's just that the magic has worn off as we've used this tech. Studies to evaluate it have gotten better and cleaned up mistakes in the past.

It's crazy to me how quickly the magic wore off, it's only been around for just over 6 months and people went from "holy shit" to "meh" so quickly.

It needs to be made open source to have its Stable Diffusion halcyon days.

Re: GPT-4 is getting worse over time, not better

#87

Post by [deleted] »

airstrike wrote: post #11
shmatt wrote: post #4

>Having the behavior of an LLM change over time is not acceptable. By now this is actually funny to read. Never rely on another companies product to make your own product, without accepting things can change overnight and shut you down As Llama2 is self hosted, you can choose which iteration to host. Much better developer experience Edit: to be clear OpenAI is unprofitable, so is Reddit, so was Stadia. Building on to…

That's a pretty defeatist take. Surely if you pay for a service you should expect the provider to be making good faith efforts to provide the same quality of service over time? Natural degradation would be fine, but purposefully sandbagging the service so it gets worse because cheaper is unacceptable. That we have become numb to the point that we collectively accept such poor behavior on the part of vendors in concre…

[deleted]

Re: GPT-4 is getting worse over time, not better

#88

Post by hospitalJail »

phillipcarter wrote: post #31

The linked twitter account is an AI influencer, so take whatever is written with a grain of salt. Their goal is to get clicks and views by saying controversial things. This topic has come up before, and my hypothesis is still that GPT-4 hasn't gotten worse, it's just that the magic has worn off as we've used this tech. Studies to evaluate it have gotten better and cleaned up mistakes in the past.

Are we talking GPT4 or chatgpt4?

Its not disputed that chatGPT4 has degraded in quality as its been 'aligned'.

Re: GPT-4 is getting worse over time, not better

#90

Post by chaxor »

haolez wrote: post #2

Yesterday, while using ChatGPT-4, it gave me a very long answer almost instantly. It felt like I was using ChatGPT-3.5, including the poor quality of the answer. In the following prompts, it became slow again, as GPT-4 is supposed to be. The quality improved as well. I think they are trying some aggressive customization on their infra to try to make it economically viable, but it's just speculation at this point.

If their statement is that they haven't changed the weights, there still an immense number of things they can toy with.

For example, the prompt may add more 'safety' language in it, which can cause strange differences to occur, or typical_p sampling values, top_p, top_k, etc, or if they do use mixture of experts, they may even be able to 'use less experts' and only run 1/4 of the models, such that the speed is improved greatly.

There's plenty of ways to make the model change output without retraining.

Post reply on HN