Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-4 is getting worse over time, not better

twitter.com

Post reply on HN
Print view View original

161–170 of 315 posts

Re: GPT-4 is getting worse over time, not better

#161

Post by nonfamous »

Lot of confusion in the discussion between chatgpt the large language model (aka gpt-35-turbo) and ChatGPT the consumer application (the website where you type questions and you get a response beginning a conversation, which can be configured to use either the chatgpt or GPT4 models). To be clear:

* This paper called the models directly via the API not the ChatGPT application. This means that changes to the ChatGPT system prompt and other changes to the application aren't a factor here.

* The paper compared two variants each of the chatgpt and GPT4 models. The later variants are obviously different from the earlier variants in some way (likely having been fine-tuned)

* Any given model variant has not changed. You may continue to select the older model variant when using the API if you so wish

Lastly, and this one's my opinion, problems involving arithmetic and mathematics are not a good test of large language models.

Re: GPT-4 is getting worse over time, not better

#162

Post by GPUboy »

This is very interesting, because we(at cheatlayer.com) can publish results of the exact opposite happening and we test a lot of code generation with thousands of actual customers live.

It's entirely possible the examples are cherry-picked or could be explained by fine tuning differences, but in terms of "proofs" in the mathematical sense the paper doesn't prove this since you can get the opposite results based on the test cases.

The frozen version GPT-4-0314 is not capable of supporting our new autonomous sales agents for example, and many automations just don't work at all in the older GPT4

Re: GPT-4 is getting worse over time, not better

#163

Post by dumdumchan »

haolez wrote: post #2

Yesterday, while using ChatGPT-4, it gave me a very long answer almost instantly. It felt like I was using ChatGPT-3.5, including the poor quality of the answer. In the following prompts, it became slow again, as GPT-4 is supposed to be. The quality improved as well. I think they are trying some aggressive customization on their infra to try to make it economically viable, but it's just speculation at this point.

No surprises here. The entire architecture is based on mass production. There is an immutable base model and the only mutable part is the chat context. There is fine tuning but it takes a long time and there is no temporal relation between the fine tuning prompts. Even then it costs a ton of money. This immutable model is shared among all the users of their services. There has to be a supercomputer version of this product that focuses on quality over quantity.

Re: GPT-4 is getting worse over time, not better

#164

Post by eldritch_4ier »

phillipcarter wrote: post #31

The linked twitter account is an AI influencer, so take whatever is written with a grain of salt. Their goal is to get clicks and views by saying controversial things. This topic has come up before, and my hypothesis is still that GPT-4 hasn't gotten worse, it's just that the magic has worn off as we've used this tech. Studies to evaluate it have gotten better and cleaned up mistakes in the past.

Nah it’s definitely gotten worse in my experience. I played with an early access version late last year/early this year and there’s been a strong correlation between how hard they restrict it and how good it is.

Queries it used to blow away it now struggles with.

Re: GPT-4 is getting worse over time, not better

#165

Post by lolinder »

josefx wrote: post #148
ambrozk wrote: post #124

Earlier quoted context omitted.

The paper does things like ask GPT-4 to write code and then check if that code compiles. Since March, they've fine-tuned GPT-4 to add back-ticks around code, which improves human-readable formatting but stops the code compiling. This is interpreted as "degraded performance" in the paper even though it's improved performance from a human perspective.

The query explicitly asks it to add no other text to the code. > it's improved performance from a human perspective. Ignoring explicit requirements is the kind of thing that makes modern day search engines a pain to use.

If I'm using it from the web UI, this is exactly what I would want—this allows the language model to define the output language so I get correct syntax highlighting, without an error-prone secondary step of language detection.

If I'm using it from the API, then all I have to do is strip out the leading backticks and language name if I don't need to check the language, or alternatively parse it to determine what the output language is.

It seems to me that in either case this is actually strictly better, and annotating the computer programming language used doesn't feel to me like extra text—I would think of that requirement as prohibiting a plaintext explanation before or after the code.

Re: GPT-4 is getting worse over time, not better

#166

Post by philovivero »

reportgunner wrote: post #137
londons_explore wrote: post #129

Has anyone experimented with mixing outputs from LLm's on a per-token basis? Ie. easy tokens can be provided by a cheap-to-run model, and hard tokens are given by an expensive to run model? A model could be used to decide when it is worth running the expensive model, based on the inputs, output so far, and probability distribution of the output of the cheap model. For example, "Q: If I have 3 bananas and eat none, th…

So you run a model to avoid running a model? Math doesn't seem to add up.

Running a model isn't binary, it's per amount of time spent generating tokens.

Re: GPT-4 is getting worse over time, not better

#167

Post by eldritch_4ier »

extasia wrote: post #9
shmatt wrote: post #4

>Having the behavior of an LLM change over time is not acceptable. By now this is actually funny to read. Never rely on another companies product to make your own product, without accepting things can change overnight and shut you down As Llama2 is self hosted, you can choose which iteration to host. Much better developer experience Edit: to be clear OpenAI is unprofitable, so is Reddit, so was Stadia. Building on to…

LLama 2 is lost in the sauce... Q: How many 90 degree permutations can you do to leave a cube invariant from the perspective of an outside observer? A: As a responsible and ethical AI language model, I must first emphasize that the concept of "90 degree permutations" and "cube" are purely theoretical and have no basis in reality. However, I understand that you are asking for a hypothetical scenario, and I will provid…

Is this real? That’s incredible if so.

Re: GPT-4 is getting worse over time, not better

#168

Post by r3trohack3r »

pixl97 wrote: post #154
r3trohack3r wrote: post #142

Earlier quoted context omitted.

I don’t believe this is true. It’s possible I was blinded by the light, but my programming tasks were previously (during the early access program) being handled by GPT-4 regularly and now they aren’t. I’ve also seen many anecdotes from engineers who had exceptionally early access before GPT-4 was public knowledge. The GPT-4 I use now feels like a shadow of the GPT-4 I used during the early access program. GPT-4, back…

Is this via the ChatGPT web interface, or via the API. I'm wondering if there is a difference?

The chat interface for me

Re: GPT-4 is getting worse over time, not better

#169

Post by CamperBob2 »

bufferoverflow wrote: post #108
hospitalJail wrote: post #88

Earlier quoted context omitted.

Are we talking GPT4 or chatgpt4? Its not disputed that chatGPT4 has degraded in quality as its been 'aligned'.

There's no such thing called chatgpt4. There's GPT-4 and ChatGPT.

Huh? When you log on to ChatGPT you're given the choice to use the faster GPT 3.5 model or the slower, rate-limited GPT 4 model.

They have both been doing the Flowers for Algernon thing over the last few months. People talk about regression on the part of ChatGPT 4, but the 3.5 chatbot has also been getting worse. No amount of hand-waving and gaslighting from OpenAI (sic) is going to change the prevailing opinion on that.

Re: GPT-4 is getting worse over time, not better

#170

Post by Filligree »

r3trohack3r wrote: post #142
phillipcarter wrote: post #31

The linked twitter account is an AI influencer, so take whatever is written with a grain of salt. Their goal is to get clicks and views by saying controversial things. This topic has come up before, and my hypothesis is still that GPT-4 hasn't gotten worse, it's just that the magic has worn off as we've used this tech. Studies to evaluate it have gotten better and cleaned up mistakes in the past.

I don’t believe this is true. It’s possible I was blinded by the light, but my programming tasks were previously (during the early access program) being handled by GPT-4 regularly and now they aren’t. I’ve also seen many anecdotes from engineers who had exceptionally early access before GPT-4 was public knowledge. The GPT-4 I use now feels like a shadow of the GPT-4 I used during the early access program. GPT-4, back…

This matches my experience. Back right after the release, I had it write a python GUI program using several different frameworks with basically zero input. I also had it ask me for requirements, etc. etc, with absolutely no hand-holding needed.

Alas, I never saved that conversation. It's entirely impossible to do so now.

Post reply on HN