Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-4 is getting worse over time, not better

twitter.com

Post reply on HN
Print view View original

51–60 of 315 posts

Re: GPT-4 is getting worse over time, not better

#51

Post by shmatt »

AbrahamParangi wrote: post #14
shmatt wrote: post #4

>Having the behavior of an LLM change over time is not acceptable. By now this is actually funny to read. Never rely on another companies product to make your own product, without accepting things can change overnight and shut you down As Llama2 is self hosted, you can choose which iteration to host. Much better developer experience Edit: to be clear OpenAI is unprofitable, so is Reddit, so was Stadia. Building on to…

I assume you run your own servers and operating system too?

Do you remember all of the iOS flashlight apps that just disappeared once Apple added it to the OS level? Perfect example of building on top of someone else's product

Re: GPT-4 is getting worse over time, not better

#53

Post by greatpostman »

superkuh wrote: post #3

Every time a LLM is fine-tuned it gets stupider and less capable compared to the bare model. openai's legal and social ass covering attempts to neuter their model's output via fine tuning have done the same.

Surprised people argue against this. Censoring the model ruins the logic capabilities inside the model, same thing happens to humans

Re: GPT-4 is getting worse over time, not better

#54

Post by _ea1k »

airstrike wrote: post #11
shmatt wrote: post #4

>Having the behavior of an LLM change over time is not acceptable. By now this is actually funny to read. Never rely on another companies product to make your own product, without accepting things can change overnight and shut you down As Llama2 is self hosted, you can choose which iteration to host. Much better developer experience Edit: to be clear OpenAI is unprofitable, so is Reddit, so was Stadia. Building on to…

That's a pretty defeatist take. Surely if you pay for a service you should expect the provider to be making good faith efforts to provide the same quality of service over time? Natural degradation would be fine, but purposefully sandbagging the service so it gets worse because cheaper is unacceptable. That we have become numb to the point that we collectively accept such poor behavior on the part of vendors in concre…

That's the thing, I'm sure there are others who'd report that it has gotten better. Even improvement inevitably makes some things "worse".

Re: GPT-4 is getting worse over time, not better

#55

Post by lhl »

tanseydavid wrote: post #33

>> Unfortunately, the latest version of GPT-4 did not generate intermediate steps and instead answered incorrectly with a simple "No." This is almost shocking to me. Can anyone confirm or deny seeing the same behavior? (i.e. refusing to do Chain-of-Thinking output)

Hmm, I tested twice with GPT-4 and it got the answer wrong both times but it did think step by step...

The first time it gets to "17077 divided by 13 is exactly 1313, a whole number." [1] and the second time it concludes "However, 17077 ÷ 131 = 130.279, which is an integer. This means 131 is a factor of 17077."[2] (lol)

Of course, Code Interpreter runs the Python code to do the math and gets it correct: https://chat.openai.com/share/ed88a2c6-c421-418c-9e23-0a00d1...

[1] https://chat.openai.com/share/c7ed951e-99b2-4009-ae00-76e11a...

[2] https://chat.openai.com/share/258c0409-03a2-4c41-a02f-76b4cc...

Re: GPT-4 is getting worse over time, not better

#56

Post by jeremyjh »

phillipcarter wrote: post #31

The linked twitter account is an AI influencer, so take whatever is written with a grain of salt. Their goal is to get clicks and views by saying controversial things. This topic has come up before, and my hypothesis is still that GPT-4 hasn't gotten worse, it's just that the magic has worn off as we've used this tech. Studies to evaluate it have gotten better and cleaned up mistakes in the past.

So you are just going to ignore the data (not anecdotes) presented in the SP?

Re: GPT-4 is getting worse over time, not better

#57

Post by simonw »

capableweb wrote: post #48

Title: "GPT-4 is getting worse over time, not better" Paper title: "How Is ChatGPT’s Behavior Changing over Time?" Paper Abstract: "GPT-3.5 and GPT-4 are the two most widely used large language model (LLM) services." When are people gonna realize that GPT-4/3.5 != ChatGPT As far as I can tell, the paper doesn't explain the methodology either, so hard to know if they're actually using "raw" GPT-4 or GPT-4 via ChatGPT.…

They do explain their methodology in some detail in the accompanying GitHub repo: https://github.com/lchen001/LLMDrift

They seem to have been taking to the API directly and requesting the two different model snapshots.

I'm not convinced by their methodology generally. It looks like everything may have been run with temperature 0.1, which I don't think reflects most real-world usage for example.

Re: GPT-4 is getting worse over time, not better

#58

Post by ecshafer »

komodo wrote: post #16

There is a Chatbot AI product called character.ai that has suffered a marked decline in quality since its launch as they battle their users to maintain the AI’s safety protocols (similar to chatGPT “jailbreaks”). I wonder if something similar could be happening here.

These fighting against people using their product in “unauthorized” ways by the ai companies doesn’t make any sense to me. Who cares if character.ai users do some weird stuff with it, or replika creates romantic relationships, or people make off color jokes in gpt. There seems to be a lot of engineering effort driven by some product managers to have the ai do very specific things which makes the product much worse in the Long term.

Re: GPT-4 is getting worse over time, not better

#59

Post by beebmam »

jeremyjh wrote: post #56
phillipcarter wrote: post #31

The linked twitter account is an AI influencer, so take whatever is written with a grain of salt. Their goal is to get clicks and views by saying controversial things. This topic has come up before, and my hypothesis is still that GPT-4 hasn't gotten worse, it's just that the magic has worn off as we've used this tech. Studies to evaluate it have gotten better and cleaned up mistakes in the past.

So you are just going to ignore the data (not anecdotes) presented in the SP?

[flagged]

Re: GPT-4 is getting worse over time, not better

#60

Post by peddling-brink »

capableweb wrote: post #48

Title: "GPT-4 is getting worse over time, not better" Paper title: "How Is ChatGPT’s Behavior Changing over Time?" Paper Abstract: "GPT-3.5 and GPT-4 are the two most widely used large language model (LLM) services." When are people gonna realize that GPT-4/3.5 != ChatGPT As far as I can tell, the paper doesn't explain the methodology either, so hard to know if they're actually using "raw" GPT-4 or GPT-4 via ChatGPT.…

Which is better? I assumed they were the same. I’ve been getting ok results with chat-gpt4, might I get better results with the api gpt4?
Post reply on HN