Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-4 is getting worse over time, not better

twitter.com

Post reply on HN
Print view View original

151–160 of 315 posts

Re: GPT-4 is getting worse over time, not better

#151

Post by Filligree »

tanaros wrote: post #138
gwd wrote: post #72

Earlier quoted context omitted.

This comment has an interesting take on it, haven't read the paper to verify the take: https://news.ycombinator.com/item?id=36781968 EDIT: FWIW I haven't noticed any such regression. I don't generally use it to find prime numbers, but I do use it for coding, and have been really impressed with what it's able to do. 8 This paper is being misinterpreted. The degradations reported are somewhat peculiar to the authors' t…

> Code generation: the change they report is that the newer GPT-4 adds non-code text to its output. They don't evaluate the correctness of the code. They merely check if the code is directly executable. So the newer model's attempt to be more helpful counted against it. In the prompt they specifically request only the Python code, no other output. An “attempt to be helpful” that directly contradicts the user’s reques…

I guess, but that still isn't the sort of degradation people have been talking about. It's not a useful data point in that regard.

Re: GPT-4 is getting worse over time, not better

#152

Post by kippinitreal »

londons_explore wrote: post #129

Has anyone experimented with mixing outputs from LLm's on a per-token basis? Ie. easy tokens can be provided by a cheap-to-run model, and hard tokens are given by an expensive to run model? A model could be used to decide when it is worth running the expensive model, based on the inputs, output so far, and probability distribution of the output of the cheap model. For example, "Q: If I have 3 bananas and eat none, th…

Clever idea. I think you would have to recompute the context (ie embed the prior tokens) every time you swapped models because the weight distributions would be different for each model. Going from big->small might make this overhead worth it, but going back from small->big would assuredly be very costly.

Re: GPT-4 is getting worse over time, not better

#153

Post by JohnFen »

insanitybit wrote: post #149
JohnFen wrote: post #133

Earlier quoted context omitted.

I never said anyone lied to or tricked anyone. I'm saying that "beta" is a term referring to pre-release testing. If people are paying for it, it's been released and is therefore no longer a beta. Calling it a "beta" at that point is just pure PR.

> If people are paying for it, it's been released and is therefore no longer a beta. no You can define "beta" whatever weird way you want but don't complain when it's not how literally everyone else uses it and don't complain when you're paying for something that uses the term the way everyone else does

My "weird way" is old-school, I admit, but it's not weird. It used to be mainstream. Google led the charge to deprive "beta" of actual meaning. I won't stop pushing back on it, because the redefinition makes it more difficult to talk about technical issues.

> don't complain when you're paying for something that uses the term the way everyone else does

I won't, because I won't pay to be a beta tester in the first place. I don't care what others do.

Re: GPT-4 is getting worse over time, not better

#154

Post by pixl97 »

r3trohack3r wrote: post #142
phillipcarter wrote: post #31

The linked twitter account is an AI influencer, so take whatever is written with a grain of salt. Their goal is to get clicks and views by saying controversial things. This topic has come up before, and my hypothesis is still that GPT-4 hasn't gotten worse, it's just that the magic has worn off as we've used this tech. Studies to evaluate it have gotten better and cleaned up mistakes in the past.

I don’t believe this is true. It’s possible I was blinded by the light, but my programming tasks were previously (during the early access program) being handled by GPT-4 regularly and now they aren’t. I’ve also seen many anecdotes from engineers who had exceptionally early access before GPT-4 was public knowledge. The GPT-4 I use now feels like a shadow of the GPT-4 I used during the early access program. GPT-4, back…

Is this via the ChatGPT web interface, or via the API.

I'm wondering if there is a difference?

Re: GPT-4 is getting worse over time, not better

#155

Post by SirMaster »

robterrell wrote: post #143
xtracto wrote: post #121

Earlier quoted context omitted.

These models have been explicitly nerfed since their first release due to copyright considerations. I've mentioned in two previous cases both for [1] code generation and [2] book summarizing. From my point of view, it is sad that these sort of socio-political constructs (copyright) are hindering innovation. The funny thing is that in say, 10 years, the "pirate" version of LLMs will be way more powerful and useful tha…

> From my point of view, it is sad that these sort of socio-political constructs (copyright) are hindering innovation Why not pay authors of the data the LLM has ingested?

Don't a lot of content creators expect payment per use? Like every time someone streams a song on sportify, the artist gets a few pennies.

So should they be paid a few pennies every time the LLM spits out a response that "used" that training data?

And I am pretty sure it's not even possible to really link the output back to training data anyways.

Re: GPT-4 is getting worse over time, not better

#157

Post by never_inline »

smolder wrote: post #131
never_inline wrote: post #23

Earlier quoted context omitted.

I would like to read on this. Do you have any sauce?

Can I recommend you use the actual word "source", and let the pointless "sauce" meme die its overdue death?

I would like to read on this. Do you have any source?

Re: GPT-4 is getting worse over time, not better

#158

Post by appletrotter »

csmpltn wrote: post #140
Vvector wrote: post #70

Earlier quoted context omitted.

ChatGPT isn't the right tool to use for checking if numbers are prime. It is tuned for conversations. I'd like to see a MathGPT or WolframGPT. The real question is if ChatGPT is worse on math and better elsewhere, or just worse overall. That is still unknown

> "ChatGPT isn't the right tool to use for checking if numbers are prime. It is tuned for conversations." Two months ago they were telling me ChatGPT is coming for everyone - programmers, accountants, technical writers, lawyers, etc. Now we're slowly back to "so here's the thing about LLMs"...

It's been well know from the start that these LLMs aren't optimized for math. I remember reading discussions when it came out.

You weren't paying attention.

Re: GPT-4 is getting worse over time, not better

#159

Post by insanitybit »

JohnFen wrote: post #153
insanitybit wrote: post #149

Earlier quoted context omitted.

> If people are paying for it, it's been released and is therefore no longer a beta. no You can define "beta" whatever weird way you want but don't complain when it's not how literally everyone else uses it and don't complain when you're paying for something that uses the term the way everyone else does

My "weird way" is old-school, I admit, but it's not weird. It used to be mainstream. Google led the charge to deprive "beta" of actual meaning. I won't stop pushing back on it, because the redefinition makes it more difficult to talk about technical issues. > don't complain when you're paying for something that uses the term the way everyone else does I won't, because I won't pay to be a beta tester in the first plac…

> because I won't pay to be a beta tester in the first place.

Apparently you will, because you started this thread with "why am I paying for this beta software?".

Post reply on HN