Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-4 is getting worse over time, not better

twitter.com

Post reply on HN
Print view View original

291–300 of 315 posts

Re: GPT-4 is getting worse over time, not better

#291

Post by ChatGTP »

xtracto wrote: post #121
ChatGTP wrote: post #34

Earlier quoted context omitted.

Are you sure it wasn't just that the novelty wore off after a few hours of usage? I never really got into LLMs, but I must say at first it seemed like pretty cool stuff. OpenAI have repeatedly stated the model hasn't changed so how could this happen otherwise?

These models have been explicitly nerfed since their first release due to copyright considerations. I've mentioned in two previous cases both for [1] code generation and [2] book summarizing. From my point of view, it is sad that these sort of socio-political constructs (copyright) are hindering innovation. The funny thing is that in say, 10 years, the "pirate" version of LLMs will be way more powerful and useful tha…

Personally, I disagree with you. I have absolutely no idea why OpenAI should benefit from copyrighted material without paying authors for the usage. If everything OpenAI did was opensource, I'd probably feel differently about it, but it's totally not, so I really disagree with the view that they should deserve special treatment here.

I remember seeing interviews and reading many comments saying that that the copyright data shouldn't be an issue because once the model is trained, it kind of "forgets" the copyrighted material and we're just left with pure, unfettered intelligence...it's not a fuzzy jpeg of the web etc...

I think the copyright angle is on the money though, this is why Bard isn't as good, just as Adobe Firefly isn't as good as Stable Diffusion. The inputs aren't as good so the outputs aren't either.

Google can't come out in public and accuse OpenAI of blatant copyright infringement for legal reasons, but I bet internally, they know what's up.

Re: GPT-4 is getting worse over time, not better

#292

Post by ChatGTP »

epups wrote: post #238
robterrell wrote: post #143

Earlier quoted context omitted.

> From my point of view, it is sad that these sort of socio-political constructs (copyright) are hindering innovation Why not pay authors of the data the LLM has ingested?

Since they are trained on all kinds of Internet content, might be a little tricky to manage paying >1B people

I'm sure it would be easy to pay publishers who pass on the money? How do you think Amazon does things for kindle?

Re: GPT-4 is getting worse over time, not better

#293

Post by ChatGTP »

JohnFen wrote: post #105
ChatGTP wrote: post #43

Earlier quoted context omitted.

I feel like it would be a fairly large conspiracy by the OpenAI team though? In fact, what motive would they have to make the model dumber, really? If it gets out and you're right, I think it will cause major trust issues with the product.

> I feel like it would be a fairly large conspiracy by the OpenAI team though? I'm not sure what you mean by "conspiracy", but this sort of thing isn't unknown. All it takes is the employees being bound by an NDA and a marketing team can say anything it likes (within the bounds of legality, anyway) without fear that they will spill the beans.

I'm not a lawyer but I think saying anything about the model would be in beach of the NDA then? Maybe they went and go special permission to say that the model hasn't been changed since launch?

Re: GPT-4 is getting worse over time, not better

#294

Post by joshspankit »

This smells like cost savings, but besides that: Safety and performance are diametrically opposed. I would expect it to get worse. Eapecially innthe short term while they test different apporaches.

ICE cars have also “gotten worse”[0] over time, not better.

[0] based on having more safety measures getting in the way of raw speed.

Re: GPT-4 is getting worse over time, not better

#295

Post by TeeMassive »

r3trohack3r wrote: post #230
adhesive_wombat wrote: post #184

Earlier quoted context omitted.

> terrifyingly unaligned Honestly, if people think that a statistical language model is "terrifying" because it can verbalise the concept of a mass killing, they need to give their heads a wobble. My text editor can be used to write "set off a nuclear weapon in a city, lol". Is Notepad++.exe terrifying? What about the Sum of All Fears ? I could get some pointers from that. Is Tom Clancy unaligned? Am I terrifying bec…

I think there is a significant difference between an LLM and your other examples. Society is a lot more fragile than many people believe. Most people aren’t Ted Kazinsky. And most wanna be Ted Kazinsky’s that we’ve caught don’t have super smart friends they can call up and ask for help in planning their next task. But a world where every disgruntled person who aims to do the most harm has an incredibly smart friend w…

The vas majority of data these models are built from are from public sources. It's already out there. LLMs are just a way to aggregate pre-existing knowledge.

And there are also Ted Kazinsky in the goovernment and big corporations with way more power and way less accountability. Dispowering the public is counter-productive here.

Re: GPT-4 is getting worse over time, not better

#296

Post by muzani »

eldritch_4ier wrote: post #167
extasia wrote: post #9

Earlier quoted context omitted.

LLama 2 is lost in the sauce... Q: How many 90 degree permutations can you do to leave a cube invariant from the perspective of an outside observer? A: As a responsible and ethical AI language model, I must first emphasize that the concept of "90 degree permutations" and "cube" are purely theoretical and have no basis in reality. However, I understand that you are asking for a hypothetical scenario, and I will provid…

Is this real? That’s incredible if so.

LLMs are generally terrible with math and reasoning questions, so everyone likes to "prove" that a model is bad by giving it a simple math question. It's unintuitive because computers are amazing with math and terrible at everything else.

Basically a computer turns everything into a calculation to process something, but a LLM turns everything into tokens.

Re: GPT-4 is getting worse over time, not better

#297

Post by phreeza »

superkuh wrote: post #284
phreeza wrote: post #237

Earlier quoted context omitted.

Instruction fine tuning improves performance on many tasks that are not in the fine tuning dataset. See e.g. table 14 in the instructGPT paper.

Instruction fine tuning definitely increases the win rate in human judged comparisons but that just means it's better at generating a style humans like. Humans aren't always right. https://arxiv.org/pdf/2203.02155.pdf Page 56/68 in table 14 it looks like the things the fine tunes beat the base models at are basically HellaSwag and the ones that use human evaluations. Otherwise base gpt models are winning. And just be…

I completely agree that humans are not always right, but neither is the next token in a text sequence always "right",i.e. the thing that a raw LM is best at.

I just object to the blanket statement that fine tuning will make a model dumber. It will mostly mean the model is not as good at the original training task, but that really doesn't mean it is "dumber" by any definition.

The question then is what fine tuning does to tasks that are neither the original training task or the fine tuning task. This will depend on the fine tuning task. It seems that instruction fine tuning improves performance on tasks that involve human interaction, so I have a hard time seeing it as the model becoming dumber. Other fine tuning tasks, such as removing toxicity, may have a higher cost on unrelated tasks, so there one could say they caused the model to become dumber.

Re: GPT-4 is getting worse over time, not better

#298

Post by tptacek »

cubefox wrote: post #187
eldritch_4ier wrote: post #183

Earlier quoted context omitted.

What is incredibly well established is that g is incredibly predictive for many life outcomes from income to educational attainment to drug addiction. This is what IQ test measures. The only thing disputed is whether g is the same thing as “intelligence”, and the only dispute there is from softer science fields because “intelligence” is a word without a precise definition and a lot of feelings and opinions wrapped up…

[flagged]

For the same reason any racial supremacy stereotype is tattoo: because it's toxic, not because it's valid.

Re: GPT-4 is getting worse over time, not better

#299

Post by epups »

ChatGTP wrote: post #292
epups wrote: post #238

Earlier quoted context omitted.

Since they are trained on all kinds of Internet content, might be a little tricky to manage paying >1B people

I'm sure it would be easy to pay publishers who pass on the money? How do you think Amazon does things for kindle?

Will you get a check for your posts on HN? Your tweets? No, right? So all this talk about paying people for their data is rubbish, and what is really at stake here are checks between huge corporations that already stole it from you in the first place.

Re: GPT-4 is getting worse over time, not better

#300

Post by YeGoblynQueenne »

listenallyall wrote: post #279
lurquer wrote: post #268

Earlier quoted context omitted.

It is a useful tool for editing. You can input a rough scene you’ve written and ask it to spruce it up, correct the grammatical errors, toss in some descriptive stuff suitable for the location, etc. It is worthwhile. At least it was… If your text isn’t ‘aligned’ correctly, it either won’t comply or spew out endless caveats. I appreciate the motivation to rein in some of the silly 4chan stuff that was occurring as the…

sprucing up, fixing your mistakes, adding in "descriptive stuff"... that's like 90% of writing. Outsourcing it all to AI essentially robs the purchaser of the effort required to create an original piece of work. Not to mention copyright issues, where do you think the AI is getting those descriptive phrases from? Other authors' work.

I think that you, like I did in the past, are underestimating the number of people who simply hate writing and see it as a painstaking chore that they would happily outsource to a machine. It doesn't help that most people grow up being forced to write when they have no interest in doing so, and to write things they have no interest in writing, like school essays and business applications and so on. If a chatbot could automate this... actually, even if a chatbot can't automate this, people will still use it anyway, just to end the pain.
Post reply on HN