Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-4 is getting worse over time, not better

twitter.com

Post reply on HN
Print view View original

281–290 of 315 posts

Re: GPT-4 is getting worse over time, not better

#281

Post by pc86 »

autoexec wrote: post #250
pc86 wrote: post #79

Earlier quoted context omitted.

Then why comment? I'm not trying to be an ass, but this is literally the only comment you've made in this thread. The linked article is a Twitter thread. If you don't have Twitter, fine, move on to the next thing you can actually comment on. Person A offers a reason why we may want to be slightly more skeptical than normal about this information. Person B suggests we can pretty easily look past that. Person C (you) i…

I do think it'd be nice if people mostly posted links that were accessible to the general public. At least paywalled articles usually have a link in the comments where they can be read. Many times tweets could even be pasted as text into a comment to make them accessible.

Twitter accounts are free and I think it takes about 60 seconds to make one, so for all intents and purposes a Twitter thread is publicly accessible.

Re: GPT-4 is getting worse over time, not better

#282

Post by jeremyjh »

modriano wrote: post #280
jeremyjh wrote: post #223

Earlier quoted context omitted.

We don’t need a conspiracy to explain OpenAI motives. They released an early access model that cost far more to operate than they could ask for in subscriptions. They were selling dollars for $0.50 and of course everyone loved it. Then they decided they should stop bleeding money and maybe even make some.

The main silicon valley business model (over at least my profession life) follows this rough pattern: 0) get enough money to start stage 1, 1) make product, 2) build userbase, 3) monetize (investors, bootstrap, acquisition, however) After a bit of googling, I couldn't find out how many paying users ChatGPT has, but many touting the number of users (100M). MSFT invested $13B [0] into OpenAI (650M paying-user-months @…

Every single GPT 4.0 user is a paying sub. Free users don't have access to it.

Re: GPT-4 is getting worse over time, not better

#283

Post by autoexec »

pc86 wrote: post #281
autoexec wrote: post #250

Earlier quoted context omitted.

I do think it'd be nice if people mostly posted links that were accessible to the general public. At least paywalled articles usually have a link in the comments where they can be read. Many times tweets could even be pasted as text into a comment to make them accessible.

Twitter accounts are free and I think it takes about 60 seconds to make one, so for all intents and purposes a Twitter thread is publicly accessible.

Last I heard they required a phone number which isn't free. I even created a couple accounts a long time ago, but they were both banned after a week or so. I suspect because they thought they were bot accounts. I used free email services like yahoo to sign up, I only ever used the website and not the app, and I never posted or interacted with anything, I just followed a small number of accounts.

Re: GPT-4 is getting worse over time, not better

#284

Post by superkuh »

phreeza wrote: post #237
superkuh wrote: post #7

Earlier quoted context omitted.

Lets not conflate a model's smartness with a model's ability to be used reliably and predictably. They're very different things. Yes, fine-tuning, for instruction, or whatever you chose, will make the models easier to work with with less going off the rails. But they'll also take a big hit on perplexity and other more robust dataset completion tests. They're objectively stupider.

Instruction fine tuning improves performance on many tasks that are not in the fine tuning dataset. See e.g. table 14 in the instructGPT paper.

Instruction fine tuning definitely increases the win rate in human judged comparisons but that just means it's better at generating a style humans like. Humans aren't always right.

https://arxiv.org/pdf/2203.02155.pdf Page 56/68 in table 14 it looks like the things the fine tunes beat the base models at are basically HellaSwag and the ones that use human evaluations. Otherwise base gpt models are winning.

And just before that they're discussing how the fine-tunes do cause performance regressions on page 55.

Re: GPT-4 is getting worse over time, not better

#285

Post by oceanplexian »

epcoa wrote: post #246
pessimizer wrote: post #204

Earlier quoted context omitted.

Humans are definitely aligned, and for the same reasons as a LLM. Socialization, being allowed to work, being allowed to speak. edit: It's a social faux pas to say "died" about a person acquainted to the listener in most situations, you have to say "passed away."

> It's a social faux pas to say "died" about a person acquainted to the listener in most situations That’s overly simplistic and an Americanism. The resurgence of the “passed away” euphemism is a recent (about 40 years) phenomenon in American English which seems to have been started out of the funeral industry as prior to that “died” was nearly universal for both news stories and obituaries. “Died” is not a social fa…

I have never use the phrase passed away.

Always died, dead, gone, or something along those lines. When someone says passed away it sounds like they are trying to feign empathy, but hey, perhaps I am not an “aligned” human and need some RLHF.

Re: GPT-4 is getting worse over time, not better

#286

Post by ssabev »

StackOverlord wrote: post #218
pwillia7 wrote: post #25

Earlier quoted context omitted.

totally agree -- Bing GPT4 still works somewhat better in my opinion but has declined in quality alongside chat gpt+ too

What about the GPT4 hosted on poe.com. It has the same output pace as it had in march.

Thought they use Claude instead of gpt

Re: GPT-4 is getting worse over time, not better

#287

Post by rsynnott »

csmpltn wrote: post #140
Vvector wrote: post #70

Earlier quoted context omitted.

ChatGPT isn't the right tool to use for checking if numbers are prime. It is tuned for conversations. I'd like to see a MathGPT or WolframGPT. The real question is if ChatGPT is worse on math and better elsewhere, or just worse overall. That is still unknown

> "ChatGPT isn't the right tool to use for checking if numbers are prime. It is tuned for conversations." Two months ago they were telling me ChatGPT is coming for everyone - programmers, accountants, technical writers, lawyers, etc. Now we're slowly back to "so here's the thing about LLMs"...

Who were 'they'? The same people who were telling you you'd be zooming around a bitcoin-powered metaverse on your Segway, then having your self-driving car bring you back to your 3d-printed house to watch 3d TV?

There's a certain type of person who sees a new thing, and goes "this will change everything". For every new thing. Very occasionally, they're right, by accident. In general, it's safest to be very, very sceptical.

Re: GPT-4 is getting worse over time, not better

#288

Post by kvetching »

bredren wrote: post #194
StackOverlord wrote: post #173

Earlier quoted context omitted.

There is degraded performance because GPT4 refuses to carry out certain tasks. To figure it out though, you must need to be able to switch between GPT4-0316 and GPT4-0614. The task it is reluctant to do include: - legal advice - psychological guidance - complex programming tasks. IMO OpenAI is just backtracking on what it released to resegment their product into multiple offerings.

>IMO OpenAI is just backtracking on what it released to resegment their product into multiple offerings. I think this is it, but also it is pulling back on the value of products if it reduces their compute costs. I'm hoping competition will be so fierce in this market that quality and opaque changes to priced LLM experiences won't be a thing for very long.

Hilarious because Google is taking the exact opposite approach.

They released Bard as a terrible model to begin with, and now you can program and have it interpret images. It's been consistently improving.

OpenAI really got sidetracked when they put out such a good model that freaked everyone out. Google saw this and decided to do the opposite to prevent too much attention. Now Google can improve quietly and nobody will notice.

Re: GPT-4 is getting worse over time, not better

#289

Post by kvetching »

ARandumGuy wrote: post #200
StackOverlord wrote: post #173

Earlier quoted context omitted.

There is degraded performance because GPT4 refuses to carry out certain tasks. To figure it out though, you must need to be able to switch between GPT4-0316 and GPT4-0614. The task it is reluctant to do include: - legal advice - psychological guidance - complex programming tasks. IMO OpenAI is just backtracking on what it released to resegment their product into multiple offerings.

I think it's probably a good idea that GPT4 avoids legal or psychological tasks. Those are areas where giving incorrect output can have catastrophic consequences, and I can see why GPT4's developers want to avoid potential liability.

It's ridiculous. We should have access to a completely neutered version if we want it, we should just have to sign for access if they are worried about begin sued.

Re: GPT-4 is getting worse over time, not better

#290

Post by epcoa »

oceanplexian wrote: post #285
epcoa wrote: post #246

Earlier quoted context omitted.

> It's a social faux pas to say "died" about a person acquainted to the listener in most situations That’s overly simplistic and an Americanism. The resurgence of the “passed away” euphemism is a recent (about 40 years) phenomenon in American English which seems to have been started out of the funeral industry as prior to that “died” was nearly universal for both news stories and obituaries. “Died” is not a social fa…

I have never use the phrase passed away. Always died, dead, gone, or something along those lines. When someone says passed away it sounds like they are trying to feign empathy, but hey, perhaps I am not an “aligned” human and need some RLHF.

I used to live in the south 30 years ago. I remember some southern baptist moms that wouldn't say the word "dead" - sometimes they'd spell it out, lol, similar to the way some people consider hell a swear word. This kind of hypersensitivity wasn't common outside of limited circles then, and it's certainly not more common today.

Similarly I wouldn't consider saying hell in public in 2023 to be a social faux pas.

Post reply on HN