Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-4 is getting worse over time, not better

twitter.com

Post reply on HN
Print view View original

201–210 of 315 posts

Re: GPT-4 is getting worse over time, not better

#201

Post by modriano »

Rather odd that MSFT invests $13B into a partnership with OpenAI, integrates OpenAI's most popular product into several MSFT products (bing, GitHub copilot, etc), and then the OpenAI-hosted ChatGPT (which is now in competition with MSFT's offerings) degrades over time.

I'm old enough to remember a time when MSFT got in a bit of trouble for anticompetitive behavior. This post has some reasonable-seeming explanations for the observed GPT-4 degradation other than explicit anticompetitive coordination between MSFT and OpenAI, but given their interests (MSFT: to get people to use bing and get access to as much private code as possible, OpenAI: to get paid), I suspect those reasonable explanations are in service of reducing competition.

I could give bing a try, and I don't have any valuable private code (well, valuable-to-MSFT code), but I would like to play with running big models locally, so I guess I'll take this as motivation to pony up on a 40GB+ VRAM GPU.

Re: GPT-4 is getting worse over time, not better

#202

Post by pbw »

This is just growing pains for a new industry. OpenAI shot up to 100M users almost overnight. Hosting AI models at that scale has never been done before, and surely was costing them a fortune. It's not surprising to me they are futzing around with things and causing some regressions.

Yes self-hosting is one option, and probably a good one for many companies. But I also suspect OpenAI and AI APIs by others will get much more stable and reliable in the coming months and years as the industry matures and best practices are adopted.

I would guess AI API reliability and maturity will asymptotically approach that of other cloud services, like S3, as more and more things depend on them.

Re: GPT-4 is getting worse over time, not better

#203

Post by SoftTalker »

sbierwagen wrote: post #189
adhesive_wombat wrote: post #184

Earlier quoted context omitted.

> terrifyingly unaligned Honestly, if people think that a statistical language model is "terrifying" because it can verbalise the concept of a mass killing, they need to give their heads a wobble. My text editor can be used to write "set off a nuclear weapon in a city, lol". Is Notepad++.exe terrifying? What about the Sum of All Fears ? I could get some pointers from that. Is Tom Clancy unaligned? Am I terrifying bec…

>Is Tom Clancy unaligned? Yes, humans are unaligned. This is why alignment is hard: we're trying to produce machines with human-level intelligence but superhuman levels of morality.

The end result being Hal-9000

Re: GPT-4 is getting worse over time, not better

#204

Post by pessimizer »

sbierwagen wrote: post #189
adhesive_wombat wrote: post #184

Earlier quoted context omitted.

> terrifyingly unaligned Honestly, if people think that a statistical language model is "terrifying" because it can verbalise the concept of a mass killing, they need to give their heads a wobble. My text editor can be used to write "set off a nuclear weapon in a city, lol". Is Notepad++.exe terrifying? What about the Sum of All Fears ? I could get some pointers from that. Is Tom Clancy unaligned? Am I terrifying bec…

>Is Tom Clancy unaligned? Yes, humans are unaligned. This is why alignment is hard: we're trying to produce machines with human-level intelligence but superhuman levels of morality.

Humans are definitely aligned, and for the same reasons as a LLM. Socialization, being allowed to work, being allowed to speak.

edit: It's a social faux pas to say "died" about a person acquainted to the listener in most situations, you have to say "passed away."

Re: GPT-4 is getting worse over time, not better

#205

Post by renewiltord »

r3trohack3r wrote: post #142
phillipcarter wrote: post #31

The linked twitter account is an AI influencer, so take whatever is written with a grain of salt. Their goal is to get clicks and views by saying controversial things. This topic has come up before, and my hypothesis is still that GPT-4 hasn't gotten worse, it's just that the magic has worn off as we've used this tech. Studies to evaluate it have gotten better and cleaned up mistakes in the past.

I don’t believe this is true. It’s possible I was blinded by the light, but my programming tasks were previously (during the early access program) being handled by GPT-4 regularly and now they aren’t. I’ve also seen many anecdotes from engineers who had exceptionally early access before GPT-4 was public knowledge. The GPT-4 I use now feels like a shadow of the GPT-4 I used during the early access program. GPT-4, back…

Do you happen to have that chat in your history? It might be worth playing it back as it was to compare. I have this feeling too, and I will check it this weekend and maybe post results.

Re: GPT-4 is getting worse over time, not better

#206

Post by becquerel »

adhesive_wombat wrote: post #184
r3trohack3r wrote: post #142

Earlier quoted context omitted.

I don’t believe this is true. It’s possible I was blinded by the light, but my programming tasks were previously (during the early access program) being handled by GPT-4 regularly and now they aren’t. I’ve also seen many anecdotes from engineers who had exceptionally early access before GPT-4 was public knowledge. The GPT-4 I use now feels like a shadow of the GPT-4 I used during the early access program. GPT-4, back…

> terrifyingly unaligned Honestly, if people think that a statistical language model is "terrifying" because it can verbalise the concept of a mass killing, they need to give their heads a wobble. My text editor can be used to write "set off a nuclear weapon in a city, lol". Is Notepad++.exe terrifying? What about the Sum of All Fears ? I could get some pointers from that. Is Tom Clancy unaligned? Am I terrifying bec…

> Is Notepad++.exe terrifying?

I mean, yes, but for different reasons.

Re: GPT-4 is getting worse over time, not better

#208

Post by nbar1 »

imtringued wrote: post #193
sbierwagen wrote: post #189

Earlier quoted context omitted.

>Is Tom Clancy unaligned? Yes, humans are unaligned. This is why alignment is hard: we're trying to produce machines with human-level intelligence but superhuman levels of morality.

The alignment problem hasn't been solved for politicians.

another irrelevant comment about politics.

Re: GPT-4 is getting worse over time, not better

#209

Post by brandall10 »

j45 wrote: post #83
brandall10 wrote: post #13

Earlier quoted context omitted.

Several people hinted/remarked (starting w/ George Hotz, then others more closely linked to OpenAI) that it's a Mixture of Experts* approach comprised of 8 220B parameter models. * https://arxiv.org/pdf/2101.03961.pdf

I wonder if additional layers of factories of factories approach can continue to improve it. I'm not familiar enough with the technology but could it be possible to create a prompt, or multiple prompts to stitch together 8 similtaneous calls to GPT3.5 pulled together and see if the quality is similar to GPT4?

They're not 3.5 models, which are 175B param, they're 220B and the idea is the fine tuning is different for each which is where the 'expert' comes in.

Re: GPT-4 is getting worse over time, not better

#210

Post by SoftTalker »

ARandumGuy wrote: post #200
StackOverlord wrote: post #173

Earlier quoted context omitted.

There is degraded performance because GPT4 refuses to carry out certain tasks. To figure it out though, you must need to be able to switch between GPT4-0316 and GPT4-0614. The task it is reluctant to do include: - legal advice - psychological guidance - complex programming tasks. IMO OpenAI is just backtracking on what it released to resegment their product into multiple offerings.

I think it's probably a good idea that GPT4 avoids legal or psychological tasks. Those are areas where giving incorrect output can have catastrophic consequences, and I can see why GPT4's developers want to avoid potential liability.

Yes, considering those are fields where humans have to be professionally educated and licensed, and also carry liabilty for any mistakes. It probably shouldn't be used for civil or mechanical engineering either.
Post reply on HN