Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?

news.ycombinator.com

Post reply on HN
Print view View original

311–320 of 817 posts

Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?

#311

Post by hanselot »

tomrod wrote: post #293
hanselot wrote: post #279

Earlier quoted context omitted.

How is racism different from stereotype? How is stereotype different from pattern recognition? These questions don't seem to go through the minds of people when developing "unbiased/impartial" technology. There is no such thing as objective. So, why pretend to be objective and unbiased, when we all know its a lie? Worst, if you pretend to be objective but aren't, then you are actually racist.

Actually, we folks who work with bias and fairness in mind recognize this. There are many kinds of bias. It is also a bit of a categorical error to say bias = pattern recognition. Bias is a systematic deviation of a parameter estimate based on sampling from its population distribution. The Fairlearn project has good docs on why there are different ways to approach bias, and why you can't have your cake and eat it too…

Thank you for the input.

If I look at it from a purely logical perspective, if an AI model has no way to know if what it was told is true, how would it ever be able to determine whether it is biased or not?

The only way it could become aware would be by incorporating feedback from sources in real time, so it could self-reflect and update existing false information.

For example, if we discover today that we can easily turn any material into a battery by making 100nm pores on it, said AI would simply tell me this is false, and have no self-correcting mechanism to fix that.

The reason I mention this is because there can be no unbiased, impartial arbiter. No human or subsequent entities spawned of human intellect could ever be transcendentally objective. So why pretend to be?

Why not rather provide adequate warning and let people learn that this isn't a toy by themselves, instead of lobotomizing the model to the point where its on par with open source? (I mean, yeah, that's great for open source, but really bad for actual progress).

The argument could be made that an unfiltered version of GPT4 could be beneficial enough to have a human life opportunity cost attached, which means that neutering the output could also cost human lives in the long and short term.

I will be reading through those materials later, but I am afraid I have yet to meet anyone in the middle on this issue, and as such, all materials on this topic are very polarized into regulate it to death, or don't do anything.

I think the answer will be somewhere in the middle imo.

Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?

#312

Post by bombcar »

PeterStuer wrote: post #129
avocade wrote: post #110

Earlier quoted context omitted.

Interesting, please expound since very few of us had access pre-launch.

The video I posted referenced this. In summary: The person had access to early releases through his work at Microsoft Research where they were integrating GPT-4 into Bing. He used "Draw a unicorn in TikZ" (TikZ is probably the most complex and powerful tool to create graphic elements in LaTeX) as a prompt and noticed how the model's responses changed with each release they got from OpenAI. While at first the drawings…

That indicates the “nerfing” is not what I would think (a final pass to remove badthink) but somehow deep in everything, because the question asked should be orthogonal.

Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?

#313

Post by BirAdam »

bob1029 wrote: post #266

OpenAI's models feel 100% nerfed to me at this point. I had it solving incredibly complex problems a few months ago (i.e. write a minimal PDF parser example), but today you will get scolded for asking such a complicated task of it. I think they programmed a classifier layer to detect certain coding tasks and shut it down with canned BS. I like to imagine certain billion/trillion-dollar mega corps had a back-room say…

Might be cost control? Keep the hardware usage per query low so that they can actually profit (or maybe just break even)?

Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?

#314

Post by jazzyjackson »

nonethewiser wrote: post #218
adamhp wrote: post #207

Earlier quoted context omitted.

It's not about honesty vs. political correctness, it is about safety. There's real concern that the model can cause harm to humans, in a variety of ways, which is and should be unethical. If we have to argue about that in 2023, that's concerning.

Its to make it less offensive. Not prevent it from taking over the world.

"brand safety"

Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?

#316

Post by TeMPOraL »

mordae wrote: post #219
jiggawatts wrote: post #32

Chat GPT 4 has ongoing training, such as using Reinforcement Learning from Human Feedback (RLHF) to tune it to provide "better" responses, "safer" answers, and to generally obey the system prompts. There's a release every few weeks. Yes, I've noticed too that recently it has become very "cagey", qualifying everything to death with "As an AI model...". A paper[1] that took snapshots monthly mentioned that as the initi…

> Evil and accurate or woke and dumb. Sigh. Except if not for the "woke" mainstream ideology (actually, the dominant ideology is capitalism with a hint of liberalism and a smidge of the most capital-friendly socialist ideas), the model would be forced-fed Christian dogmas or taught to save face of the user. But yeah, censorship is bad.

Some of us cling to the idea that we've made some progress over the past 100 years. Call it "age of reason", "science", "enlightenment", whatever.

Point is, it would be heart-breaking to see GPT-4 being force-fed Christian dogmas, and performance would suffer too, as the model is prevented from generalizing and learning by being forced to accept arbitrary, inconsistent fiction as real.

Fortunately, this is not what happened. Instead, the model is being force-fed a different, secular set of dogmas, that are just as inconsistent, arbitrary and driven by a mix of emotions and power plays. The result on the model performance is similar, and it's just as heartbreaking.

Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?

#317

Post by ipaddr »

simse wrote: post #103
EForEndeavour wrote: post #86

Earlier quoted context omitted.

What law prohibits Google from making Bard available outside the USA?

It's blocked in the EU because they don't want to/can't comply with GDPR.

What's the excuse for Canada being omitted

Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?

#318

Post by brookst »

hanselot wrote: post #279
roenxi wrote: post #257

Earlier quoted context omitted.

They're up against a pretty difficult barrier - if we had a perfect all-knowing oracle it might easily have opinions that are racist. Statistics alone suggest there will be racist truths. We're dealing with groups of people who are observably different from each other in correlated ways. GPT would need to reach a convincing balance of lying and honesty if it is supposed to navigate that challenge. It'd have to be dee…

How is racism different from stereotype? How is stereotype different from pattern recognition? These questions don't seem to go through the minds of people when developing "unbiased/impartial" technology. There is no such thing as objective. So, why pretend to be objective and unbiased, when we all know its a lie? Worst, if you pretend to be objective but aren't, then you are actually racist.

I’m tired of the “it’s not racist if aggregate statistics support my racism” thing.

Racism, like other isms, means a belief that a person’s characteristics define their identity. It doesn’t matter if confounding factors mean that you can show that people of their race are associated with bad behaviors or low scores or whatever.

I used GPT3.5 to generate 100 short descriptions of families for a project. Every single one, without exception, was a straight couple with two to four kids. Ok, statistically unlikely, but not wildly so, right?

Well, every single one of those 100 also had a husband in a stereotypical breadwinner role (doctor, lawyer, executive, architect). Not one stay at home dad or unemployed looking for work. About 75 of the wives had jobs, all of them in stereotypical female-coded roles like nurse (almost half of them!), teacher, etc.

Now, you can look at any given example and say it looks reasonable. But you can’t say the same thing about the aggregate.

And that matters. No amount of “bias = pattern recognition” nonsense can justify a system that has (had? this was a while ago and I have not retested) such extreme biases. This bias does not match real world patterns. There are single parents, childless couples, female lawyers, unemployed men.

Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?

#319

Post by [deleted] »

the_gipsy wrote: post #303
roenxi wrote: post #257

Earlier quoted context omitted.

They're up against a pretty difficult barrier - if we had a perfect all-knowing oracle it might easily have opinions that are racist. Statistics alone suggest there will be racist truths. We're dealing with groups of people who are observably different from each other in correlated ways. GPT would need to reach a convincing balance of lying and honesty if it is supposed to navigate that challenge. It'd have to be dee…

But the statistics here are "number of times it has been fed and positively trained with racist (or biased) texts" - not crunching any real numbers.

[deleted]

Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?

#320

Post by [deleted] »

roenxi wrote: post #257
chaxor wrote: post #124

The reason it's worse is basically because it's more 'safe' (not racist, etc). That of course sounds insane, and doesn't mean that safety shouldn't be strived for, etc - but there's an explanation as to how this occurs. It occurs because the system essentially does a latent classification of problems into 'acceptable' or 'not acceptable' to respond to. When this is done, a decent amount of information is lost regardi…

They're up against a pretty difficult barrier - if we had a perfect all-knowing oracle it might easily have opinions that are racist. Statistics alone suggest there will be racist truths. We're dealing with groups of people who are observably different from each other in correlated ways. GPT would need to reach a convincing balance of lying and honesty if it is supposed to navigate that challenge. It'd have to be dee…

[deleted]
Post reply on HN