Live data from Hacker News

Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?

news.ycombinator.com

751–760 of 817 posts

Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?

#751

Earlier quoted context omitted.

> Crime stats, average IQ across groups, stereotype accuracy, etc. If you measured these stats for Irish Americans in 1865 you'd also see high crime and low IQ. If you measure these stats with recent black immigrants from Africa, you see low crime and high IQ. These statistical differences are not caused by race. An all-knowing oracle wouldn't need to hold "opinions that are racist" to understand them.

But for accuracy it doesn't matter if the relationship is causal, it matters whether the correlation is real. If in some country - for the sake of discussion, outside of Americas - a distinct ethnic group is heavily discriminated against, gets limited access to education and good jobs, and because of that has a high rate of crime, any accurate model should "know" that it's unlikely that someone from that group is a d…

> should acknowledge that someone named XYZ is much more likely to be a felon

The obvious problem comes with the questions why is that true and what do we do with that information. Information is, sadly, not value-neutral. We see "XYZ is a felon" and it implies specific causes (deviance in the individual and/or community) and solutions (policing, incarceration, continued surveillance), which are in fact embedded in the very definition of "felon". (Felony, and crime in general, are social and governmental constructs.)

Here's the same statement, phrased in a way that is not racist and taboo:

Someone named XYZ is much more likely to be watched closely by the police, much more likely to be charged with a crime, and much less likely to be able to defend himself against that charge. He is far more likely to be affected by the economic instability that comes with both imprisonment and a criminal record, and is therefore likely to resort to means of income that are deemed illegal, making him a risk for re-imprisonment.

That's a little long-winded, so we can reduce it to the following:

Someone named XYZ is much more likely to be a victim of overpolicing and the prison-industrial complex.

Of course, none of this is value-neutral either; it in many ways implies values opposite to the ones implied by the original statement.

All of this is to say: You can't strip context, and it's a problem to pretend that we can.

Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?

#752
post #124

The reason it's worse is basically because it's more 'safe' (not racist, etc). That of course sounds insane, and doesn't mean that safety shouldn't be strived for, etc - but there's an explanation as to how this occurs. It occurs because the system essentially does a latent classification of problems into 'acceptable' or 'not acceptable' to respond to. When this is done, a decent amount of information is lost regardi…

Do you have any sources for that? How would making ChatGPT less likely to return a racist answer or hate speech affect its ability to return code? After a question has been classified into a coding problem, presumably ChatGPT servers could now continue to solve the problem as usual. Maybe running ChatGPT is really expensive, and they nerfed in order to reign in costs. That would explain why the answers we get are les…

[deleted]

Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?

#753

Earlier quoted context omitted.

Who has the necessary resources to run, let alone train the model?

I think a lot of people are unaware that these models have an enormous human training component performed through companies such as Amazon Mechanical Truk and dataannotation.tech. Called Human Intelligence Tasks, a large number of people have been working in this area for close to a decade. Dataannotation Tech claims to have over 100k workers. From Cloud Research, "How Many Amazon Mechanical Turk Workers Are There in…

They don't want to know how the sausage is made.

Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?

#754
post #293

Earlier quoted context omitted.

Actually, we folks who work with bias and fairness in mind recognize this. There are many kinds of bias. It is also a bit of a categorical error to say bias = pattern recognition. Bias is a systematic deviation of a parameter estimate based on sampling from its population distribution. The Fairlearn project has good docs on why there are different ways to approach bias, and why you can't have your cake and eat it too…

When did people start to use “folks” in this unnatural way.

I think "us folks" is more standard than "we folks" but it's no different in meaning.

Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?

#755

Earlier quoted context omitted.

Yikes One is never former CIA, once you're in, you're in, even if you leave. Although he is a CompSci grad, he's also a far-right Republican. A spook who leans far right sitting atop OpenAI is worse than Orwell's worst nightmares coming to fruition.

Will Hurd is a liberal Republican. He supports Dreamers. Very early critic of Donald Trump.

Early critic of Donald Trump means nothing - Lindsey Graham was too, but has resorted to kissing Trump's ass for the last 7 years. You could say the same for Mitt Romney - an early critic who spoke against candidate Trump, but voted for candidate Trump, and voted in lockstep with President Trump.

A liberal Republican? Will Hurd's voting record speaks otherwise. In the 115th Congress, Hurd voted with Donald Trump 94.8% of the time. In the 116th Congress, that number dropped to 64.8%. That's an 80.4% average across Trump's presidency. [0] Agreeing with Donald Trump 4 times out of 5 across all legislative activities over 4 years isn't really being critical of him or his administration.

[0] https://projects.fivethirtyeight.com/congress-trump-score/wi...

It's like calling Dick Cheney a liberal because one of his daughters is lesbian, even though he supports all sort of other far-right legislation.

Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?

#756

Earlier quoted context omitted.

The API chat endpoint dramatically changes its responses every few weeks. You can spend hours crafting a prompt and then a week later the responses to that same prompt can become borderline useless. Writing against the ChatGPT API is like working against an API that breaks every other week with completely undocumented changes.

> The API chat endpoint dramatically changes its responses every few weeks. You can spend hours crafting a prompt and then a week later the responses to that same prompt can become borderline useless. Welcome to statistical randomness?

... with temperature = 0

Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?

#757

Earlier quoted context omitted.

No wonder, is this just the chat interface or the API too? I guess gpt4 was never sustainable at $20 a month. Annoying to be charged the same subscription and the product made inferior.

I wonder what the unfilitered one is like. Are they sitting on a near-perfect arbiter of truth? That would be worth hiding.

No.

Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?

#759
post #675

Earlier quoted context omitted.

It's a big pain when trying to build things on top of the GPT-4 API. We had some experiments that were reliably, reproducibly achieving a goal, and then one day it suddenly stops working properly; then the student managed a different prompt that worked (again, reproducibly, with proper clean restarts from fresh context), and within a few days it broke. I understand that there is a desire to tweak the model and improv…

same for me - also the api itself is very unstable sometimes the same prompt finish’s within a minute, sometimes our client timesout after 10 minutes and sometimes the api sends a 502 bad gateway after 5-10 minutes. the very same request then runs fine within a few minutes after a delay of 5 minutes. the results vary very much, even with a temperature of 0.1 requests that needs responses with over ~2k tokens almost a…

They have such a huge load, I’m not at all surprised

Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?

#760
post #657
post #293

Earlier quoted context omitted.

Actually, we folks who work with bias and fairness in mind recognize this. There are many kinds of bias. It is also a bit of a categorical error to say bias = pattern recognition. Bias is a systematic deviation of a parameter estimate based on sampling from its population distribution. The Fairlearn project has good docs on why there are different ways to approach bias, and why you can't have your cake and eat it too…

This is interesting, thanks for the links. It seems like the dimensions of fairness and group classifications are often cribbed from the United States Protected Classes list in practice with a few culturally prescribed additions. What can be done to ensure that 'fairness' is fair? That is, when we decide what groups/dimensions to consider, how do we determine if we are fair in doing so? Is it even possible to determi…

Bit of a tangent topic I think -- any specification of group classification and fairness will have the same issues presented.

If we want to remove stereotypes, I reckon better data is required to piece out the attributes that can be causally inferred to be linked to poorer outcomes.

As likely not even the Judeo-Christian version of God can logically be that omniscient, occasional stereotypes and effusively communal forgiveness of edge cases are about the best we'll ever arrive to in policy.

Post reply on HN