Yep, it's struggling for me with simple problems like splitting full names and so on. Few weeks ago, it was bang on — and the task is literally the same
Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?
551–560 of 817 posts
Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?
#552To me, it feels like it's started giving superficial responses and encouraging follow-up elsewhere -- I wouldn't be surprized if its prompt has changed to something to that effect. Before, if I had an issue with a library or debugging issue, it would try to be helpful and walk me through potential issues, and ask me to 'let it know' if it worked or not. Now it will try to superficially diagnose the problem and then a…
>To me, it feels like it's started giving superficial responses and encouraging follow-up elsewhere -- I wouldn't be surprized if its prompt has changed to something to that effect. That's the vibe I've been getting. The responses feel a little cagier at times than they used to. I assume it's trying to limit hallucinations in order to increase public trust in the technology, and as a consequence it has been nerfed a…
[flagged]
Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?
#553I don't think it's any worse at all. I think what most people are expressing here is reaching the limits of the technology and realizing that it's not magic.
I suspect it's the same people complaining how Google's gotten worse. One time, I went to the hotel restaurant. I ordered something; it was amazing. Figured I'd do it again a few days later. It wasn't bad, but it wasn't as good. There wasn't any specific difference I could put my finger on other than that the novelty wore off.
Google has gotten worse. So what you're saying is people complaining about GPT-4 are correct?
Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?
#554It is worse than it was in april by leaps and bounds. pretty sure they nerfed it by request of... ?
Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?
#555It’s so annoying when GPT4 refuses to do what it’s asked.
GPT3.5 immediately does as its requested and much faster too.
Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?
#556Earlier quoted context omitted.
The video I posted referenced this. In summary: The person had access to early releases through his work at Microsoft Research where they were integrating GPT-4 into Bing. He used "Draw a unicorn in TikZ" (TikZ is probably the most complex and powerful tool to create graphic elements in LaTeX) as a prompt and noticed how the model's responses changed with each release they got from OpenAI. While at first the drawings…
That indicates the “nerfing” is not what I would think (a final pass to remove badthink) but somehow deep in everything, because the question asked should be orthogonal.
There's a section in the GPT-4 release docs where they talk about how the safety stuff changes the accuracy for the worse.
Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?
#557Earlier quoted context omitted.
I’m tired of the “it’s not racist if aggregate statistics support my racism” thing. Racism, like other isms, means a belief that a person’s characteristics define their identity. It doesn’t matter if confounding factors mean that you can show that people of their race are associated with bad behaviors or low scores or whatever. I used GPT3.5 to generate 100 short descriptions of families for a project. Every single o…
> No amount of “bias = pattern recognition” nonsense can justify a system that has (had? this was a while ago and I have not retested) such extreme biases One possible explanation is that when you ask for 100 example families the task is parsed as "pick the most likely family composition and add a bit of randomness" and "repeat the aforementioned task" 100 times. If phrased like that it would be surprising to find on…
> You're right about the difference between one-by-one prompts and prompts that create a population. I switched to sets of 10 at a time and it got better.
But still, when you ask for "make up a family", the model should not interpret that as "pick the most likely family".
I disagree with your opinion that it's hard. GPT does not work by creating a pool of possible families and then sampling them; it works by picking the next set of words based on the prompt and probabilities. If "Dr. Laura Nguyen and Robert Smith, an unemployed actor" is 1% likely, it should come up 1% of the time. The sampling is built in to the system.
Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?
#558Earlier quoted context omitted.
I think a lot of people are unaware that these models have an enormous human training component performed through companies such as Amazon Mechanical Truk and dataannotation.tech. Called Human Intelligence Tasks, a large number of people have been working in this area for close to a decade. Dataannotation Tech claims to have over 100k workers. From Cloud Research, "How Many Amazon Mechanical Turk Workers Are There in…
Here's an account of a person in Africa that helped train (wading thru gnarly explicit content in the process): https://www.bigtechnology.com/p/he-helped-train-chatgpt-it-t...
Another thing that people don't know is that a lot of the safe-ified output is hand crafted. Part of "safety" is that a human has to identify the offensive content, decide what's offensive about it, and write a response blurb to educate the user and direct them to safety.
Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?
#559Earlier quoted context omitted.
You can lobby in every country, what do you mean
It only takes one country to host an open source AI model thanks to the internet.
Unless your country of residence decides to block access to that model, or the model's country of residence decides to apply a customs-like regime to its internet.
Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?
#560I believe they introduced a sort of rate limiting where the expectation will be that the user is doing more thinking and due diligence and asking more precise questions, so that they're not just attempting to get a lot of hand holding with very broad prompts when they can otherwise think about what code they're being given back from more specific and structured questioning. This is useful because it will preserve the value proposition of AI while avoiding a machine just being milked for code by people who want to just throw code at the wall and see what sticks. Milking a model for code is fun in some ways but it won't scale from a cost-to-run point of view, and it also will incentivize no longer thinking deeply and critically about what is being built.
If you are incorporating machine models into your development process very heavily, make sure you're still building deep knowledge and deep context of your codebase. Otherwise you'll atrophy in that department because you falsely believe the machine maintains the context for you.
That isn't feasible right now because the cost of maintaining huge context per user to enable milking rather than enhanced-thinking is too high.
Also consider that we don't want to enable milking. People who have no idea what they're doing will just throw a lot of code at the wall and then when there's a huge mess they'll be asking engineers to fix it. We need to be careful with who has this kind of access on our teams in general. Someone who is non-technical should not be given a firecracker before they've ever even just turned on a microwave or a stove.