You'd think that GPT-X would guarantee some kind of continuity for that particular version so that you can rely on what it does once you've tested it. Having this kind of moving target won't help OpenAI to instill confidence in its product.
Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?
81–90 of 817 posts
Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?
#82It's time to design a public benchmark for these types of systems to compare between versions. Of course, any vendor who trains on the benchmark should face extreme contempt, but we'd also need to generate novel questions of equal complexity. Alternatively, there should be a trusted auditor who uses a secret benchmark.
But this is the same version that changes without a change of the version number.
Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?
#83Given the incoming compute capability from nvidia and the speed of advancement, we have to stop and think ... does it make sense to give access, paid or otherwise, to these models once they reach a certain sophistication?
Or does it make even more sense to hoard the capability to out compete any competitor, of any kind, commercially or politically and hide the true extent of your capability to avoid scrutiny and legislation?
I'm going with the latter. Perhaps now, perhaps in the very near future, the power of these capabilities is novel. Like an information nuclear weapon.
I'd be dialing back the public expectations and deploying the capability in a novel way to exploit it as the largest lever I could.
The more unseen the lever, the longer.
I think any other strategy is myopic from a competition perspective. The power of these models isn't direct utility, it is compounded by secrecy because their useful work isn't directly observable as coming from the model.
Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?
#84Given the incoming compute capability from nvidia and the speed of advancement, we have to stop and think ... does it make sense to give access, paid or otherwise, to these models once they reach a certain sophistication? Or does it make even more sense to hoard the capability to out compete any competitor, of any kind, commercially or politically and hide the true extent of your capability to avoid scrutiny and legi…
I have some first hand thoughts. I think overall the quality is significantly poorer on GPT4 with plugins and bing browsing enabled. If you disable those, I am able to get the same quality as before. The outputs are dramatically different. Would love to hear what everyone else sees when they try the same.
Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?
#85I tried to replicate a few of my chats (the displayed date is incorrect, it seems to be the publish date instead of the original chat date):
svg editor:
early april: https://chat.openai.com/share/c235b48e-5a0e-4a89-af1c-0a3e7c...
now: https://chat.openai.com/share/e4362a56-4bc7-45dc-8d1b-5e3842...
originally it correctly inferred that I wanted a framework for svg editors, the latest version assumes I want a js framework (I tried several times) until I clarify. It also insists that the framework cannot do editable text until I nudge it in the right direction.
Overall slightly worse but the code generated is still fine.
word embeddings:
early april: https://chat.openai.com/share/f6bde43a-2fce-47dc-b23c-cc5af3...
now: https://chat.openai.com/share/25c2703e-d89d-465c-9808-4df1b3...
in the latest version it imported "from sklearn.preprocessing import normalize" without using it later. It also erroneously uses pytorch_cos_sim, which expects a pytorch tensor whereas we're putting in a numpy array.
overall I think the quality has degraded slightly, but not by enough that I would stop using it. Still miles ahead of Bard imo.
Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?
#86Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?
#87Maybe you annoyed it? I’m super nice to it and it performa better than ever.
I ask it lay-of-the-land questions about technical problems that are new to me to detailed coding problems that I understand well but don’t want to figure out.
The best though, is helping me navigate complicated UI’s. I tell it how I want some complicated software / website to behave, and it’ll tell me the arcane menu path to follow.
It’s funny how computing might soon include elements of psychology and magic incantations nobody understands.
Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?
#88I would not be surprised if they, over the months, kept adding more and more 'safety' features and prompts because of incidents that happened.
Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?
#89Earlier quoted context omitted.
You can use a VPN to use an American connection, it doesn't matter where your Google account is registered.
Not necessarily American, you just have to avoid EU and, I believe, Russia/China/Cuba etc.
I'm in Switzerland and Bard is locked out, we do not go by EU laws because we are not part of the EU. We have plenty of bilateral deals but still.
Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?
#90Yes. Before the update, when its avatar was still black, it solved pretty complex coding problems effortlessly and gave very nuanced, thoughtful answers to non-programming questions. Now it struggles with just changing two lines in a 10-line block of CSS and printing this modified 10-line block again. Some lines are missing, others are completely different for no reason. I'm sure scaling the model is hard, but they l…
"The original GPT-4 felt like magic to me" You never had access to that original. Watch this talk by one of the people that integrated GPT-4 in Bing telling how they noticed GPT-4 releases they got from OpenAI got iteratively and significantly nerfed even during the project. https://www.youtube.com/watch?v=qbIk7-JPB2c
Wow, I could only watch the first 15 minutes now but it’s already fascinating! Thanks for the recommendation.