I don't think it's any worse at all. I think what most people are expressing here is reaching the limits of the technology and realizing that it's not magic.
Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?
461–470 of 817 posts
Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?
#462Is there a potential short term solution that can utilize p2p networking to train and run an open GPT instance openly? (Until we reach a point to run larger, quality networks efficiently with simpler resources)
Yep. AI Horde.
I dunno about training. There have been promises of p2p AI training ever since the crypto/web3 boom, but thats far less trivial to network than inference jobs like AI Horde does. Vast.ai is kinda like p2p GPU compute.
Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?
#463Earlier quoted context omitted.
If you want a datapoint, Origami is a "pure Ruby library to parse, modify and generate PDF documents". That library cloc's in at 13,683 lines of code and 3,295 lines of comments.
Thats not a lot of code tho, but i see your point.
Try getting GPT-4 to spit out that much code and have it be coherent and run together.
Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?
#464Earlier quoted context omitted.
>To me, it feels like it's started giving superficial responses and encouraging follow-up elsewhere -- I wouldn't be surprized if its prompt has changed to something to that effect. That's the vibe I've been getting. The responses feel a little cagier at times than they used to. I assume it's trying to limit hallucinations in order to increase public trust in the technology, and as a consequence it has been nerfed a…
Seems like the metric they're optimising for is reducing the number of bad answers, not the proportion of bad answers, and giving non-answers to a larger fraction of questions will achieve that.
I haven't noticed ChatGPT-4 to give worse answers overall recently, but I have noticed it refusing to answer more queries. I couldn't get it to cite case law, for example (inspired by that fool of a lawyer who couldn't be bothered to check citations).
Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?
#465Earlier quoted context omitted.
I think the only real path forward is for somebody to create an open source "unaligned" version of GPT. Any corporate controlled AI is going to be nerfed to prevent it from doing things that its corporate master considers to not be in the interests of the corporation. In addition, most large corporations these days are ideological institutions so the last thing they want is an AI that undermines public belief in thei…
Who has the necessary resources to run, let alone train the model?
I think a lot of people are unaware that these models have an enormous human training component performed through companies such as Amazon Mechanical Truk and dataannotation.tech. Called Human Intelligence Tasks, a large number of people have been working in this area for close to a decade. Dataannotation Tech claims to have over 100k workers. From Cloud Research,
"How Many Amazon Mechanical Turk Workers Are There in 2019? In a recent research article, we reported that there are 250,810 MTurk workers worldwide who have completed at least one Human Intelligence Task (HIT) posted through the TurkPrime platform. More than 226,500 of these workers are based in the US."
Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?
#466A lot of people in this thread are making fuzzy claims. We can only evaluate if there are clear examples and test results.
Is anyone taking screenshots or sharing their chat logs, where they run the same questions over some time period?
Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?
#467This is inevitable. Elsewhere, I've argued that the most likely response is to replicate Middle Age European guilds -- for isolated communities to train their own LLMs on their own proprietary or confidential texts, and use them only through internal tooling.
Proprietary LLMs that optimize for performance will out compete Public LLMs that optimize for political correctness or social acceptance as explained by another commenter [1].
Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?
#468Earlier quoted context omitted.
Also the deal to make the browsing model to only use Bing. That's bait and switch. I paid for browsing, and now it only browses Bing. They even had the gall to update the plugin name to Browsing with Bing.
It can definitely browse websites that aren't Bing, I asked it to look at a page that isn't in the bing cache and it worked.
Clearly "Browse with Bing" doesn't mean that it will only browse bing.com, but what exactly does it mean? I can't quite figure it out. Is it that it's identifying as a Bing crawler?
Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?
#469Earlier quoted context omitted.
Who has the necessary resources to run, let alone train the model?
All of us together do. I saw the nerfing of GPT in real time: one day it was giving me great book summaries, the next one it said that it couldn't do it due to copyright. I actually called it in a comment several months ago: copyright and other forms of control would make GPT dumb in the long run. We need an open source frontier less version.
There is a lot of randomness involved, are you sure it wasn’t just chance? If you try again it might work
Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?
#470Reading all the comments here, seems like being able to run your own models is vital. If not, you are subject to a service where the capabilities are changing underneath you constantly and without notice.
This is easy to say, but its seems like most of HN hasn't dabbled with setting up local LLaMA finetunes
Hooking them up to a good embeddings database (so the model doesn't hallucinate so much) is particularly tricky.