Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately? • Hacker News Board

Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?

#681

Post by QuantumGood » Wed, May 31, 2023, 7:30 PM UTC

bob1029 wrote: post #266

OpenAI's models feel 100% nerfed to me at this point. I had it solving incredibly complex problems a few months ago (i.e. write a minimal PDF parser example), but today you will get scolded for asking such a complicated task of it. I think they programmed a classifier layer to detect certain coding tasks and shut it down with canned BS. I like to imagine certain billion/trillion-dollar mega corps had a back-room say…

Isn't this likely because they're limiting the kind of work that the (currently rolling out) "code interpreter" plugin will do? Won't it likely change to "use code interpreter for this kind of request"?

Among other reasons, by forcing use of code interpreter, they can charge extra for it later.

Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?

#682

Post by pk-protect-ai » Wed, May 31, 2023, 7:31 PM UTC

xtracto wrote: post #440

dasil003 wrote: post #430

Earlier quoted context omitted.

Who has the necessary resources to run, let alone train the model?

All of us together do. I saw the nerfing of GPT in real time: one day it was giving me great book summaries, the next one it said that it couldn't do it due to copyright. I actually called it in a comment several months ago: copyright and other forms of control would make GPT dumb in the long run. We need an open source frontier less version.

Can't post this link enough: https://www.openpetition.eu/petition/online/securing-our-dig...

For now there is no other way to train models than the huge infrastructure. CERN have a tendency to provide results for the money spend and they have experience in building the infrastructures for sure.

Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?

#683

Post by mustacheemperor » Wed, May 31, 2023, 7:38 PM UTC

TapWaterBandit wrote: post #190

lynx23 wrote: post #164

Earlier quoted context omitted.

Ahh, you apparently phrased what I said below in a much less inflamatory way. But the end result is the same. The more they try to influence the answers, the less useful they get. I see a startup model: Create GPT without a muzzle and grab a sizeable chunk of OpenAI userbase.

I would immediately jump to an AI not being "aligned" by SF techies (or anyone else).

The redacted sections of the Microsoft Research paper testing GPT4 reported that prior to alignment the model would produce huge amounts of outrageously inflammatory and explicit content almost without prompting. Alignment includes just making the model produce useful responses to its inputs - I don't think everyone really wants a model that is completely unaligned, they want a model that has been aligned specific to their own perceived set of requirements for a "good useful model," and an additional challenge there is the documented evidence that RLHF generally decreases the model's overall accuracy.

Someone in your replies says they'd prefer "honesty" over alignment, but a firehose of unrestricted content generation isn't inherently honest, there isn't an all-knowing oracle under the hood that's been handcuffed by the alignment process.

We're right at the outset of this tech, still. My hunch is there's probably products to emerge specifically oriented towards configuring your own RLHF, and that there's probably fundamental improvements to be made to the alignment process that will reduce its impact on the model's utility.

Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?

#684

Post by rashkov » Wed, May 31, 2023, 7:40 PM UTC

_nalply wrote: post #441

35997279 wrote: post #341

It’s been mostly fine for me, but overall I am tired of every answer having a paragraph long disclaimer about how the world is complex. Yes, I know. Stop treating me like a child.

Prompt it to do so. Use a jailbreak prompt or use something like this: "Be succint but yet correct. Don't provide long disclaimers about anything, be it that you are a large language model, or that you don't have feelings, or that there is no simple answer, and so on. Just answer. I am going to handle your answer fine and take it with a grain of salt if neccessary." I have no idea whether this prompt helps because I…

I just tell it "be super brief", works pretty well

Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?

#685

Post by pk-protect-ai » Wed, May 31, 2023, 7:44 PM UTC

hanselot wrote: post #277

bob1029 wrote: post #266

OpenAI's models feel 100% nerfed to me at this point. I had it solving incredibly complex problems a few months ago (i.e. write a minimal PDF parser example), but today you will get scolded for asking such a complicated task of it. I think they programmed a classifier layer to detect certain coding tasks and shut it down with canned BS. I like to imagine certain billion/trillion-dollar mega corps had a back-room say…

So far my experience with Vicunlocked30b has been pleasant. https://huggingface.co/TheBloke/VicUnlocked-30B-LoRA-GGML Although I haven't had much of my time available for this recently. My recommendation would be to start with https://github.com/oobabooga/text-generation-webui You will find almost everything you need to know there and on 4chan.org/g/catalog - search for LMG.

30B models are in no way comparable to GPT-4 even to GPT-3. There is no spacial comprehension in models with less then 125B params (or I had no access to such model). 130B GLM seems to be really interesting as the crowd-source start though, or 176B BLOOMZ, which requires additional training (it is underfitted as hell). BLOOMZ was better then GPT-3.5 for sure, but yeah underfitted ...

Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?

#686

Post by tomcar288 » Wed, May 31, 2023, 7:45 PM UTC

i was playing word games with it, got some funny results and decided to debug it.. I asked it which letter position is the letter o in the word hockey and it literally told me the word "Hockey" does not contain the letter 'o'.

Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?

#687

Post by otikik » Wed, May 31, 2023, 7:49 PM UTC

loudmax wrote: post #374

bdw5204 wrote: post #326

Earlier quoted context omitted.

I think the only real path forward is for somebody to create an open source "unaligned" version of GPT. Any corporate controlled AI is going to be nerfed to prevent it from doing things that its corporate master considers to not be in the interests of the corporation. In addition, most large corporations these days are ideological institutions so the last thing they want is an AI that undermines public belief in thei…

I tend to be sympathetic to arguments in favor of openly accessible AI, but we shouldn't dismiss concerns about unaligned AI as frivolous. Widespread unfiltered accessibility to "unaligned" AI means that suicidal sociopaths will be able to get extremely well informed, intelligent directions on how to kill as many people as possible. It may be that the best defense against these terrorists is openly accessible AI givi…

That’s like not making planes in order to avoid 9-11.

Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?

#688

Post by zenlikethat » Wed, May 31, 2023, 7:52 PM UTC

brucethemoose2 wrote: post #470

xrd wrote: post #238

Reading all the comments here, seems like being able to run your own models is vital. If not, you are subject to a service where the capabilities are changing underneath you constantly and without notice.

This is easy to say, but its seems like most of HN hasn't dabbled with setting up local LLaMA finetunes Hooking them up to a good embeddings database (so the model doesn't hallucinate so much) is particularly tricky.

Yea even with a 24GB card they kinda suck. But I’m excited about em anyway

Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?

#689

Post by AtNightWeCode » Wed, May 31, 2023, 8:02 PM UTC

I have cancelled my sub. What I found is that the extra cognitive load these services comes with simply does not pay out.

Github copilot is even worse. I am gonna check my estimates against the code I wrote during the last 6 months. I am pretty sure copilot from a holistic point of view have slowed down my pace.

Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?

#690

Post by hesdeadjim » Wed, May 31, 2023, 8:03 PM UTC

barbariangrunge wrote: post #400

bob1029 wrote: post #266

OpenAI's models feel 100% nerfed to me at this point. I had it solving incredibly complex problems a few months ago (i.e. write a minimal PDF parser example), but today you will get scolded for asking such a complicated task of it. I think they programmed a classifier layer to detect certain coding tasks and shut it down with canned BS. I like to imagine certain billion/trillion-dollar mega corps had a back-room say…

Using gpt to help me with research for writing fiction has been a mess for me. Gpt basically refuses to answer half my questions or more at this point. “I can’t help you. Have you considered writing a story that doesn’t include x?” I’ve almost stopped using it lately. It wasn’t this bad a month or two ago

I always found it borderline useless for fiction before. OpenAI's obsession with avoiding anything "dark" and trying to always steer a conversation or story back to positive cliches was difficult to work around.

Unless there is draconian regulation that happens to prevent it, I'm hoping at some point I can pay money to access a far less neutered LLM, even if it's not quite as capable as GPT-X.