Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?

news.ycombinator.com

Post reply on HN
Print view View original

771–780 of 817 posts

Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?

#771

Post by com2kid »

krainboltgreene wrote: post #740
com2kid wrote: post #676

Earlier quoted context omitted.

The API chat endpoint dramatically changes its responses every few weeks. You can spend hours crafting a prompt and then a week later the responses to that same prompt can become borderline useless. Writing against the ChatGPT API is like working against an API that breaks every other week with completely undocumented changes.

> The API chat endpoint dramatically changes its responses every few weeks. You can spend hours crafting a prompt and then a week later the responses to that same prompt can become borderline useless. Welcome to statistical randomness?

No, these are clear creative differences.

I submit the same prompt dozens of times a day and run the output through a parser. It'll work fine for weeks then I have to change the prompt because now 20% of what is returned doesn't follow the format I've specified.

A couple months ago the stories ChatGPT 3.5 returned were simple, a few sentences in each paragraph, then a conclusion. Sometimes there were interesting plot twists, but the writing style was very distinct. Same prompt now gets me dramatically different results, characters are described with so much detail that the AI runs out of tokens before the story can be finished.

Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?

#772

Post by deadeye »

deanCommie wrote: post #767
deadeye wrote: post #765

Earlier quoted context omitted.

[flagged]

> In fact I'm happy to let anyone identify as anything, as long as I'm not compelled to pretent along with them. If a person legally changes their name (forget gender, only name), and you refuse to use it, and insist on using the old name even after requests to stop, at some point that would become considered malicious and become harassment. But ultimately because society and science deems that "name" is not somethin…

[flagged]

Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?

#773

Post by lee101 »

scottfr wrote: post #134
bbotond wrote: post #40

Earlier quoted context omitted.

“Bard isn’t currently supported in your country. Stay tuned!”

The Bard model (Bison) is available without region lock as part of Google Cloud Platform. In addition to being able to call it via an API, they have a similar developer UI to the OpenAI playground to interactively experiment with it. https://console.cloud.google.com/vertex-ai/generative/langua...

[dead]

Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?

#774

Post by mcny »

boringuser2 wrote: post #647
caddemon wrote: post #638

Earlier quoted context omitted.

I've heard such mixed things about Bard lately, I wonder if it depends on the application one is trying to use it for? And yeah there's definitely good reason to work on scalability but they are charging such a cheap rate to begin with, it seems like there could be a middle ground here. Increasing the cost of the full compute power to the point of profitability and leaving it up as an option wouldn't prevent them fro…

It seems like google has been pumping Bard as a competitor to ChatGPT, but every time I use it for trivial tasks, it completely hallucinates something absurd after showing only a modicum of what could be perceived to be "understanding". My mileu is programming, general tech stuff, philosophy, literature, science, etc. -- a wide berth. The only sample I probably don't have it representative for is producing fiction wr…

The next time it gives you a wrong answer and you know the correct answer, try saying something like “that is incorrect can you please try again” or something like that.

Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?

#775

Post by jonathan-kosgei »

bbotond wrote: post #14

Yes. Before the update, when its avatar was still black, it solved pretty complex coding problems effortlessly and gave very nuanced, thoughtful answers to non-programming questions. Now it struggles with just changing two lines in a 10-line block of CSS and printing this modified 10-line block again. Some lines are missing, others are completely different for no reason. I'm sure scaling the model is hard, but they l…

I just tried a comparison of ChatGPT, Claude and Bard to write a python function I needed for work and ChatGPT (using GPT-4) whined and moaned about what a gargantuan task it was and then did the wrong thing. Claude and Bard gave me what I expected.

Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?

#776

Post by inciampati »

PeterStuer wrote: post #129
avocade wrote: post #110

Earlier quoted context omitted.

Interesting, please expound since very few of us had access pre-launch.

The video I posted referenced this. In summary: The person had access to early releases through his work at Microsoft Research where they were integrating GPT-4 into Bing. He used "Draw a unicorn in TikZ" (TikZ is probably the most complex and powerful tool to create graphic elements in LaTeX) as a prompt and noticed how the model's responses changed with each release they got from OpenAI. While at first the drawings…

I experienced the same thing as a user of the public service. The system could at one point draw something approximating a unicorn in tikz. Now, its renditions are extremely weak, to the point of barely resembling any four-legged animal.

Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?

#777

Post by inciampati »

pmarreck wrote: post #712
renewiltord wrote: post #556

Earlier quoted context omitted.

There's a section in the GPT-4 release docs where they talk about how the safety stuff changes the accuracy for the worse.

this, more than anything, makes me want to run my own open-source model without these nearsighted restrictions

Indeed, this is the most important step we need to make together. We must learn to build, share, and use open models that behave like gpt-4. This will happen, but we should encourage it.

Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?

#778

Post by jeeeberz »

deadeye wrote: post #772
deanCommie wrote: post #767

Earlier quoted context omitted.

> In fact I'm happy to let anyone identify as anything, as long as I'm not compelled to pretent along with them. If a person legally changes their name (forget gender, only name), and you refuse to use it, and insist on using the old name even after requests to stop, at some point that would become considered malicious and become harassment. But ultimately because society and science deems that "name" is not somethin…

[flagged]

[dead]

Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?

#779

Post by theaussiestew »

tmikaeld wrote: post #750
theaussiestew wrote: post #745

Earlier quoted context omitted.

Impressive, this'll cut down on my work a lot. When I say Bing, I meant Bing AI which also uses GPT-4. Can you share some of the prompts you've been using? I'm assuming you don't need to paste in context around the library, you simply ask it to use TipTap and it'll do that?

Yeah I won't be using Edge just to use AI. It takes a bit of back-and-forth, just be clear about which version of tiptap it should write extensions for, the new v2 is very different from v1 and since the cutoff is 2021, it's missing a bit of information. But in general, it knows the public api very well, so markers and dom works great!

Very impressive, hearing this just made my job much easier.

Re: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?

#780

Post by benxh »

napier wrote: post #770
benxh wrote: post #741

Earlier quoted context omitted.

To be honest, I've been asking myself the same thing, technically the amount of "good quality" data in libgen is huge, way larger than the books3 dataset. However it would probably run afoul of copyright. Then again, a huge amount of data that LLMs go through is copyrighted.

Training on copyright data is arguably considered fair use in quite a few jurisdictions to various extents and levels of precedent, and entirely legal for entities based in Japan.

Yes, but the acquisition of that data itself is illegal in almost all jurisdictions, since libgen is treated as a piracy website. Now if there were a pipeline to access books from Amazon or the Google Books project for training it would be a different story.

Still, for certain languages, only libgen and public piracy websites contain any scientific or fiction material in digital formats. E.g. my native language doesn't have easily accessible e-books at all, unless you go through illegal means.

I hope somebody undertakes the steps necessary to train on the entirety of libgen. The amount of high quality tokens in libgen should be substantial.

Post reply on HN