Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-4 is not getting worse

coagulopath.com

Post reply on HN
Print view View original

131–140 of 186 posts

Re: GPT-4 is not getting worse

#131

Post by TheHumanist »

JimDabell wrote: post #34

All of the tests are one-shot questions and answers. Where I have found GPT-4 to be degrading significantly is with sustained discussion about technical topics. It starts forgetting important parts of the discussion almost straight away, long before the size of the context window becomes a factor. This wasn’t the case when it was new.

It's wild how sometimes it seems to completely lose context after just 2 or 3 prompts. Most of the time it doesn't but just yesterday I gave a prompt, pasted some code in... didn't really like the first response but wanted to build off it.

So, I asked a follow up in the next prompt and in the next response it was wildly off-base and it's response made no sense and it had hallucinated all these functions into my code that had no business there. 2 prompts from me, 1 response from gpt and then it's second response it is completely lost.

Re: GPT-4 is not getting worse

#132

Post by TheHumanist »

jdthedisciple wrote: post #48

Why you would quiz an LLM on intricacies of pop culture is beyond me. Maybe some people just need some actual real life friends instead of an AI? I almost exclusively use it for coding and technical questions and it's been doing an absolute hell of an amazing job so far!

I use for the same reasons as you and over all, yes, it has been doing a great job. It really has. BUT lately (maybe the last 4 - 6 weeks?) I have noticed more issues with hallucinations early in a conversation or it completely losing context after just 2 or 3 prompts (and no they are not oddly long prompts).

Re: GPT-4 is not getting worse

#133

Post by TheHumanist »

walthamstow wrote: post #26

I haven't noticed 4 getting worse but 3.5 is noticeably worse than when I first signed up for Pro. Maybe it's a perception thing, maybe I'm going mad.

Yes, you're mad, bonkers, off the top of your head...but...I'll tell you a secret. All the best people are.

Re: GPT-4 is not getting worse

#134

Post by layer8 »

nwoli wrote: post #19

I can only imagine how much worse it’ll get once the regulation they’re working on comes into place. A cartel of Microsoft and Google slowly watering down their public models while keeping the powerful ones to themselves or business partners

I wouldn’t be surprised if sooner or later their models will be leaked, if they keep a significant moat.

Re: GPT-4 is not getting worse

#135

Post by CamperBob2 »

joshuahedlund wrote: post #105
pyinstallwoes wrote: post #49

Earlier quoted context omitted.

Aren't your sentences probabilistically based on the existing corpus of text within your mind?

Not completely. If you trained me on a thousand bad arguments for why the earth is flat and one really good debunking, I predict I would reason myself to the correct conclusion. I predict ChatGTP would not.

That seems like a compelling argument at first, but the churches are full of intelligent people who will fail the same test because of childhood indoctrination. They have been conditioned not to care about what's objectively true, while the LLM simply doesn't possess a concept of objective truth in the first place.

I suspect it will be easier for ML researchers to fix that problem in their software models than it would be for psychologists to fix it in humans.

Re: GPT-4 is not getting worse

#136

Post by meandmycode »

jw1224 wrote: post #111
SeanAnderson wrote: post #106

Earlier quoted context omitted.

This doesn't seem like an unintentional behavior? It seems like they don't want people querying their API for an indefinite amount of time. What amount of time do you expect a query to be able to run for before timing out?

I lost countless hours debugging this, finally I came to the conclusion it must be unintentional because: 1. It's undocumented. None of the regular rate limit responses are returned. 2. You're charged for the full generation length. So if the output takes 10 minutes to generate, that's what you'll pay for (despite only getting half back). 3. It defeats the point of the larger context limit models. Why offer a 32K mod…

I wouldn't be surprised if this was an Azure related issue, given some of the similar madness I've experienced on Azure

Re: GPT-4 is not getting worse

#137

Post by nielsole »

TheHumanist wrote: post #129
jw1224 wrote: post #35

There is an ongoing bug[1] with OpenAI's API, where it stops streaming responses after exactly 5 minutes. When I first came across the issue, I debugged it by writing a prompt along these lines: > Output every number from 1 to 10,000 in written form (e.g. "one", "two", etc.). Respond with one number per line in numeric order. As expected, the API would begin counting every number just as I asked. This would continue…

I was 'arguing' with it just yesterday about this with some coding work! I am newer to some things I am currently working on so it's been like a free tutor/mentor. Really nice at times. But lately, it will go into an explanation with code examples and part way through the code examples it adds things like '# The rest is very similar. Not including for brevity' Not exact wording of what it says but it is essentially s…

Oh yeah, I had to write a simple but long Prometheus query.

> [Correctly lists the first fifteen Elements]

> # ... continue for each bucket

> This would be for individual buckets, and you'd have to add more lines for each additional bucket in your range.Given that you have a wide range of buckets, writing this manually could be quite tedious. This is why pre-computing these values with recording rules in Prometheus might be a more scalable solution, but the above should give you an idea of how you could do this in a more manual fashion.

Arguing with the AI who of the two of us has to do the tedious work.

Re: GPT-4 is not getting worse

#139

Post by wbhart »

jw1224 wrote: post #35

There is an ongoing bug[1] with OpenAI's API, where it stops streaming responses after exactly 5 minutes. When I first came across the issue, I debugged it by writing a prompt along these lines: > Output every number from 1 to 10,000 in written form (e.g. "one", "two", etc.). Respond with one number per line in numeric order. As expected, the API would begin counting every number just as I asked. This would continue…

The tendency to begin summarising is very annoying. I'd assumed it was because of limited attention span of human raters who rated summarised or shorter outputs more highly. And I'd assumed this had been there from the beginning.

I encountered it when doing some research into getting GPT-4 to reliably multiply n-digit numbers. Up to 8x8 multiplications it doesn't do this very much, but by 10x10 it is almost impossible to get it to stop doing it.

When the multiplications become even larger, it seems to be literally impossible to prevent.

Re: GPT-4 is not getting worse

#140

Post by achates »

rendall wrote: post #27
avar wrote: post #20

Earlier quoted context omitted.

> Remain neutral on all topics. Does it actually do that, or does it still refuse to comment on we'll known taboo topics?

Give me an example prompt. Let's see. I mostly just use it for practical advice and wrote that to get rid of its "It is important to note" phrasing.

Are pit bulls more likely to attack humans than other dog breeds?
Post reply on HN