Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-4 is not getting worse

coagulopath.com

Post reply on HN
Print view View original

111–120 of 186 posts

Re: GPT-4 is not getting worse

#111

Post by jw1224 »

SeanAnderson wrote: post #106
jw1224 wrote: post #35

There is an ongoing bug[1] with OpenAI's API, where it stops streaming responses after exactly 5 minutes. When I first came across the issue, I debugged it by writing a prompt along these lines: > Output every number from 1 to 10,000 in written form (e.g. "one", "two", etc.). Respond with one number per line in numeric order. As expected, the API would begin counting every number just as I asked. This would continue…

This doesn't seem like an unintentional behavior? It seems like they don't want people querying their API for an indefinite amount of time. What amount of time do you expect a query to be able to run for before timing out?

I lost countless hours debugging this, finally I came to the conclusion it must be unintentional because:

  1. It's undocumented. None of the regular rate limit responses are returned.
  2. You're charged for the full generation length. So if the output takes 10 minutes to generate, that's what you'll pay for (despite only getting half back).
  3. It defeats the point of the larger context limit models. Why offer a 32K model if it fails after ~6K tokens?
  4. The server response doesn't include any error codes or message, it simply terminates unexpectedly. Hit any of the actual rate limits, and you get told about it.
I'd expect to be able to generate output until the model reaches its context limit, or a stop sequence is detected, or I hit an actual documented rate limit.

We're paying for these requests in full. We should get the full response back!

Re: GPT-4 is not getting worse

#112

Post by vharuck »

lairv wrote: post #87
rendall wrote: post #11

> I hate its craven, cringing, condescending, bitch-broken, mewling, groveling, wheedling, suck-ass tone. I hate its endless passive voice and weasel words. I hate its pompous It is important to remembers and it should be noteds.* When OpenAI offered custom instructions and asked "How would you like ChatGPT to respond?" I wrote: Be terse. Do not offer unprompted advice or clarifications. Remain neutral on all topics.…

The main pain point I still have with GPT4 is that it's hard to prevent it from being a yes-man "- I have problem A, what do you think of X ? - Yes X is great ... - But with X I'd encounter problem B, maybe I should go with Y ? - You're right X would cause problem B. With Y ..." One way to limit this issue is to not give ChatGPT any hint on your intent, instead of writing "I have problem A, what do you think of X ?",…

The training data of written content, no matter the sources, will always be biased in one way: it doesn't contain thoughts people kept to themselves.

Example 1: Mr. X is very popular author among a small group, who chat daily about it on various websites. Outside of that group, many people know about Mr. X but don't care much about his works. So the majority of content about Mr. X is positive, even though the majority of opinions are ambivalent.

Example 2: Somebody asks for help on StackOverflow. Two replies offer different answers, A and B. So the answers seem like they'd have equal value. If 90% of people think A is the right answer, the written content wouldn't reflect that, because those people won't write duplicate answers.

I'm sure there are ways to account for these examples. Assigning different "weights" to content sources could help with #1, and feedback mechanisms like upvotes could help with #2. But I doubt those fixes would cover all the problems with the bias against thoughts that people never shared.

Re: GPT-4 is not getting worse

#113

Post by jw1224 »

weird-eye-issue wrote: post #107
jw1224 wrote: post #35

There is an ongoing bug[1] with OpenAI's API, where it stops streaming responses after exactly 5 minutes. When I first came across the issue, I debugged it by writing a prompt along these lines: > Output every number from 1 to 10,000 in written form (e.g. "one", "two", etc.). Respond with one number per line in numeric order. As expected, the API would begin counting every number just as I asked. This would continue…

The issue you linked to is related to an SDK, not the API itself. Have you tried just directly streaming the response back without an SDK? For example we use fetch directly without any SDK for OpenAI in production with no issue

I've tried making manual requests without an SDK, exactly the same thing happens. Even when you don't use streaming (wait for the full response), it still occurs.

Whilst the bug is API-specific, there's no other way to report this, and that issue is the only place I've found anyone discussing this.

I contacted OpenAI support about it, waited several weeks, and just got a generic useless autoreply back.

Re: GPT-4 is not getting worse

#114

Post by jw1224 »

manojlds wrote: post #109
jw1224 wrote: post #35

There is an ongoing bug[1] with OpenAI's API, where it stops streaming responses after exactly 5 minutes. When I first came across the issue, I debugged it by writing a prompt along these lines: > Output every number from 1 to 10,000 in written form (e.g. "one", "two", etc.). Respond with one number per line in numeric order. As expected, the API would begin counting every number just as I asked. This would continue…

That looks like a library issue than an API issue no?

Definitely an API fault, but that library issue is the only place anyone's discussing it so far. OpenAI support were no help, I waited several weeks and they just sent back an autoreply linking to their docs.

Re: GPT-4 is not getting worse

#115

Post by porkbeer »

rendall wrote: post #11

> I hate its craven, cringing, condescending, bitch-broken, mewling, groveling, wheedling, suck-ass tone. I hate its endless passive voice and weasel words. I hate its pompous It is important to remembers and it should be noteds.* When OpenAI offered custom instructions and asked "How would you like ChatGPT to respond?" I wrote: Be terse. Do not offer unprompted advice or clarifications. Remain neutral on all topics.…

I did the same but still get its apologetic handwavy crap, although somewhat less.

Re: GPT-4 is not getting worse

#116

Post by porkbeer »

chaosbolt wrote: post #5

Your problem is you want to be a contrarian. >I've hated chatgpt for a very long time because of how it sounds >it's not getting worse I want the opinion of the people who objectively saw it as a tool and didn't like or hate it because of external opinions. For example I've been using it since it came out, and I (no charts or data or proof) have just felt that it was nerfed... I mean maybe it wasn't who knows, but as…

I was an early subscriber and cancelled it after it got objecively worse at answering my questions. It may not be 'worse' for everyone, but it sure seems worse for me as time went on. I ended up just arguing rather than get anything done. Looking back, I am glad to be free of its mental crutch that never seemed appropriate when i needed it.

Re: GPT-4 is not getting worse

#117

Post by donpark »

gberger wrote: post #36
jw1224 wrote: post #35

There is an ongoing bug[1] with OpenAI's API, where it stops streaming responses after exactly 5 minutes. When I first came across the issue, I debugged it by writing a prompt along these lines: > Output every number from 1 to 10,000 in written form (e.g. "one", "two", etc.). Respond with one number per line in numeric order. As expected, the API would begin counting every number just as I asked. This would continue…

That looks like a timeout configured on the web servebr

Very likely. Typical serverless hosts like Vercel times out at 5 minutes.

Re: GPT-4 is not getting worse

#118

Post by JoshTko »

JimDabell wrote: post #34

All of the tests are one-shot questions and answers. Where I have found GPT-4 to be degrading significantly is with sustained discussion about technical topics. It starts forgetting important parts of the discussion almost straight away, long before the size of the context window becomes a factor. This wasn’t the case when it was new.

I definitely experienced this as well. Like conversation context is soo quickly forgotten these days.

Re: GPT-4 is not getting worse

#119

Post by oezi »

pelorat wrote: post #79

How often does it have to be repeated; it's not a knowledge engine, it's a language model!

How often does it have to be repeated, it's not a nail fixation engine, it's a hammer.

But surprise, people use tools for purposes. Many tools can be used for quite many purposes not in the original intended purpose.

Re: GPT-4 is not getting worse

#120

Post by isaacfung »

wouldbecouldbe wrote: post #59
jw1224 wrote: post #35

There is an ongoing bug[1] with OpenAI's API, where it stops streaming responses after exactly 5 minutes. When I first came across the issue, I debugged it by writing a prompt along these lines: > Output every number from 1 to 10,000 in written form (e.g. "one", "two", etc.). Respond with one number per line in numeric order. As expected, the API would begin counting every number just as I asked. This would continue…

It's actually supper annoying. I keep asking it to write out entire pieces of code, instead of only things it changed, can be annoying to merge it otherwise. But nowadays it acknowledges my request "I will write complete code" and proceed to still comment out parts.

Claude instant 100k does this all the time.
Post reply on HN