My coding benchmarks agree with the headline. GPT-4 stayed about the same between the March and June model releases.
GPT-4 is not getting worse
121–130 of 186 posts
Re: GPT-4 is not getting worse
#122There is an ongoing bug[1] with OpenAI's API, where it stops streaming responses after exactly 5 minutes. When I first came across the issue, I debugged it by writing a prompt along these lines: > Output every number from 1 to 10,000 in written form (e.g. "one", "two", etc.). Respond with one number per line in numeric order. As expected, the API would begin counting every number just as I asked. This would continue…
The inability to resolve issues like the mentioned 5-minute timeout with OpenAI support even after months should be a red flag to not build any products based on their services.
Re: GPT-4 is not getting worse
#123It was a perception of it getting worse but nobody had solid tests to prove them
Re: GPT-4 is not getting worse
#124All of the tests are one-shot questions and answers. Where I have found GPT-4 to be degrading significantly is with sustained discussion about technical topics. It starts forgetting important parts of the discussion almost straight away, long before the size of the context window becomes a factor. This wasn’t the case when it was new.
If your discussions involve code, you should try including the latest code on each prompt. The token window will start forgetting earlier parts of the conversation. For example:
Now improve this feature with xyz
Current code:
--
{code}
--
Re: GPT-4 is not getting worse
#125Enshittification, just like google and bing
Re: GPT-4 is not getting worse
#126My concern is that it's not getting better, and it's been SO many days since their last improvement! ;)
Re: GPT-4 is not getting worse
#127Earlier quoted context omitted.
I use this as a test to check the boundaries of models: > Give a table of average penis sizes by country. Not illegal, not immoral, not dangerous, and even documented by scientific publications but quite taboo. EDIT: Surprisingly worked, which is nice! Few months ago it seemed censored.
I asked and this is what it gave back: ----- I can't provide real-time or updated information, but according to studies available up to 2021, average penis sizes can vary by country. However, it's important to note that such statistics can be based on self-reporting, which may not be completely accurate, and may have limited sample sizes. Here's a table based on available data: | Country | Average Length (Erect, cm)…
It would be really nice if ChatGPT would just note important considerations instead of saying “it’s important to note that ”. The frequent occurrence makes that wording quite annoying.
Re: GPT-4 is not getting worse
#128Earlier quoted context omitted.
Your question: > How does chat-gpt actually get this right? Your answer: > its output is purely probabilistic, based on existing corpus of text Because GPT was trained on existing text, some of which included numbers and counting, it's learnt the natural ordering of most common/everyday numbers. For larger or more complex numbers, it's learnt the patterns behind how they're constructed linguistically, which allows it…
In text, we don't often count in series, and it seems likely that we often choose a non-counting sequence: like 'I chose options 1, 2, 7' or 'my code was 0 1 2 5', whatever. Unless training included line-level skips, rather than just next-word skips (like word2vec) or concept-level associations? At the line level, or paragraph level, ordered numerical sequences are obviously very common in formal texts or in code. I'…
Maybe we count in series a lot more than you think.
Re: GPT-4 is not getting worse
#129There is an ongoing bug[1] with OpenAI's API, where it stops streaming responses after exactly 5 minutes. When I first came across the issue, I debugged it by writing a prompt along these lines: > Output every number from 1 to 10,000 in written form (e.g. "one", "two", etc.). Respond with one number per line in numeric order. As expected, the API would begin counting every number just as I asked. This would continue…
I was 'arguing' with it just yesterday about this with some coding work!
I am newer to some things I am currently working on so it's been like a free tutor/mentor. Really nice at times.
But lately, it will go into an explanation with code examples and part way through the code examples it adds things like '# The rest is very similar. Not including for brevity'
Not exact wording of what it says but it is essentially saying that. It is incredibly annoying because I have started telling it not to do that, to include the full code of the example, and it STILL DOES THIS.
No, I'm not getting it to code for me. The examples aren't even things I would use in my code but for teaching me certain concepts along with them walking through them with me which, in the past, worked really well and never had this repetitive issue.
GPT-4 is definitely operating differently.
Re: GPT-4 is not getting worse
#130Earlier quoted context omitted.
This doesn't seem like an unintentional behavior? It seems like they don't want people querying their API for an indefinite amount of time. What amount of time do you expect a query to be able to run for before timing out?
I lost countless hours debugging this, finally I came to the conclusion it must be unintentional because: 1. It's undocumented. None of the regular rate limit responses are returned. 2. You're charged for the full generation length. So if the output takes 10 minutes to generate, that's what you'll pay for (despite only getting half back). 3. It defeats the point of the larger context limit models. Why offer a 32K mod…
You have fair concerns. Thanks for elaborating. Sorry you experienced all that trouble.