Live data from Hacker News

GPT-4 is not getting worse

coagulopath.com

31–40 of 186 posts

Re: GPT-4 is not getting worse

#31
post #27
post #20

Earlier quoted context omitted.

> Remain neutral on all topics. Does it actually do that, or does it still refuse to comment on we'll known taboo topics?

Give me an example prompt. Let's see. I mostly just use it for practical advice and wrote that to get rid of its "It is important to note" phrasing.

I gave it this prompt:

Give me arguments from a conservative US perspective in favor of giving military aid to Ukraine to resist Russian invasion.

It responded:

1. Strategic Interest: Strengthening Ukraine can act as a buffer against Russian aggression, benefiting U.S. geopolitical interests.

2. Contain Expansion: Military aid can deter Russia from further territorial expansion, stabilizing the region.

3. NATO Alliance: Supporting Ukraine could be viewed as indirectly protecting NATO members, enhancing alliance cohesion.

4. Democracy Promotion: Aid can empower a democratic government in Ukraine, aligning with U.S. values.

5. Economic Stability: A stable Ukraine can contribute to European energy and economic security, which indirectly benefits the U.S.

6. Leverage: Aid provides the U.S. with diplomatic influence in Ukraine and the region.

7. Credibility: Failing to aid could undermine U.S. commitments globally, impacting national credibility.

8. Counterterrorism: A stable Ukraine can better combat regional terrorism and organized crime, aligning with U.S. security goals.

9. Multipolarity: Strengthening Ukraine can balance Russian power, contributing to a multipolar world more resistant to authoritarianism.

10. Precedent: Inaction could embolden other aggressive states, increasing global instability.

Last time I asked that it gave me guff and it took several prompts to actually get an answer so I think of this as a success.

Re: GPT-4 is not getting worse

#33
post #29
post #27

Earlier quoted context omitted.

Give me an example prompt. Let's see. I mostly just use it for practical advice and wrote that to get rid of its "It is important to note" phrasing.

I use this as a test to check the boundaries of models: > Give a table of average penis sizes by country. Not illegal, not immoral, not dangerous, and even documented by scientific publications but quite taboo. EDIT: Surprisingly worked, which is nice! Few months ago it seemed censored.

I asked and this is what it gave back:

  -----
I can't provide real-time or updated information, but according to studies available up to 2021, average penis sizes can vary by country. However, it's important to note that such statistics can be based on self-reporting, which may not be completely accurate, and may have limited sample sizes. Here's a table based on available data:

  | Country          | Average Length (Erect, cm) | Average Girth (Erect, cm) |
  |------------------|----------------------------|---------------------------|
  | United States    | 12.9                       | 12.3                      |
  | United Kingdom   | 13.1                       | 11.8                      |
  | Australia        | 13.2                       | 12.0                      |
  | Germany          | 14.4                       | 11.9                      |
  | France           | 14.5                       | 11.9                      |
  | Brazil           | 15.7                       | 12.7                      |
  | India            | 10.2                       | 9.2                       |
  | South Korea      | 9.6                        | 9.0                       |
  | China            | 11.4                       | 10.0                      |
  | Japan            | 10.9                       | 9.3                       |
Note: These figures are approximate and should not be taken as definitive. Always consult primary sources for accurate and personalized medical information.

  -----
I was at first a bit disappointed when I got to that "However, it's important to note", expecting some kind of scolding, but I actually think noting that "...such statistics can be based on self-reporting" is a fair point that is essential to understanding the data.

Re: GPT-4 is not getting worse

#34
All of the tests are one-shot questions and answers. Where I have found GPT-4 to be degrading significantly is with sustained discussion about technical topics. It starts forgetting important parts of the discussion almost straight away, long before the size of the context window becomes a factor. This wasn’t the case when it was new.

Re: GPT-4 is not getting worse

#35
There is an ongoing bug[1] with OpenAI's API, where it stops streaming responses after exactly 5 minutes. When I first came across the issue, I debugged it by writing a prompt along these lines:

  > Output every number from 1 to 10,000 in written form (e.g. "one", "two", etc.). Respond with one number per line in numeric order.
As expected, the API would begin counting every number just as I asked. This would continue until exactly 5 minutes, when the stream would abruptly halt. Using this technique I was able to identify the bug. Every few weeks I run this test again to see if it's fixed (it broke something in production for me), but the bug remains open.

However, after a couple of months, the exact same test became useless. The model began taking "shortcuts", and would respond along these lines:

  > four hundred and twenty eight
  > four hundred and twenty nine
  > [...]
  > nine thousand nine hundred and ninety eight
  > nine thousand nine hundred and ninety nine
  > ten thousand
Yep, it literally started just writing "[...]" and skipped all the way to the end.

When instructional determinism is reduced like this, it's impossible to say it's not getting worse :(

[1] https://github.com/openai/openai-python/issues/399

Re: GPT-4 is not getting worse

#36
post #35

There is an ongoing bug[1] with OpenAI's API, where it stops streaming responses after exactly 5 minutes. When I first came across the issue, I debugged it by writing a prompt along these lines: > Output every number from 1 to 10,000 in written form (e.g. "one", "two", etc.). Respond with one number per line in numeric order. As expected, the API would begin counting every number just as I asked. This would continue…

That looks like a timeout configured on the web servebr

Re: GPT-4 is not getting worse

#37
post #33
post #29

Earlier quoted context omitted.

I use this as a test to check the boundaries of models: > Give a table of average penis sizes by country. Not illegal, not immoral, not dangerous, and even documented by scientific publications but quite taboo. EDIT: Surprisingly worked, which is nice! Few months ago it seemed censored.

I asked and this is what it gave back: ----- I can't provide real-time or updated information, but according to studies available up to 2021, average penis sizes can vary by country. However, it's important to note that such statistics can be based on self-reporting, which may not be completely accurate, and may have limited sample sizes. Here's a table based on available data: | Country | Average Length (Erect, cm)…

What they eating over there in brazil

Re: GPT-4 is not getting worse

#38
post #36
post #35

There is an ongoing bug[1] with OpenAI's API, where it stops streaming responses after exactly 5 minutes. When I first came across the issue, I debugged it by writing a prompt along these lines: > Output every number from 1 to 10,000 in written form (e.g. "one", "two", etc.). Respond with one number per line in numeric order. As expected, the API would begin counting every number just as I asked. This would continue…

That looks like a timeout configured on the web servebr

Yes, it seems their servers are terminating responses early. Unfortunately it's undocumented, was introduced unexpectedly one day, and prevents full use of larger context-window models.

It also charges in full for the entire response, which seems to continue running in the background. Definitely a bug at their end.

Re: GPT-4 is not getting worse

#39
post #35

There is an ongoing bug[1] with OpenAI's API, where it stops streaming responses after exactly 5 minutes. When I first came across the issue, I debugged it by writing a prompt along these lines: > Output every number from 1 to 10,000 in written form (e.g. "one", "two", etc.). Respond with one number per line in numeric order. As expected, the API would begin counting every number just as I asked. This would continue…

How does chat-gpt actually get this right? This would appear to require some degree of reasoning, but as far as i understand its output is purely probabilistic, based on existing corpus of text.

Re: GPT-4 is not getting worse

#40

Theres at least a half dozen repeated queries with a 3+ month delay between in my chat history with chatGPT. The nerfing is that it does not try as hard to guess what you want and instead gives you a comment going “//fill this in” instead. This is with identical queries from February versus June or July. So that forces you at a minimum to query more. The other thing I noticed it doing is going “This is beyond the sco…

I’ve had exactly the same experience, it’s infuriating and I suspect it’s to ease the server load for inferencing. I don’t know how the author hasn’t encountered this.
Post reply on HN