31–40 of 315 posts
This topic has come up before, and my hypothesis is still that GPT-4 hasn't gotten worse, it's just that the magic has worn off as we've used this tech. Studies to evaluate it have gotten better and cleaned up mistakes in the past.
This is almost shocking to me. Can anyone confirm or deny seeing the same behavior? (i.e. refusing to do Chain-of-Thinking output)
I've been paying for GPT-4 since 3 hours after its release. The decrease in quality was noticeable just one week later (on top of the cap changes from 50 messages every 4 hours to 25 messages every 3 hours) I originally assumed that this was due to the increase in demand. It never went back to being as sharp as it was during those first hours of usage
OpenAI have repeatedly stated the model hasn't changed so how could this happen otherwise?
One thing that I have confirmed is while the abstract and intro talk about evaluating "code generation" as if GPT-4 code generation is getting worse, In is 3.3/Figure 4 it says they judge correctness only if it's passing raw code: "We call it directly executable if the online judge accepts the answer" not whether the code snippet is actually correct (!). The latest model outputs code as triple ticked in Markdown: "In June, however, they added extra triple quotes before and after the code snippet, rendering the code not executable." I mean, this is important if you're passing code directly into an API I suppose, but I don't think this should be properly extracted to judge code generation capability.
(I've had access to the Code Interpreter for several months now so I can't really say so much about the base GPT-4 model since I default to that most of the time for its programming abilities, but I use it basically every day and subjectively, I have not found the June update to make the CI model less useful).
One other potentially interesting data-point is that while the original GPT-4 Technical Report (https://arxiv.org/pdf/2303.08774v3.pdf) gave the Human Eval pass@1 score as 67%, independent testing from 3/15 (presumably on the 0314 model) seems was 85.36% (https://twitter.com/amanrsanger/status/1635751764577361921). And this current paper https://arxiv.org/abs/2305.01210 (well worth reading for those interested in LLM coding capabilities) scored GPT-4's pass@1 at 88.4%, which point towards coding capabilities improving since launch, not regressing.
Yesterday, while using ChatGPT-4, it gave me a very long answer almost instantly. It felt like I was using ChatGPT-3.5, including the poor quality of the answer. In the following prompts, it became slow again, as GPT-4 is supposed to be. The quality improved as well. I think they are trying some aggressive customization on their infra to try to make it economically viable, but it's just speculation at this point.
I heard GPT-4 described as "eight GPT-3's in a trenchcoat" but I'm not sure how accurate that is.
My favorites misstep from GPT-4 was when my friend asked it about the difference between vet bulb temperatures and dry bulb. You see that typo correctly (he was dictating): > The main difference is in what they're measuring. Temperature measurement at a vet is usually taken to determine an animal's body temperature, often done rectally or via the ear. It is direct and generally provides an absolute temperature value.…
>Having the behavior of an LLM change over time is not acceptable. By now this is actually funny to read. Never rely on another companies product to make your own product, without accepting things can change overnight and shut you down As Llama2 is self hosted, you can choose which iteration to host. Much better developer experience Edit: to be clear OpenAI is unprofitable, so is Reddit, so was Stadia. Building on to…
That's a pretty defeatist take. Surely if you pay for a service you should expect the provider to be making good faith efforts to provide the same quality of service over time? Natural degradation would be fine, but purposefully sandbagging the service so it gets worse because cheaper is unacceptable. That we have become numb to the point that we collectively accept such poor behavior on the part of vendors in concre…
I've been paying for GPT-4 since 3 hours after its release. The decrease in quality was noticeable just one week later (on top of the cap changes from 50 messages every 4 hours to 25 messages every 3 hours) I originally assumed that this was due to the increase in demand. It never went back to being as sharp as it was during those first hours of usage
Are you sure it wasn't just that the novelty wore off after a few hours of usage? I never really got into LLMs, but I must say at first it seemed like pretty cool stuff. OpenAI have repeatedly stated the model hasn't changed so how could this happen otherwise?
My favorites misstep from GPT-4 was when my friend asked it about the difference between vet bulb temperatures and dry bulb. You see that typo correctly (he was dictating): > The main difference is in what they're measuring. Temperature measurement at a vet is usually taken to determine an animal's body temperature, often done rectally or via the ear. It is direct and generally provides an absolute temperature value.…
> It seems like there's a bit of a typo in your question. I think you may be referring to the difference between "wet bulb" temperatures and "dry bulb" temperatures. [...]