Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI: Increased errors across API and ChatGPT

status.openai.com

Post reply on HN
Print view View original

51–60 of 67 posts

Re: OpenAI: Increased errors across API and ChatGPT

#51

Post by lazysheepherd »

willsmith72 wrote: post #43
lazysheepherd wrote: post #36

Earlier quoted context omitted.

Or maybe they are just AB testing and aggressively optimizing the response generation? LLMs are known to be compute/energy hungry to execute. It is a developing technology, if not downright experimental. Therefore, this explanation is very likely. I cannot see the reason to call this a conspiracy.

AB testing on what? AB tests need to produce some results which are then compared. How would releasing different versions in production help with that? It would make more sense if that was internal and the responses were then graded. A failed canary release would be more likely, where they released this version to a small amount of people not realising it was bad

On top of my mind: responses have feedback buttons below them.

You can simply deploy different versions and compare the neutral + positive / negative feeback ratio.

It would be sinful if they did not add other metrics like how many times the user had to correct and update their prompt before ending the chat, etc.

Data, data, data...

Re: OpenAI: Increased errors across API and ChatGPT

#52

Post by JoshuaDavid »

rurp wrote: post #33
speedgoose wrote: post #20

Earlier quoted context omitted.

Why do you use an internet search engine when you can walk to the library?

The question at hand is pretty easy to test manually and the information you get is much more useful. You will get to see the exact behavior for yourself, can easily build on the test case as related questions come up, and you know the information you are getting is correct rather than a hallucination. Copying information from ChatGPT is the newer version of blindly copying answers from StackOverflow. It often works…

> The question at hand is pretty easy to test manually and the information you get is much more useful.

This approach can be hazardous to the health of the product you're building. For example, if you take this approach to answer the question of "what happens if I have two connections to a MySQL database, start a transaction in one of them and insert a row (but don't commit) and then issue a SELECT which would show the inserted row", then you will see consistent results across all of the experiments you run with that particular database, but you could easily end up with bugs that only show up when the transaction isolation level changes from how you tested it.

Whereas if you search for or ask that question, the answers you get will likely mention that transaction isolation levels are a thing.

You might also be able to get this level of knowledge by reading the manual, though there will still be things that are not included in the manual but do come up regularly in discussions on the wider internet.

Re: OpenAI: Increased errors across API and ChatGPT

#53

Post by bun_at_work »

timeon wrote: post #44
sidibe wrote: post #3

Not working for me. Am I going to have to read docs or search Google like some boomer?

With Google'n'co you at least know when search is wrong.

Don't people often fall into the "vaccines cause autism" trap from Google?

Re: OpenAI: Increased errors across API and ChatGPT

#54

Post by nuz »

Filligree wrote: post #23
BrunoJo wrote: post #16

You may want to try https://lemonfox.ai/ as a OpenAI API alternative. I think relying on open-source models is a great alternative.

The solution to 'GPT-4 sometimes breaks' isn't to use something that never works...

Finetuned local models can work just as well or better than gpt-4 in many use cases

Re: OpenAI: Increased errors across API and ChatGPT

#55

Post by [deleted] »

dvfjsdhgfv wrote: post #35
swatcoder wrote: post #18

Earlier quoted context omitted.

Well, were it possible, I'd say go back in time and study your tools so that you're not spending the journeyman period of your career ricocheting between tutorials and faqs. Failing that, read the documentation. Failing that, stand up a quick experiment. Somehow, we survived before ChatGPT and even before saturated question boards. Those strategies are still available to you and well worth learning

I see your point but the world changes so fast. Back in my day you just needed to learn C, understand algorithms and so on and then you could get deeper in an area or two. Today, you need to understand and be able to proficiently use so many technologies that you can feel lost. And this is what happens when, say, you loose a job you've been doing for 10-15 years. You need to re-learn the world. And a lifetime is not…

[deleted]

Re: OpenAI: Increased errors across API and ChatGPT

#56

Post by __loam »

captainkrtek wrote: post #24
timthelion wrote: post #17

I think we need a new type of status page or at least a public version number on llms, yesterday for me GPT4 started giving nonsense super generic answers, like it was hardly reading what I wrote, and today it is back to top notch performance. I think they were trying to make the model more efficient or something but I just saw a massive decrease in the quality of output. From my side though, there is no version numb…

On a recent interview of Sam Altman (Hard Fork podcast) he mentioned that due to the load they have been trying to make optimizations, disable certain features, etc. so it’s not outside the realm of possibility that some tweak caused this.

I think one of the harder things about developing these models is that regressions are hard to figure out or even detect.

Re: OpenAI: Increased errors across API and ChatGPT

#57

Post by esotericimpl »

__loam wrote: post #56
captainkrtek wrote: post #24

Earlier quoted context omitted.

On a recent interview of Sam Altman (Hard Fork podcast) he mentioned that due to the load they have been trying to make optimizations, disable certain features, etc. so it’s not outside the realm of possibility that some tweak caused this.

I think one of the harder things about developing these models is that regressions are hard to figure out or even detect.

[dead]

Re: OpenAI: Increased errors across API and ChatGPT

#58

Post by jjcon »

Filligree wrote: post #23
BrunoJo wrote: post #16

You may want to try https://lemonfox.ai/ as a OpenAI API alternative. I think relying on open-source models is a great alternative.

The solution to 'GPT-4 sometimes breaks' isn't to use something that never works...

Have you never used the open source models? They are getting really good - better than 3.5 for sure not as good as 4 except when domain trained in my opinion

Re: OpenAI: Increased errors across API and ChatGPT

#59

Post by toomuchtodo »

bigfudge wrote: post #48
linsomniac wrote: post #40

Earlier quoted context omitted.

The "good old days weren't always good". I'm tired of either limiting myself to the information I have on the top of my head, the LLMs are really helping allow me to be creative and stretch out to do things that are just beyond my bread and butter, or things that I do infrequently.

Exactly this. I -could- become an expert in the intricacies of every tool I touch, or I could use chat gpt and move on to solving the next problem.

LLMs are the great equalizer of our time.

Re: OpenAI: Increased errors across API and ChatGPT

#60

Post by morsch »

bun_at_work wrote: post #2

> OK I have a table in postgresql and I am adding a trigger such that when an insert happens on that table, an insert happens on another table. The second table has a constraint. What happens to the first insert if the second insert violates the constraint? How can I get help with this now? Google result 1: https://stackoverflow.com/questions/77148711/create-a-trigge... Google result 2: https://dba.stackexchange.com/…

So ChatGPT says -- to me, a minute ago, ymmv -- it will rollback the first insert. Now what? Do you believe it? Cool. I wouldn't. I would confirm its claim, either by Googling or by trying it myself.

Also, when I asked it "what if I use PostgreSQL's non-transactional triggers", which I thought I just made up, it told me it wouldn't roll back the first insert: Non-transactional triggers are executed as part of the statement that triggered them, but they don't participate in the transaction control. So now I don't know what to think.

Post reply on HN