Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A ChatGPT mistake cost us $10k

asim.bearblog.dev

Post reply on HN
Print view View original

161–170 of 526 posts

Re: A ChatGPT mistake cost us $10k

#161

Post by yareal »

This post mortem is sort of classically underdone. It describes a step that was taken that was in the path of the error, but is not the root cause. The root cause here is not "we copy pasted from chat gpt and it hallucinates", but rather a "our systems allowed this failure to get to production". Which in turn should be met with why? Because we didn't have tests or qa that covers this path. Why?

Keep. Asking. Why.

ChatGPT didn't fail, your system allowed ChatGPT to fail. Answering why is the interesting thing to discuss and blog about.

Re: A ChatGPT mistake cost us $10k

#162

Post by zeroonetwothree »

brigadier132 wrote: post #81
viraptor wrote: post #28

Earlier quoted context omitted.

They spent 5 days. The bug type is pretty common and could easily be done by a developer. (It's a similar class to the singleton default argument issue that many people complain about) Meh, I don't mind the cautionary tale and don't think chatgpt was even relevant. It's actually a tricky bug, because usual tests wouldn't catch it (db wiped for good isolation) and many ways of manual testing would restart the service…

> db wiped for good isolation Why? In fact, not having good isolation would have caught this bug. Generate random emails for each test. Why would you test on a completely new db as if that is what will happen in the real world?

It makes your tests more robust. Generally you don’t want tests that are too sensitive to external state since they will fail spuriously and become useless.

Re: A ChatGPT mistake cost us $10k

#163

Post by ggorlen »

hfusdvdkchfbfb wrote: post #144

[flagged]

There wouldn't be much industry if everyone who trusted ChatGPT and other ways of quickly getting code up (copy-pasting Stack Overflow, "try random stuff until it works" debugging, hopping on calls with random freelancers, etc) followed your advice.

Many programmers I've encountered in early stage tech startups (and in general) are not craftspeople--they're scrambling to get a product to market as quickly as possible and quality and process are very much secondary. Many are working in unfamiliar languages by necessity, or are relatively new or even untrained as professional programmers. I mentor such folk regularly. (Actually, these untrained hackers are often "better" at programming in many respects than senior engineers with 10 years of experience, but that's another story).

If the company survives long enough, they might pay off the tech debt later. OP's team just got unlucky doing the same strategy many other startups are doing nowadays and are willing to admit it.

To be clear, I'm not excusing the mistake or endorsing the process they followed, only noting that their actions aren't out of the ordinary (other than admitting to the mistake) and empathy is due.

Re: A ChatGPT mistake cost us $10k

#164

Post by PhasmaFelis »

zeroonetwothree wrote: post #156
PhasmaFelis wrote: post #109

Earlier quoted context omitted.

> It read like no one really knew what they were doing. In my experience, hardly anyone in software does know what they're doing, for sufficiently rigorous values of "know what you're doing." We all read about other people's stupid mistakes, and think "haha, I would never have done that, because I know about XYZ!" And then we go off and happily make some equally stupid mistake because we don't know about ABC.

There’s a difference between not knowing what you are doing and making a mistake.

An awful lot of mistakes are made because one didn't know something that would have enabled one to avoid it. Not knowing what you don't know is difficult to work around.

Re: A ChatGPT mistake cost us $10k

#165

Post by throw46365 »

PhasmaFelis wrote: post #109
userbinator wrote: post #21

Earlier quoted context omitted.

It read like no one really knew what they were doing. "We just let it generate the code and everything seemed to work" is certainly not a good way to market your company.

> It read like no one really knew what they were doing. In my experience, hardly anyone in software does know what they're doing, for sufficiently rigorous values of "know what you're doing." We all read about other people's stupid mistakes, and think "haha, I would never have done that, because I know about XYZ!" And then we go off and happily make some equally stupid mistake because we don't know about ABC.

I dunno. I tend to annoy people when taking on jobs by telling people what I am concerned about and do not understand, and then sharing with them the extent to which I have managed to allay my own concerns through research.

I turn down a lot of jobs I don't feel confident with; maybe more than I should.

An LLM never will.

Re: A ChatGPT mistake cost us $10k

#167

Post by bogota »

userbinator wrote: post #21
Aurornis wrote: post #5

On one hand, thanks for being honest about a story of how this bug came to be. On the other hand, I don’t think advertising the fact that the company introduced a major bug from copy and pasting ChatGPT code around and that they spent a week being unable to even debug why it was failing. I don’t know much about this startup, but this blog post had the opposite effect of all of the other high quality post-mortem posts…

It read like no one really knew what they were doing. "We just let it generate the code and everything seemed to work" is certainly not a good way to market your company.

This is getting more common. I have already had people try to tell me how something works from a chat gpt summary. This would have led to us taking a completely different direction… 5 minutes of reading the actual docs and I found out they were wrong.

Now at a new company i have caught several people copy pasting gpt code that is just horrendous.

It seems like this is where the industry is headed. The only thing i have found gpt to be good at is solving interview questions although it still uses phantom functions about 50% of the time. The future is bumming me out.

Re: A ChatGPT mistake cost us $10k

#168

Post by whakim »

cmeacham98 wrote: post #15

I understand how the mistake was made, it seems relatively easy to slip by even when writing code without ChatGPT. But what I don't understand is how this wasn't caught after the first failure? Does this company not have any logging? Shouldn't the fact the backend is attempting to reuse UUIDs be immediately obvious from observing the error?

Yeah, I agree with this here. I think it's totally reasonable that something as specific as multiple Stripe subscriptions wouldn't be exercised by normal unit testing; as mentioned in the post, this wouldn't have been an easy error to reproduce via an acceptance test; and I think the focus on ChatGPT is overblown (by both the OP and everyone else) and mistakenly passing a String instead of a Callable to a function that accepts either happens all the time. My gut instinct is that not using an ORM would have prevented this particular issue, but that may just be my bias against ORMs speaking; one could easily imagine a similar bug occurring in a non-database context. My real conclusion is that all the folks crowing that they would have definitely caught this bug are either much better engineers than I am, or (more likely) are just a bit deluded about their own abilities.

I am also very confused about the apparent lack of logging or recourse to logging. It's been a while, but if I recall correctly ECS should automatically propagate the resulting Duplicate Key exceptions which were presumably occurring to CloudWatch without a bunch of additional configuration - was that not happening? If it was happening, did no one think to go check what types of Exceptions were happening overnight?

Re: A ChatGPT mistake cost us $10k

#169

Post by threecheese »

koito17 wrote: post #40
Aurornis wrote: post #5

On one hand, thanks for being honest about a story of how this bug came to be. On the other hand, I don’t think advertising the fact that the company introduced a major bug from copy and pasting ChatGPT code around and that they spent a week being unable to even debug why it was failing. I don’t know much about this startup, but this blog post had the opposite effect of all of the other high quality post-mortem posts…

More importantly, what was the motivation behind a rewrite from TypeScript to Python? From the article Our project was originally full stack NextJS but we wanted to first migrate everything to Python/FastAPI. Seems like this entire mess could've been avoided if they had stuck with their existing codebase, which seemed to have been satisfying their business requirements.

My guess, when I read it, was this would permit them to independently scale the backend on some commodity capacity provider, and then their Nextjs frontend becomes just another React app. OP didn’t mention what their product was, but if it’s AI-adjacent then a python backend doesn’t sound like a terrible idea.

Re: A ChatGPT mistake cost us $10k

#170

Post by throw46365 »

zeroonetwothree wrote: post #156
PhasmaFelis wrote: post #109

Earlier quoted context omitted.

> It read like no one really knew what they were doing. In my experience, hardly anyone in software does know what they're doing, for sufficiently rigorous values of "know what you're doing." We all read about other people's stupid mistakes, and think "haha, I would never have done that, because I know about XYZ!" And then we go off and happily make some equally stupid mistake because we don't know about ABC.

There’s a difference between not knowing what you are doing and making a mistake.

The "zen" of LLMs is that they do not see a real distinction between these two things, or either of these two things and success ;-)
Post reply on HN