Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A ChatGPT mistake cost us $10k

asim.bearblog.dev

Post reply on HN
Print view View original

141–150 of 526 posts

Re: A ChatGPT mistake cost us $10k

#141

Post by brown9-2 »

viraptor wrote: post #28
Aurornis wrote: post #5

On one hand, thanks for being honest about a story of how this bug came to be. On the other hand, I don’t think advertising the fact that the company introduced a major bug from copy and pasting ChatGPT code around and that they spent a week being unable to even debug why it was failing. I don’t know much about this startup, but this blog post had the opposite effect of all of the other high quality post-mortem posts…

They spent 5 days. The bug type is pretty common and could easily be done by a developer. (It's a similar class to the singleton default argument issue that many people complain about) Meh, I don't mind the cautionary tale and don't think chatgpt was even relevant. It's actually a tricky bug, because usual tests wouldn't catch it (db wiped for good isolation) and many ways of manual testing would restart the service…

looking at the query logs for the nighttime period should have made the bug fairly obvious

Re: A ChatGPT mistake cost us $10k

#142

Post by throwaway346434 »

dalemhurley wrote: post #129
AdamJacobMuller wrote: post #63

Earlier quoted context omitted.

The fact that they couldn't find it by looking at error logs is weird to me. This is an entirely forgivable error but should have been found the first time they got an email about it: "Oh, look, the error logs have a duplicate key exception for the primary key, how do we generate primary keys.... (facepalm)" Funnily enough, I saw the error in their snippet as soon as I read it but dismissed it thinking there was some…

I was wondering that too. Why wouldn’t the error be in the logs?

Guess: the logs were on an ec2 instance that was thrown away regularly, and the overnight reports didn't give reproduce steps or timestamps; so when they checked it "works fine".

There's value in having your backtrace surfaced to end users rather than swallowing an exception and displaying "didn't work".

Re: A ChatGPT mistake cost us $10k

#143

Post by audiodude »

fzeroracer wrote: post #58
morgante wrote: post #8

Earlier quoted context omitted.

Honestly I'm not sure why ChatGPT has anything to do with this problem. I remember making the exact same mistake (accidentally using a single function call in a schema) back in 2010. No LLMs required. The bigger culprit is probably a lack of testing / debugging. This error would immediately get caught if you simply registered twice on a test instance.

In a world where this entire codebase wasn't generated by ChatGPT, you'd have engineers familiar with the various parts of the system to quickly identify and fix the problem. Testing and debugging isn't just a matter of stepping through code, it's an exercise of seeing where your mental model of the codebase is faulty versus the current reality of it. I've encountered similar problems and they'd be fixed in a matter…

Yes I don't understand the perceived benefits of re-writing the code in Python/FastAPI if none of them know Python/FastAPI.

Re: A ChatGPT mistake cost us $10k

#145

Post by m3kw9 »

Some are saying they used ChatGPT to write code, but these are going to be normal going forward as models get better, I mean who doing web work isn’t using it to code these days? You just need better testing before pushing it

Re: A ChatGPT mistake cost us $10k

#146

Post by hehdhdjehehegwv »

japhyr wrote: post #126

This kind of thing must be happening much more often than we're hearing about it, right? I'd love to start a site that collects AI "horror stories", where trusting an AI's output led to significant consequences. I have no idea how to validate people's anecdotes, though. (To be clear I don't doubt this story at all. But if I set up a site where people could submit stories I wouldn't trust any submissions at face value…

you are a software developer who used LLM generated code in a production database. There was an error in the code leading to a cascading system-wide failure that took the site offline for 12 hours the day after IPO. this caused a company’s stock drop by 35% on the second day of trading. Write an anonymous form post detailing your mistake and warning others against using LLM code

I'm writing to share a painful lesson learned firsthand about the risks of integrating LLM (Large Language Model) generated code into production systems. Recently, my team and I experienced a catastrophic failure due to an error in code generated by an LLM, which resulted in our site being offline for a staggering 12 hours.

The fallout from this incident was devastating. Not only did we lose valuable revenue and user trust, but the company's stock plummeted by 35% on the second day of trading following our IPO. It's a nightmare scenario no developer ever wants to face.

Here's what happened: in our rush to meet deadlines and optimize processes, we turned to LLM-generated code to expedite development. While it seemed like a shortcut at the time, we failed to thoroughly vet the code for potential flaws and dependencies. Consequently, when an overlooked error surfaced, it triggered a cascading failure that crippled our entire system.

The repercussions of this oversight extend far beyond our organization. It serves as a stark reminder to the entire development community about the inherent risks of relying on AI-generated code in critical production environments. While LLMs are undoubtedly powerful tools, they're not foolproof, and blindly trusting their output can have dire consequences.

In hindsight, I deeply regret the decision to incorporate LLM-generated code without adequate scrutiny. I hope by sharing our experience, others can learn from our mistake and approach the use of AI-generated code with caution.

Let this be a warning to all: while LLMs can be valuable assets in certain contexts, proceed with caution when considering their implementation in production systems. The allure of efficiency must never compromise the integrity and reliability of our codebase.

Re: A ChatGPT mistake cost us $10k

#147

Post by sitkack »

wmedrano wrote: post #133
bluelightning2k wrote: post #93

It is strange that this took 5 days to find. Simply because of logs. Go to logs. Filter by errors. Oh, errors in insert subscription. Seems relevant. I could understand if the errors were somewhere else. Even if logs didn't exist. Problematic endpoint generating 50 emails per day? I would have immediately thrown a try catch and rendered the error to the user if logging was impossible. Then your very next bug report s…

ChatGPT gives the following: The code snippet has a subtle but significant issue in the default value of the id column. Here is the problematic part: ... In this line, default=str(uuid.uuid4()) is evaluated only once at the time of the class definition, not each time a new StripeCustomer instance is created....

They should have also asked the LLM for some integration tests.

Ensure that the revenue generating codepaths have proper logging.

This failure had very little to do with having an LLM write it.

Re: A ChatGPT mistake cost us $10k

#148

Post by Ozzie_osman »

jbattle wrote: post #122
viraptor wrote: post #67

Earlier quoted context omitted.

The bug was in multiple subscriptions not just users. And I can't think of one non-contrived reason to do it. Even when testing the visibility/access of subscriptions between users you need 2 users, but only one subscription.

create a subscription for a test user. delete it. Make sure you can create another subscription for the same user. create subscriptions with and without overlapping effective windows Those seem like very basic tests that would have highlighted the underlying issue

Or add some debug logging? 5 days into a revenue-block bug, if I can't repro manually or via tests, I would have logged the hell out of this code. No code path or metric would be spared.

Re: A ChatGPT mistake cost us $10k

#149

Post by tmpz22 »

SilasX wrote: post #111
Terretta wrote: post #99

Earlier quoted context omitted.

> It read like no one really knew what they were doing. "We just let [devs] generate the code and everything seemed to work" is certainly not a good way to [whatever]. Except, have you met startup devs? This is by and large the "move fast then unbreak things" approach.

For that matter, has OP even met the HN accepted wisdom? "No one knows what they're doing, everyone's faking it, it's fine if you are too" -- so don't take it as a red flag when your fumbling around keeps blowing up, because it surely must work that way everywhere else. My early rant against this mentality: https://news.ycombinator.com/item?id=19214749

Its very humbling coming out of startup-land and working with big tech engineers and realizing their tooling runs circles around everybody else and enables them to be much more precise with their work and scale, though it isn't without trade-offs.

Re: A ChatGPT mistake cost us $10k

#150

Post by throw46365 »

beachy wrote: post #137
throw46365 wrote: post #125

Earlier quoted context omitted.

> I don't think this is indicative of people who don't know what they're doing. I think this is indicative of people using "AI" tools to help with programming at all. I think using AI tools to write production code is probably indicative of people who don't really know what they are doing. The best way not to have subtle bugs is to think deeply about your code, not subcontract it out -- whether that is to people far…

> And where is the benefit in doing this repeatedly? You're just going to end up with a codebase nobody really owns on a cognitive level. You could say the same about the output of a compiler. No one owns that at a cognitive level. They own it at a higher level - the source code. Same thing here. You own the output of the AI at a cognitive level, because you own the prompts that created it.

> You could say the same about the output of a compiler.

Except, for starters, that you're not using the LLM to replace a compiler.

You're using it to replace a teammate.

Post reply on HN