Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A ChatGPT mistake cost us $10k

asim.bearblog.dev

Post reply on HN
Print view View original

51–60 of 526 posts

Re: A ChatGPT mistake cost us $10k

#51

Post by viraptor »

hipadev23 wrote: post #29

This is a great example of why SQLAlchemy is a terrible ORM, and chatGPT is not alone in making the same mistake as millions of engineers, in fact likely where it learned the mistake. default = python code evaluates the default value as necessary for each new record server_default = the initial CREATE TABLE uses this computed (from python) value, thus the hardcoded UUID. They also could have done server_default=text(…

I'm not sure what you think SQLalchemy can fix here? There has to be an option to pass a static default value and the library does not have a visibility into the parse tree. What's the proposed solution?

Re: A ChatGPT mistake cost us $10k

#53

Post by 7thpower »

I have a lot of employees ask about how they will create value in light of AI that can do more and more of the things that have been central to their careers, and the answer is usually that they will do different things than before, and perhaps more of them, by leveraging these tools but also that they are responsible for the quality of work product, the tool is not.

That’s always been the case, but there is so much more surface area for human and tool interactions now that we have tools that are so generalized.

Good for them for sharing the story, countless others have them but not sharing them.

Re: A ChatGPT mistake cost us $10k

#54

Post by viraptor »

tazjin wrote: post #46

A good chance to learn why people who write reliable software almost universally like static type systems.

This code would pass static type validation, there's nothing wrong with it at that level. It gets a default value to use and does exactly that.

Re: A ChatGPT mistake cost us $10k

#55

Post by nicce »

koito17 wrote: post #40
Aurornis wrote: post #5

On one hand, thanks for being honest about a story of how this bug came to be. On the other hand, I don’t think advertising the fact that the company introduced a major bug from copy and pasting ChatGPT code around and that they spent a week being unable to even debug why it was failing. I don’t know much about this startup, but this blog post had the opposite effect of all of the other high quality post-mortem posts…

More importantly, what was the motivation behind a rewrite from TypeScript to Python? From the article Our project was originally full stack NextJS but we wanted to first migrate everything to Python/FastAPI. Seems like this entire mess could've been avoided if they had stuck with their existing codebase, which seemed to have been satisfying their business requirements.

> NextJS

There is well-hidden vendor-lock when using NextJS, at least.

Re: A ChatGPT mistake cost us $10k

#56

Post by jackspawn »

asddubs wrote: post #31
Closi wrote: post #20

Earlier quoted context omitted.

Your take-away is that this is ChatGPT's fault rather than a failure of testing?

i can see missing this during testing since it relies on doing the flow twice. though i have a hard time imagining a team of people not figuring this out in less than 5 days

to be fair, they didnt think it was a problem at the beginning, which can happen... if you made a mistake with your logging/monitoring

Re: A ChatGPT mistake cost us $10k

#57

Post by hn_throwaway_99 »

viraptor wrote: post #28
Aurornis wrote: post #5

On one hand, thanks for being honest about a story of how this bug came to be. On the other hand, I don’t think advertising the fact that the company introduced a major bug from copy and pasting ChatGPT code around and that they spent a week being unable to even debug why it was failing. I don’t know much about this startup, but this blog post had the opposite effect of all of the other high quality post-mortem posts…

They spent 5 days. The bug type is pretty common and could easily be done by a developer. (It's a similar class to the singleton default argument issue that many people complain about) Meh, I don't mind the cautionary tale and don't think chatgpt was even relevant. It's actually a tricky bug, because usual tests wouldn't catch it (db wiped for good isolation) and many ways of manual testing would restart the service…

TBH, while I definitely could see this being an easy bug to write, something is definitely wrong if it took 5 days to identify the root cause of this bug.

That is, I'm struggling to understand how a dive into the logs wouldn't show that all of these inserts were failing with duplicate key constraint violations. At that point at least I'd think you'd be able to narrow down the bug to a problem with key generation, at which point you're 90% of the way there.

I also don't agree that "usual tests wouldn't catch it (db wiped for good isolation)". I'd think that you'd have at least one test case that inserted multiple users within that single test.

Re: A ChatGPT mistake cost us $10k

#58

Post by fzeroracer »

morgante wrote: post #8
chrisjj wrote: post #2

Thanks for telling. Bookmarked for the next time we're told ChatGPT's code error rate is acceptable because we review its code just like an intern's.

Honestly I'm not sure why ChatGPT has anything to do with this problem. I remember making the exact same mistake (accidentally using a single function call in a schema) back in 2010. No LLMs required. The bigger culprit is probably a lack of testing / debugging. This error would immediately get caught if you simply registered twice on a test instance.

In a world where this entire codebase wasn't generated by ChatGPT, you'd have engineers familiar with the various parts of the system to quickly identify and fix the problem.

Testing and debugging isn't just a matter of stepping through code, it's an exercise of seeing where your mental model of the codebase is faulty versus the current reality of it.

I've encountered similar problems and they'd be fixed in a matter of hours, not days.

Re: A ChatGPT mistake cost us $10k

#59

Post by Lunrtick »

hipadev23 wrote: post #29

This is a great example of why SQLAlchemy is a terrible ORM, and chatGPT is not alone in making the same mistake as millions of engineers, in fact likely where it learned the mistake. default = python code evaluates the default value as necessary for each new record server_default = the initial CREATE TABLE uses this computed (from python) value, thus the hardcoded UUID. They also could have done server_default=text(…

They could also have done default=uuid.uuid4 to have a new id each time, or default=lambda : str(uuid.uuid4()). It's not really related to whether or not it's a database default.

It seems quite unfair to place the blame on SQLAlchemy here, or even Python.

Even a statically typed language wouldn't prevent this kind of issue - the author of the code is the only person who can decide when they mean "use this exact string each time" or "use this function to give me a new string each time".

I suppose a column description API could follow the dataclass style definition, with different argument names for default and default_func. That would (I think) prevent this from happening.

Re: A ChatGPT mistake cost us $10k

#60

Post by sjansen »

la64710 wrote: post #25

I dont get it , line 56 every time it was invoked would call the uuid4 function to generate a unique id isn’t it? Or was the issue due to the uuid4 function not getting invoked for any reason?

Line 56 is executed as the file is loaded. Simplified, the line is essentially:

  id = Column(default=str(uuid.uuid4()))
As written, a UUID is generated once and used to set a class-level attribute. Each Python process would generate a unique value, so it wouldn't be immediately obvious. Most of the time Python's ability to run code as a file is loaded is helpful, but this is one the well known gotchas.

Although I'm not a SQL Alchemy user, I assume the fix is essentially the same as it would be for Django. So the correct code would have been essentially:

  id = Column(default=uuid.uuid4)
Instead of executing `uuid4()` and caching a single UUID value, it would have executed `uuid4()` each time a new object was created.
Post reply on HN