Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A ChatGPT mistake cost us $10k

asim.bearblog.dev

Post reply on HN
Print view View original

41–50 of 526 posts

Re: A ChatGPT mistake cost us $10k

#41

Post by jorams »

Aurornis wrote: post #5

On one hand, thanks for being honest about a story of how this bug came to be. On the other hand, I don’t think advertising the fact that the company introduced a major bug from copy and pasting ChatGPT code around and that they spent a week being unable to even debug why it was failing. I don’t know much about this startup, but this blog post had the opposite effect of all of the other high quality post-mortem posts…

Yeah this is not a good thing to advertise.

- They were under large time constraints, but decided a full rewrite to a completely different stack was a good idea.

- They copy-pasted a whole bunch of code, tested it manually once locally, once in production, and called it a day.

- The debugging procedure for this issue so significant it made them dread waking up involved... testing it once and moving on. Every day.

The bug is pretty easy to miss, but should also be trivial to diagnose the moment you look at the error message, and trivial to reproduce if you just try more than once.

Re: A ChatGPT mistake cost us $10k

#42

Post by crooked-v »

viraptor wrote: post #28
Aurornis wrote: post #5

On one hand, thanks for being honest about a story of how this bug came to be. On the other hand, I don’t think advertising the fact that the company introduced a major bug from copy and pasting ChatGPT code around and that they spent a week being unable to even debug why it was failing. I don’t know much about this startup, but this blog post had the opposite effect of all of the other high quality post-mortem posts…

They spent 5 days. The bug type is pretty common and could easily be done by a developer. (It's a similar class to the singleton default argument issue that many people complain about) Meh, I don't mind the cautionary tale and don't think chatgpt was even relevant. It's actually a tricky bug, because usual tests wouldn't catch it (db wiped for good isolation) and many ways of manual testing would restart the service…

> and could easily be done by a developer

...who didn't know how the ORM they were using worked. That's what makes them look so bad here: nobody knew how it worked, not even at the surface level of knowing what the SQL actually generated by the tool looks like.

Re: A ChatGPT mistake cost us $10k

#44

Post by jackspawn »

viraptor wrote: post #28
Aurornis wrote: post #5

On one hand, thanks for being honest about a story of how this bug came to be. On the other hand, I don’t think advertising the fact that the company introduced a major bug from copy and pasting ChatGPT code around and that they spent a week being unable to even debug why it was failing. I don’t know much about this startup, but this blog post had the opposite effect of all of the other high quality post-mortem posts…

They spent 5 days. The bug type is pretty common and could easily be done by a developer. (It's a similar class to the singleton default argument issue that many people complain about) Meh, I don't mind the cautionary tale and don't think chatgpt was even relevant. It's actually a tricky bug, because usual tests wouldn't catch it (db wiped for good isolation) and many ways of manual testing would restart the service…

agree, but this sounds like it would produce logs/error messages which could then lead to a solution, quicker... if the logs were captured and propagated sufficiently

Re: A ChatGPT mistake cost us $10k

#45

Post by la64710 »

donatj wrote: post #34
la64710 wrote: post #25

I dont get it , line 56 every time it was invoked would call the uuid4 function to generate a unique id isn’t it? Or was the issue due to the uuid4 function not getting invoked for any reason?

I'm not a Pythonista, but I think the deal is that line 56 was only executed once, at class definition, so every time the server spun up, you got a new uuid that could only use once

Ah so it would get invoked one time for the first user that resulted in a task (container ) initialization on the backed ECS cluster and once all 40 of the tasks were running new users will just reuse one of the running containers and already instantiated object in memory.

Re: A ChatGPT mistake cost us $10k

#47

Post by TheCleric »

la64710 wrote: post #25

I dont get it , line 56 every time it was invoked would call the uuid4 function to generate a unique id isn’t it? Or was the issue due to the uuid4 function not getting invoked for any reason?

Since this looks like table definition code it likely would run once per db migration. So in all likelihood it ran once when they created the database tables which itself essentially ended up with a definition like “default=89abcdef-0000-0000-0000-fedcba987654”

Re: A ChatGPT mistake cost us $10k

#48

Post by YaBa »

Imagine trusting ChatGPT for something important. OMG. My apologies, But I cannot feel any kind of sorry. Even for my private projects, I hardly trust on it since it allucinates a lot.

Re: A ChatGPT mistake cost us $10k

#50

Post by dgfitz »

koito17 wrote: post #40
Aurornis wrote: post #5

On one hand, thanks for being honest about a story of how this bug came to be. On the other hand, I don’t think advertising the fact that the company introduced a major bug from copy and pasting ChatGPT code around and that they spent a week being unable to even debug why it was failing. I don’t know much about this startup, but this blog post had the opposite effect of all of the other high quality post-mortem posts…

More importantly, what was the motivation behind a rewrite from TypeScript to Python? From the article Our project was originally full stack NextJS but we wanted to first migrate everything to Python/FastAPI. Seems like this entire mess could've been avoided if they had stuck with their existing codebase, which seemed to have been satisfying their business requirements.

I’m sure they are influenced by the likes of Reddit and Twitter rewriting their stack. I mean, that’s what has to be done, right? /s
Post reply on HN