A ChatGPT mistake cost us $10k

#1

Post by asim-shrestha » Sun, Jun 09, 2024, 8:56 PM UTC

A ChatGPT mistake cost us $10k
asim.bearblog.dev

Re: A ChatGPT mistake cost us $10k

#2

Post by chrisjj » Sun, Jun 09, 2024, 9:26 PM UTC

Thanks for telling.

Bookmarked for the next time we're told ChatGPT's code error rate is acceptable because we review its code just like an intern's.

Re: A ChatGPT mistake cost us $10k

#3

Post by nikonyrh » Sun, Jun 09, 2024, 9:33 PM UTC

I'm not familiar with this library, how does `text("(now())")` evaluate and why there are extra parentheses? And should that be a lambda expression as well, so that `create_date` isn't just the timestamp when the python process was started?

Re: A ChatGPT mistake cost us $10k

#4

Post by internetter » Sun, Jun 09, 2024, 9:35 PM UTC

"Note: I want to preface this by saying yes the practices here are bad and could have been avoided. This was from a different time under large time constraints. Please read with that in mind"

These "constraints" are why I'm terrified of subscribing to software

Re: A ChatGPT mistake cost us $10k

#5

Post by Aurornis » Sun, Jun 09, 2024, 9:39 PM UTC

On one hand, thanks for being honest about a story of how this bug came to be.

On the other hand, I don’t think advertising the fact that the company introduced a major bug from copy and pasting ChatGPT code around and that they spent a week being unable to even debug why it was failing.

I don’t know much about this startup, but this blog post had the opposite effect of all of the other high quality post-mortem posts I’ve read lately. Normally the goal of these posts is to demonstrate your team’s rigor and engineering skills, not reveal that ChatGPT is writing your code and your engineers can’t/won’t debug it for a very long time despite knowing that it’s costing them signups.

Re: A ChatGPT mistake cost us $10k

#6

Post by ergonaught » Sun, Jun 09, 2024, 9:39 PM UTC

ChatGPT didn’t make the mistake that cost you.

Re: A ChatGPT mistake cost us $10k

#7

Post by muzani » Sun, Jun 09, 2024, 9:40 PM UTC

internetter wrote: post #4

"Note: I want to preface this by saying yes the practices here are bad and could have been avoided. This was from a different time under large time constraints. Please read with that in mind" These "constraints" are why I'm terrified of subscribing to software

The alternative is writing it yourself, which adds constraints to everything else :)

Re: A ChatGPT mistake cost us $10k

#8

Post by morgante » Sun, Jun 09, 2024, 9:40 PM UTC

chrisjj wrote: post #2

Thanks for telling. Bookmarked for the next time we're told ChatGPT's code error rate is acceptable because we review its code just like an intern's.

Honestly I'm not sure why ChatGPT has anything to do with this problem.

I remember making the exact same mistake (accidentally using a single function call in a schema) back in 2010. No LLMs required.

The bigger culprit is probably a lack of testing / debugging. This error would immediately get caught if you simply registered twice on a test instance.

Re: A ChatGPT mistake cost us $10k

#9

Post by DataDive » Sun, Jun 09, 2024, 9:40 PM UTC

proper title: database modeling bug costs company 10K

but then perhaps coding the entire thing with ChatGPT saved the company more than 10K, so they came out well ahead

or maybe tons of other bugs lurk that will cost the company well over 10K over the long run

Re: A ChatGPT mistake cost us $10k

#10

Post by oxidant » Sun, Jun 09, 2024, 9:41 PM UTC

nikonyrh wrote: post #3

I'm not familiar with this library, how does `text("(now())")` evaluate and why there are extra parentheses? And should that be a lambda expression as well, so that `create_date` isn't just the timestamp when the python process was started?

My guess is that "now()" is the DB function that returns the current timestamp.