Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A ChatGPT mistake cost us $10k

asim.bearblog.dev

Post reply on HN
Print view View original

381–390 of 526 posts

Re: A ChatGPT mistake cost us $10k

#381

Post by rvnx »

spencerchubb wrote: post #209
wavemode wrote: post #65

No, a lack of monitoring cost you $10K. Your app was throwing a database exception and nobody was alerted that this was not only happening, but happening continuously and in large volumes. Such an alert would have made this a 5-minute investigation rather than 5 days. If you haven't fixed that alerting deficiency, then you haven't really fixed anything.

I agree this is more of a monitoring mistake, and little to do with chatgpt

It could have happened with any programmer writing the code (ChatGPT or not)

Re: A ChatGPT mistake cost us $10k

#382

Post by rvnx »

animex wrote: post #306

No, ChatGPT made you the money that your app generated since you had no ability to implement it otherwise/without ChatGPT. Your inability to code, debug, log, monitor cost you the $10k. ChatGPT is net positive in this story.

Especially for a $20/month cost...

Re: A ChatGPT mistake cost us $10k

#384

Post by rnts08 »

wavemode wrote: post #65

No, a lack of monitoring cost you $10K. Your app was throwing a database exception and nobody was alerted that this was not only happening, but happening continuously and in large volumes. Such an alert would have made this a 5-minute investigation rather than 5 days. If you haven't fixed that alerting deficiency, then you haven't really fixed anything.

This gets more and more common, companies and founders does not think about the infrastructure since they believe that their cloud provider of choice is going to do it for them with it's magic.

As soon as you expect paying customers in your system you need to have someone with the knowledge and experience to deal with infrastructure. That means logging, monitoring, alerting, security etc.

DevOps.. amateurs.

Re: A ChatGPT mistake cost us $10k

#385

Post by cube00 »

klabb3 wrote: post #120
DataDive wrote: post #9

proper title: database modeling bug costs company 10K but then perhaps coding the entire thing with ChatGPT saved the company more than 10K, so they came out well ahead or maybe tons of other bugs lurk that will cost the company well over 10K over the long run

In a typical web based app, schema is the one and only thing that you should be paranoid about. Writing it by hand is important for the same reason typing your password to confirm a big transaction is important. The time and thought going into it is worth its weight in gold. I would rather write it with one finger twice over, than giving it to ChatGPT.

Nullable strings for all!

Re: A ChatGPT mistake cost us $10k

#386

Post by fragmede »

primitivesuave wrote: post #314

I spotted the error instantly. With all due respect to your team - this has nothing to do with ChatGPT and everything to do with using a programming model that your team does not have sufficient expertise in. Even if this error managed to slip by code review, it would have been caught with virtually any monitoring solution, many of which take less than 5 minutes to set up.

Interestingly, you know who else spotted the error? ChatGPT-4o. Annoyingly you can't share a chat with an image in it, but pasting in the image of the bad code, and prompting "whats wrong with the code" got ChatGPT to tell me that:

* UUID Generation in Primary Key: The default parameter should use the callable uuid.uuid4 directly instead of str(uuid.uuid4()). SQLAlchemy will call the function to generate the value.

* Date Default Value: server_default=text("(now())") might not work as expected. Use func.now() for server-side defaults in SQLAlchemy.

* Import Statements: Ensure uuid and text from sqlalchemy are imported.

* Column Definitions: Consider using DateTime(timezone=True) for datetime columns to handle time zones.

It then provided me with corrected code that does

    id = Column(String, primary_key=True, default=lambda: str(uuid.uuid4()), unique=True, nullable=False)
where the addition of lambda: fixes the problem.

Re: A ChatGPT mistake cost us $10k

#387

Post by TheDong »

randomdata wrote: post #358
TheDong wrote: post #303

Earlier quoted context omitted.

In python it's harder to ignore errors than in Go. In go, I've definitely seen: tx, err := db.Tx() defer tx.Commit() // silently ignores the error on committing, which is the important one That would have masked this error so it didn't get logged by the application. In python, if you ignore an exception entirely, like I did that error above, you instead get an exception logged by default. Python's exceptions also inc…

> I've definitely seen What did they say was the thinking behind it? defer tx.Rollback() would make sense, but defer tx.Commit() is nonsensical, regardless of whether or not the error is handled. It seems apparent that the problem there isn't forgetting to check an error, but that someone got their logic all mixed up, confusing rollback with commit.

"defer tx.Commit() is nonsensical"

It's not pure nonsense, it works in the happy path, and it matches the pattern of how people often handle file IO in go.

    f, err := os.OpenFile(...)
    defer f.Close()
... which is another place most gophers ignore errors incorrectly. Just like the "defer tx.Commit()" example, it's collocating the idea of setup and cleanup together.

Those two patterns are so similar, python handles them in the same way:

    with db.begin() as conn: # implicit transaction, gets automatically committed

    with open(...) as f: # implicit file open + close pair, automatically closed
You're of course right that go requires more boiler-plate to do the right thing, but the wrong code has no compiler errors and works in the happy path, and fits how people think about the problem in other languages with sane RAII constructs, so of course people will write it.

Re: A ChatGPT mistake cost us $10k

#388

Post by [deleted] »

primitivesuave wrote: post #314

I spotted the error instantly. With all due respect to your team - this has nothing to do with ChatGPT and everything to do with using a programming model that your team does not have sufficient expertise in. Even if this error managed to slip by code review, it would have been caught with virtually any monitoring solution, many of which take less than 5 minutes to set up.

[deleted]

Re: A ChatGPT mistake cost us $10k

#389

Post by lispm »

kazinator wrote: post #257

I've written less than 1000 lines of Python in total probably, but I correctly spotted the problem. Python has this misfeature whereby it didn't correctly crib Common Lisp's evaluation strategy for the expressions that give default values to optional function arguments. When you have an argument like foo=obj.whatever() the obj.whatever() is evaluated (would you believe it!) at the time the definition of the function…

Good explanation...

Re: A ChatGPT mistake cost us $10k

#390

Post by IshKebab »

asddubs wrote: post #26
IshKebab wrote: post #16

More like how switching to Python (and not meeting Python's high testing requirements) cost you $10k. This is a well known Python foot-gun and a developer could easily have made that mistake too.

agreed, though if a developer had manually made the mistake they might have realized the problem in less than 5 days. copy paste a bunch of ai generated code into your project and no one can try to deduce where the problem might lie once something goes wrong though a little logging would also have gone a long way. I don't really get how this could have taken 5 days to find, since they knew exactly where the problem w…

Agreed. Also Pylint has a lint for exactly this mistake so they didn't even set up the standard linting / static type checking tools which are absolutely a must with Python.
Post reply on HN