Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A ChatGPT mistake cost us $10k

asim.bearblog.dev

Post reply on HN
Print view View original

421–430 of 526 posts

Re: A ChatGPT mistake cost us $10k

#421

Post by Akronymus »

arjvik wrote: post #329
primitivesuave wrote: post #314

I spotted the error instantly. With all due respect to your team - this has nothing to do with ChatGPT and everything to do with using a programming model that your team does not have sufficient expertise in. Even if this error managed to slip by code review, it would have been caught with virtually any monitoring solution, many of which take less than 5 minutes to set up.

To be fair, if I wasn't looking for this bug I never would have spotted it. That being said, you're entirely right that any monitoring or even the most basic manual testing should have instantly caught this.

Those kinda issues are why I ALWAYS make an integration test with calling the same insert multiple times.

I did step into that particular trap more than once (passing the result, rather than the function)

Re: A ChatGPT mistake cost us $10k

#422

Post by Akronymus »

KennyBlanken wrote: post #352
primitivesuave wrote: post #314

I spotted the error instantly. With all due respect to your team - this has nothing to do with ChatGPT and everything to do with using a programming model that your team does not have sufficient expertise in. Even if this error managed to slip by code review, it would have been caught with virtually any monitoring solution, many of which take less than 5 minutes to set up.

It's not some innocent mistake. The title is purposefully clickbait / keyword-y, implying that it was chatgpt that made the 'mistake' for SEO and to generate panicked clicks. "We made a programming error in our use of an LLM, didn't do any QA, and it cost us $10k" doesn't generate the C-suite "oh shit what if ChatGPT fucks up, what's our exposure!?" reaction. There's a million middle and upper management posting this…

> It's like the Mr. Beast open-mouth-surprised expression thumbnail nonsense; you feel incredibly compelled to click it.

Sponsorblock is great for combating that. (Altough, I conciously avoid channels that mostly do clickbait anyways)

Re: A ChatGPT mistake cost us $10k

#423

Post by nicce »

throw156754228 wrote: post #417
nicce wrote: post #86

Earlier quoted context omitted.

Simply put - if you want get the best out of the framework, you need to host it in Vercel. Otherwise, there are better options for frameworks. No need to ”fight it”. You will find many issues from GitHub which are not considered because they would make the framework ”better” or easier to use on other clouds.

>there are better options Do you mind naming some NextJS alternatives without potential vendor lock in? Mulling a change in my fe.

For example, Remix has got a lot of traction recently. It is also backed by Shopify where the business model does not conflict.

https://remix.run/

Re: A ChatGPT mistake cost us $10k

#426

Post by shawabawa3 »

cuppypastexprt wrote: post #350
animex wrote: post #306

No, ChatGPT made you the money that your app generated since you had no ability to implement it otherwise/without ChatGPT. Your inability to code, debug, log, monitor cost you the $10k. ChatGPT is net positive in this story.

Looking at this team's project at github.com/reworkd, it clearly tells the maturity of the product as well as the team. Emoji driven development. Emoji's for all commit messages. Monkeys, bananas, rockets, fireworks, you name it, they have it in their commit message.

the emoji's seem fine

instead of `bug: fix blah`, it's `:bug:: fix blah`, which, honestly actually seems clearer and easier to parse at a glance

edit: hacker news doesn't support unicode emojis

Re: A ChatGPT mistake cost us $10k

#427

Post by _xnmw »

throw156754228 wrote: post #410
wavemode wrote: post #65

No, a lack of monitoring cost you $10K. Your app was throwing a database exception and nobody was alerted that this was not only happening, but happening continuously and in large volumes. Such an alert would have made this a 5-minute investigation rather than 5 days. If you haven't fixed that alerting deficiency, then you haven't really fixed anything.

Should've been picked up at unit, way before monitoring.

The error didn't show up until multiple entities were created. I can forgive someone for not having unit tests for scaling issues.

Re: A ChatGPT mistake cost us $10k

#428

Post by tgv »

kidme5 wrote: post #361
kidme5 wrote: post #335

Earlier quoted context omitted.

$10k.. peanuts. Elon might lose $500B to his xAI mistake that came out today: https://grook.ai/share?id=e269e88a7b1a71eff4f176c864b30161&x...

post: https://news.ycombinator.com/item?id=40628584

Which is flagged for some reason.

Re: A ChatGPT mistake cost us $10k

#429

Post by probably_wrong »

Closi wrote: post #20
chrisjj wrote: post #2

Thanks for telling. Bookmarked for the next time we're told ChatGPT's code error rate is acceptable because we review its code just like an intern's.

Your take-away is that this is ChatGPT's fault rather than a failure of testing?

There is a comment higher up in this thread [1] about how "ChatGPT 4-o can spot the error immediately" which I believe exemplifies the point the parent is making, namely, that any criticism of ChatGPT is met with "that's only because you didn't use enough of it", aka the "more cowbell" defense.

If HN is to be believed you should use ChatGPT to generate your data insertion code, to port it to a different language, and to ask for which error it made when you asked before. What to do if an error slips through in this version is always an exercise left for the reader.

[1] https://news.ycombinator.com/item?id=40630906

Re: A ChatGPT mistake cost us $10k

#430

Post by debugnik »

4hg4ufxhy wrote: post #346
internetter wrote: post #4

"Note: I want to preface this by saying yes the practices here are bad and could have been avoided. This was from a different time under large time constraints. Please read with that in mind" These "constraints" are why I'm terrified of subscribing to software

Having worked with some legacy subscription code it can be quite nasty. We had race conditions where we would charge users twice This has made me paranoid that any time I see timeout or error related to money I assume it went through and come back later.

A local cinema's shitty website charged me and, immediately after, my wifi disconnected for a few seconds. This somehow crashed their server entirely for several minutes, failing to send me the tickets even when it got back up.

It took me several threatening emails to make them understand they had already taken my money and I wasn't going to try again until I got a refund. Now I'm paranoid any time I purchase online at mediocre shops.

Post reply on HN