Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A ChatGPT mistake cost us $10k

asim.bearblog.dev

Post reply on HN
Print view View original

471–480 of 526 posts

Re: A ChatGPT mistake cost us $10k

#471

Post by KronisLV »

neonsunset wrote: post #466
KronisLV wrote: post #462

Earlier quoted context omitted.

> You don't need to make DTOs when you don't have to, using AutoMapper is considered a bad practice and is heavily discouraged (if you do have to use a tool like that, there are alternatives like Mapperly which are zero-cost to use and will give you built-time information on what doesn't map without having to run the application). The thing is, that you'll probably have entities mapped against the database schema wit…

C# differs quite a bit from Java particularly in surrounding libraries, and there is a learning curve to things...the expectation that it's another Java ends up misleading many people :) I'm sure MapStruct would also require you to handle differences in data presentation, in a similar way you would have to do with Automapper (or Mapperly). .NET generally puts more emphasis on "boilerplate-free happy path + predictabl…

Thank you so much for sharing your experiences, lots of useful suggestions here!

Re: A ChatGPT mistake cost us $10k

#472

Post by sokoloff »

kissgyorgy wrote: post #470
sokoloff wrote: post #440

Earlier quoted context omitted.

I don’t find the source of the error being a careless human writing original code without proper review vs a careless human copy/pasting code without proper review to be significantly different.

The difference is that everyone knows about hallucinations, so an LLM never can be trusted by default, they still trusted it blindly.

Is that really worse than trusting blindly the code of a human, about which "everyone knows about" the bugs that humans write as well.

Re: A ChatGPT mistake cost us $10k

#473

Post by appplication »

kidme5 wrote: post #335
animex wrote: post #306

No, ChatGPT made you the money that your app generated since you had no ability to implement it otherwise/without ChatGPT. Your inability to code, debug, log, monitor cost you the $10k. ChatGPT is net positive in this story.

$10k.. peanuts. Elon might lose $500B to his xAI mistake that came out today: https://grook.ai/share?id=e269e88a7b1a71eff4f176c864b30161&x...

What an I supposed to see here? What’s the mistake/tie in to him losing something

Re: A ChatGPT mistake cost us $10k

#475

Post by latexr »

bluelightning2k wrote: post #93

It is strange that this took 5 days to find. Simply because of logs. Go to logs. Filter by errors. Oh, errors in insert subscription. Seems relevant. I could understand if the errors were somewhere else. Even if logs didn't exist. Problematic endpoint generating 50 emails per day? I would have immediately thrown a try catch and rendered the error to the user if logging was impossible. Then your very next bug report s…

> Ironically they should have asked ChatGPT for help debugging

Why are you assuming they didn’t? This is an AI company using AI to build their product and trusting it without proper code review, testing, or guardrails. Clearly they’re all in on AI hype. This took them days to solve, so not only would I bet they asked ChatGPT, I’d wager multiple people tried it multiple times.

Re: A ChatGPT mistake cost us $10k

#476

Post by withinboredom »

f35caleb wrote: post #467
huygens6363 wrote: post #456

Earlier quoted context omitted.

Creating more than one entry is not a “scaling issue”.

Agreed. I'm chuckling because I was wrestling with this exact same bug on a FastAPI project last night. I caught it because I have a habit of submitting API endpoints multiple times with the same data to see how the database reacts. Got a key collision when I tried to submit the endpoint the second time and figured out that setup didn't create a new UUID each time. Unit tests are good, yes. Monitoring is also good. B…

I generally follow this process:

1. Get it working: write the code for the desired behavior, not worrying about making it beautiful, testable, whatever.

2. Get it working well: manually testing and finding edge cases, refactoring to get it testable and writing tests to solidify behavior.

3. Get it working fast: optimizing it to be as fast as I need it to be (can sometimes skip this step). No tests should change here, but only new tests.

Re: A ChatGPT mistake cost us $10k

#477

Post by stouset »

Kesty wrote: post #405
stouset wrote: post #293

Earlier quoted context omitted.

Also facepalms here: UUIDs as strings and UUIDv4. UUIDs are just 128-bit values. They might be conventionally encoded for humans as hex, but storing them as 36-byte (plus a few more for length) strings is a pointless waste of both space and performance.

They don't have logs and commit directly to production 10/20 times a day. I don't think 128 bits vs 36 byte performance it's a main concern right now

Probably not, but this is something that would have taken all of ten seconds to get right. And the size and performance impact is multiplied at every tier of their application.

It’s also not just the size itself. Despite being fixed-size in practice, these are variable-sized strings in application code which now means gajillions of pointless allocations and indirection for everything. There are a ton of knock-on performance consequences here, all on the most heavily-used columns in your data model.

Worst of all should they actually succeed, this is going to be absolutely excruciating to fix.

Re: A ChatGPT mistake cost us $10k

#478

Post by sgarland »

zo1 wrote: post #80
TheRoque wrote: post #74

Earlier quoted context omitted.

In their defense, I find SQLAlchemy syntax quite horrible, and I always have to look up everything. It also got a 2.0 release recently which changes some syntax (good luck guessing which version ChatGPT will use), and makes the process even more annoying.

SQLAlchemy syntax is ridiculously obvious and straightforward as long as you're not doing anything weird. The takeaway here is that they weren't mature enough to realize they were, in fact, doing something "weird". I.e. Using UUIDs for PKs, because hey "Netflix does it so we have to too! Oh and we need an engineering blog to advertise it". Edit. More clarity about why the UUID is my point of blame: If they had used a…

Unfortunately, UUID as PK is an extremely common pattern these days, because devs love to believe that they’ll need a distributed DB, and also that you can’t possibly use integers with such a setup. The former is rarely true, the latter is blatantly false.

Re: A ChatGPT mistake cost us $10k

#479

Post by Edd1CC »

huygens6363 wrote: post #336
wavemode wrote: post #65

No, a lack of monitoring cost you $10K. Your app was throwing a database exception and nobody was alerted that this was not only happening, but happening continuously and in large volumes. Such an alert would have made this a 5-minute investigation rather than 5 days. If you haven't fixed that alerting deficiency, then you haven't really fixed anything.

IMO way sooner. Some low hanging fruit tests would catch this before it’d even hit git.

Yeah if one instance created one ID, then any integration tests creating more than one user would have failed. There were no testing or logging on a system with live users while doing a refactor between two dynamic languages

Re: A ChatGPT mistake cost us $10k

#480

Post by TheNewsIsHere »

wg0 wrote: post #448
Edd1CC wrote: post #445

They had 8 AWS tasks running 5 instances each with code written in TypeScript and Python, with frameworks like next.js, with $40 revenue and only a few weeks dev time? What the actual fuck hahahaha This is made worse when they edit saying the reason the codes crap is because of time constraints, but spent their time refactoring across languages and spinning up a distributed system FOR NO REASON. That is self imposed…

Because they have half a million dollar to star with and 1.2 million dollars on top and then free AWS credits to burn.

They even said as much in TFA - "[...] overkill, yes we know, but to be fair we had AWS credits".

Fully acknowledging the irony I am about to invoke - this is why I hate startup culture. Not startups, but this ridiculous culture of "well the VC gave us a million bucks and that bought us $100,000 in AWS credits, so let's just use it."

As someone who has built my company fully on my own dime (and the dimes of two colleagues), it's easy enough to burn piles of money in AWS (or any other cloud) when you're making an attempt at being judicious. Spinning up eight backends (edit: running five instances each, no less!) just because you have money, despite the fact that you know you don't need that much compute, is just insane. If for no other reason than you're just throwing your own credits away.

Post reply on HN