Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A ChatGPT mistake cost us $10k

asim.bearblog.dev

Post reply on HN
Print view View original

201–210 of 526 posts

Re: A ChatGPT mistake cost us $10k

#201

Post by whakim »

hamasho wrote: post #102
SmellTheGlove wrote: post #69

Earlier quoted context omitted.

This is a pretty common mistake with sqlalchemy whether you’re using ChatGPT or not. I learned the same lesson years ago, although I caught it while testing. I write plenty of python and I just don’t often pass functions in as parameters. In this case you need to! For something like this where you’re generating a unique id and probably need it in every model, it’s better to write a new Base model that includes things…

I wonder are there linters to detect those types of mistakes for SQLAlchemy. Even though I'm aware of such pitfalls, it's nice if linters can catch them cause I'm not confident to cache them all the time during code review. Some linters like pyright can identify dangerous defaults in function call, like `def hello(x=[]): pass` (mutable values shouldn't be a default). Linter plugins for widely-used and critical librar…

The mutable-default-arguments issue is easy for a third-party to linter to catch because it doesn't require any specific knowledge about primary keys and databases. Are there static typing plugins for other common packages that would catch issues like this?

Re: A ChatGPT mistake cost us $10k

#203

Post by Neywiny »

la64710 wrote: post #25

I dont get it , line 56 every time it was invoked would call the uuid4 function to generate a unique id isn’t it? Or was the issue due to the uuid4 function not getting invoked for any reason?

Interesting that more people weren't asking for more explanation on this. I feel like the article did a poor job but other commenters explained it well. More commonly I see this with `def f(x=list()):...` or similar. Linters will even warn about it. It really shows a lack of Python knowledge, so I'll jump on the bandwagon of rewriting this with such pressure was a bad idea.

Re: A ChatGPT mistake cost us $10k

#204

Post by atrettel »

jorams wrote: post #41
Aurornis wrote: post #5

On one hand, thanks for being honest about a story of how this bug came to be. On the other hand, I don’t think advertising the fact that the company introduced a major bug from copy and pasting ChatGPT code around and that they spent a week being unable to even debug why it was failing. I don’t know much about this startup, but this blog post had the opposite effect of all of the other high quality post-mortem posts…

Yeah this is not a good thing to advertise. - They were under large time constraints, but decided a full rewrite to a completely different stack was a good idea. - They copy-pasted a whole bunch of code, tested it manually once locally, once in production, and called it a day. - The debugging procedure for this issue so significant it made them dread waking up involved... testing it once and moving on. Every day. The…

I'd rather they admit a mistake and learn a lesson from it even if it isn't a good thing to advertise. That said, I agree that you are identifying a more important issue here but I also think you are being a bit too subtle about even if I agree with what you are saying. The real lesson that they should have learned from this ordeal is to never push code directly into production --- period. The article never mentions using a testbed or sandbox beforehand, and I kinda feel like they learned a good lesson but it may in fact be the wrong lesson to learn here.

Re: A ChatGPT mistake cost us $10k

#205

Post by steelbrain »

nicce wrote: post #113
steelbrain wrote: post #107

Earlier quoted context omitted.

FWIW, we host on Cloudflare and use their API to resize images on the fly and we're fine. Not so much a "lock-in" if other vendors can fill-in, is it?

You likely needed to do more extra work than needed, when compared to some other options. The lock-in here is the added developer time and complexity vs. just paying premium.

I disagree with your threshold for what makes something a lock-in but I admire your ideology of less friction in portability

Re: A ChatGPT mistake cost us $10k

#206

Post by thruway516 »

hehdhdjehehegwv wrote: post #191
SmellTheGlove wrote: post #69

Earlier quoted context omitted.

This is a pretty common mistake with sqlalchemy whether you’re using ChatGPT or not. I learned the same lesson years ago, although I caught it while testing. I write plenty of python and I just don’t often pass functions in as parameters. In this case you need to! For something like this where you’re generating a unique id and probably need it in every model, it’s better to write a new Base model that includes things…

Depending on your viewpoint, ORM are by nature a mistake. The time people spend learning the quirks of an ORM is much better put into learning SQL.

Honestly same can be said about a lot of frameworks. You will pry my vanilla JS debugged with print statements hand-coded in vi from these hands only when they're cold and dead.

Re: A ChatGPT mistake cost us $10k

#207

Post by throwaway2037 »

throw46365 wrote: post #125
mort96 wrote: post #103

Earlier quoted context omitted.

Eh I imagine they looked over the code as well, doing code review -- and at first glance, the code looks reasonable. I certainly wasn't able to catch the bug even though I tried to find it (and I was given a tiny collection of lines and the knowledge that there's a bug there!). If anything, I think this says something about how dangerous ChatGPT and similar tools are: reading code is harder than writing code, and whe…

> I don't think this is indicative of people who don't know what they're doing. I think this is indicative of people using "AI" tools to help with programming at all. I think using AI tools to write production code is probably indicative of people who don't really know what they are doing. The best way not to have subtle bugs is to think deeply about your code, not subcontract it out -- whether that is to people far…

    > I think using AI tools to write production code is probably indicative of people who don't really know what they are doing.
People said the same to me for using Microsoft IntelliSense 20 years ago. AI tools for programming are absolutely the future.

Re: A ChatGPT mistake cost us $10k

#209

Post by spencerchubb »

wavemode wrote: post #65

No, a lack of monitoring cost you $10K. Your app was throwing a database exception and nobody was alerted that this was not only happening, but happening continuously and in large volumes. Such an alert would have made this a 5-minute investigation rather than 5 days. If you haven't fixed that alerting deficiency, then you haven't really fixed anything.

I agree this is more of a monitoring mistake, and little to do with chatgpt

Re: A ChatGPT mistake cost us $10k

#210

Post by krick »

userbinator wrote: post #21
Aurornis wrote: post #5

On one hand, thanks for being honest about a story of how this bug came to be. On the other hand, I don’t think advertising the fact that the company introduced a major bug from copy and pasting ChatGPT code around and that they spent a week being unable to even debug why it was failing. I don’t know much about this startup, but this blog post had the opposite effect of all of the other high quality post-mortem posts…

It read like no one really knew what they were doing. "We just let it generate the code and everything seemed to work" is certainly not a good way to market your company.

I don't really care about them marketing their company, but, Jesus, seriously, that's how software is going to be written now. TBH, I'm not really sure if it's that much different from how it was, but it sounds just... fabulous.
Post reply on HN