Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A ChatGPT mistake cost us $10k

asim.bearblog.dev

Post reply on HN
Print view View original

81–90 of 526 posts

Re: A ChatGPT mistake cost us $10k

#81

Post by brigadier132 »

viraptor wrote: post #28
Aurornis wrote: post #5

On one hand, thanks for being honest about a story of how this bug came to be. On the other hand, I don’t think advertising the fact that the company introduced a major bug from copy and pasting ChatGPT code around and that they spent a week being unable to even debug why it was failing. I don’t know much about this startup, but this blog post had the opposite effect of all of the other high quality post-mortem posts…

They spent 5 days. The bug type is pretty common and could easily be done by a developer. (It's a similar class to the singleton default argument issue that many people complain about) Meh, I don't mind the cautionary tale and don't think chatgpt was even relevant. It's actually a tricky bug, because usual tests wouldn't catch it (db wiped for good isolation) and many ways of manual testing would restart the service…

> db wiped for good isolation

Why? In fact, not having good isolation would have caught this bug. Generate random emails for each test. Why would you test on a completely new db as if that is what will happen in the real world?

Re: A ChatGPT mistake cost us $10k

#82

Post by dragonwriter »

viraptor wrote: post #51
hipadev23 wrote: post #29

This is a great example of why SQLAlchemy is a terrible ORM, and chatGPT is not alone in making the same mistake as millions of engineers, in fact likely where it learned the mistake. default = python code evaluates the default value as necessary for each new record server_default = the initial CREATE TABLE uses this computed (from python) value, thus the hardcoded UUID. They also could have done server_default=text(…

I'm not sure what you think SQLalchemy can fix here? There has to be an option to pass a static default value and the library does not have a visibility into the parse tree. What's the proposed solution?

One solution would be to have a separate default (static value) and default_factory (callable that returns a value) keyword argument, and make a static default on a PK (or other unique) column, by default, be an error.

Re: A ChatGPT mistake cost us $10k

#83

Post by __jonas »

nicce wrote: post #55
koito17 wrote: post #40

Earlier quoted context omitted.

More importantly, what was the motivation behind a rewrite from TypeScript to Python? From the article Our project was originally full stack NextJS but we wanted to first migrate everything to Python/FastAPI. Seems like this entire mess could've been avoided if they had stuck with their existing codebase, which seemed to have been satisfying their business requirements.

> NextJS There is well-hidden vendor-lock when using NextJS, at least.

that’s a bold claim, could you give an example?

Re: A ChatGPT mistake cost us $10k

#84

Post by __jonas »

dgfitz wrote: post #50
koito17 wrote: post #40

Earlier quoted context omitted.

More importantly, what was the motivation behind a rewrite from TypeScript to Python? From the article Our project was originally full stack NextJS but we wanted to first migrate everything to Python/FastAPI. Seems like this entire mess could've been avoided if they had stuck with their existing codebase, which seemed to have been satisfying their business requirements.

I’m sure they are influenced by the likes of Reddit and Twitter rewriting their stack. I mean, that’s what has to be done, right? /s

Is there a trend there of moving from Next to FastAPI? I would be surprised.

Perhaps they are doing some AI thing and want to have python everywhere.

Re: A ChatGPT mistake cost us $10k

#85

Post by joshstrange »

> Our project was originally full stack NextJS but we wanted to first migrate everything to Python/FastAPI.

Tell me you had no business being invested in without telling me.

I’m going to be harsh here but I honestly have no clue how else to respond. You wrote your backend in Node/Typescript and then decided to change it to Python. What in the world would make that a good idea? No seriously, there is absolutely nothing sane about that decision. In top of that, you used ChatGPT to do the conversion for some of your DB models, was that just for speed or because you didn’t know what you were doing (new language/framework?).

Also, you say you had credits to burn (god this industry is so messed up sometimes) so why rewrite? Clearly not for cost/performance and Node to Python seems like a very lateral move all things considered.

I’m completely flabbergasted as to why you would rewrite your backend like this.

Re: A ChatGPT mistake cost us $10k

#86

Post by nicce »

brigadier132 wrote: post #78
nicce wrote: post #55

Earlier quoted context omitted.

> NextJS There is well-hidden vendor-lock when using NextJS, at least.

There is no vendor lockin with nextjs

Simply put - if you want get the best out of the framework, you need to host it in Vercel. Otherwise, there are better options for frameworks. No need to ”fight it”.

You will find many issues from GitHub which are not considered because they would make the framework ”better” or easier to use on other clouds.

Re: A ChatGPT mistake cost us $10k

#87

Post by joshstrange »

contextnavidad wrote: post #43

> Our project was originally full stack NextJS but we wanted to first migrate everything to Python/FastAPI This is the eye opener for me, how is a startup justifying a re-write when they don't even have customers?

Dear god, I thought I was taking crazy pills. After saying the same thing (a rewrite this early is insane) I was scanning the comments and no one else was pointing this out. I have no clue what would drive someone to rewrite this early (with or without customers) for what is effectively a lateral move (node to python). If you had hundreds of customers and wanted to rewrite in Go or similar then maybe (I still question even that).

Re: A ChatGPT mistake cost us $10k

#88

Post by viraptor »

hn_throwaway_99 wrote: post #75
viraptor wrote: post #67

Earlier quoted context omitted.

The bug was in multiple subscriptions not just users. And I can't think of one non-contrived reason to do it. Even when testing the visibility/access of subscriptions between users you need 2 users, but only one subscription.

What? This doesn't make any sense: 1. First, if you look at the code they posted, they had the same bug on line 45 where they create new Stripe customers. 2. The issue is not multiple subscriptions per user (again, if you look at the code, you'll see each Subscription has one foreign key user_id column). The problem is if you had multiple subscriptions (each from different users) created from the same backend instanc…

Not every user needs a stripe customer. I'm creating the stripe entries only on subscription in my app.

Your second point is true, but I don't see what it changes. Most automated unit/integration testing would just wipe the database between tests and needing two subscribed users in a single test is not that likely.

Re: A ChatGPT mistake cost us $10k

#89

Post by OutOfHere »

This is why you should always rush your engineers, never giving them enough time to validate or understand what ChatGPT just spewed. Good job. /s

Meanwhile, I go through the tedious process of understanding ChatGPT's code letter-by-letter, also reading the docs, searching StackOverflow, even offering and rewarding bounties on StackOverflow, all to see if the code makes a shred of sense.

Re: A ChatGPT mistake cost us $10k

#90

Post by nikonyrh »

crote wrote: post #33
nikonyrh wrote: post #3

I'm not familiar with this library, how does `text("(now())")` evaluate and why there are extra parentheses? And should that be a lambda expression as well, so that `create_date` isn't just the timestamp when the python process was started?

I'm not familiar with the library either, but that seems to be a SQL expression executed on the database server. It's basically a copy-paste from the official documentation[0]. So no, not a lambda expression, because it's not computed in Python. As to the extra parentheses: I bet that's a force-of-habit thing to prevent potential issues. For example, it seems Sqlite requires them for exactly this kind of default defi…

Aah that makes sense, thanks!
Post reply on HN