40 servers for one paying customer might seem like a bit of total annihilating overkill.
Not if you’re burning someone else’s venture capital.
A ChatGPT mistake cost us $10k
331–340 of 526 posts
Re: A ChatGPT mistake cost us $10k
#332Earlier quoted context omitted.
I think deploying and then going to sleep is the red flag here. They should have deployed the change at 9am or something and had the workday to monitor issues.
You can deploy and go to sleep if you have monitoring and alerting and someone getting paged. It shouldn't be a human monitoring for issues anyway, so the only reason to choose 9am over bedtime should be that you don't want to risk a late night page, not that someone will actually be checking up actively on the deployment.
For a big launch, you can have your entire team looking at issues in realtime, rather than a small number of people responding to pages in the middle of the night. I think if it's worth it to you there's a big difference.
Re: A ChatGPT mistake cost us $10k
#333I have to agree it feels navie to publish this.
"We're too lazy to write our own code, or even test that it works, please give us money VCs/Users."
Chat GPT isn't to blame here, chat GPT is like a tireless intern. It isn't to blame if the CTO pushes it's code straight to prod.
This is giving me a start up idea, code review as a service!
Re: A ChatGPT mistake cost us $10k
#334> We copy pasted the code it generated, saw everything worked fine, tried it in production, saw it also worked, and went on our merry way.
Oh.. this is not good. How did you see it worked fine? You did not try inserting new customers?
Re: A ChatGPT mistake cost us $10k
#335No, ChatGPT made you the money that your app generated since you had no ability to implement it otherwise/without ChatGPT. Your inability to code, debug, log, monitor cost you the $10k. ChatGPT is net positive in this story.
$10k.. peanuts. Elon might lose $500B to his xAI mistake that came out today:
https://grook.ai/share?id=e269e88a7b1a71eff4f176c864b30161&x...
Re: A ChatGPT mistake cost us $10k
#336No, a lack of monitoring cost you $10K. Your app was throwing a database exception and nobody was alerted that this was not only happening, but happening continuously and in large volumes. Such an alert would have made this a 5-minute investigation rather than 5 days. If you haven't fixed that alerting deficiency, then you haven't really fixed anything.
IMO way sooner. Some low hanging fruit tests would catch this before it’d even hit git.
Re: A ChatGPT mistake cost us $10k
#337Earlier quoted context omitted.
The bug was in multiple subscriptions not just users. And I can't think of one non-contrived reason to do it. Even when testing the visibility/access of subscriptions between users you need 2 users, but only one subscription.
create a subscription for a test user. delete it. Make sure you can create another subscription for the same user. create subscriptions with and without overlapping effective windows Those seem like very basic tests that would have highlighted the underlying issue
Have a look at stripe API. You don't delete subscriptions. You change them to free plan instead / cancel and then resume later. This for would not result in deletion of the entry. You also can update to change the billing date, so no overlapping subscriptions are needed. Neither test would result in the described bug.
Re: A ChatGPT mistake cost us $10k
#338Earlier quoted context omitted.
The StripeCustomer table has the same issue. There's both an `id` column and a unique `customerId` column. Presumably the `id` column is useless and could be removed. Also, is there a way to set up foreign key constraints on `userId` with this ORM? That seems like another oversight.
In that blog they say they have now added very robust unit and integration tests, so I don't think this is an issue.
I'd be concerned said "robust" tests were also lifted from ChatGPT.
Re: A ChatGPT mistake cost us $10k
#339Earlier quoted context omitted.
I totally disagree with this. You might as well argue that we shouldn't use code-completion of any kind because you might accidentally pick the wrong dependency or import. Or perhaps we shouldn't use any third-party libraries at all because you can use them to write reasonable-looking but incorrect code? Heck, why even bother using a programming language at all since we don't "own" how it's interpreted or compiled? U…
It is possible to use autocompletion correctly. It is possible to use libraries correctly. It is not possible to use AI correctly. It is only possible to correct its inevitable mistakes.
AI can provably write code without mistakes?
Re: A ChatGPT mistake cost us $10k
#340I have to agree it feels navie to publish this. "We're too lazy to write our own code, or even test that it works, please give us money VCs/Users." Chat GPT isn't to blame here, chat GPT is like a tireless intern. It isn't to blame if the CTO pushes it's code straight to prod. This is giving me a start up idea, code review as a service!
Given ChatGPT can pick up this bug when you feed it line by line [1], you might have the next startup here.