311–320 of 526 posts
Did anyone else assume that this was about the Bear app, because there is no branding or link back to the author's actual project? Also, did anyone click on the double ^ at the bottom, hoping to either go back up to the top or find something about the author's company, only to find that they had upvoted the post by mistake? I'm wondering if (by this count) 167 other people might have done that.
I understand how the mistake was made, it seems relatively easy to slip by even when writing code without ChatGPT. But what I don't understand is how this wasn't caught after the first failure? Does this company not have any logging? Shouldn't the fact the backend is attempting to reuse UUIDs be immediately obvious from observing the error?
This sort of process failure is why there’s a Correction of Errors (postmortem) process at Amazon: https://aws.amazon.com/blogs/mt/why-you-should-develop-a-cor... Specifically asking why did it take so long to detect and why did it take so long to diagnose is useful in these situations.
Type 1 tries to find the error message and figure out what it really, really means by breaking down the error message and system.
Type 2 does trial and error on random related things until the problem goes away.
I hate to say that I've seen way more type 2s engineers than type 1, but maybe I’m working at the wrong companies.
No, a lack of monitoring cost you $10K. Your app was throwing a database exception and nobody was alerted that this was not only happening, but happening continuously and in large volumes. Such an alert would have made this a 5-minute investigation rather than 5 days. If you haven't fixed that alerting deficiency, then you haven't really fixed anything.
I think deploying and then going to sleep is the red flag here. They should have deployed the change at 9am or something and had the workday to monitor issues.
Now granted, if they'd carried on doing stuff (but not redeployed production) it may have shown up in say mid-aftenoon, or not depending on volume.
And of course the general guideline of not deploying anything to production early in the day, or on Friday, is still valid.
Earlier quoted context omitted.
It is correct, it's one of the most surprising things about Python and it causes a number of mistakes, even for experts. The easiest way to see this is by running something like this and seeing what gets printed out and when: print("1. start") def function(arg=print("2. func definition")): print("4. func call") print("3. after definition") function() function() function() You should see that the print statement in th…
My canonical example is that you want a function to have an argument that defaults to an empty list: def func(arg=[]): arg.append(1) print(arg) func() func() func() shoving a print into a function definition is weird and not something you'd do normally. But someone who doesn't know this footgun is going to write a function that defaults to an empty list, and then tear their hair out when things are broken.
Earlier quoted context omitted.
Just give it to GPT-5 for a refactor, easy!
It terrifies me that I have heard people say this unironically.
Forget knowing anything, just come up with a nice pitch deck and let the LLM write the stack.
Not wholly surprised these people are YC backed. I’ve got the impression YC don’t place much weight on technical competence, assuming you can just hire the talent if you know how to sell.
Well, now replace “hire some talent” with “get a GPT subscription and YOLO”, and you get the foundation these companies of tomorrow are going to be built on.
Which hey, maybe that’s right and they know something I don’t.
I have no idea why people are letting chatGPT do anything without pouring over everything it says first, and at that point why bother.
No, ChatGPT made you the money that your app generated since you had no ability to implement it otherwise/without ChatGPT. Your inability to code, debug, log, monitor cost you the $10k. ChatGPT is net positive in this story.
> Our project was originally full stack NextJS but we wanted to first migrate everything to Python/FastAPI.
> What happened was that as part of our backend migration, we were translating database models from Prisma/Typescript into Python/SQLAlchemy. This was really tedious. We found that ChatGPT did a pretty exceptional job doing this translation and so we used it for almost the entire migration.
ChatGPT wasn't a net positive if they wouldn't have tried to do this migration up-front without it.
Possibly they had better error logging in the other stack, possibly they didn't, possibly they needed it less because they were actually writing the code for it themselves and knew how it worked.
("Write all the code a second time before turning on monetization" is itself an interesting decision, of course.)
I agree, the headline is... misleading. Yes, ChatGPT made a mistake, but the issue is multiple. Perhaps it would be better if the headline was something like "A ChatGPT mistake taught us a $10k lesson".
I love using ChatGPT as much as the next person, but I also have run into more than a few circumstances on simple code where it's simply... made shit up. Like AWS (Boto3) functions that don't exist at all. So any code that comes out of it gets tested and understood by me. I'll ask it to explain and dig into the docs when it does things I don't understand.
That being said, the valuable lesson is in QA, debugging, logging and alerting. It's something that isn't a surprise a small (couple person, few months) startup would have done well. Often the developers of these projects are DEVELOPERS and not DevOps/SysAdmins/DBAs. The code gets written like developers do and not instrumented like a DevOps engineer would. Most get away with this for a long time (honestly, most companies get away with this for far too long).
So great write up, good lesson.