I don’t understand how you can move fast in software development without at least some rudimentary observability in place (logs). You’d see a 500 and likely an IntegrityError exception and that would give you a huge clue you’re not setting your PK correctly.
A ChatGPT mistake cost us $10k
181–190 of 526 posts
Re: A ChatGPT mistake cost us $10k
#182I understand how the mistake was made, it seems relatively easy to slip by even when writing code without ChatGPT. But what I don't understand is how this wasn't caught after the first failure? Does this company not have any logging? Shouldn't the fact the backend is attempting to reuse UUIDs be immediately obvious from observing the error?
IMO this is the real issue.
I guarantee you that they _will_ have another production bug like this sometime in the future (every fast paced project will). You'd hope this next one wont take 5 days to identify.
Re: A ChatGPT mistake cost us $10k
#183Re: A ChatGPT mistake cost us $10k
#184Earlier quoted context omitted.
There’s a difference between not knowing what you are doing and making a mistake.
An awful lot of mistakes are made because one didn't know something that would have enabled one to avoid it. Not knowing what you don't know is difficult to work around.
[deleted]
Re: A ChatGPT mistake cost us $10k
#185Earlier quoted context omitted.
It read like no one really knew what they were doing. "We just let it generate the code and everything seemed to work" is certainly not a good way to market your company.
Eh I imagine they looked over the code as well, doing code review -- and at first glance, the code looks reasonable. I certainly wasn't able to catch the bug even though I tried to find it (and I was given a tiny collection of lines and the knowledge that there's a bug there!). If anything, I think this says something about how dangerous ChatGPT and similar tools are: reading code is harder than writing code, and whe…
I totally disagree with this. You might as well argue that we shouldn't use code-completion of any kind because you might accidentally pick the wrong dependency or import. Or perhaps we shouldn't use any third-party libraries at all because you can use them to write reasonable-looking but incorrect code? Heck, why even bother using a programming language at all since we don't "own" how it's interpreted or compiled? Ultimately I agree that using third-party tools saves time at the cost of potentially introducing some types of bugs. (Note that said tools may also help you avoid other types of bugs!) But it's clearly a tradeoff (and one where we've collectively disagreed with you the vast, vast majority of the time) and boiling that down to AI=bad misses the forest for the trees.
Re: A ChatGPT mistake cost us $10k
#186On one hand, thanks for being honest about a story of how this bug came to be. On the other hand, I don’t think advertising the fact that the company introduced a major bug from copy and pasting ChatGPT code around and that they spent a week being unable to even debug why it was failing. I don’t know much about this startup, but this blog post had the opposite effect of all of the other high quality post-mortem posts…
That's an alright takeaway: the team made a rookie mistake and then they made a PR mistake by oversharing.
Otherwise, I think this comment thread is a classic example why company engineering blogs choose to be boring. Better ten articles that have some useful information, than a single article that allows the commentariat to pile on and ruin your reputation.
Re: A ChatGPT mistake cost us $10k
#187Earlier quoted context omitted.
> db wiped for good isolation Why? In fact, not having good isolation would have caught this bug. Generate random emails for each test. Why would you test on a completely new db as if that is what will happen in the real world?
It makes your tests more robust. Generally you don’t want tests that are too sensitive to external state since they will fail spuriously and become useless.
Of course your tests shouldn't be sensitive to external state. Why would other tests running affect your test?
Re: A ChatGPT mistake cost us $10k
#188> Our project was originally full stack NextJS but we wanted to first migrate everything to Python/FastAPI. Tell me you had no business being invested in without telling me. I’m going to be harsh here but I honestly have no clue how else to respond. You wrote your backend in Node/Typescript and then decided to change it to Python. What in the world would make that a good idea? No seriously, there is absolutely nothin…
> Tell me you had no business being invested in without telling me. Check out their comment history to see who invested in them.
Whats even more ironic is that their business is about extracting, fixing, and repairing data using AI. They had a chance to dogfood here and missed it.
Re: A ChatGPT mistake cost us $10k
#189Re: A ChatGPT mistake cost us $10k
#190On one hand, thanks for being honest about a story of how this bug came to be. On the other hand, I don’t think advertising the fact that the company introduced a major bug from copy and pasting ChatGPT code around and that they spent a week being unable to even debug why it was failing. I don’t know much about this startup, but this blog post had the opposite effect of all of the other high quality post-mortem posts…
This is a pretty common mistake with sqlalchemy whether you’re using ChatGPT or not. I learned the same lesson years ago, although I caught it while testing. I write plenty of python and I just don’t often pass functions in as parameters. In this case you need to! For something like this where you’re generating a unique id and probably need it in every model, it’s better to write a new Base model that includes things…
Ah, but it’s not a characteristic of SQLAlchemy tho. It’s how Python evaluates statements. Both Peewee and the Django ORM work on the same principle with default values.
The intent is to pass a callable, not to call a function and populate an argument with what it returns.