Earlier quoted context omitted.
The idea of inheriting a ChatGPT code base no one understands now makes it worse.
Just give it to GPT-5 for a refactor, easy!
It terrifies me that I have heard people say this unironically.
A ChatGPT mistake cost us $10k
221–230 of 526 posts
Re: A ChatGPT mistake cost us $10k
#222Earlier quoted context omitted.
It read like no one really knew what they were doing. "We just let it generate the code and everything seemed to work" is certainly not a good way to market your company.
> We just let it generate the code and everything seemed to work Everyone using C++20 compilers: side-glancing monkey.
The difference is that you can read the code that ChatGPT generates.
Re: A ChatGPT mistake cost us $10k
#223Two more problems identified solely from the screenshot: * you have two competing subscription id columns. * a uuid is not a string, it is a 128 bit integer. If your database limits to 64 bit integers then use a 64 bit integer for the id instead of a string, or use an array of 128 bytes.
Good catch, I agree that subscription_id looks hallucinated.
Re: A ChatGPT mistake cost us $10k
#224Earlier quoted context omitted.
It read like no one really knew what they were doing. "We just let it generate the code and everything seemed to work" is certainly not a good way to market your company.
This is getting more common. I have already had people try to tell me how something works from a chat gpt summary. This would have led to us taking a completely different direction… 5 minutes of reading the actual docs and I found out they were wrong. Now at a new company i have caught several people copy pasting gpt code that is just horrendous. It seems like this is where the industry is headed. The only thing i ha…
Like people who post "here's what ChatGPTx said" instead of their own answer. Quite literally, what is the point?
However, I don't think it's really bad for the technical industries long term. It probably does mean that some companies with loose internal quality control and enough shiftless employees pasting enough GPT spew without oversight will go to the wall because their software became unmaintainable and not useful, but this already happens. It's probably not hugely worse than the flood of bootcamp victims who wrote fizzbuzz in Python, get recruited by a credulous, cheap or desperate company and proceed to trash the place if not adequately supervised. If you can't detect bad work being committed, that's mostly on the company, not ChatGPT. Yes, it may make it harder, a bit, but it was oversight you should already have been prepared to have, doubly so if you can't trust employee output. It also probably implies strong QA, which is already a prerequisite of a solid product company.
Normal interest rates coming back will cut away companies that waste everyone's time and money by overloading themselves on endlessly compounding technical debt.
Re: A ChatGPT mistake cost us $10k
#225Earlier quoted context omitted.
This is getting more common. I have already had people try to tell me how something works from a chat gpt summary. This would have led to us taking a completely different direction… 5 minutes of reading the actual docs and I found out they were wrong. Now at a new company i have caught several people copy pasting gpt code that is just horrendous. It seems like this is where the industry is headed. The only thing i ha…
Like people who post "here's what ChatGPTx said" instead of their own answer. Quite literally, what is the point? However, I don't think it's really bad for the technical industries long term. It probably does mean that some companies with loose internal quality control and enough shiftless employees pasting enough GPT spew without oversight will go to the wall because their software became unmaintainable and not use…
Is the idea here that normal (read: low?) interest rates will let companies spend more time getting things right?
Re: A ChatGPT mistake cost us $10k
#226Did anyone else assume that this was about the Bear app, because there is no branding or link back to the author's actual project?
Also, did anyone click on the double ^ at the bottom, hoping to either go back up to the top or find something about the author's company, only to find that they had upvoted the post by mistake? I'm wondering if (by this count) 167 other people might have done that.
Re: A ChatGPT mistake cost us $10k
#227Earlier quoted context omitted.
This is a pretty common mistake with sqlalchemy whether you’re using ChatGPT or not. I learned the same lesson years ago, although I caught it while testing. I write plenty of python and I just don’t often pass functions in as parameters. In this case you need to! For something like this where you’re generating a unique id and probably need it in every model, it’s better to write a new Base model that includes things…
Ah, but it’s not a characteristic of SQLAlchemy tho. It’s how Python evaluates statements. Both Peewee and the Django ORM work on the same principle with default values. The intent is to pass a callable, not to call a function and populate an argument with what it returns.
Correct, it's not specific to sqlalchemy - I'm just saying I notice this a lot with sqlalchemy. Probably because it was the first significant bug I had to figure out how to fix when I introduced it in one of my first apps. I guess we never forget the first time we shot ourselves in the foot.
Re: A ChatGPT mistake cost us $10k
#228Earlier quoted context omitted.
Eh I imagine they looked over the code as well, doing code review -- and at first glance, the code looks reasonable. I certainly wasn't able to catch the bug even though I tried to find it (and I was given a tiny collection of lines and the knowledge that there's a bug there!). If anything, I think this says something about how dangerous ChatGPT and similar tools are: reading code is harder than writing code, and whe…
I'm a some-time Django developer and... I caught the bug instantly. Once I saw it was model/ORM code it was the first thing I looked for. I say that not to brag because (a) default args is a known python footgun area already and (b) I'd hope most developers with any real Django or SQLAlchemy experience would have caught this pretty quick. I guess I'm just suggesting that maybe domain experience is actually worth some…
I missed it, because I was really confused what the models were doing: why is there an id and a subscription_id? Are the user_id fields related?
Re: A ChatGPT mistake cost us $10k
#229On one hand, thanks for being honest about a story of how this bug came to be. On the other hand, I don’t think advertising the fact that the company introduced a major bug from copy and pasting ChatGPT code around and that they spent a week being unable to even debug why it was failing. I don’t know much about this startup, but this blog post had the opposite effect of all of the other high quality post-mortem posts…
That's an alright takeaway: the team made a rookie mistake and then they made a PR mistake by oversharing. Otherwise, I think this comment thread is a classic example why company engineering blogs choose to be boring. Better ten articles that have some useful information, than a single article that allows the commentariat to pile on and ruin your reputation.
A mistake is interesting if the mistake itself or the RCA is interesting - using sloppy methods isn't really that interesting on its face.
Re: A ChatGPT mistake cost us $10k
#230On one hand, thanks for being honest about a story of how this bug came to be. On the other hand, I don’t think advertising the fact that the company introduced a major bug from copy and pasting ChatGPT code around and that they spent a week being unable to even debug why it was failing. I don’t know much about this startup, but this blog post had the opposite effect of all of the other high quality post-mortem posts…
That's an alright takeaway: the team made a rookie mistake and then they made a PR mistake by oversharing. Otherwise, I think this comment thread is a classic example why company engineering blogs choose to be boring. Better ten articles that have some useful information, than a single article that allows the commentariat to pile on and ruin your reputation.
I think it’s an unfair takeaway. I have over a decade of experience and still had to stare at the line to find the bug. If that makes them incompetent, I stand with them. It’s a bug I’ve seen people make in other contexts, not just chatbots.
The AI angle is probably why people are piling on. There’s a latent fear that AI will take our jobs, and this is a great way to skewer home that we’re still needed. For now.
The one thing I will say is that it probably wouldn’t take me days to track it down. But that’s only because I have lots of experience dealing with bugs the way that The Wolf deals with backs of cars. When you’re trying to run a startup on top of everything else, it can be easy to miss.
I’m happy they gave us a glimpse of early stage growing pains, and I don’t think this was a PR fumble. It shows that lots of people want what they’re making, which is roughly the only thing that matters.