Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A ChatGPT mistake cost us $10k

asim.bearblog.dev

Post reply on HN
Print view View original

171–180 of 526 posts

Re: A ChatGPT mistake cost us $10k

#171

Post by dragonwriter »

morgante wrote: post #8
chrisjj wrote: post #2

Thanks for telling. Bookmarked for the next time we're told ChatGPT's code error rate is acceptable because we review its code just like an intern's.

Honestly I'm not sure why ChatGPT has anything to do with this problem. I remember making the exact same mistake (accidentally using a single function call in a schema) back in 2010. No LLMs required. The bigger culprit is probably a lack of testing / debugging. This error would immediately get caught if you simply registered twice on a test instance.

> Honestly I'm not sure why ChatGPT has anything to do with this problem.

Because if humans had written and reviewed the code, multiple team members would have had to have learned Python, and SQLAlchemy specifically, which, even if the mistake was initially made as many times as ChatGPT did, there would have been multiple independent opportunities for it to be caught and questioned and the relevant knowledge shared during development.

ChatGPT may be able to crank out immense volumes of superficially functional code, but if its your only “teammate” that understands the libraries used and touches the code, its a huge single point of failure.

Re: A ChatGPT mistake cost us $10k

#172

Post by zeroonetwothree »

userbinator wrote: post #62

During the work day, this was fine. We probably committed 10-20 times a day That's... scary, to put it mildly. I wonder how many of those are fixes to things broken by previous commits. Then again, I work on software where the average is far less than one commit per day, although it's a mature product. Nonetheless, "slow down and think" is probably good advice in this case.

I suppose it depends how many people work on it and what stage it’s in. Without knowing that’s it’s hard to say whether 20 makes more sense

Re: A ChatGPT mistake cost us $10k

#173

Post by hipadev23 »

viraptor wrote: post #51
hipadev23 wrote: post #29

This is a great example of why SQLAlchemy is a terrible ORM, and chatGPT is not alone in making the same mistake as millions of engineers, in fact likely where it learned the mistake. default = python code evaluates the default value as necessary for each new record server_default = the initial CREATE TABLE uses this computed (from python) value, thus the hardcoded UUID. They also could have done server_default=text(…

I'm not sure what you think SQLalchemy can fix here? There has to be an option to pass a static default value and the library does not have a visibility into the parse tree. What's the proposed solution?

> I'm not sure what you think SQLalchemy can fix here

Separate table definition (DDL) from row-insertion (DML) definitions along with their corresponding defaults.

Re: A ChatGPT mistake cost us $10k

#174

Post by jowea »

Lunrtick wrote: post #59
hipadev23 wrote: post #29

This is a great example of why SQLAlchemy is a terrible ORM, and chatGPT is not alone in making the same mistake as millions of engineers, in fact likely where it learned the mistake. default = python code evaluates the default value as necessary for each new record server_default = the initial CREATE TABLE uses this computed (from python) value, thus the hardcoded UUID. They also could have done server_default=text(…

They could also have done default=uuid.uuid4 to have a new id each time, or default=lambda : str(uuid.uuid4()). It's not really related to whether or not it's a database default. It seems quite unfair to place the blame on SQLAlchemy here, or even Python. Even a statically typed language wouldn't prevent this kind of issue - the author of the code is the only person who can decide when they mean "use this exact strin…

I could imagine some sort of static analysis saying that using a constant value instead of a function is a mistake for columns with unique set.

Re: A ChatGPT mistake cost us $10k

#175

Post by sanderjd »

tmpz22 wrote: post #149
SilasX wrote: post #111

Earlier quoted context omitted.

For that matter, has OP even met the HN accepted wisdom? "No one knows what they're doing, everyone's faking it, it's fine if you are too" -- so don't take it as a red flag when your fumbling around keeps blowing up, because it surely must work that way everywhere else. My early rant against this mentality: https://news.ycombinator.com/item?id=19214749

Its very humbling coming out of startup-land and working with big tech engineers and realizing their tooling runs circles around everybody else and enables them to be much more precise with their work and scale, though it isn't without trade-offs.

Yeah but a lot of that is just the accrual of improvements that is possible with a lot of resources over a long period of time.

People working in "big tech" aren't fundamentally better at building reliable tools and systems; the time and resource constraints are entirely different.

Re: A ChatGPT mistake cost us $10k

#176

Post by throw46365 »

japhyr wrote: post #126

This kind of thing must be happening much more often than we're hearing about it, right? I'd love to start a site that collects AI "horror stories", where trusting an AI's output led to significant consequences. I have no idea how to validate people's anecdotes, though. (To be clear I don't doubt this story at all. But if I set up a site where people could submit stories I wouldn't trust any submissions at face value…

Free idea: a website called But Humans Also, where one collects bad justifications for applying LLMs.

the dot-com is available!

Re: A ChatGPT mistake cost us $10k

#178

Post by jowea »

bluepnume wrote: post #95
tazjin wrote: post #76

Earlier quoted context omitted.

Sure, you can follow along the old meme: https://twitter.com/vbhvsgr/status/1419369352164372482 Though in practice in decent languages it's much less likely you'd write your own `any -> any, any`-typed library for whatever (in this case DB interactions), and use a strongly typed one in which this would at least have been a much more explicit mistake to make.

But this isn't an `any -> any` case. They passed in a default value, as a string, which is the correct type for a default value for this column. Even with very strong typing they wouldn't have got a type error here right?

You could make a special primary key column creation function that rejects static values.

Re: A ChatGPT mistake cost us $10k

#179

Post by hn_throwaway_99 »

viraptor wrote: post #88
hn_throwaway_99 wrote: post #75

Earlier quoted context omitted.

What? This doesn't make any sense: 1. First, if you look at the code they posted, they had the same bug on line 45 where they create new Stripe customers. 2. The issue is not multiple subscriptions per user (again, if you look at the code, you'll see each Subscription has one foreign key user_id column). The problem is if you had multiple subscriptions (each from different users) created from the same backend instanc…

Not every user needs a stripe customer. I'm creating the stripe entries only on subscription in my app. Your second point is true, but I don't see what it changes. Most automated unit/integration testing would just wipe the database between tests and needing two subscribed users in a single test is not that likely.

> needing two subscribed users in a single test is not that likely.

Apparently not.

Re: A ChatGPT mistake cost us $10k

#180

Post by droopyEyelids »

bogota wrote: post #167
userbinator wrote: post #21

Earlier quoted context omitted.

It read like no one really knew what they were doing. "We just let it generate the code and everything seemed to work" is certainly not a good way to market your company.

This is getting more common. I have already had people try to tell me how something works from a chat gpt summary. This would have led to us taking a completely different direction… 5 minutes of reading the actual docs and I found out they were wrong. Now at a new company i have caught several people copy pasting gpt code that is just horrendous. It seems like this is where the industry is headed. The only thing i ha…

It’sa Tower of Babel like effect
Post reply on HN