[deleted]
A ChatGPT mistake cost us $10k
291–300 of 526 posts
Re: A ChatGPT mistake cost us $10k
#292So I understand right, there are two solutions that would have handled this before it even got to prod or at least found it in prod fast. 1. Bunch of tests that simulate exactly the scenario of signups. Hundreds of them actually inserting db records with maybe some kind of dummy stripe code. 2. Logs of the actual uuid for each person. The second would never have been used since the tests would have caught this bug. B…
Im guessing they used all those credits to set up those instances, but never took the extra step to add log ingestion or any kind of monitoring. Unique constraint violations peaking should have at least sent some kind of mail or slack notification a few hours after release (putting aside the "it didnt happen during the day because we push to prod several times daily" - which is insane in its own right).
Nothing here really sounds like GPTs fault to me. The issue is something that could easily have been done by a human and missed in PR.
Re: A ChatGPT mistake cost us $10k
#293Earlier quoted context omitted.
They didn’t even know there was an error until the customers came ringing. You always want to know what errors happened before your customers do, logging, alerting, any monitoring at all would have helped them here.
Experience should tell you to always take a hard look at anything UUIDs.
Also facepalms here: UUIDs as strings and UUIDv4.
UUIDs are just 128-bit values. They might be conventionally encoded for humans as hex, but storing them as 36-byte (plus a few more for length) strings is a pointless waste of both space and performance.
Re: A ChatGPT mistake cost us $10k
#294> Our project was originally full stack NextJS but we wanted to first migrate everything to Python/FastAPI. Tell me you had no business being invested in without telling me. I’m going to be harsh here but I honestly have no clue how else to respond. You wrote your backend in Node/Typescript and then decided to change it to Python. What in the world would make that a good idea? No seriously, there is absolutely nothin…
> Tell me you had no business being invested in without telling me. Check out their comment history to see who invested in them.
The lesson I learned from the dot-com era is that people who are dependent on the hype to make profit will crane their necks to believe the hype.
Salespeople, executives, engineers, it doesn’t matter.
Every day on HN reminds me a little more of 1998.
Re: A ChatGPT mistake cost us $10k
#295Two more problems identified solely from the screenshot: * you have two competing subscription id columns. * a uuid is not a string, it is a 128 bit integer. If your database limits to 64 bit integers then use a 64 bit integer for the id instead of a string, or use an array of 128 bytes.
The StripeCustomer table has the same issue. There's both an `id` column and a unique `customerId` column. Presumably the `id` column is useless and could be removed.
Also, is there a way to set up foreign key constraints on `userId` with this ORM? That seems like another oversight.
Re: A ChatGPT mistake cost us $10k
#296I've written less than 1000 lines of Python in total probably, but I correctly spotted the problem. Python has this misfeature whereby it didn't correctly crib Common Lisp's evaluation strategy for the expressions that give default values to optional function arguments. When you have an argument like foo=obj.whatever() the obj.whatever() is evaluated (would you believe it!) at the time the definition of the function…
> When you have an argument like foo=obj.whatever(), the obj.whatever() is evaluated at the time the definition of the function is being processed, not at the time when the function is being called. This can't be correct, surely? What if .whatever() relies on internal state that changes after obj is initialized (or after the function surrounding foo is declared, not sure what you're saying)?
There is an obj visible at the point of the function definition. When that function definition takes place, obj.whatever() is called, and the value stashed away. Thus the call obj.whatever() has to work at that time. The stashed value is retrieved whenever the corresponding function argument is missing.
Re: A ChatGPT mistake cost us $10k
#297Earlier quoted context omitted.
It read like no one really knew what they were doing. "We just let it generate the code and everything seemed to work" is certainly not a good way to market your company.
Eh I imagine they looked over the code as well, doing code review -- and at first glance, the code looks reasonable. I certainly wasn't able to catch the bug even though I tried to find it (and I was given a tiny collection of lines and the knowledge that there's a bug there!). If anything, I think this says something about how dangerous ChatGPT and similar tools are: reading code is harder than writing code, and whe…
I don't know Python or SQLAlchemy that great, though I do have the benefit of it being cut down to a small amount of code and being told there was a bug there. That said, I didn't see the actual bug, but I did mentally flag that as something I ought to look up how it actually behaved. It's suspicious that some columns used `default` with what looks like Python code while others used `server_default` with what appears to be strings that look more like database engine code. If I was actually responsible for this, I'd want to dig into why there is that difference and where and when that code actually runs.
It's also the case that "code review" covers a lot of things, from quickly skimming the code and saying eh, it's probably fine, to deeply reading and ensuring that you fully understand the behavior in all possible cases of every line of code. The latter is much more effective, but probably not nearly as common as it ought to be.
Re: A ChatGPT mistake cost us $10k
#298Earlier quoted context omitted.
> When you have an argument like foo=obj.whatever(), the obj.whatever() is evaluated at the time the definition of the function is being processed, not at the time when the function is being called. This can't be correct, surely? What if .whatever() relies on internal state that changes after obj is initialized (or after the function surrounding foo is declared, not sure what you're saying)?
https://stackoverflow.com/questions/1132941/the-mutable-defa... and https://www.valentinog.com/blog/tirl-python-default-argument... basically, having a default argument value in a function definition means to evaluate it during definition time of that function, not when the function is invoked. This is a foot gun.
if the default argument is an object it’s reused between invocations. Hence why setting default parameters to empty list / empty dict is flagged by static analysis suites
Re: A ChatGPT mistake cost us $10k
#299I have seen the same mistake made in code created by humans. Many times, especially in react / typescript/ JavaScript, someone will forget to use a lambda. I felt the blog post failed to articulate the root cause of the issue and went straight to blaming ChatGPT. When you rush and make large or non peer code reviewed commits to main it is going to happen. The real issue was when you rush, take shortcuts and don’t ade…
My mental model for ChatGPT is that it’s an entry-level engineer that will never be promoted to a terminal level and will eventually be let go.
However, this engineer can type infinitely fast, which means it might be useful if used very carefully.
Anyway, letting such a person near financially important code would lead to similar issues, and in both cases, I’d question the judgment of the person that decided to deploy the code at all, let alone without much testing.
Re: A ChatGPT mistake cost us $10k
#300I've written less than 1000 lines of Python in total probably, but I correctly spotted the problem. Python has this misfeature whereby it didn't correctly crib Common Lisp's evaluation strategy for the expressions that give default values to optional function arguments. When you have an argument like foo=obj.whatever() the obj.whatever() is evaluated (would you believe it!) at the time the definition of the function…
While you are correct, that is not what is happening in the blog post. Their issue was inside a class definition, not a function.