A ChatGPT mistake cost us $10k • Hacker News Board

Re: A ChatGPT mistake cost us $10k

#271

Post by sen » Mon, Jun 10, 2024, 3:43 AM UTC

japhyr wrote: post #126

This kind of thing must be happening much more often than we're hearing about it, right? I'd love to start a site that collects AI "horror stories", where trusting an AI's output led to significant consequences. I have no idea how to validate people's anecdotes, though. (To be clear I don't doubt this story at all. But if I set up a site where people could submit stories I wouldn't trust any submissions at face value…

Just like Reddit, you’d have users using AI to generate fake posts to farm karma.

User generated content in general these days is completely poisoned.

Re: A ChatGPT mistake cost us $10k

#272

Post by whoknowsidont » Mon, Jun 10, 2024, 3:45 AM UTC

kfarr wrote: post #151

dalemhurley wrote: post #128

I have seen the same mistake made in code created by humans. Many times, especially in react / typescript/ JavaScript, someone will forget to use a lambda. I felt the blog post failed to articulate the root cause of the issue and went straight to blaming ChatGPT. When you rush and make large or non peer code reviewed commits to main it is going to happen. The real issue was when you rush, take shortcuts and don’t ade…

Yeah ChatGPT is a red herring -- it doesn't matter what generates the code, it's what you do with it.

Surely current events explains why ChatGPT is topical?

Re: A ChatGPT mistake cost us $10k

#273

Post by nomilk » Mon, Jun 10, 2024, 3:47 AM UTC

Part of the skill in using LLMs is knowing when and how to use them, how to set the 'temperature' (how 'creative' it will be in its response), and how to write a prompt that is less prone to illusory responses.

My eyes were opened one relaxing morning when sipping my coffee and pondering how to tidy up a database column by migrating from string to enum.

I asked ChatGPT for its thoughts and its response seemed perfunctory and on point, until at one particular line, tucked in the otherwise sensible migration file [1], it casually recommended deleting all users whose value for that attribute wasn't among those specified by the enum. I spat my coffee out and learned a very valuable lesson that morning!

[1] https://imgur.com/a/ejIdCH6

Re: A ChatGPT mistake cost us $10k

#274

Post by fragmede » Mon, Jun 10, 2024, 3:50 AM UTC

wavemode wrote: post #65

No, a lack of monitoring cost you $10K. Your app was throwing a database exception and nobody was alerted that this was not only happening, but happening continuously and in large volumes. Such an alert would have made this a 5-minute investigation rather than 5 days. If you haven't fixed that alerting deficiency, then you haven't really fixed anything.

Right? The log message would have said the id isn't unique, and then it would have taken much less time to debug this problem.

Programming when everything works is easy, it's handling the problems that makes it hard.

Re: A ChatGPT mistake cost us $10k

#275

Post by viraptor » Mon, Jun 10, 2024, 3:55 AM UTC

brigadier132 wrote: post #187

zeroonetwothree wrote: post #162

Earlier quoted context omitted.

It makes your tests more robust. Generally you don’t want tests that are too sensitive to external state since they will fail spuriously and become useless.

Of course your tests shouldn't be sensitive to external state. Why would other tests running affect your test?

They shouldn't. But they do. We're not perfectly spherical developers and we all make mistakes. Sometimes it's also extremely tricky to figure out what state is leaking, especially is it's an access race issue and happens only for some tests and very rarely. If you haven't seen that happening, you just need to work on larger projects.

Re: A ChatGPT mistake cost us $10k

#276

Post by cuppypastexprt » Mon, Jun 10, 2024, 3:55 AM UTC

> We had eight ECS tasks on AWS, all running five instances of our backend (overkill, yes we know, but to be fair we had AWS credits).

Yes, that's a very fair reasoning. YC did the right thing by investing in this company. Fits very well with the rest of their portfolio.

Re: A ChatGPT mistake cost us $10k

#277

Post by qup » Mon, Jun 10, 2024, 3:55 AM UTC

nomilk wrote: post #273

Part of the skill in using LLMs is knowing when and how to use them, how to set the 'temperature' (how 'creative' it will be in its response), and how to write a prompt that is less prone to illusory responses. My eyes were opened one relaxing morning when sipping my coffee and pondering how to tidy up a database column by migrating from string to enum. I asked ChatGPT for its thoughts and its response seemed perfunc…

That's one way to avoid bugs

Re: A ChatGPT mistake cost us $10k

#278

Post by sneak » Mon, Jun 10, 2024, 3:56 AM UTC

wavemode wrote: post #65

No, a lack of monitoring cost you $10K. Your app was throwing a database exception and nobody was alerted that this was not only happening, but happening continuously and in large volumes. Such an alert would have made this a 5-minute investigation rather than 5 days. If you haven't fixed that alerting deficiency, then you haven't really fixed anything.

TBH, if the backend were written in Go, this probably wouldn’t have happened to the extent it did. Somewhere in a log a descriptive error would have shown up.

One of the reasons I use Go whenever possible is that it removes a lot of the classic Python footguns. If you are going to rewrite your backend from Javascript, why would you rewrite it in another untyped, error-prone language?

Re: A ChatGPT mistake cost us $10k

#279

Post by fragmede » Mon, Jun 10, 2024, 3:56 AM UTC

crabmusket wrote: post #244

sanderjd wrote: post #175

Earlier quoted context omitted.

Yeah but a lot of that is just the accrual of improvements that is possible with a lot of resources over a long period of time. People working in "big tech" aren't fundamentally better at building reliable tools and systems; the time and resource constraints are entirely different.

And the stakes! This outage might have cost the OP $10k. A similar snafu at a larger company might have cost tens of millions or more.

The big tech tooling probably cost tens of millions of dollars to create, and probably had a couple $10k mistakes on the way to getting it written and running.

Re: A ChatGPT mistake cost us $10k

#280

Post by beala » Mon, Jun 10, 2024, 3:56 AM UTC

furyofantares wrote: post #256

Felt like a clickbait headline to me, but there's no link back to the project. So I guess not. Definitely respect for telling an embarrassing story even if I disagree wholly with the title (both the implication that the ChatGPT mistake is to blame, and that it cost them $10k.) Anyway I believe the product in question is https://agentgpt.reworkd.ai

I had the same question. Everyone is talking about how this is bad for the company's reputation... but it wasn't immediately clear to me what the company is.

I also eventually landed on reworkd.ai after some googling. The blog is called "asim" and the OP's username is "asim-shrestha". That lead me to this: https://www.ycombinator.com/companies/reworkd They are S23, which is mention in the blog.