An AI Vibe Coding Horror Story

fwiw i know tobias and its very very unlikely he made this up. my guess its intentionally vague to not leak any information about the culprit which i guess is fair.

I think the issue here is less about AI misbehaving and more about people doing things they should not be doing without thinking too hard about the consequences.

There are going to be a lot of accidents like this because it's just really easy to do. And some people are inevitably going to do silly things.

But it's not that different from people doing stupid things with Visual Basic back in the day. Or responding to friendly worded emails with the subject "I love you". Putting CDs/USB drives in work PCs with viruses, worms, etc.

That's what people do when you give the useful tools with sharp edges.

> All "access control" logic lived in the JavaScript on the client side, meaning the data was literally one command away from anyone who looked

This is the top!

This is a typical example of someone using Coding Agents without being a developer: AI that isn't used knowingly can be a huge risk if you don't know what you're doing.

AI used for professional purposes (not experiments) should NOT be used haphazardly.

And this also opens up a serious liability issue: the developer has the perception of being exempt from responsibility and this also leads to enormous risks for the business.

I'm also curious how much effort it would be to setup some OWASP tools with an agent and crawl for company tools. I'm sure I'm not the first one to think of this, but for local businesses it would give a solid rep, I suppose.

I have a feeling that next year's theme will be security. People have turned off their brain when it comes to tech.

We don’t blame companies selling 3D Design software or 3D printers or mortar and cement, or graph paper and pencils. When people abuse those tools and build huts or houses or bridges that fall down, we usually blame the user for not having appropriate professional qualifications, accreditation, and experience. (Very occasionally we blame bugs in simulation software tools).

AI is a tool. It’s not intelligent, and it works at a much bigger scale than bricks and mortar, but it’s still just a tool. There’s lots we can blame AI companies for, but abuse of the tool isn’t a clear-cut situation. We should blame them for misleading marketing. But we should also blame users (who are often highly intelligent - eg doctors) for using it outside their ability. Much like doctors are fed up of patients using AI to try to act like doctors, software engineers are now finding out what it’s like when clients try to use AI to act like software engineers.