Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An AI Vibe Coding Horror Story

tobru.ch

Post reply on HN
Print view View original

171–180 of 224 posts

Re: An AI Vibe Coding Horror Story

#171

Post by hamasho »

The worst blunder I made was when I explored cloud resources to improve the product's performance.

I created a GCP project (my-app-dev) for exploring how to scale up the cloud service. I added several resources to mock the production, like compute instances/cloud SQL/etc, then populated the data and run several benchmarks.

I changed the specs, number of instances and replicas, and configs through gcloud command.

  $ gcloud compute instances stop instance-1 --project=my-app-dev
  $ gcloud compute instances set-machine-type instance-1 --machine-type=c3-highcpu-176 --project=my-app-dev
  $ gcloud sql instances patch db-1 --tier=db-custom-32-131072 --project=my-app-dev
But for some reason, at one point codex asked to list all projects; I couldn't understand the reason, but it seemed harmless so I approved the command.

  $ gcloud projects list
  PROJECT_ID     NAME       PROJECT_NUMBER
  my-app-test    my app     123456789012
  my-app-dev     my app     234567890123
And after this, for whatever reason it changed the target project from the dev (my-app-dev) to the production (my-app) without asking or me realizing.

Of course I checked every commands. I couldn't YOLO while working on cloud resources, even in dev environment. But I focused on the subommands and its content and didn't even think it had changed the project ID along the way.

It continued to suggest more and more aggressive commands for testing, and I approved them brain-deadly...

  $ gcloud sql instances patch db-1 --database-flags=max_connections=500 --project=my-app
  $ gcloud compute instances delete instance-1 --project=my-app
  $ echo 'DELETE FROM users WHERE username="test";' \
      | gcloud sql connect my-db 
   --user=user --database=my-db --project=my-app
  $ wrk -t4 -c200 -d30s \
      "http://$(gcloud compute instances describe instance-1 \
      --project=my-app \
      --format='get(networkInterfaces[0].accessConfigs[0].natIP)')"
It took a shamefully long time to realize codex was actually operating on production, so I DDoSed and SQL-injected to the production...

Fortunately, it didn't do anything irreversible. But it was one of the most terrifying moments in my career.

Re: An AI Vibe Coding Horror Story

#172

Post by embedding-shape »

randomeel wrote: post #156
embedding-shape wrote: post #136

Earlier quoted context omitted.

At one point I worked as a customer support agent outsourced to Apple via the company. Apple forced us to us some very outdated browser UIs, basically for filling in forms, across maybe 4-5 different services in some cases. The machines we were given by this outsourcing company of course where Apple computers, fairly locked down. But one thing they hadn't locked doll wn, was installing extensions in Safari, and given…

Hope you are doing better now

Oh yeah, night and day :) Pretty much the best that could have happen to me, in retrospect.

Re: An AI Vibe Coding Horror Story

#173

Post by BitsAndObjects »

kjksf wrote: post #166
BitsAndObjects wrote: post #62

Earlier quoted context omitted.

One of the key functions of a professional body is to ensure all members are aware of existing and new laws, standards and codes of practice. And to ensure different grades of engineer are aware of different levels of the standards. And that sector-specific laws and standards are accredited accordingly. High profile convictions are not a good way of dealing with this. Not in the short or long term. Sure they have an…

Nothing would be more effective at killing open source and commercial software business that requiring everyone that writes and ships software to users, directly or indirectly (e.g. an open-source library) to have License To Program from Software Licensing Organization. > aware of existing and new laws, standards and codes of practice Yeah, because software business is not at all ruled by fads. 1997: you have to foll…

What complete nonsense. Professional bodies don't mandate fads. Get a grip.

Re: An AI Vibe Coding Horror Story

#174

Post by keysersoze33 »

The takeaway is to vet new companies one is dealing with - even just calling them up and asking if they've AI generated any system which deals with customer/patient data.

This is going to get more common (state sponsored hackers are going to have a field day)

Re: An AI Vibe Coding Horror Story

#175

Post by BitsAndObjects »

consumer451 wrote: post #116
freakynit wrote: post #57

I think vibe-coding is cool, but it runs into limits pretty fast (at least right now). It kinda falls apart once you get past a few thousand lines of code... and real systems aren't just big, they're actually messy...shit loads of components, services, edge cases, things breaking in weird ways. Getting all of that to work together reliably is a different game altogether. And you still need solid software engineering…

There are all kinds of memory hacks, tools that index your code, etc. The thing I have found that makes things work much better is, wait for it... Jira. Everyone loves to hate on Jira, but it is a mature platform for managing large projects. First, I use the Jira Rovo MCP (or cli, I don't wanna argue about that) to have Claude Code plan and document my architecture, features, etc. I then manually review and edit all…

Doesn't require Jira but yes, specification-first is the way to get better (albeit still not reliably good) results out of AI tools. Some people may call this "design-first" or "architecture-first". The point is really to think through what is being built before asking AI to write the implementation (i.e. code), and to review the code to make sure it matches the intended design.

Most people run into problems (with or without AI) when they write code without knowing what they're trying to create. Sometimes that's useful and fun and even necessary, to explore a problem space or toy with ideas. But eventually you have to settle on a design and implement it - or just end up with an unmaintainable mess of code (whether it's pure-human or AI-assisted mess doesn't matter lol).

Re: An AI Vibe Coding Horror Story

#176

Post by bootsmann »

anal_reactor wrote: post #93
aledevv wrote: post #36

> All "access control" logic lived in the JavaScript on the client side, meaning the data was literally one command away from anyone who looked This is the top! This is a typical example of someone using Coding Agents without being a developer: AI that isn't used knowingly can be a huge risk if you don't know what you're doing. AI used for professional purposes (not experiments) should NOT be used haphazardly. And th…

The problem isn't AI, the problem is lack of an intelligent person somewhere in this whole situation. Way before AI I've seen a medical company create a service where frontend would tell backend what SQL queries to execute.

“You’re just holding it wrong”

Re: An AI Vibe Coding Horror Story

#177

Post by Ekaros »

shivaniShimpi_ wrote: post #125

Every other field that's figured out high stakes failure models eventually landed on the same solution - make sure two people that understand the details are looking at it - pilots have copilots surgeons with checklists and nuclear plants have independent verification. Software was always the exception, cause when it broke it mostly just broke for you, vibe coding is not going to change the equation, it barely remove…

We do have code reviews for pull requests. But on average I would guess there is great amount of complacency there. I suppose old proper QA phase was best answer we had. But that is expensive and slow.

Re: An AI Vibe Coding Horror Story

#178

Post by BitsAndObjects »

hamasho wrote: post #171

The worst blunder I made was when I explored cloud resources to improve the product's performance. I created a GCP project (my-app-dev) for exploring how to scale up the cloud service. I added several resources to mock the production, like compute instances/cloud SQL/etc, then populated the data and run several benchmarks. I changed the specs, number of instances and replicas, and configs through gcloud command. $ gc…

This is part of the reason deployments to production cloud environments should:

1. Only be allowed via CI/CD

2. All infra should be defined as code

3. Any deployment to production should be a delayed process that also has a human-approval step in the workflow (at least one, if not more)

(Exactly where that review step is placed depends on your organisation - culture, size, etc.)

And anyone that does need to touch production should do so from an isolated VM with temporary credentials. Developers shouldn't routinely have production access from their terminal. This last aspect is easy and cheap to set up on AWS. I presume it's also possible in Google Cloud.

Re: An AI Vibe Coding Horror Story

#179

Post by rsynnott »

bjoli wrote: post #84
fainpul wrote: post #55

Earlier quoted context omitted.

> AEPD […] known to be brutal. Nice. I wish more countries had something like that. Many of these organizations are lethargic and have to be forced into action by civilian efforts or the press.

AEPD are well known, even in the rest of the world. They have a different strategy compared to other countries. Ireland's DPC are also heavy handed, but focus on large companies mostly. France's CNIL is also not bad. They are particularly hard against things like "you accidentally sign up for x y z services when only wanting to sign up to service A". Gdpr in the EU is also miles ahead of what the US has, or at least…

> Ireland's DPC are also heavy handed, but focus on large companies mostly.

Also, generally, very, very, VERY slow. The massive fines you hear about are usually for behaviour _years_ ago.

Re: An AI Vibe Coding Horror Story

#180

Post by voidUpdate »

trick-or-treat wrote: post #75
voidUpdate wrote: post #23

Earlier quoted context omitted.

It is physically possible for a consultant to write bad code. But you'd hope that a consultant could understand that medical data is extremely important to keep secure, and actually write it to have some level of security

Sure, but you'd hope that the LLM could understand that too.

And yet it seems it didn't
Post reply on HN