I believe this is in response to PocketOS. When I read the original post, I was trying to figure out how they even built a workflow that had AI so close to the self-destruct button. This post's explanation about it probably being fully vibe-coded makes sense. How else would the system be so fragile and for the agent to have such far reach? They built a house of cards.
AI didn't delete your database, you did
211–220 of 329 posts
Re: AI didn't delete your database, you did
#212So the question of “why does a public facing api that can delete your database even exit”?
If you worked in cloud environments - every database had public facing api that can delete it.
For the rest or it - yeah, running autonomous pipelines in production which decide what to run and what not to run seems fine until it isn’t.
But every database deployed in a cloud environment has an api that can delete it. Even if you say you’re running on vms - there exist api that can delete the disk, the vm, the network config, etc.
Re: AI didn't delete your database, you did
#213Just skip straight to the Twitter post, it's way better than this secondary article.
We had no idea — and Railway's token-creation flow gave us no warning — that the same token had blanket authority across the entire Railway GraphQL API, including destructive operations like volumeDelete" [...] Railway's volume backups are stored in the same volume.
Re: AI didn't delete your database, you did
#214I recently wrote a blog post where I argued that there are a few principles we should consistently follow when talking about AI: https://susam.net/inverse-laws-of-robotics.html To summarise them: 1. Do not anthropomorphise AI systems. 2. Do not blindly trust the output of AI systems. 3. Retain full human responsibility and accountability for any consequences arising from the use of AI systems. I would like to see the…
> Retain full human responsibility and accountability for any consequences arising from the use of AI systems So if the tool doesn't do what it's supposed to be doing we should blame the user instead of the company that made the tool?
What do you think an LLM is "supposed" to do?
At the end of the day it's just a big weighted graph traversal. Its output is a result of many combined probabilities. It's not deterministic and even if it was the input range is so massive that it would be impossible to comprehensively test.
You cannot possibly know an LLM will do what you command it to. It's impossible by design. LLMs are inherently unpredictable. They can still be useful, but that unpredictability needs to be accounted for to use them safely.
Re: AI didn't delete your database, you did
#215Earlier quoted context omitted.
I agree that LLMs could be more open about their dangers and that people are bad at judging risks sometimes. Still I think a band saw has very little warning on it and by it's design there is very little anyone can do about me cutting off my finger if I am not careful. LLM companies can do very little about the unpredictability of LLMs. So we have to choose how for we will let it go. In the end the LLM only produces…
A band saw is always a screaming band of bladed death. An LLM is sometimes a buddy, sometimes a mentor, and only sometimes a guy that drops your database.
[dead]
Re: AI didn't delete your database, you did
#216This applies to all infra. Why can you delete a network load balancer that is still getting traffic? Why can you delete a VM that is getting non-trivial network traffic? Why can you delete a database that has sessions / requests in the last hour? Why can you drop a table that has queries in the last hour?
Someone will add safeguards for all that stuff and it ends up making it way harder to get real work done. I know in theory all of it can be done well, but in practice it's harder than it might sound. I've seen this at work the most with slow rollouts. They said it was for prod only, then it became applied to staging and dev somehow. They said you can force push in emergencies, but approximately 0 people on any given…
If you automate it all it’s fine. The automation has no problem waiting around for traffic to drain out of something before decommissioning it.
Re: AI didn't delete your database, you did
#217AI companies: here's a knife so you can automate bread cutting!
User: I tried to cut some bread and it cut my finger instead.
AI companies: not my problem!
HN: The AI didn't cut your finger, you did, idiot.
Re: AI didn't delete your database, you did
#218Earlier quoted context omitted.
Someone will add safeguards for all that stuff and it ends up making it way harder to get real work done. I know in theory all of it can be done well, but in practice it's harder than it might sound. I've seen this at work the most with slow rollouts. They said it was for prod only, then it became applied to staging and dev somehow. They said you can force push in emergencies, but approximately 0 people on any given…
If you automate it all it’s fine. The automation has no problem waiting around for traffic to drain out of something before decommissioning it.
Then you're back to square 1. The only way to win here is to require user supervision and make it simple and easy to use.
Re: AI didn't delete your database, you did
#219Earlier quoted context omitted.
> Retain full human responsibility and accountability for any consequences arising from the use of AI systems So if the tool doesn't do what it's supposed to be doing we should blame the user instead of the company that made the tool?
You are quoting a point from my summary and extrapolating what my post might be saying. Even in that quote, I do not say that the user must be responsible. The point is that responsibility and accountability should remain with some humans. Depending on the case, those humans may be the people who manufactured the tool, the people who deployed it or the people who took bad output from the tool and applied it to the re…
> Even in that quote, I do not say that the user must be responsible.
I didn't say that. I made a question so you could elaborate which human you were referring to.
Re: AI didn't delete your database, you did
#220I think the perspective here is completely wrong. The problem is that people are now building our world around tooling that eschews accountability . Over a decade ago now, I had a conversation with Gerald Sussman which had enormous influence on me: https://dustycloud.org/blog/sussman-on-ai/ > At some point Sussman expressed how he thought AI was on the wrong track. He explained that he thought most AI directions were…
The key quote is in the increasingly prescient 1979 IBM training manual: "A computer can never be held accountable, therefore a computer must never make a management decision."
That manual aged much more gracfully than the 1930s "Songs of the IBM," featuring lines like "The name of T.J. Watson means a courage none can stem / And we feel honored to be here to toast the I.B.M.," and of course classic American standards like "To G.H. Armstrong, Sales Manager, ITR and IS Divisions."