AI didn't delete your database, you did • Hacker News Board

Re: AI didn't delete your database, you did

#101

Post by susam » Tue, May 05, 2026, 3:36 PM UTC

pier25 wrote: post #84

susam wrote: post #49

I recently wrote a blog post where I argued that there are a few principles we should consistently follow when talking about AI: https://susam.net/inverse-laws-of-robotics.html To summarise them: 1. Do not anthropomorphise AI systems. 2. Do not blindly trust the output of AI systems. 3. Retain full human responsibility and accountability for any consequences arising from the use of AI systems. I would like to see the…

> Retain full human responsibility and accountability for any consequences arising from the use of AI systems So if the tool doesn't do what it's supposed to be doing we should blame the user instead of the company that made the tool?

You are quoting a point from my summary and extrapolating what my post might be saying.

Even in that quote, I do not say that the user must be responsible. The point is that responsibility and accountability should remain with some humans. Depending on the case, those humans may be the people who manufactured the tool, the people who deployed it or the people who took bad output from the tool and applied it to the real world.

Did you read the actual section at https://susam.net/inverse-laws-of-robotics.html#non-abdicati...>? It has more nuance than what the summary alone can capture.

Re: AI didn't delete your database, you did

#102

Post by jmalicki » Tue, May 05, 2026, 3:36 PM UTC

lowbloodsugar wrote: post #64

paroneayea wrote: post #36

I think the perspective here is completely wrong. The problem is that people are now building our world around tooling that eschews accountability . Over a decade ago now, I had a conversation with Gerald Sussman which had enormous influence on me: https://dustycloud.org/blog/sussman-on-ai/ > At some point Sussman expressed how he thought AI was on the wrong track. He explained that he thought most AI directions were…

Humans aren’t any better. That’s why we have OSHA etc. I think you’re hoping for a formal logic based AI and I’ll wager no such thing will ever exist - and if it do, it would try to kill us all.

Formal logic AI systems have existed and were popular in the 1980s. One of the problems is that they don't work - in the real world there are no firm facts, everything is squishy, and when you try to build a large system you end up making tons of exceptions for special cases until it becomes completely untenable.

Non-deterministic systems that work probabilistically are just superior in function to that, even if it makes us all deeply uncomfortable.

Re: AI didn't delete your database, you did

#103

Post by PessimalDecimal » Tue, May 05, 2026, 3:37 PM UTC

sam0x17 wrote: post #92

paroneayea wrote: post #36

I think the perspective here is completely wrong. The problem is that people are now building our world around tooling that eschews accountability . Over a decade ago now, I had a conversation with Gerald Sussman which had enormous influence on me: https://dustycloud.org/blog/sussman-on-ai/ > At some point Sussman expressed how he thought AI was on the wrong track. He explained that he thought most AI directions were…

There used to be a lot of research into using deep NNs to train decision trees, which are themselves much less of a black box and can actually be reasoned about. I wonder where that all went?

History is littered with great ideas that lost people's interest and focus. A sad realization is that the focus may never return to them either.

Re: AI didn't delete your database, you did

#104

Post by locknitpicker » Tue, May 05, 2026, 3:38 PM UTC

BadBadJellyBean wrote: post #71

paroneayea wrote: post #36

I think the perspective here is completely wrong. The problem is that people are now building our world around tooling that eschews accountability . Over a decade ago now, I had a conversation with Gerald Sussman which had enormous influence on me: https://dustycloud.org/blog/sussman-on-ai/ > At some point Sussman expressed how he thought AI was on the wrong track. He explained that he thought most AI directions were…

My team and I are firm that we are the ones accountable. LLMs are a tool like every other. Only that it's non deterministic. But I am the one using the tool. I am the one giving the tool access. I am the one who has to keep everything safe. I have shot myself in the foot using gparted in the past by wiping the wrong disk. gparted wasn't to blame. I was. Letting LLMs work freely without supervision sounds great but it…

> My team and I are firm that we are the ones accountable. LLMs are a tool like every other.

Except it is definitely not.

LLMs alone have highly non-deterministic even at a high-level, where they can even pursuit goals contrary to the user's prompts. Then, when introduced in ReAct-type loops and granted capabilities such as the ability to call tools then they are able to modify anything and perform all sorts of unexpected actions.

To make matters worse, nowadays models not only have the ability to call tools but also to generate code on the fly whatever ad-hoc script they want to run, which means that their capabilities are not limited to the software you have installed in your system.

This goes way beyond "regular tool" territory.

Re: AI didn't delete your database, you did

#105

Post by justinhj » Tue, May 05, 2026, 3:39 PM UTC

paroneayea wrote: post #36

I think the perspective here is completely wrong. The problem is that people are now building our world around tooling that eschews accountability . Over a decade ago now, I had a conversation with Gerald Sussman which had enormous influence on me: https://dustycloud.org/blog/sussman-on-ai/ > At some point Sussman expressed how he thought AI was on the wrong track. He explained that he thought most AI directions were…

Very informative post. I think however we are not at the point AI can be taken to court. We know it can hallucinate, we know that context can fill up or obfuscate a rule and cause behaviour we explicitly didn't want.

If you give the AI agency to execute some task, you are still responsible. In the near term we should focus on tooling for auditing and sandboxing, and human in the loop confirmations.

Re: AI didn't delete your database, you did

#106

Post by seethishat » Tue, May 05, 2026, 3:39 PM UTC

This reminds me of a James Micken's quote from "This World of Ours" in response to security people admonishing users for clicking links in email:

    "It’s not clear what else there is to do with computers besides click on things..."

If you have an API with exposed endpoints, it's not clear to the AI bot what else there is to do with the API besides call the endpoints.

Re: AI didn't delete your database, you did

#107

Post by fyrabanks » Tue, May 05, 2026, 3:40 PM UTC

BadBadJellyBean wrote: post #71

paroneayea wrote: post #36

I think the perspective here is completely wrong. The problem is that people are now building our world around tooling that eschews accountability . Over a decade ago now, I had a conversation with Gerald Sussman which had enormous influence on me: https://dustycloud.org/blog/sussman-on-ai/ > At some point Sussman expressed how he thought AI was on the wrong track. He explained that he thought most AI directions were…

My team and I are firm that we are the ones accountable. LLMs are a tool like every other. Only that it's non deterministic. But I am the one using the tool. I am the one giving the tool access. I am the one who has to keep everything safe. I have shot myself in the foot using gparted in the past by wiping the wrong disk. gparted wasn't to blame. I was. Letting LLMs work freely without supervision sounds great but it…

Thank you. Exactly this.

There were so many fundamental problems with the infrastructure even before the person gave a poor prompt to an agent.

If you're using the same API key for staging and prod--and just storing it somewhere randomly to forget about--you're setting yourself up for failure with or without AI.

Re: AI didn't delete your database, you did

#108

Post by nradov » Tue, May 05, 2026, 3:40 PM UTC

abdullahkhalids wrote: post #73

Steve16384 wrote: post #54

Earlier quoted context omitted.

It's taking "computer says no" to the next level. Computers do exactly what they're told, but who told them? The person entering data? The original programmer or designer of the system? The author of whatever language text was used to feed the ai? Even before AI, it was very difficult to determine who is accountable, and now it's even more obfuscated.

This also applies qualitatively to physical devices. It takes some effort to determine if a vehicular accident was caused by a fault in the vehicle or a driver error or environmental causes. Some key inherent differences with older engineering fields is that software can be more complex than physical devices and their functionality can be obfuscated because it is written as text but distributed as binaries. However,…

Software is already subject to enough regulation. The stuff that's actually safety critical like medical devices or avionics is already heavily regulated.

Re: AI didn't delete your database, you did

#109

Post by dpoloncsak » Tue, May 05, 2026, 3:40 PM UTC

locknitpicker wrote: post #104

BadBadJellyBean wrote: post #71

Earlier quoted context omitted.

My team and I are firm that we are the ones accountable. LLMs are a tool like every other. Only that it's non deterministic. But I am the one using the tool. I am the one giving the tool access. I am the one who has to keep everything safe. I have shot myself in the foot using gparted in the past by wiping the wrong disk. gparted wasn't to blame. I was. Letting LLMs work freely without supervision sounds great but it…

> My team and I are firm that we are the ones accountable. LLMs are a tool like every other. Except it is definitely not. LLMs alone have highly non-deterministic even at a high-level, where they can even pursuit goals contrary to the user's prompts. Then, when introduced in ReAct-type loops and granted capabilities such as the ability to call tools then they are able to modify anything and perform all sorts of unexp…

Isn't the next sentence there literally 'Only that it's non deterministic'?

Re: AI didn't delete your database, you did

#110

Post by kennywinker » Tue, May 05, 2026, 3:40 PM UTC

pc86 wrote: post #46

tantalor wrote: post #22

Earlier quoted context omitted.

New rule: Roll a 1 on a D20 -> you accidentally delete your own database

It seems closer to "roll two or three successive 1s on a D100 and have your LLM hooked directly into your production systems and have your LLM user have DELETE permissions" and probably 1 or 2 other things I'm forgetting.

It pulled an api key from an unrelated file. It wasn’t given delete permission, it found it.

Not picking on you specifically, but in general the comments here have me wondering if AI has stolen our basic reading comprehension, or if we were always this bad.

Anyway, take “LLM user had delete permission” off your list and add “deleting the production db also deletes all the backups” to the list.