Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI didn't delete your database, you did

idiallo.com

Post reply on HN
Print view View original

81–90 of 329 posts

Re: AI didn't delete your database, you did

#81

Post by madeofpalk »

paroneayea wrote: post #36

I think the perspective here is completely wrong. The problem is that people are now building our world around tooling that eschews accountability . Over a decade ago now, I had a conversation with Gerald Sussman which had enormous influence on me: https://dustycloud.org/blog/sussman-on-ai/ > At some point Sussman expressed how he thought AI was on the wrong track. He explained that he thought most AI directions were…

People are eschewing their own accountability, blaming the tools instead for their poor decision making and lack of access controls.

Why is it possible for you to fat-finger your way to deleting production database locally?

Re: AI didn't delete your database, you did

#82

Post by voncheese »

2ndorderthought wrote: post #40
paroneayea wrote: post #36

I think the perspective here is completely wrong. The problem is that people are now building our world around tooling that eschews accountability . Over a decade ago now, I had a conversation with Gerald Sussman which had enormous influence on me: https://dustycloud.org/blog/sussman-on-ai/ > At some point Sussman expressed how he thought AI was on the wrong track. He explained that he thought most AI directions were…

Accountability is the prevailing missing ingredient in us society.

To expand on this a little more, the absence of accountability contributes to the loss of learning. Mistakes and errors will always happen, whether they are sourced by humans or machines. But something (the human or the machine) has to be able to take accountability to have the opportunity to learn and improve so the chances of the same mistake happening again go down.

Since machines don't yet have the ability to take accountability, it falls on the human to do that. And organizations must enable / enforce this so they too can learn and improve.

Without that, there's a lot of dependency being pushed on the machine to (cross fingers) not make the same mistake again.

Re: AI didn't delete your database, you did

#84

Post by pier25 »

susam wrote: post #49

I recently wrote a blog post where I argued that there are a few principles we should consistently follow when talking about AI: https://susam.net/inverse-laws-of-robotics.html To summarise them: 1. Do not anthropomorphise AI systems. 2. Do not blindly trust the output of AI systems. 3. Retain full human responsibility and accountability for any consequences arising from the use of AI systems. I would like to see the…

> Retain full human responsibility and accountability for any consequences arising from the use of AI systems

So if the tool doesn't do what it's supposed to be doing we should blame the user instead of the company that made the tool?

Re: AI didn't delete your database, you did

#85

Post by philipallstar »

paroneayea wrote: post #36

I think the perspective here is completely wrong. The problem is that people are now building our world around tooling that eschews accountability . Over a decade ago now, I had a conversation with Gerald Sussman which had enormous influence on me: https://dustycloud.org/blog/sussman-on-ai/ > At some point Sussman expressed how he thought AI was on the wrong track. He explained that he thought most AI directions were…

> The problem is that people are now building our world around tooling that eschews accountability.

If you tell Terraform the wrong thing it will remove your database and not be accountable either.

Re: AI didn't delete your database, you did

#86

Post by wongarsu »

figmert wrote: post #41
oersted wrote: post #30

Some details from the original post for context: They had a Railway token in an unrelated file (unclear if it was a local secret) for managing custom domains. It turns out that token has full admin access to Railway. The AI deleted a single relevant volume by id. The author is rather vague about what exactly it asked it to do, he just says there was a “credentials mismatch” and Claude took the initiative to fix it by…

> The author is rather vague about what exactly it asked it to do, he just says there was a “credentials mismatch” and Claude took the initiative to fix it by deleting the volume. But it’s likely that they are somewhat downplaying their culpability by being vague. I was just talking to my girlfriend saying I've realised that I've not written a single line of code, nor have I debugged myself for at least the past 3 mo…

I have had Claude go "oh, this query fails because the field I just added isn't in your sqlite database file, let me just delete it so it gets recreated". So I wouldn't rule out that Claude tries deleting a volume if it believes that will fix things and believes it isn't a production system.

That said, Claude seems to have gotten a lot more careful about these kinds of things in the last couple months

Re: AI didn't delete your database, you did

#87

Post by lokar »

This applies to all infra.

Why can you delete a network load balancer that is still getting traffic?

Why can you delete a VM that is getting non-trivial network traffic?

Why can you delete a database that has sessions / requests in the last hour?

Why can you drop a table that has queries in the last hour?

Re: AI didn't delete your database, you did

#89

Post by lokar »

ebiester wrote: post #77

First, no matter what you do, if a human has write access to the production database, the database can be deleted. Second, there is a legitimate reason to destroy a database in development and automation. The biggest problem I see is often treating your development data like pets not cattle. You absolutely need to have safeguards that this cannot be run in production, but if a human has access to the credentials to r…

You don't need the same permissions in prod and dev.

And in both cases, the humans don't need direct access to the raw CSP API. Use a local proxy that adds more safety checks. In dev, sure, delete away.

In prod, check a bunch of things first (like, has it been used recently?). Humans do not need direct access to delete production resources (you can have a break-glass setup for exceptional emergencies).

Re: AI didn't delete your database, you did

#90

Post by bilbo0s »

onlyrealcuzzo wrote: post #66
2ndorderthought wrote: post #40

Earlier quoted context omitted.

Accountability is the prevailing missing ingredient in us society.

> The problem is that people are now building our world around tooling that eschews accountability. Management has doing a wonderful job of eschewing accountability for decades. It's a lot of people's dream to be able to say, yeah, our product doesn't work, but it's not OUR fault, and the client just shrug and grumble ai ai ai, and just put up with it because they know they can't get a better service anywhere else. I…

Well just to be clear from a legal perspective, in the case of AI, as long as AI is "property", the owners, developers, and/or users will be held liable for things like the hypothetical fatal car accident that Sussman posits.

Currently, from a legal perspective, AI is considered a "tool" without legal persona. So you sue the developer, the owner, or the user of the AI. (Just kidding, any lawyer worth his/her salt will sue all three! But you get the point.)

Legally speaking, AI will probably be viewed that way for a long time. There are too many issues agitating against viewing it any other way. Owners will not give up property rights. No will to overbear. On and on and on.

Post reply on HN