Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI didn't delete your database, you did

idiallo.com

Post reply on HN
Print view View original

181–190 of 329 posts

Re: AI didn't delete your database, you did

#181

Post by skydhash »

BadBadJellyBean wrote: post #145
pjc50 wrote: post #124

Earlier quoted context omitted.

This is kind of the reverse of https://en.wikipedia.org/wiki/Poka-yoke . A lot of tools have affordances built in to make "right" things easy and "wrong" or unsafe things harder. LLMs .. well, the text interface is uniquely flat. Everything is seemingly as easy as everything else. I worry about the use of humans as sacrificial accountability sinks. The "self-driving car" model already has this: a car which drives its…

I agree that LLMs could be more open about their dangers and that people are bad at judging risks sometimes. Still I think a band saw has very little warning on it and by it's design there is very little anyone can do about me cutting off my finger if I am not careful. LLM companies can do very little about the unpredictability of LLMs. So we have to choose how for we will let it go. In the end the LLM only produces…

It’s possible to make AI safe, but that also throws most of the gains out of the windows, especially if the artifact is a diff which can take time to review. In IT, you often have to give access to possible malicious users, you just have to scope what they can do.

But the push is agentic everything, where AI needs to be everywhere, not in its own sandbox.

Re: AI didn't delete your database, you did

#182

Post by Barrin92 »

kenjackson wrote: post #170
Barrin92 wrote: post #141

Earlier quoted context omitted.

>The fallacy here is the assumption that humans know why we do what we do. Much like modern LLMs we have an explanation Human brains and cognition do not work like LLMs, but that aside that's irrelevant. Existing machines can explain what they did, that's why we built them. As Dijkstra points out in his essay on 'the foolishness of natural language programming', the entire point of programming is: ( https://www.cs.ut…

That is absurd as a suggestion of it being the entire point of programming. In fact, it goes back to my original point - I have no idea why Djikstrs would say something so non-sensical, and likely neither did he.

what do you mean "likely neither did he", I literally linked you the piece in which he said it. And of course he of all people would make that (correct) point, because he was always the strongest advocate of the virtue of formal correctness of programming languages, again from his article:

"A short look at the history of mathematics shows how justified this challenge is. Greek mathematics got stuck because it remained a verbal, pictorial activity, Moslem "algebra", after a timid attempt at symbolism, died when it returned to the rhetoric style, and the modern civilized world could only emerge —for better or for worse— when Western Europe could free itself from the fetters of medieval scholasticism —a vain attempt at verbal precision!— thanks to the carefully, or at least consciously designed formal symbolisms that we owe to people like Vieta, Descartes, Leibniz, and (later) Boole."

LLMs are nothing else but the exact reversal of this. To go from the system of computation that Boole gave you to treating your computer like a genie you perform incantations on, it's literally sending you back to the medieval age.

Re: AI didn't delete your database, you did

#183

Post by pc86 »

Nicook wrote: post #70
pc86 wrote: post #25

Earlier quoted context omitted.

There really shouldn't be any "serious and unfixable way" to break things, especially in a modern company that uses technology in any meaningful way. The fact it's even possible to get into an unrecoverable state is the primary issue.

That's literally always possible? The idea is to put up walls and fail-safes to minimize the chance.

You should always be able to either undo what you did or rebuild to identical infrastructure and nearly-identical data.

Re: AI didn't delete your database, you did

#184

Post by engeljohnb »

locknitpicker wrote: post #144
keerthiko wrote: post #127

Earlier quoted context omitted.

I think you are misinterpreting gp as saying "LLMs are a tool [like every other tool]" to mean "LLMs have similar properties to other tools" — when I believe they meant "LLMs are a tool. other tools are also tools," where the operative implication of "tool" is not about scope of capabilities or how deterministic its output is (these aren't defining properties of the concept of "tool"), but the relationship between 't…

> - a tool is activated with operator intent (at some point in the call-chain) This again is where the simplistic assumption breaks down. Just because you can claim that a person kick started something, that does not mean that person is aware and responsible for all its doing. Let's put things in perspective: if you install a mobile app from the app store, are you responsible and accountable for every single thing th…

>Just because you can claim that a person kick started something

Kick started what? If you decided to give an LLM access to your database, it's completely on you when you when it does something you don't want. You should've known better.

If all you "kickstart" is an LLM generating text that you can use however you decide, there will never be anything to worry about from the LLM.

> Let's put things in perspective: if you install a mobile app from the app store, are you responsible and accountable for every single thing the app does in your system?

Yes, and it bothers me that others don't feel the same. You vetted the app, you installed the app, and you gave it permission to do whatever on your system. Of course you're responsible.

Re: AI didn't delete your database, you did

#185

Post by tejohnso »

danbruc wrote: post #172
paroneayea wrote: post #36

I think the perspective here is completely wrong. The problem is that people are now building our world around tooling that eschews accountability . Over a decade ago now, I had a conversation with Gerald Sussman which had enormous influence on me: https://dustycloud.org/blog/sussman-on-ai/ > At some point Sussman expressed how he thought AI was on the wrong track. He explained that he thought most AI directions were…

If an AI driven car drives off the side of the road, I want to know why it did that. I could take the software developer to court, but I would much rather take the AI to court. How would that work? You have the AI explain its reasoning - and trust that this is accurate - and then you decide whether that is acceptable behavior. If not, you ban the AI from driving because it will deterministically or at least statistic…

That'd be great for the corporations. Take the AI to court, not us. The AI the gets punished (whatever that means...let's say banned) and the corporation continues without accountability. They could then create another AI and do the same thing all over again.

Or maybe the accountability flows upward from the AI to the corp that created it? Sounds nice, but we know that accountability doesn't work that way in practice.

I think I'd rather have the corporation primarily accountable in the first place rather than have the AI take the bulk of the blame and then hope the consequences fall into place appropriately.

Re: AI didn't delete your database, you did

#186

Post by engeljohnb »

edot wrote: post #20

This is why you don’t hire interns! They can delete things and cause havoc! The same people who would blame AI for their failing to properly configure permissions would also blame interns for deleting production whatever. Blame should go up, praise should go down. People always invert these.

Interns are human. Humans can always be held accountable. A computer never can. Therefore, no one should leave a computer in charge of human decisions.

Re: AI didn't delete your database, you did

#187

Post by traderj0e »

amluto wrote: post #51
stingraycharles wrote: post #35

Earlier quoted context omitted.

> This is why you don’t hire interns! I’d like to rephrase this as: this is why you don’t give interns permissions to delete your prod database. This is a process failure, not an AI failure. I honestly don’t understand why people blame AI here, when you literally gave AI permissions to do exactly this. It’s like blaming AWS for exposing some database to the public. That’s just not AWS’ fault. Neither is this the faul…

There is a major issue with current AI tools that they want to effectively grant access to everything their user has access to. The whole sandbox structure is wrong (although various people have vibe coded assorted improvements).

If you pretend you have an intern with their own machine and run the AI agents on that machine, you have the same separation.

Re: AI didn't delete your database, you did

#188

Post by traderj0e »

lokar wrote: post #87

This applies to all infra. Why can you delete a network load balancer that is still getting traffic? Why can you delete a VM that is getting non-trivial network traffic? Why can you delete a database that has sessions / requests in the last hour? Why can you drop a table that has queries in the last hour?

Someone will add safeguards for all that stuff and it ends up making it way harder to get real work done. I know in theory all of it can be done well, but in practice it's harder than it might sound.

I've seen this at work the most with slow rollouts. They said it was for prod only, then it became applied to staging and dev somehow. They said you can force push in emergencies, but approximately 0 people on any given team know how to do this reliably, and it still takes way longer even in --force --now --breakglass --yesimeanit mode. So the end result is longer MTTR. It maybe prevents some kinds of outages, but also you're less likely to manually monitor a rollout when it takes longer.

Re: AI didn't delete your database, you did

#189

Post by keerthiko »

locknitpicker wrote: post #144
keerthiko wrote: post #127

Earlier quoted context omitted.

I think you are misinterpreting gp as saying "LLMs are a tool [like every other tool]" to mean "LLMs have similar properties to other tools" — when I believe they meant "LLMs are a tool. other tools are also tools," where the operative implication of "tool" is not about scope of capabilities or how deterministic its output is (these aren't defining properties of the concept of "tool"), but the relationship between 't…

> - a tool is activated with operator intent (at some point in the call-chain) This again is where the simplistic assumption breaks down. Just because you can claim that a person kick started something, that does not mean that person is aware and responsible for all its doing. Let's put things in perspective: if you install a mobile app from the app store, are you responsible and accountable for every single thing th…

There can be more than one person or entity to be held accountable, depending on the details of impact

If I install a powerful/dangerous app, and I come under harm, I have some accountability — most of it if it's due to user error (eg: I install termux and `rm -rf /`).

If it's malware, and Google/Apple approved said app to their store which is where I got it from, when their whole value proposition for walled-garden storefronts is protecting users, then they have significant accountability.

If the app requests more permissions than necessary for stated goals, and/or intentionally harms users via misrepresentation or misdirection (malware), the app publisher should also be held accountable (by the storefront, legally, etc).

I'm also unclear what angle you are arguing: are you stating that because tools have gotten so complicated that the end user may not understand how it all works, no one should be considered responsible or held accountable? Or that the tool (currently a non-entity) itself should be held accountable somehow? Or that no one other than the distributor of the tool should be accountable?*

Re: AI didn't delete your database, you did

#190

Post by chrisweekly »

pjc50 wrote: post #124
BadBadJellyBean wrote: post #71

Earlier quoted context omitted.

My team and I are firm that we are the ones accountable. LLMs are a tool like every other. Only that it's non deterministic. But I am the one using the tool. I am the one giving the tool access. I am the one who has to keep everything safe. I have shot myself in the foot using gparted in the past by wiping the wrong disk. gparted wasn't to blame. I was. Letting LLMs work freely without supervision sounds great but it…

This is kind of the reverse of https://en.wikipedia.org/wiki/Poka-yoke . A lot of tools have affordances built in to make "right" things easy and "wrong" or unsafe things harder. LLMs .. well, the text interface is uniquely flat. Everything is seemingly as easy as everything else. I worry about the use of humans as sacrificial accountability sinks. The "self-driving car" model already has this: a car which drives its…

^ which approach makes no logical sense; an inattentive or even partly-attentive driver simply cannot resume control and react accordingly within even 2 seconds.
Post reply on HN