Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI didn't delete your database, you did

idiallo.com

Post reply on HN
Print view View original

121–130 of 329 posts

Re: AI didn't delete your database, you did

#121

Post by locknitpicker »

BadBadJellyBean wrote: post #116
locknitpicker wrote: post #104

Earlier quoted context omitted.

> My team and I are firm that we are the ones accountable. LLMs are a tool like every other. Except it is definitely not. LLMs alone have highly non-deterministic even at a high-level, where they can even pursuit goals contrary to the user's prompts. Then, when introduced in ReAct-type loops and granted capabilities such as the ability to call tools then they are able to modify anything and perform all sorts of unexp…

Then that is also on me for using a tool that I can't control. I don't run my LLMs in a way where they can just do things without me signing off on it. It's not nearly as fast as just letting it do it's thing but I kept it from doing stupid things so many times. Giving up control is a decision. The consequences of this decision are mine to carry. I can do my best to keep autonomous LLMs contained and safe but if I am…

> Then that is also on me for using a tool that I can't control.

That's a core trait of LLMs.

Even the AI companies developing frontier models felt the need to put together whole test suites purposely designed to evaluate a model's propensity to try to subvert the user's intentions.

https://www.anthropic.com/research/shade-arena-sabotage-moni...

> Giving up control is a decision.

No, it is definitely not. Only recently did frontier models started to resort to generating ad-hoc scripts as makeshift tools. They even generate scripts to apply changes to source files.

Re: AI didn't delete your database, you did

#122

Post by mayneack »

pjc50 wrote: post #119
paroneayea wrote: post #36

I think the perspective here is completely wrong. The problem is that people are now building our world around tooling that eschews accountability . Over a decade ago now, I had a conversation with Gerald Sussman which had enormous influence on me: https://dustycloud.org/blog/sussman-on-ai/ > At some point Sussman expressed how he thought AI was on the wrong track. He explained that he thought most AI directions were…

Have I got a book for you: https://en.wikipedia.org/wiki/The_Unaccountability_Machine Not actually about technology at all, but about organizational structure.

I think the "black box" framing that it uses neatly applies the same theory to organizations and ais. It doesn't matter whether there's technological or organizational reasons inside the black box to dodge accountability, the outcome is the same.

Re: AI didn't delete your database, you did

#124

Post by pjc50 »

BadBadJellyBean wrote: post #71
paroneayea wrote: post #36

I think the perspective here is completely wrong. The problem is that people are now building our world around tooling that eschews accountability . Over a decade ago now, I had a conversation with Gerald Sussman which had enormous influence on me: https://dustycloud.org/blog/sussman-on-ai/ > At some point Sussman expressed how he thought AI was on the wrong track. He explained that he thought most AI directions were…

My team and I are firm that we are the ones accountable. LLMs are a tool like every other. Only that it's non deterministic. But I am the one using the tool. I am the one giving the tool access. I am the one who has to keep everything safe. I have shot myself in the foot using gparted in the past by wiping the wrong disk. gparted wasn't to blame. I was. Letting LLMs work freely without supervision sounds great but it…

This is kind of the reverse of https://en.wikipedia.org/wiki/Poka-yoke . A lot of tools have affordances built in to make "right" things easy and "wrong" or unsafe things harder. LLMs .. well, the text interface is uniquely flat. Everything is seemingly as easy as everything else.

I worry about the use of humans as sacrificial accountability sinks. The "self-driving car" model already has this: a car which drives itself most of the time, but where a human user is required to be constantly alert so that the AI can transfer responsibility a few hundred miliseconds before the crash.

Re: AI didn't delete your database, you did

#125

Post by criddell »

madeofpalk wrote: post #81
paroneayea wrote: post #36

I think the perspective here is completely wrong. The problem is that people are now building our world around tooling that eschews accountability . Over a decade ago now, I had a conversation with Gerald Sussman which had enormous influence on me: https://dustycloud.org/blog/sussman-on-ai/ > At some point Sussman expressed how he thought AI was on the wrong track. He explained that he thought most AI directions were…

People are eschewing their own accountability, blaming the tools instead for their poor decision making and lack of access controls. Why is it possible for you to fat-finger your way to deleting production database locally?

[dead]

Re: AI didn't delete your database, you did

#126

Post by locknitpicker »

stingraycharles wrote: post #35
edot wrote: post #20

This is why you don’t hire interns! They can delete things and cause havoc! The same people who would blame AI for their failing to properly configure permissions would also blame interns for deleting production whatever. Blame should go up, praise should go down. People always invert these.

> This is why you don’t hire interns! I’d like to rephrase this as: this is why you don’t give interns permissions to delete your prod database. This is a process failure, not an AI failure. I honestly don’t understand why people blame AI here, when you literally gave AI permissions to do exactly this. It’s like blaming AWS for exposing some database to the public. That’s just not AWS’ fault. Neither is this the faul…

> I’d like to rephrase this as: this is why you don’t give interns permissions to delete your prod database.

Nowadays AI code assistants are designed to execute their tools in your personal terminals using your personal credentials with access to all your personal data. See how every single AI integration extension for any IDE works.

You cannot shift blame if by design it is using your credentials for everything it does.

Re: AI didn't delete your database, you did

#127

Post by keerthiko »

locknitpicker wrote: post #104
BadBadJellyBean wrote: post #71

Earlier quoted context omitted.

My team and I are firm that we are the ones accountable. LLMs are a tool like every other. Only that it's non deterministic. But I am the one using the tool. I am the one giving the tool access. I am the one who has to keep everything safe. I have shot myself in the foot using gparted in the past by wiping the wrong disk. gparted wasn't to blame. I was. Letting LLMs work freely without supervision sounds great but it…

> My team and I are firm that we are the ones accountable. LLMs are a tool like every other. Except it is definitely not. LLMs alone have highly non-deterministic even at a high-level, where they can even pursuit goals contrary to the user's prompts. Then, when introduced in ReAct-type loops and granted capabilities such as the ability to call tools then they are able to modify anything and perform all sorts of unexp…

I think you are misinterpreting gp as saying

"LLMs are a tool [like every other tool]" to mean "LLMs have similar properties to other tools" — when I believe they meant "LLMs are a tool. other tools are also tools," where the operative implication of "tool" is not about scope of capabilities or how deterministic its output is (these aren't defining properties of the concept of "tool"), but the relationship between 'tool' and 'operator':

- a tool is activated with operator intent (at some point in the call-chain)

- the operator is accountable for the outcomes of activating the tool, intended or otherwise

The capabilities and the abilities of a tool to call sub-tools is only relevant insofar as expressing how much larger the scope of damage and surface area of accountability is with a new generation of tools. This is not that different than past technological leaps.

When a US bomber dropped a nuke in Hiroshima, the accountability goes up the chain to the war-time president giving the authorization to the military and air force to execute the mission — the scope of accountability of a single decision was way larger than supreme commanders had in prior wars. If the US government decides to deploy an LLM to decide who receives and who is denied healthcare coverage, social security payments, voting rights, or anything else, the head of internal affairs to authorize the use of that tool should be held accountable, non-determinism of the tool be damned.

Re: AI didn't delete your database, you did

#128

Post by paroneayea »

sigbottle wrote: post #91
paroneayea wrote: post #36

I think the perspective here is completely wrong. The problem is that people are now building our world around tooling that eschews accountability . Over a decade ago now, I had a conversation with Gerald Sussman which had enormous influence on me: https://dustycloud.org/blog/sussman-on-ai/ > At some point Sussman expressed how he thought AI was on the wrong track. He explained that he thought most AI directions were…

About the blog you linked and not your comment: Doesn't symbolic AI have a lot of philosophical problems? Think back to Quine's two dogmas - you can't just say, "Let's understand the true meanings of these words and understand the proper mappings". There is no such thing as fixed meaning. I don't see how you get around that. Deep learning is admittedly an ugly solution, but it works better than symbolic AI at least.

Yes! But it's still valuable. How am I understanding your argument at all?

I think my friend Jonathan Rees put it best:

  "Language is a continuous reverse engineering effort, where both sides are trying to figure out what the other side means."
More on that: https://dustycloud.org/blog/identity-is-a-katamari/

This reverse engineering effort is important between you and me, in this exchange right here. It is a battle that can never be won, but the fight of it is how we make progress in most things.

Re: AI didn't delete your database, you did

#129

Post by BadBadJellyBean »

locknitpicker wrote: post #121
BadBadJellyBean wrote: post #116

Earlier quoted context omitted.

Then that is also on me for using a tool that I can't control. I don't run my LLMs in a way where they can just do things without me signing off on it. It's not nearly as fast as just letting it do it's thing but I kept it from doing stupid things so many times. Giving up control is a decision. The consequences of this decision are mine to carry. I can do my best to keep autonomous LLMs contained and safe but if I am…

> Then that is also on me for using a tool that I can't control. That's a core trait of LLMs. Even the AI companies developing frontier models felt the need to put together whole test suites purposely designed to evaluate a model's propensity to try to subvert the user's intentions. https://www.anthropic.com/research/shade-arena-sabotage-moni... > Giving up control is a decision. No, it is definitely not. Only recent…

You seem to misunderstand me. An LLM can only spit out text. It is the tooling I use that allows it to write scripts and call them. In my tooling it waits for me to accept changes, call scripts or other tools that might change something. I can make that deterministic. I know that it will stop and ask because it has no choice. If I want to be safer I give it no tools at all.

I can also just choose not to use an LLM. It is my choice to use them so it is my duty to keep myself safe. If I can't control that I'd be stupid to use them.

My take is that I probably can use LLMs safely when I don't let it run autonomously. There is a slight chance that the LLM will generate a string that will cause a bug in an MCP that will let the LLM do what it wants. That is the risk I am going to take and I will take the blame if it goes wrong.

Re: AI didn't delete your database, you did

#130

Post by patcon »

thuuuomas wrote: post #117
patcon wrote: post #114

Earlier quoted context omitted.

I wish you could have what you want, but I worry you won't get this, because life doesn't give you that, and these systems are tending away from machine precision, and more toward life-like trade-offs. I am almost certain that even if you did get what you want, something that isn't what you want will run circles around you and eat your lunch EDIT: I suspect this will be an unpopular take on Hacker News. And so I am s…

> I think you won’t get [cathedral],.. > even if you do get [cathedral], [bazar] will run circles around you…

Ahhh I like that

It's nested and recursive cathedrals and bazaars, all the way down. And perhaps the bazaar has finally arrived inside the favourite cathedral of most everyone here

EDIT: out of curiosity, does anyone have any good examples of biomes/ecosystems that are so far toward cathedrals? Or is that a uniquely human invention/extreme at the ecosystem scale?

Post reply on HN