Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI didn't delete your database, you did

idiallo.com

Post reply on HN
Print view View original

171–180 of 329 posts

Re: AI didn't delete your database, you did

#171

Post by yonatan8070 »

amluto wrote: post #51
stingraycharles wrote: post #35

Earlier quoted context omitted.

> This is why you don’t hire interns! I’d like to rephrase this as: this is why you don’t give interns permissions to delete your prod database. This is a process failure, not an AI failure. I honestly don’t understand why people blame AI here, when you literally gave AI permissions to do exactly this. It’s like blaming AWS for exposing some database to the public. That’s just not AWS’ fault. Neither is this the faul…

There is a major issue with current AI tools that they want to effectively grant access to everything their user has access to. The whole sandbox structure is wrong (although various people have vibe coded assorted improvements).

Another issue I've noticed is they're sometimes very resourceful. For example when Codex can't directly edit file due to sandboxing restrictions, rather than asking "hey can I apply this diff on the file", it'd ask for permission to run a `cat EOF` command to re-write the whole file, which the UI doesn't surface properly (just shows the first line...).

This sounds similar to what's described in the "Claude deleted my DB post", it decided "I need to do X", then searched for whatever would let it do X, regardless of intended purpose.

Re: AI didn't delete your database, you did

#172

Post by danbruc »

paroneayea wrote: post #36

I think the perspective here is completely wrong. The problem is that people are now building our world around tooling that eschews accountability . Over a decade ago now, I had a conversation with Gerald Sussman which had enormous influence on me: https://dustycloud.org/blog/sussman-on-ai/ > At some point Sussman expressed how he thought AI was on the wrong track. He explained that he thought most AI directions were…

If an AI driven car drives off the side of the road, I want to know why it did that. I could take the software developer to court, but I would much rather take the AI to court.

How would that work? You have the AI explain its reasoning - and trust that this is accurate - and then you decide whether that is acceptable behavior. If not, you ban the AI from driving because it will deterministically or at least statistically repeat the same behavior in similar scenarios? Fine, I guess, that will at least prevent additional harm. But is this really all that you want? The AI - at least as we have them today - did not create itself and choose any of its behaviors, the developers did that. Would you not want to hold them responsible if they did not properly test the AI before releasing it, if they cut corners during development? In the same way you might hold parents responsible for the action of their children in certain circumstances?

Re: AI didn't delete your database, you did

#173

Post by srdjanr »

Brendinooo wrote: post #44

The one counterpoint I'd offer is that it's very obvious that these companies are tuning LLMs to be more decisive to get stuff done autonomously. If they wanted, they could be putting in similar efforts to be more cautious and stop at the right times to ask for help. So yeah, of course we're ultimately responsible for how we use the tools. But I definitely think it's a two way street. To attempt an analogy, it's like…

It's a tradeoff.

LLMs stopping and asking more would make them less useful. I'd much rather let an agent run for 1 hour, than it wanting my input every 15 mins, even if results are somewhat worse.

The real solution for security is a proper sandbox.

Re: AI didn't delete your database, you did

#174

Post by rafterydj »

aeturnum wrote: post #72
paroneayea wrote: post #36

I think the perspective here is completely wrong. The problem is that people are now building our world around tooling that eschews accountability . Over a decade ago now, I had a conversation with Gerald Sussman which had enormous influence on me: https://dustycloud.org/blog/sussman-on-ai/ > At some point Sussman expressed how he thought AI was on the wrong track. He explained that he thought most AI directions were…

When I was a masters student in STS[1], one of my concepts for a thesis was arguing that one of the primary uses of software was to shift or eschew agency and risk. Basically the reverse of the famous IBM "a computer can not be held responsible" slide. Instead, now companies prefer computers be responsible because when they do illegal things they tend to be in a better legal position. If you want to build as tool tha…

That's really interesting. Are there any things you advocate for with respect to curtailing those practices? I hesitate to throw all liability on the individual, but I don't see how we can even legislate this category of behavior, much less enforce regulations on them.

Re: AI didn't delete your database, you did

#175

Post by AmbroseBierce »

edot wrote: post #20

This is why you don’t hire interns! They can delete things and cause havoc! The same people who would blame AI for their failing to properly configure permissions would also blame interns for deleting production whatever. Blame should go up, praise should go down. People always invert these.

Well, a significant difference is nobody is selling the concept of interns as the end-all solution of humanity's problems, unlike AI.

Re: AI didn't delete your database, you did

#176

Post by HarHarVeryFunny »

Yes, of course any company is responsible for what they ship, regardless of what tools were used to develop it.

However, at least in the US, it is usual for companies to advise against use of their products in a way that may cause harm, and we certainly don't see that from the LLM vendors. We see them claim the tech to be near human level, capable of replacing human software developers (a job that requires extreme responsibility), and see them withholding models that they say are dangerous (encouraging you to think that the ones they release are safe).

Where are the warnings that "product may fail to follow instructions", and "may fail to follow safety instructions"? Where is the warning not to give the LLM agency and let it control anything where there are financial/safety/etc consequences to failure to follow instructions?

Re: AI didn't delete your database, you did

#177

Post by [deleted] »

paroneayea wrote: post #36

I think the perspective here is completely wrong. The problem is that people are now building our world around tooling that eschews accountability . Over a decade ago now, I had a conversation with Gerald Sussman which had enormous influence on me: https://dustycloud.org/blog/sussman-on-ai/ > At some point Sussman expressed how he thought AI was on the wrong track. He explained that he thought most AI directions were…

[deleted]

Re: AI didn't delete your database, you did

#178

Post by xboxnolifes »

philipallstar wrote: post #85
paroneayea wrote: post #36

I think the perspective here is completely wrong. The problem is that people are now building our world around tooling that eschews accountability . Over a decade ago now, I had a conversation with Gerald Sussman which had enormous influence on me: https://dustycloud.org/blog/sussman-on-ai/ > At some point Sussman expressed how he thought AI was on the wrong track. He explained that he thought most AI directions were…

> The problem is that people are now building our world around tooling that eschews accountability. If you tell Terraform the wrong thing it will remove your database and not be accountable either.

But nobody would try to excuse their mistake with "terraform deleted my database". Or if a small handful of people did try, every single other person would call them out.

Re: AI didn't delete your database, you did

#179

Post by amluto »

yonatan8070 wrote: post #171
amluto wrote: post #51

Earlier quoted context omitted.

There is a major issue with current AI tools that they want to effectively grant access to everything their user has access to. The whole sandbox structure is wrong (although various people have vibe coded assorted improvements).

Another issue I've noticed is they're sometimes very resourceful. For example when Codex can't directly edit file due to sandboxing restrictions, rather than asking "hey can I apply this diff on the file", it'd ask for permission to run a `cat EOF` command to re-write the whole file, which the UI doesn't surface properly (just shows the first line...). This sounds similar to what's described in the "Claude deleted my…

I amused myself by removing codex-rs’s web search tool and then asking it to search for “foo”. It wrote a Python script to do the search.

Re: AI didn't delete your database, you did

#180

Post by amluto »

zahlman wrote: post #153
amluto wrote: post #51

Earlier quoted context omitted.

There is a major issue with current AI tools that they want to effectively grant access to everything their user has access to. The whole sandbox structure is wrong (although various people have vibe coded assorted improvements).

If you want them to be able to write code and then run tests on that code, it can be a bit difficult to restrict access meaningfully....

Only for code that can’t be tested in an isolated environment, and designing code that can’t be tested in an isolated environment is generally a mistake for quite a few reasons.
Post reply on HN