"Dog ate my homework"
AI didn't delete your database, you did
301–310 of 329 posts
Re: AI didn't delete your database, you did
#302Using AI is a mistake. It might delete your database.
Giving an LLM the ability to interact with your system is, in fact, a mistake. One that it turns out a lot of people are foolish enough to make, and they don't care at all about the predictable consequences of that mistake.
Re: AI didn't delete your database, you did
#303I recently wrote a blog post where I argued that there are a few principles we should consistently follow when talking about AI: https://susam.net/inverse-laws-of-robotics.html To summarise them: 1. Do not anthropomorphise AI systems. 2. Do not blindly trust the output of AI systems. 3. Retain full human responsibility and accountability for any consequences arising from the use of AI systems. I would like to see the…
> Retain full human responsibility and accountability for any consequences arising from the use of AI systems So if the tool doesn't do what it's supposed to be doing we should blame the user instead of the company that made the tool?
In this case, yes.
That's not without precedent. There are all sorts of tools where our society has decided to presumptively/usually blame the user when the tool is involved in a disaster. Like, it's not always/never, but the difference is pretty stark: if the power's out and a restaurant is closed, you usually blame the power provider for your cancelled reservation. If the power's out and people die in a hospital, you blame the hospital for not having backups.
GP's proposing that AI be in the second category of "presumptively the responsibility of the user", I think.
Re: AI didn't delete your database, you did
#304Earlier quoted context omitted.
I think you are misinterpreting gp as saying "LLMs are a tool [like every other tool]" to mean "LLMs have similar properties to other tools" — when I believe they meant "LLMs are a tool. other tools are also tools," where the operative implication of "tool" is not about scope of capabilities or how deterministic its output is (these aren't defining properties of the concept of "tool"), but the relationship between 't…
> - a tool is activated with operator intent (at some point in the call-chain) This again is where the simplistic assumption breaks down. Just because you can claim that a person kick started something, that does not mean that person is aware and responsible for all its doing. Let's put things in perspective: if you install a mobile app from the app store, are you responsible and accountable for every single thing th…
A few years back, I discovered my router had joined a botnet. The only reason I made this discovery was because of third-party external DNS logs.
Upon investigation, I also discovered that all 3 routers I owned were pwned. So I threw them out the window and tried making do with my ISP's equipment.
My ISP can't provide adequate service on theirs and it's worse than COTS routers, so I purchased a bleeding edge WiFi 7 router. Now there are the two literal black boxes on my network. They do their job and I don't know what else. I can't know.
It could be C2 or it could be a backdoor shell or some kind of server that collects illicit material, and torrents it out? Borrow your HDD for some CSAM sir? It could be a residential proxy that just steals part of my connection for some other paying customer. Are they infringing TOS? How would I know? Check their ID and verify their age??
I, and 99% of consumers with an ISP, have no way of telling when our routers or IoTs are pwned. A silent botnet or two is extremely likely. They're nigh undetectable, and can't be mitigated or defended, except by fastidious updates and upgrades.
My new router was literally triggering printouts on my old printer, because it was so damn "proactive" about "network security scans" and the old trusty printer couldn't tell the difference between a red-team intrusion, and a legit request to print something out!
Likewise even someone with a singular Windows or Mac directly plugged into their ISP could be in a botnet, and it's hard to know. Everyone who's got a smart TV or something with a Linux kernel and an Ethernet, could be doing more than was asked of it. It's the worst kind of malware that alerts the user to its presence. It's a shoddy install if your AV can detect and clean it. If it's stealthy enough then there's no telling.
It's because the vendors own these devices. They deploy the software. They control the builds. The vendors are responsible for what these machines are doing in our hands. Who really, really knows all that goes on when we click that green button? Was it a Joomla or a scam or a legit bank request? Who dafuq knows or cares anymore? Is it an apt analogy that they're selling us herds of animals and farms, and we know nothing of ranching? "Oh feed yourself; should be easy you got everything there" until the coyotes and locusts come? Or like having children who seem to be in school and doing alright, but where do they go at night? Sell drugs? Who knows, I'm not their father, they just live here?
Are they responsible for knowing and mitigating them? Our ISPs don't seem to care or notify us or disconnect us when it happens. Why should we? Why take responsibility?
Re: AI didn't delete your database, you did
#305The whole life delete my database fiasco is being looked at the wrong way. Why did tooling have access to alter or drop? Why did tooling, in any way have more permissions than were m I nimallt necessary to do the job? Decades ago we embraced POLA. What happened to basic hygiene? Sure the agent "screwed up", but it never should have had this access in the first place.
It actively found the wrong API key, that gave it the access it needed...
Still sounds like a basic security issue to me.
Re: AI didn't delete your database, you did
#306Earlier quoted context omitted.
I don’t know why people still consider the US the ideal country for starting companies. Everything seems to evolve around taking people to court.
Because it rarely does end up in courts. But having a fair and strong judicial system is a feature not a bug. The parent points out, in the end there must be a way to resolve accountability and ideally it's done in a manner where both parties can be heard and make a case. Find me a better system than a judicial system for this? Mobs?
I don't know, a system where people still trust a handshake? And a judicial system that is less punitive?
So basically Europe.
Re: AI didn't delete your database, you did
#307Earlier quoted context omitted.
> - a tool is activated with operator intent (at some point in the call-chain) This again is where the simplistic assumption breaks down. Just because you can claim that a person kick started something, that does not mean that person is aware and responsible for all its doing. Let's put things in perspective: if you install a mobile app from the app store, are you responsible and accountable for every single thing th…
>Just because you can claim that a person kick started something Kick started what? If you decided to give an LLM access to your database, it's completely on you when you when it does something you don't want. You should've known better. If all you "kickstart" is an LLM generating text that you can use however you decide, there will never be anything to worry about from the LLM. > Let's put things in perspective: if…
> Kick started what? If you decided to give an LLM access to your database, it's completely on you when you when it does something you don't want. You should've known better.
You don't decide anything. You prompt a coding assistant to apply a change to a repository and without intervention it asserts there's a typo in a table name and renames it. The agent validates the change by running tests and integration tests fail because they are pointing to the old table name. The agent then fixes the issue by applying the change to the database.
Congratulations, you just dropped a table.
I don't think you fully understand how agents and coding assistants work. By design they are completely autonomous and work by reusing your own personal credentials. As they are completely autonomous, they can apply arbitrary changes. I mean, code assistants nowadays write their own tools on the fly. Why do you even presume that people explicitly grant permissions? That's not how it works at all.
If you wish to criticize a topic, the very least you must do is get acquainted with the topic. Otherwise you'll spend your time arguing with your misplaced beliefs instead if the actual problem.
> Yes, and it bothers me that others don't feel the same.
This is a problem you need to overcome, because you have clearly a distorted view of the whole problem domain and also personal responsibility. I recommend you spend a few minutes researching legal precedents associated with malware, because you will quickly learn that runninh arbitrary code you didn't explicitly authorized and acts against your best interests is widely considered a criminal act against the user.
Re: AI didn't delete your database, you did
#308Earlier quoted context omitted.
> - a tool is activated with operator intent (at some point in the call-chain) This again is where the simplistic assumption breaks down. Just because you can claim that a person kick started something, that does not mean that person is aware and responsible for all its doing. Let's put things in perspective: if you install a mobile app from the app store, are you responsible and accountable for every single thing th…
> if you install a mobile app from the app store, are you responsible and accountable for every single thing the app does in your system? Yes. I can try to vet the app to the best of your abilities and beyond that it's a tradeoff between how likely is it to cause harm and do the benefits outweigh these harms. Of course everyone is differently qualified to do this but my argument is more about professionals. Managers…
> Yes. I can try to vet the app to the best of your abilities and beyond that it's a tradeoff between how likely is it to cause harm and do the benefits outweigh these harms.
No, you don't. If you install malware you are not suddenly held responsible for what has been done to you. Even EULAs you are forced to accept don't shift the responsibility away from bad actors.
Re: AI didn't delete your database, you did
#309Earlier quoted context omitted.
Maybe we can just not give it access to production databases ever? Not picking on you, but AI maximalism has infected tech to the point where we talk about how to stop AI from deleting prod instead of seeing that giving AI access to prod is a foolish idea to begin with.
I mean that it’s easy to be careful around a bandsaw because it’s clearly dangerous. The danger with LLMs is that they don’t seem overtly dangerous so you just go right ahead and throw your whole arm in there.
I don't know. The dangers of LLMs are quite well documented by now. It's definitely not a secret.
Re: AI didn't delete your database, you did
#310Earlier quoted context omitted.
> A lot of tools have affordances built in to make "right" things easy and "wrong" or unsafe things harder. This is true for almost anything handed to laypeople, but not for a lot of professional tools. Even a plain battery powered drill has very few protections against misuse. A soldering iron has none. Neither do sewing needles; sewing machines barely do, in the sense that you can't stick your fingers in a gap too…
An LLM is a large and complex machine, not a screwdriver. Large and complex [physical] machines are built with safeguards to prevent misuse, injury, etc by regulation.
A bulldozer is a large and complex physical machine, yet it has (almost¹) no safeguards against misuse or injury. It's all operator training. Lathes tend to not have doors/enclosures, in particular large ones. You get taught where to not put your fingers, and to wear safety goggles. Cranes don't have a lot of safeguards either, you better know how to attach things; hardhats aren't gonna do sh*t if you get a ton of concrete dropped on you.
etc. pp.
I'm not sure where this "tools are made to be safe" belief comes from. This is only the case in "consumer" environments. Of course you don't intentionally make things unnecessarily unsafe, but — in a professional environment there is an expectiation that the operator had training and knows what they're doing.
Maybe that's what we're missing: training in safe AI use. With a certificate that has to be periodically renewed. At the current rate things are going, I'd say 3 months is a good renewal cycle ;D.
(¹ it beeps when it goes backwards. Honestly, I'm not sure that counts for much.)