When I was setting up the generation pipeline for my own AI project, I had to hardcode strict.
AI didn't delete your database, you did
231–240 of 329 posts
Re: AI didn't delete your database, you did
#232I think the perspective here is completely wrong. The problem is that people are now building our world around tooling that eschews accountability . Over a decade ago now, I had a conversation with Gerald Sussman which had enormous influence on me: https://dustycloud.org/blog/sussman-on-ai/ > At some point Sussman expressed how he thought AI was on the wrong track. He explained that he thought most AI directions were…
My team and I are firm that we are the ones accountable. LLMs are a tool like every other. Only that it's non deterministic. But I am the one using the tool. I am the one giving the tool access. I am the one who has to keep everything safe. I have shot myself in the foot using gparted in the past by wiping the wrong disk. gparted wasn't to blame. I was. Letting LLMs work freely without supervision sounds great but it…
Tesla has been sued for a similar reason "full self-driving".
AI companies are selling their products as "perfect" ("better than humans...").
I agree in part with you but I also agree that they are selling a hammer which can blow-up without notice.
Re: AI didn't delete your database, you did
#233I think the perspective here is completely wrong. The problem is that people are now building our world around tooling that eschews accountability . Over a decade ago now, I had a conversation with Gerald Sussman which had enormous influence on me: https://dustycloud.org/blog/sussman-on-ai/ > At some point Sussman expressed how he thought AI was on the wrong track. He explained that he thought most AI directions were…
> The problem is that people are now building our world around tooling that eschews accountability.
If by "now" you mean "for the past few decades", I think you've got it spot on, at least per the very interesting https://en.wikipedia.org/wiki/The_Unaccountability_Machine
Re: AI didn't delete your database, you did
#234I recently wrote a blog post where I argued that there are a few principles we should consistently follow when talking about AI: https://susam.net/inverse-laws-of-robotics.html To summarise them: 1. Do not anthropomorphise AI systems. 2. Do not blindly trust the output of AI systems. 3. Retain full human responsibility and accountability for any consequences arising from the use of AI systems. I would like to see the…
>1. Do not anthropomorphise AI systems. This is maddeningly difficult IMX.
Give it a name, but something non-human like "thingbot" or "tacosplosion."
"Hey tacosplosion, generate me an exploding taco image."
Re: AI didn't delete your database, you did
#235I recently wrote a blog post where I argued that there are a few principles we should consistently follow when talking about AI: https://susam.net/inverse-laws-of-robotics.html To summarise them: 1. Do not anthropomorphise AI systems. 2. Do not blindly trust the output of AI systems. 3. Retain full human responsibility and accountability for any consequences arising from the use of AI systems. I would like to see the…
> Retain full human responsibility and accountability for any consequences arising from the use of AI systems So if the tool doesn't do what it's supposed to be doing we should blame the user instead of the company that made the tool?
Your comment is a perfect example of not caring about nuance. More charitably, it comes from a place of naivete about how LLMs work.
LLMs are non-deterministic [0]. They can't be trusted to fully follow your prompts. As such, you have to be careful about what permissions they have.
Like...I use Claude Code. I allow it to run some shell commands that only read (grep, ls, find, etc.). I will never allow it to run Python code without checking with me first. Yeah, it slows me down when I have to answer its prompt for permission to run Python, but the alternative is outright dangerous.
Compare this with any other tool, say, something as simple as `rm`. I expect that if I call `rm some.file`, it will only delete that file. If it deletes anything else, that's absolutely the fault of the tool, and I should not bear any responsibility for mistakes the tool makes as long as my input was correct.
I do not give LLMs that same latitude. LLMs operate probabilistically and have far more degrees of freedom in how they interpret and act on your input, so you hold them (and yourself) to a different standard of scrutiny and accountability.
[0] Technically, LLMs are actually completely deterministic. Run any given input through the neural network, and you'll get the exact same output [1], but that output is a list of probabilities of the next potential token. Top-k sampling, temperature, and other options essentially randomize the chosen token, making them non-deterministic in practice, though APIs will often allow you to disable all that and make them deterministic.
[1] Even this statement isn't quite true because floating point math is not associative.
Re: AI didn't delete your database, you did
#236Earlier quoted context omitted.
My team and I are firm that we are the ones accountable. LLMs are a tool like every other. Only that it's non deterministic. But I am the one using the tool. I am the one giving the tool access. I am the one who has to keep everything safe. I have shot myself in the foot using gparted in the past by wiping the wrong disk. gparted wasn't to blame. I was. Letting LLMs work freely without supervision sounds great but it…
Tesla has been sued for a similar reason "full self-driving". AI companies are selling their products as "perfect" ("better than humans..."). I agree in part with you but I also agree that they are selling a hammer which can blow-up without notice.
I do agree that the companies could do a better job telling about the dangers, but let's be real here. It's hardly a secret that LLMs can be erratic. It's not news.
Other companies also tell me their product is the best thing since sliced bread. I still try to find the flaws. That's part of my job. But suddenly with LLMs we just blindly trust the companies? I don't think you.
I don't blindly give up my brain and my agency and no one else should. It's fun and educational to play around with LLMs. Find the what they are good at. But always remember that you can't predict what it will do. So maybe don't blindly trust it.
Re: AI didn't delete your database, you did
#237Some details from the original post for context: They had a Railway token in an unrelated file (unclear if it was a local secret) for managing custom domains. It turns out that token has full admin access to Railway. The AI deleted a single relevant volume by id. The author is rather vague about what exactly it asked it to do, he just says there was a “credentials mismatch” and Claude took the initiative to fix it by…
I see the value in Heroku, even though everyone on HN keeps saying it's bad now. Skeptical of other newer things. Firebase defaults have also been insane from the start.
Re: AI didn't delete your database, you did
#238Some details from the original post for context: They had a Railway token in an unrelated file (unclear if it was a local secret) for managing custom domains. It turns out that token has full admin access to Railway. The AI deleted a single relevant volume by id. The author is rather vague about what exactly it asked it to do, he just says there was a “credentials mismatch” and Claude took the initiative to fix it by…
One thing AI can power nicely is the anti-SaaS movement. Being able to just boot a cheap PC and test out any of the open source packages is so infinitely easier than piling into all the random credential Bazaars. But that won't take away the inability of the LLM from confusing whats in dev, whats in production, whats in localhost and whats remote; I've been working on getting a tools/skill for opencode that works wit…
AI can also make using AWS directly easier
Re: AI didn't delete your database, you did
#239Earlier quoted context omitted.
This is kind of the reverse of https://en.wikipedia.org/wiki/Poka-yoke . A lot of tools have affordances built in to make "right" things easy and "wrong" or unsafe things harder. LLMs .. well, the text interface is uniquely flat. Everything is seemingly as easy as everything else. I worry about the use of humans as sacrificial accountability sinks. The "self-driving car" model already has this: a car which drives its…
> A lot of tools have affordances built in to make "right" things easy and "wrong" or unsafe things harder. This is true for almost anything handed to laypeople, but not for a lot of professional tools. Even a plain battery powered drill has very few protections against misuse. A soldering iron has none. Neither do sewing needles; sewing machines barely do, in the sense that you can't stick your fingers in a gap too…
Exactly this. I was talking about professionals. People who should know better. If we as professionals give away our agency and our accountability we make ourselves obsolete. If I just tell the LLM what to do and hope it doesn't go south then the Manager could probably do that as well.
And if a non professional did it they should ask themselves why we have professionals. Maybe there was a reason and maybe they do have value.
Re: AI didn't delete your database, you did
#240The article author did not even bother to read the article they were basically replying to. Otherwise he would have noticed that the main points the OP was complaining about were not about the agent, but the hosting provider providing an API allowing destructive operations easily, using tokens with no scopes, with backups stored in the same volume as main data, etc. So this article is actually agreeing with the complaints of the original article, just more generically and without spending an effort on it, doing that with a tone that implies the original article writer is an idiot.