271–280 of 329 posts
I recently wrote a blog post where I argued that there are a few principles we should consistently follow when talking about AI: https://susam.net/inverse-laws-of-robotics.html To summarise them: 1. Do not anthropomorphise AI systems. 2. Do not blindly trust the output of AI systems. 3. Retain full human responsibility and accountability for any consequences arising from the use of AI systems. I would like to see the…
Earlier quoted context omitted.
I agree that LLMs could be more open about their dangers and that people are bad at judging risks sometimes. Still I think a band saw has very little warning on it and by it's design there is very little anyone can do about me cutting off my finger if I am not careful. LLM companies can do very little about the unpredictability of LLMs. So we have to choose how for we will let it go. In the end the LLM only produces…
A band saw is always a screaming band of bladed death. An LLM is sometimes a buddy, sometimes a mentor, and only sometimes a guy that drops your database.
Not picking on you, but AI maximalism has infected tech to the point where we talk about how to stop AI from deleting prod instead of seeing that giving AI access to prod is a foolish idea to begin with.
I think the perspective here is completely wrong. The problem is that people are now building our world around tooling that eschews accountability . Over a decade ago now, I had a conversation with Gerald Sussman which had enormous influence on me: https://dustycloud.org/blog/sussman-on-ai/ > At some point Sussman expressed how he thought AI was on the wrong track. He explained that he thought most AI directions were…
When I was a masters student in STS[1], one of my concepts for a thesis was arguing that one of the primary uses of software was to shift or eschew agency and risk. Basically the reverse of the famous IBM "a computer can not be held responsible" slide. Instead, now companies prefer computers be responsible because when they do illegal things they tend to be in a better legal position. If you want to build as tool tha…
It's something people already did with corporations and employee handbooks, not unique to software, just one of many kinds of tasks being automated.
What's interesting is that in this article, the author describes making an understandable mistake (accidentally deleting Trunk aka main from source) and how their team was able to easily recover from that due to the nature of SVN. The actual "AI deleted my database" story is really more of a "Railways' database 'backup' strategy is insane and opaque and Railway promoting AI infrastructure orchestration without guardr…
> "Railways' database 'backup' strategy is insane and opaque and Railway promoting AI infrastructure orchestration without guardrails is dangerous." Yes. However, if you choose to build on their platform you bear the responsibility to understand how it works. You could have chosen a different platform, or no platform. Instead you chose Railway. Given that, it's your responsibility to know how to use it safely.
Imo both share fault. Railway purports to be an abstraction anyone can use without expertise. Without expertise, how can a customer determine if Railway actually is an "expert".
In other areas like medicine, engineering, and trades the government or private entities step in with licensure or certification to act as an intermediary.
Earlier quoted context omitted.
Normally, people don't suddenly go insane, snap and start deliberately deleting things in production. Sure, it happens, but very, very rarely.
People make bad decisions all the time . Insanity is not required. My remark was pointing out the larger failure mode of people acting contrary to the good of the team for personal gain, eg creating a problem and blaming someone else to reduce their chances of competing for a promotion. But to your point, an SDE doesn’t need to be insane to bypass a 2 PR and force a change into production. They just need to be panick…
One of the best thing about digital computers, compared to humans, was that they can't be the first or the third thing you mentioned; unfortunately, they absolutely are the second ("the machine does exactly what you told it to do, not what you want it to do"), and at inhuman speeds. Presumably, AI would (need, actually — Nick Bostrom puts a fairly reasonable argument for that in his "Superintelligence") fix that second bullet point, and then everything will be peachy.
Instead, we have people on the internet arguing that it's not a problem, since people too have this same problem. Which is a problem. But not a problem. Ugh.
Some details from the original post for context: They had a Railway token in an unrelated file (unclear if it was a local secret) for managing custom domains. It turns out that token has full admin access to Railway. The AI deleted a single relevant volume by id. The author is rather vague about what exactly it asked it to do, he just says there was a “credentials mismatch” and Claude took the initiative to fix it by…
One thing AI can power nicely is the anti-SaaS movement. Being able to just boot a cheap PC and test out any of the open source packages is so infinitely easier than piling into all the random credential Bazaars. But that won't take away the inability of the LLM from confusing whats in dev, whats in production, whats in localhost and whats remote; I've been working on getting a tools/skill for opencode that works wit…
In my experience, Claude Code with Opus 4.7 tends to assume things are production unless explicitly told otherwise.
>there's no way to force the LLM into a protocols that limits their damage to a remote system or a narrow scope of tools
Might not be able to force it but prompting and context help. An AGENTS.md that explicitly calls out what is and isn't production helps (at least with Claude Code)
Not sure about OpenCode but in Claude Code, memories also help (more injected context)
Earlier quoted context omitted.
> "Railways' database 'backup' strategy is insane and opaque and Railway promoting AI infrastructure orchestration without guardrails is dangerous." Yes. However, if you choose to build on their platform you bear the responsibility to understand how it works. You could have chosen a different platform, or no platform. Instead you chose Railway. Given that, it's your responsibility to know how to use it safely.
Meta-ignornance Imo both share fault. Railway purports to be an abstraction anyone can use without expertise. Without expertise, how can a customer determine if Railway actually is an "expert". In other areas like medicine, engineering, and trades the government or private entities step in with licensure or certification to act as an intermediary.
Earlier quoted context omitted.
Your comment is a perfect example of not caring about nuance. More charitably, it comes from a place of naivete about how LLMs work. LLMs are non-deterministic [0]. They can't be trusted to fully follow your prompts. As such, you have to be careful about what permissions they have. Like...I use Claude Code. I allow it to run some shell commands that only read (grep, ls, find, etc.). I will never allow it to run Pytho…
> LLMs are non-deterministic For someone who complains about a lack of nuance it's surprising you're completely missing my point. The lack of trust is precisely my point. Either AI companies are made accountable for providing un unreliable service or they need to stop selling and marketing these LLMs as if they were infallible.
Maybe you should change your argument style and actually articulate a point rather than taking the insufferable approach of asking a "gotcha" question that ends up getting misunderstood.
If it's one person misunderstanding you, then its on them. If everyone is misunderstanding you, then it's on you.
Earlier quoted context omitted.
If you automate it all it’s fine. The automation has no problem waiting around for traffic to drain out of something before decommissioning it.
Then you're back to square 1. The only way to win here is to require user supervision and make it simple and easy to use.
The basic setup has worked well for me at several companies with many large teams on-call.
I think the perspective here is completely wrong. The problem is that people are now building our world around tooling that eschews accountability . Over a decade ago now, I had a conversation with Gerald Sussman which had enormous influence on me: https://dustycloud.org/blog/sussman-on-ai/ > At some point Sussman expressed how he thought AI was on the wrong track. He explained that he thought most AI directions were…
Perhaps what would be even better is to document better the process, work, and data that go into making each individual"AI" model. Regardless of whether that AI model is a "black box" or can self explain its behavior we would then have absolute metrics and comparable information to retroactively explain its "decisions". This would not be entirely dissimilar to how we explain individual humans behavior with psychology (although obviously also very different).