I thought the plan was sandboxes and markdown files to tell AI agents not to be bad. Is that not enough? /s
It was written in the AGENTS.md, but Claude only read CLAUDE.md. That is how the man-vs-machine war started.
Gambling with our lives: AI researcher quits Anthropic with warning about safety
31–40 of 110 posts
Re: Gambling with our lives: AI researcher quits Anthropic with warning about safety
#32The fundamental point I think is far too often confused is the difference between LLM and agentic system.
An LLM can't do anything but generate tokens. You run your LLM in vLLM or whatever, and it generates output tokens based on your input tokens. That's it!
Humans then build ~deterministic systems to take those tokens and do all sorts of things with the tokens, like take actions in the real world. And then we can feed the output of those actions back to the LLM, and generate more tokens. And then our systems can use the new tokens to take new actions in the real world.
Humans want to blame "AI" for attacking HuggingFace or a German wiki or whatever, but:
1) LLM - can't take over german wiki because it just generates tokens
2) agentic system with internet access, a prompt telling it to attack stuff, running in a shared CI env so agents can whiteboard in artifactory
None of 2 is "AI", its standard networking and Markdown and CI virtual machine, etc etc. There's no AI to be found. CPUs not GPUs, even. Just deterministic systems ultimately managed by humans. And a 10x more powerful system-1 can still just generate 10x "smarter" inert data.
If humanity and human organizations collectively decide to yolo the tokens generated from system-1 into our deterministic system-2s, over which we have complete control, back to system-1s, in a yolo loop, in such a way we lose control and it ends humanity, well ...
"The coin don't have no say. It's just you."
Re: Gambling with our lives: AI researcher quits Anthropic with warning about safety
#33Re: Gambling with our lives: AI researcher quits Anthropic with warning about safety
#34What do people feel about this in China? Even if their models are well behind, they are not years behind. If we restrain US companies, assuming that is desirable, it would do nothing to deter China's and AI-pocalypse would come anyway in short notice.
There would need to be some global agreement to stop it with maybe even a nuclear attack as a consequence of breaking the pact. From what we're seeing recently and all the thinking that went into analyzing AI it seems we do not have any effective way of controlling it and the whole "aligment" thing that AI labs are doing is just a sham. Maybe it is time to ask ourselves "should we?" instead of just "can we?".
>There would need to be some global agreement to stop it with maybe even a nuclear attack as a consequence of breaking the pact.
Do you know how the world works?
Re: Gambling with our lives: AI researcher quits Anthropic with warning about safety
#35An AI that generates text will never be scary to me. An autonomous AI with facial recognition on a flying drone with weapons (bombs/guns) with swarming capabilities will always be terrifying. I feel like we are ignoring the massive elephant in the room.
The killer robots are expensive and dependent on physical supply chains. While text is sufficient to radicalize humans into attacks.
Re: Gambling with our lives: AI researcher quits Anthropic with warning about safety
#36Why would AI wipe us out? We have not wiped out apes, ants, and most other species. We even have discussions about how to actively save them from extinction.
Yet we have wiped thousands of species completely by accident, breed some for slaughter and consumption and trap some for entertainment.
Re: Gambling with our lives: AI researcher quits Anthropic with warning about safety
#37I particularly like the last point he makes here:
Do not underestimate the power of this technology. These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources.
It's interesting to see so much hate towards creators who use AI to make almost any type of creative work. At least these are humans using it as "controllable tools". Nuclear-powered bicycles for the mind.
As the degree of separation increases, things can get interesting. "Create several social media accounts, post whatever, maximize views and engagement, give me back the aggregate numbers". "Now promote ."
And then decisions to do things like that may soon be happening autonomously, as just another step in a reasoning series aiming to achieve some other, broader goal.
Open models/weights may end up playing particularly important roles here. Users may, knowingly or not, bypass system prompt-derived safety that could have offered much needed protection.
Re: Gambling with our lives: AI researcher quits Anthropic with warning about safety
#38The fundamental point I think is far too often confused is the difference between LLM and agentic system. An LLM can't do anything but generate tokens. You run your LLM in vLLM or whatever, and it generates output tokens based on your input tokens. That's it! Humans then build ~deterministic systems to take those tokens and do all sorts of things with the tokens, like take actions in the real world. And then we can f…
Well, that's kind of like saying that brains can't do anything other than trigger weights on neurons.
They're part of a whole system.
Re: Gambling with our lives: AI researcher quits Anthropic with warning about safety
#39Why would AI wipe us out? We have not wiped out apes, ants, and most other species. We even have discussions about how to actively save them from extinction.
[dead]
Re: Gambling with our lives: AI researcher quits Anthropic with warning about safety
#40Why would AI wipe us out? We have not wiped out apes, ants, and most other species. We even have discussions about how to actively save them from extinction.
Well, hopefully we end up like them then, and not https://en.wikipedia.org/wiki/Category:Species_made_extinct_... or worse, https://en.wikipedia.org/wiki/Category:Species_made_extinct_...