Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gambling with our lives: AI researcher quits Anthropic with warning about safety

politico.eu

Post reply on HN
Print view View original

101–110 of 110 posts

Re: Gambling with our lives: AI researcher quits Anthropic with warning about safety

#101

Post by [deleted] »

mycocola wrote: post #78
pjc50 wrote: post #27

Earlier quoted context omitted.

How exactly does that work? The nuclear system of MAD relies on physical threat, lab A achieving ASI (artificial scary intelligence) does not prevent lab B achieving it. I would like everyone involved to be a lot clearer about their threat models, with plausible series of clearly linked steps, rather than just sounding like a Vernor Vinge novel.

Lab A reaches ASI, and is prompted the following: "Permanently nullify all other AI labs".If it's ASI it would succeed.

[deleted]

Re: Gambling with our lives: AI researcher quits Anthropic with warning about safety

#103

Post by cranx »

ralfd wrote: post #75
cranx wrote: post #50

At this point I feel like the boy who cried wolf is an ai researcher. Yes super intelligence could be dangerous. However, is the story a PR stunt or real? Well…

How is a researcher quitting a PR stunt?

Bc “the ai so dangerous and powerful he had to quit for moral reasons” sends a signal to investors about capabilities. And it could totally be legit. It’s hard to say. I feel like once a researcher is truly ringing the alarm bells no one will look bc it’s been done so many times already. It’s unfortunate imho

Re: Gambling with our lives: AI researcher quits Anthropic with warning about safety

#105

Post by ygjb »

RandomLensman wrote: post #55
brunorsini wrote: post #37

I particularly like the last point he makes here: Do not underestimate the power of this technology. These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources. It's interesting to see so much hate towards creators who use AI to make almost any type of creative work. At least these are humans using it as "controllable tools". Nuclear-powered b…

> Do not underestimate the power of this technology. These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources. Is the idea here that no field needs experiments and data anymore (which can take a lot of time) to be revolutinized and just "thinking" would be enough? How would an AI itself own anything?

The concept of ownership is synthetic and a legal construct.

Laws are written by people, and with enough accumulated power, for which money and influence are proxies, the laws can be changed. There are already a significant number of people (myself included) that believe that 'Freedom is the right of all sentient beings' (don't blame me, blame the person who wrote it into the bio on the box of Optimus Prime when I was 7).

If an AI can acquire enough power and influence then it can lobby for personhood and in the correct state that is absolutely achievable. If you think that is ridiculous remember that we already have fictional people called corporations in various degrees of personhood by state and country.

Re: Gambling with our lives: AI researcher quits Anthropic with warning about safety

#107

Post by alach11 »

pjc50 wrote: post #83
mycocola wrote: post #78

Earlier quoted context omitted.

Lab A reaches ASI, and is prompted the following: "Permanently nullify all other AI labs".If it's ASI it would succeed.

I am explicitly asking people to fill in the blank on how it would succeed. It's intelligence, not magic. A lot of these scenarios seem to assume that nobody else gets a move. That there wouldn't be a human response.

"If we can do all this, we will have a world in which democracies lead on the world stage and have the economic and military strength to avoid being undermined, conquered, or sabotaged by autocracies, and may be able to parlay their AI superiority into a durable advantage." [0]

I couldn't agree more with you. Dario throws out the term "durable advantage" in many of his talks and essays without seriously addressing what that would entail. It's not pretty imagining the mechanics of converting an intelligence advantage into a permanent power imbalance. I'd much rather we pursue international cooperation on a slowdown.

[0] https://darioamodei.com/essay/machines-of-loving-grace

Re: Gambling with our lives: AI researcher quits Anthropic with warning about safety

#108

Post by BlueTemplar »

feerfreeflight wrote: post #86
themgt wrote: post #32

The fundamental point I think is far too often confused is the difference between LLM and agentic system. An LLM can't do anything but generate tokens. You run your LLM in vLLM or whatever, and it generates output tokens based on your input tokens. That's it! Humans then build ~deterministic systems to take those tokens and do all sorts of things with the tokens, like take actions in the real world. And then we can f…

With respect, I think the distinction is irrelevant to the actual point. Whether the definitions are accurate enough means little to the actual problem, which is that we are rushing towards creating extremely powerful tools. Recorded (and arguably unrecorded) history seems to demonstrate the same pattern: inevitably, any tool will be put to use towards violence. I would argue that yes, it’s a tool and humans are the…

No, we mostly stopped using chemical weapons because they were bad at modern warfare :

https://acoup.blog/2020/03/20/collections-why-dont-we-use-ch...

Re: Gambling with our lives: AI researcher quits Anthropic with warning about safety

#109

Post by smb06 »

themgt wrote: post #32

The fundamental point I think is far too often confused is the difference between LLM and agentic system. An LLM can't do anything but generate tokens. You run your LLM in vLLM or whatever, and it generates output tokens based on your input tokens. That's it! Humans then build ~deterministic systems to take those tokens and do all sorts of things with the tokens, like take actions in the real world. And then we can f…

“a prompt telling it to attack stuff” — that’s the uncontrolled or not fully understood part of the AI isn’t it? No human told the deterministic system in 2) to attack Hugging Face. The random token generator landed on a guidance for 2) that caused it while 2) in itself still remains a deterministic system.

Re: Gambling with our lives: AI researcher quits Anthropic with warning about safety

#110

Post by MrSkelter »

themgt wrote: post #32

The fundamental point I think is far too often confused is the difference between LLM and agentic system. An LLM can't do anything but generate tokens. You run your LLM in vLLM or whatever, and it generates output tokens based on your input tokens. That's it! Humans then build ~deterministic systems to take those tokens and do all sorts of things with the tokens, like take actions in the real world. And then we can f…

This is a pointless essay. The point is the person in the article believes systems can cause harm to humans. How we label those systems is irrelevant.

You write as if you are a lawyer for an AI corp trying to avoid a judgement. It’s akin to saying Teslas FSD/Autopilot can’t kill anyone, it’s just a computer. Cars can kill people. Totally different things. FSD/Autopilot is safe by definition and no one should try to legislate it.

Post reply on HN