Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I resigned from Anthropic today

twitter.com

Post reply on HN
Print view View original

911–920 of 1001 posts

Re: I resigned from Anthropic today

#911

Post by SepiaSapient »

danaris wrote: post #514
epihelix wrote: post #304

Earlier quoted context omitted.

Yes, but the really weird thing is that they seem to: a) believe that what they're creating is a basilisk, and b) keep trying harder to do this while staring right at it I think they're very deluded about (a) -- but if they do actually believe this (and it really seems like a decent proportion of Anthropic truly does), then why keep doing (b)? That seems to be why this individual resigned, but I'm surprised it's not…

With Roko's Basilisk, if you believe in it, the most rational thing to do is to put forth every effort to bring it into being. Because if you don't, then you will be one of its targets when it does, inevitably, come into being. (I am not a basilisk believer. I think this is all absolute horseshit. But to understand someone's motivations, one must think like them.)

>Because if you don't, then you will be one of its targets when it does, inevitably, come into being.

Only if you don't spend 5 minutes thinking about the Self as a concept. Resurrecting someone by perfect simulation is a cool plot device for a novel like Altered Carbon (which ironically is really about wealth inequality) but not a serious hypothetical.

Even if we ignore that, no, super intelligent MechaGrok17 cannot make a perfect copy of you posthumously from your embarrassing Character.AI chats and IoT toilet data, the copy it still wouldn't be YOU. It would (supposedly) think and behave like you, but for all purposes it would be the Basilisk torturing a future stranger, or your descendants. I'm very much a fan of people not being potentiality tortured, so the rational action is to stop the creation Basilisk, or even just delay it so you aren't around for it. There's a bunch of other big assumptions that makes Roko's Basilisk really stupid, but they are unimportant. It's just theory crafting, like talking about the lore of video games or marvel comics. It doesn't help you understand their real motivations.

Roko's Basilisk is just recontextualized protestant Hell in a sci-fi coat of paint. It's the childhood anxieties of American nerds about hell and sin. The problem is that they are still culturally protestant. God doesn't exist or he's hands off, but in their minds there's still a open position for the wrathful and proactive God. If you think that you actually can create God for realsies, your very rational™ prediction of its behavior will coincidentally line up with your childhood fears.

Why is the spooky AI mad at you? Because you didn't worship it correctly.

Why do you obey it? Because if I don't it will torture me for eternity.

How will it do that? God is omnipotent, don't think about it.

Why don't try to stop it's creation? “I am the Alpha and the Omega,” says the Lord God, “who is, and who was, and who is to come, the Almighty.”

All the scifi scaffolding and stuff like acausal trade is paint over Timmy's first panic attack on Sunday school 30 years ago.

Re: I resigned from Anthropic today

#912

Post by igor47 »

brookst wrote: post #183
underyx wrote: post #181

Earlier quoted context omitted.

"So you think in 3 years AI is going to solve longstanding math problems because it was used to write some coherent sentences?" — people with the same amount of foresight in 2023

Are you saying at anything that can solve longstanding math problems necessarily has the means, motive, and capability to kill 8 billion people in just 3 years?

No dude, it was an analogy. Sorry to pick on you, you're just another ignorant uninformed take in this thread, but come on. This technology is progressing at an insane rate, the shit it's doing now is science fiction from just a year or two ago, and people just keep on moving the goal posts, it's maddening. The technology is unpredictable and that in itself has risks. You must realize that the window of possibility is widening the further into the future we go! Can you pick people with epistemology you trust and see if there's anything you can learn in this moment? Can you try to challenge your own ideas instead of retreating into comfortable certitude? There are real risks here, they are worth taking seriously, and serious people are doing so!

Re: I resigned from Anthropic today

#913

Post by philipallstar »

SupremumLimit wrote: post #793
philipallstar wrote: post #781

Earlier quoted context omitted.

> Capitalism cannot do otherwise. I really think this sort of claim needs to stop. Capitalism, like "patriarchy" is not a force in its own right. Capitalism is a system that encourages value creation for one's customers. That's not perfect, but it's still best so far.

"value creation for one's customers" - that is a "not even wrong" level of misunderstanding of the system dynamics of capitalism. The primary driver is generation of _profit_ for owners of the means of production. Many things fall out of that - the tendency to monopolise, for example.

> "value creation for one's customers" - that is a "not even wrong" level of misunderstanding of the system dynamics of capitalism. The primary driver is generation of _profit_ for owners of the means of production. Many things fall out of that - the tendency to monopolise, for example.

I don't see why you're talking like this. It's "not even wrong" in the sense that it's entirely correct. Monarchy and socialism grab money through taxes, and if you think monopolies are bad, wait til you're forced to use the state service that has no motivation to do well because there's no competition. It's not only a monopoly, it's a monopoly that takes your money by force.

Capitalism requires investment and risk and competition to drive value up for customers who can choose. So you need to produce value for your customers to survive, and produce a profit after costs and taxes.

Re: I resigned from Anthropic today

#914

Post by dwaltrip »

isolatedsystem wrote: post #484
emtel wrote: post #439

Earlier quoted context omitted.

Do you think people should continue developing AI up until the point that there is evidence that AI is facilitating biological weapons development? I think we should stop before then. But that necessarily means that there will not be evidence at the time that we stop.

This is rubbish. By that token, computer development is also facilitating biological weapons development. A better MacOS (or Windows, I don't know) leads to better weapons. They should clearly stop developing computers and OSes. Developers of nice test-tubes are also facilitating bioweapons. Your local O-ring manufacturer, your local medical-grade freezer manufacturer etc. are all culpable. The problem is the bioweap…

An OS developer is just as culpable as a researcher directly training AIs that (1) are capable of building bio-weapons and (2) have shown consistent and increasingly severe misalignment? Among other things...

That doesn't make sense.

Re: I resigned from Anthropic today

#915

Post by IAmBroom »

kakacik wrote: post #726
rurban wrote: post #619

Earlier quoted context omitted.

Didn't you follow the robotics advances in the Ukrainian - Russian battlefield? There is now a death zone of 50km, only controlled by drones and automatic weapons.

> There is now a death zone of 50km, only controlled by drones and automatic weapons. If there would be such 50km death zone, front line would not move a nanometer in a year, would it. You yourself contradict above in your next post. No need for being too dramatic, facts are enough here. In reality, frontline is moving constantly albeit by small chunks, russians are advancing a bit, getting beating elsewhere and so o…

There is such a thing as a No Man's Land[0]. It can move, but if you don't dare cross it for even a few days, it's very real. It doesn't have to be glacial, and it currently exists in Ukraine. Breakout pockets don't mean it is porous everywhere.

[0] Ironically, the French phrase - from ground zero for the first known No Man's Land - is "le No Man's Land".

Re: I resigned from Anthropic today

#916

Post by red-iron-pine »

rurban wrote: post #619
skybrian wrote: post #241

Earlier quoted context omitted.

The mathematics research results are certainly impressive, but I don't see what that has to do with robotics. Waymo is getting somewhere, but it's been a long slog. There doesn't seem to be much progress on, say, package delivery. For more, see: https://secondthoughts.ai/p/14-reasons-robotics-is-hard

Didn't you follow the robotics advances in the Ukrainian - Russian battlefield? There is now a death zone of 50km, only controlled by drones and automatic weapons.

50km is just the "very likely to get hit" part of the gray zone

ISR drones and other tools can go as far as 200km, albeit with worsening visibility.

in most cases everything meaningful in that 200k is seen, and the only limitation is range of the drones, artillery, missiles, and potential countermeasures.

Re: I resigned from Anthropic today

#917

Post by dwaltrip »

copperx wrote: post #153
matherial wrote: post #40

Unlike most other commenters, I applaud him for acting on his principles. If you sincerely believe that, of course you should act. You might not succeed, but your voice might be the one that tips the scales and starts a broader movement. This doesn't mean I agree with him. The fears of doomsday caused by rapid takeoff have been with us since day 1 and the mechanism is always basically "AI invents magic that sets it f…

To borrow on the 1990s Slashdot meme: 1. Invent transformer architecture. 2. Scale it up. 3. ??? 4. Machines become sentient and kill us all. OpenAI and Anthropic pinky promise that they have figured out #3 and they're not BSing just to get more funding, no. But because we live in a culture of fear, everyone eats it up no questions asked.

#3 is actually decently well mapped out. You just don't find it plausible or misunderstand it. I'd appreciate if you wrote your actual arguments against it.

At lower capability levels, the patterns are very clear and have been studied to death. E.g. Why LLMs say they have correctly fixed a broken test when they haven't. What we saw with Hugging Face is literally the exact same problem, just scaled up and with more capable agents. This shit was predicted decades ago...

No one can say exactly how it will play out as the complexity increases, but the risks are becoming extremely obvious.

I personally think it's extremely unlikely to "kill everyone", but there are many outcomes far short of that which seem quite plausible and rather undesirable. Russian roulette is not a smart game.

If you read the METR report and aren't scared at all, then I'd love to know why. It would help me sleep better. So please share.

TL;DR; increasing capabilities, reward hacking, and unsafe training regimes.

Re: I resigned from Anthropic today

#918

Post by ryandrake »

gpm wrote: post #299
ls612 wrote: post #216

Earlier quoted context omitted.

Historically it almost always takes actual fatalities rather than near misses to ground an aircraft, and aviation is famous for its obsession with safety compared to other industries.

Airplanes have pretty bounded damage. Generally you kill at most a few hundred people. Even weaponized a few thousand. This is a risk profile that allows risk taking with near misses and waiting until something goes wrong to fix it (though doing so is rightfully uncomfortable and frequently unethical). The people worrying about AI risk are worrying about "it goes wrong once and kills billions of people". That's not a…

Humanity does not have a great track record for "globally coordinated, collective action to solve/prevent global catastrophe." Look at how Climate Change is going.

Re: I resigned from Anthropic today

#919

Post by red-iron-pine »

jimmytucson wrote: post #427
Kim_Bruning wrote: post #106

Earlier quoted context omitted.

I'm somewhat skeptical of some of the crazier ideas too. But the hugging face incident was actually very large. It was not a single agent, it was not a single target, and it was not a single event. If nothing else, that's a bit of a warning as to what can happen next time (By accident, or if a government decides to go on purpose). For now let's assume the worst that can happen is that some important/significant chunk…

> For now let's assume the worst that can happen is that some important/significant chunk of (transitively) internet connected stuff goes haywire all at once. That's probably your upper limit of what can go wrong for now. If we have to disconnect from the internet to stop some kind of mold outbreak, we can't get the weather or transfer money or access healthcare or teach an elementary school class or buy stuff from s…

that's a big problem for the modern world that depends on the internet, but boy howdy we did all of those things without the internet.

ya see they had these things called cheques...

Re: I resigned from Anthropic today

#920

Post by dinfinity »

ericmay wrote: post #889
dinfinity wrote: post #872

Earlier quoted context omitted.

> if someone was 100% certain This is unreasonable because nobody can know 100% certain that the AI built at Anthropic will kill everyone. It is also a big-ass strawman because the author never claimed he was 100% certain that AI will kill all humans. You're arguing from an irrelevant hypothetical (in which, I might add, most people would still not automatically change into murderous psychopaths as you apparently thi…

> This is unreasonable because nobody can know 100% certain that the AI built at Anthropic will kill everyone. It is also a big-ass strawman because the author never claimed he was 100% certain that AI will kill all humans. Ok then stop arguing about my hypothetical if you don't want to engage with it. Those of us who are interested in talking about it can do so in peace without distracting comments. > change into mu…

I'm not the one reacting to reasonable behavior with some edgy comment containing broken logic.

The author of the Twitter thread is not some egotistical attention seeker and never said "omg I’m quitting they’re going to kill everyone". His logic for quitting his job and not acting like a psychopath is sound. Yours is not.

Post reply on HN