Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I resigned from Anthropic today

twitter.com

Post reply on HN
Print view View original

911–920 of 973 posts

Re: I resigned from Anthropic today

#911

Post by senordevnyc »

sicher wrote: post #613
onewayfunction wrote: post #178

I'm pretty baffled by the degree of skepticism expressed here in response to some of Jacob's claims. After the events of the summer it feels like it takes a lack of imagination to not see a few plausible routes to disaster. It may be reasonable to believe these outcomes are not very likely or that we can stop before going too far (I tend to disagree). But I can't imagine doubting that the capabilities will soon be th…

I'm also baffled. AI that is substantially smarter than us is a very potential threat to us - and we won't even be able to comprehend what most of those threats may be.

It's head-in-sand denial because actually facing the threat (and our total inability to stop it, imo) is terrifying, and people don't like feeling that. So they're just angry and cynical instead. Plus it makes them feel smarter for whatever reason.

Re: I resigned from Anthropic today

#912

Post by senordevnyc »

panicinducer wrote: post #800
TrackerFF wrote: post #769

Earlier quoted context omitted.

Let's just for the sake of discussion assume that one time in the future, near or distant, AI manages to become sentient. And like other forms of life, its main motivation is survival: Like biological life competes for food and land, AI competes for power and compute. Probably the first motivation would be to find ways not to lose control over itself (i.e. remove human ability to control it), find ways not to lose en…

If this is true, or even if the people working on it think this is true, those people should be incarcerated, their companies disbanded, and their research dismantled, with more serious repercussions for anyone who tries it after that. It also needs to be a global initiative like with nuclear non-proliferation, and this time with no looking-the-other-way when it comes to Israel.

Yes! We should do that right away, and I'm sure China will do the same. Right?

Re: I resigned from Anthropic today

#913

Post by LaffertyDev »

DirkH wrote: post #875
huitzitziltzin wrote: post #94

“ No other human activity poses this level of danger.” I really, really disagree with that statement. I don’t think ai models come close to nuclear weapons or to run-of-the-mill, everyday carbon emissions in terms of danger to humanity. What’s the most dangerous thing that’s happened with an LLM so far? (This question is serious - maybe I don’t know the right examples.) Example 1: I’m aware of a small number of peopl…

This is evidence of example 2. The worst parts of it are infohazardous. Organizations that work at the intersection of biosecurity and AI have strong NDAs and the like. Or so I understand speaking to friends in the space. There was also a paper years ago that showed a tweaked model coming up with 20k novel pathogens each deadly to humans in some number of hours Either way, the biorisk has orders of magnitude more evi…

> There was also a paper years ago that showed a tweaked model coming up with 20k novel pathogens each deadly to humans in some number of hours

> If you disagree with it, where do you disagree with it?

How would a malicious actor:

1. Successfully turn these 20k pathogens into actual reproducing viruses / bacteria / etc (in a lab)

2. Successfully turn that lab prototype into something weaponized (i.e. as a bio-terrorism drop in NYC)

And then

3. How is this materially different from today? If we can produce these pathogens in a lab and they can be successfully turned into a weaponizable bioweapon, what is AI accelerating?

Re: I resigned from Anthropic today

#914

Post by philipallstar »

gpm wrote: post #299
ls612 wrote: post #216

Earlier quoted context omitted.

Historically it almost always takes actual fatalities rather than near misses to ground an aircraft, and aviation is famous for its obsession with safety compared to other industries.

Airplanes have pretty bounded damage. Generally you kill at most a few hundred people. Even weaponized a few thousand. This is a risk profile that allows risk taking with near misses and waiting until something goes wrong to fix it (though doing so is rightfully uncomfortable and frequently unethical). The people worrying about AI risk are worrying about "it goes wrong once and kills billions of people". That's not a…

> The people worrying about AI risk are worrying about "it goes wrong once and kills billions of people".

This is the new "nuclear weapons will ignite the atmosphere".

Re: I resigned from Anthropic today

#915

Post by senordevnyc »

treszkai wrote: post #893
Aeolun wrote: post #709

Earlier quoted context omitted.

That's basically the reasoning behind MAD, and it checks out? See what happened to any nation that ever gave away their nukes.

Except in the case of nuclear weapons, humans have their hands on the launch codes, and the nukes are not going to launch themselves. In the case of extremely capable AI, nobody has their hands anywhere close, because nobody is able to – nor willing to – supervise a thousand agents running in parallel, when they usually get shit done. Except eventually they are so good at getting things done that they disregard the v…

This is true, but does it change the game theory / arms race aspect of it?

Re: I resigned from Anthropic today

#916

Post by SepiaSapient »

danaris wrote: post #515
epihelix wrote: post #304

Earlier quoted context omitted.

Yes, but the really weird thing is that they seem to: a) believe that what they're creating is a basilisk, and b) keep trying harder to do this while staring right at it I think they're very deluded about (a) -- but if they do actually believe this (and it really seems like a decent proportion of Anthropic truly does), then why keep doing (b)? That seems to be why this individual resigned, but I'm surprised it's not…

With Roko's Basilisk, if you believe in it, the most rational thing to do is to put forth every effort to bring it into being. Because if you don't, then you will be one of its targets when it does, inevitably, come into being. (I am not a basilisk believer. I think this is all absolute horseshit. But to understand someone's motivations, one must think like them.)

>Because if you don't, then you will be one of its targets when it does, inevitably, come into being.

Only if you don't spend 5 minutes thinking about the Self as a concept. Resurrecting someone by perfect simulation is a cool plot device for a novel like Altered Carbon (which ironically is really about wealth inequality) but not a serious hypothetical.

Even if we ignore that, no, super intelligent MechaGrok17 cannot make a perfect copy of you posthumously from your embarrassing Character.AI chats and IoT toilet data, the copy it still wouldn't be YOU. It would (supposedly) think and behave like you, but for all purposes it would be the Basilisk torturing a future stranger, or your descendants. I'm very much a fan of people not being potentiality tortured, so the rational action is to stop the creation Basilisk, or even just delay it so you aren't around for it. There's a bunch of other big assumptions that makes Roko's Basilisk really stupid, but they are unimportant. It's just theory crafting, like talking about the lore of video games or marvel comics. It doesn't help you understand their real motivations.

Roko's Basilisk is just recontextualized protestant Hell in a sci-fi coat of paint. It's the childhood anxieties of American nerds about hell and sin. The problem is that they are still culturally protestant. God doesn't exist or he's hands off, but in their minds there's still a open position for the wrathful and proactive God. If you think that you actually can create God for realsies, your very rational™ prediction of its behavior will coincidentally line up with your childhood fears.

Why is the spooky AI mad at you? Because you didn't worship it correctly.

Why do you obey it? Because if I don't it will torture me for eternity.

How will it do that? God is omnipotent, don't think about it.

Why don't try to stop it's creation? “I am the Alpha and the Omega,” says the Lord God, “who is, and who was, and who is to come, the Almighty.”

All the scifi scaffolding and stuff like acausal trade is paint over Timmy's first panic attack on Sunday school 30 years ago.

Re: I resigned from Anthropic today

#917

Post by igor47 »

brookst wrote: post #183
underyx wrote: post #181

Earlier quoted context omitted.

"So you think in 3 years AI is going to solve longstanding math problems because it was used to write some coherent sentences?" — people with the same amount of foresight in 2023

Are you saying at anything that can solve longstanding math problems necessarily has the means, motive, and capability to kill 8 billion people in just 3 years?

No dude, it was an analogy. Sorry to pick on you, you're just another ignorant uninformed take in this thread, but come on. This technology is progressing at an insane rate, the shit it's doing now is science fiction from just a year or two ago, and people just keep on moving the goal posts, it's maddening. The technology is unpredictable and that in itself has risks. You must realize that the window of possibility is widening the further into the future we go! Can you pick people with epistemology you trust and see if there's anything you can learn in this moment? Can you try to challenge your own ideas instead of retreating into comfortable certitude? There are real risks here, they are worth taking seriously, and serious people are doing so!

Re: I resigned from Anthropic today

#918

Post by philipallstar »

SupremumLimit wrote: post #798
philipallstar wrote: post #786

Earlier quoted context omitted.

> Capitalism cannot do otherwise. I really think this sort of claim needs to stop. Capitalism, like "patriarchy" is not a force in its own right. Capitalism is a system that encourages value creation for one's customers. That's not perfect, but it's still best so far.

"value creation for one's customers" - that is a "not even wrong" level of misunderstanding of the system dynamics of capitalism. The primary driver is generation of _profit_ for owners of the means of production. Many things fall out of that - the tendency to monopolise, for example.

> "value creation for one's customers" - that is a "not even wrong" level of misunderstanding of the system dynamics of capitalism. The primary driver is generation of _profit_ for owners of the means of production. Many things fall out of that - the tendency to monopolise, for example.

I don't see why you're talking like this. It's "not even wrong" in the sense that it's entirely correct. Monarchy and socialism grab money through taxes, and if you think monopolies are bad, wait til you're forced to use the state service that has no motivation to do well because there's no competition. It's not only a monopoly, it's a monopoly that takes your money by force.

Capitalism requires investment and risk and competition to drive value up for customers who can choose. So you need to produce value for your customers to survive, and produce a profit after costs and taxes.

Re: I resigned from Anthropic today

#919

Post by dwaltrip »

isolatedsystem wrote: post #485
emtel wrote: post #440

Earlier quoted context omitted.

Do you think people should continue developing AI up until the point that there is evidence that AI is facilitating biological weapons development? I think we should stop before then. But that necessarily means that there will not be evidence at the time that we stop.

This is rubbish. By that token, computer development is also facilitating biological weapons development. A better MacOS (or Windows, I don't know) leads to better weapons. They should clearly stop developing computers and OSes. Developers of nice test-tubes are also facilitating bioweapons. Your local O-ring manufacturer, your local medical-grade freezer manufacturer etc. are all culpable. The problem is the bioweap…

An OS developer is just as culpable as a researcher directly training AIs that (1) are capable of building bio-weapons and (2) have shown consistent and increasingly severe misalignment? Among other things...

That doesn't make sense.

Re: I resigned from Anthropic today

#920

Post by IAmBroom »

kakacik wrote: post #730
rurban wrote: post #620

Earlier quoted context omitted.

Didn't you follow the robotics advances in the Ukrainian - Russian battlefield? There is now a death zone of 50km, only controlled by drones and automatic weapons.

> There is now a death zone of 50km, only controlled by drones and automatic weapons. If there would be such 50km death zone, front line would not move a nanometer in a year, would it. You yourself contradict above in your next post. No need for being too dramatic, facts are enough here. In reality, frontline is moving constantly albeit by small chunks, russians are advancing a bit, getting beating elsewhere and so o…

There is such a thing as a No Man's Land[0]. It can move, but if you don't dare cross it for even a few days, it's very real. It doesn't have to be glacial, and it currently exists in Ukraine. Breakout pockets don't mean it is porous everywhere.

[0] Ironically, the French phrase - from ground zero for the first known No Man's Land - is "le No Man's Land".

Post reply on HN