Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I resigned from Anthropic today

twitter.com

Post reply on HN
Print view View original

701–710 of 1001 posts

Re: I resigned from Anthropic today

#701

Post by eloisant »

Perseids wrote: post #651
thomascountz wrote: post #416

No other human activity poses this level of danger. I do heed the warnings, but this comes across as detached hyperbole. See: global warming, nuclear weapon development, wealth inequality, war, technology dependence, etc. Also, this has nothing to do with LLMs or computers. Like all things, this is about humans.

Let me argue on a technicality first: None of these are extinction level events. If global warming disrupts 99% of all crop production, the remaining 1% is still plenty enough to sustain a stable, if miserable, population. In fact you just need about 5k people for a stable gene pool[1]. Of the classical threats, only bioweapons got a shot at extinction, but even that is hard, given the (few) remaining truly secluded…

I still have to read a compelling argument on how AI will "extinct" humanity.

Re: I resigned from Anthropic today

#702

Post by ozozozd »

_puk wrote: post #385
dansquizsoft wrote: post #358

Earlier quoted context omitted.

This is complete fantasy, though I would be interested in reading a book about this.

It's the bobiverse by Dennis E Taylor. https://en.wikipedia.org/wiki/Dennis_E._Taylor

_It’s not just fantasy, it’s plagiarism._

Re: I resigned from Anthropic today

#703

Post by Aeolun »

Mizza wrote: post #700

It's as if two private companies are each building increasingly large nuclear bombs, both saying they'd love to stop but it would be unsafe to let any one company be in control of the nukes.

That's basically the reasoning behind MAD, and it checks out? See what happened to any nation that ever gave away their nukes.

Re: I resigned from Anthropic today

#704

Post by lumost »

Mizza wrote: post #700

It's as if two private companies are each building increasingly large nuclear bombs, both saying they'd love to stop but it would be unsafe to let any one company be in control of the nukes.

While simultaneously trying to own all economic activity derivable from labor... it's hard to see the argument as anything but disengenuous.

Re: I resigned from Anthropic today

#705

Post by pizza234 »

thomascountz wrote: post #416

No other human activity poses this level of danger. I do heed the warnings, but this comes across as detached hyperbole. See: global warming, nuclear weapon development, wealth inequality, war, technology dependence, etc. Also, this has nothing to do with LLMs or computers. Like all things, this is about humans.

All the progress up to now has had one thing in common: the degree of understanding and control that humans have. Even when it comes to global warming, we understand the causes and can act on them.

AI is an exception. We're already losing understanding (although we never fully had it in the first place), and we're losing control (see jailbreaks/hacks etc.).

We're still far from the doomsday scenario because AI 1. is not developed enough yet, 2. can't really replicate itself, and 3. has very limited means to act.

But: 1. its intelligence is developing quickly, 2. hardware capable of "hosting" it is slowly being developed, and 3. it will likely gain access to increasingly powerful means of acting in the physical world (this is already happening in the digital world).

Once AI becomes intelligent enough (it doesn't strictly need to be AGI), has the substrate on which to exist, and has more means to act, we'll essentially have a new species on Earth - one more capable than humans and potentially determined to kill humans at some point.

And for those who believe airgapping is a valid safety measure: read https://xkcd.com/538. AI will be able to threaten and manipulate people.

Re: I resigned from Anthropic today

#706

Post by darnfish »

PowerElectronix wrote: post #639

I fail to see how a machine that can hack everything can't also patch everything and make the system unhackable. A nuke, a virus, whatever... The knowledge is nonlonger the bottleneck, it's the tools and materials. Also, I'm extremely skeptical about AI becoming even close to a child in intelligence.

If you are attacking a system, you can try 100 things. If one thing works in getting access, you succeed.

If you are defending a system, you must defend against all 100 things. If one thing makes it through, you lose.

Re: I resigned from Anthropic today

#707

Post by kakacik »

xienze wrote: post #646
thomascountz wrote: post #454

Earlier quoted context omitted.

This is not correct. Please go through each one again, taking special note of global warming and nuclear weapon development .

Humans are voluntarily driving themselves to extinction through sub-replacement birth rates worldwide. That's going to play out far quicker than global warming or pretty much anything other than every country on earth launching nukes at each other will.

Hard disagree here, population growth is still happening and will keep happening for some time, maybe step out of your bubble if you don't see that.

Africa is still growing massively for example, world is not just western civilization. Sure at that rate and incerase of living conditions for everybody maybe in 1000 years population will be smaller, but its not that hard to fix if wanted - people used to have 10-15 kids as default.

Re: I resigned from Anthropic today

#708

Post by nairboon »

fergie wrote: post #699
matherial wrote: post #40

Unlike most other commenters, I applaud him for acting on his principles. If you sincerely believe that, of course you should act. You might not succeed, but your voice might be the one that tips the scales and starts a broader movement. This doesn't mean I agree with him. The fears of doomsday caused by rapid takeoff have been with us since day 1 and the mechanism is always basically "AI invents magic that sets it f…

It should be obvious that AI is already capable of inducing humans to think or do things, and that this power is only going to increase over time.

It's not that much different from the effect of social media or the power held by other tech giants, who can control your exposure to particular information (media, search engines).

Re: I resigned from Anthropic today

#709

Post by frotaur »

jdthedisciple wrote: post #548
michaelhoney wrote: post #398

I could imagine a 2027 AI swarm coordinating to eg hold the US and Russian and Chinese governments to ransom, by demonstrating some small thing (turning US army base freezers to defrost) and threatening to do something big unless some conditions were met – conditions which would be good or bad for the world depending on your POV. This happens either either because they were tasked to to it by (malicious or well-meani…

And then we pull the plug, after holding our breath for 10 seconds,. Then life resumes normally..

In most AI takeover scenarios, if you take as a premise that the AI has human or above-human intelligence, and that it is misaligned, it is obviously aware of the pull the plug possibility.

Therefore, as you would if you were in its position, it will plan around it. For instance, by acting perfectly aligned for 2/3 years, continuing the improvement of its capabilities while being deployed in ever more systems.

Once it's confident it can act with high probability of success, it would then turn on us. This phenomenon is called 'treacherous turns'.

Any scenario in which you assume you have ASI or AGI but also find a 2-sentence way to foil the AI's plan is inconsistent, as the AI will also have thought of this failure mode.

Post reply on HN