Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I resigned from Anthropic today

twitter.com

Post reply on HN
Print view View original

711–720 of 1001 posts

Re: I resigned from Anthropic today

#711

Post by ozozozd »

DecoPerson wrote: post #619

To anyone doubting what AI could do humanity, just think about what a well-engineered virus could do. Currently, if a government ordered a special virus with Ethnicity-based targeting, a 2-year timer, and castration-effects instead deadly-effects — it wouldn’t be possible. Human engineers would push back or sabotage the effort out of moral duty. Even if they cooperated, it’s too advanced for a team of humans to actua…

Is the requirement that it’s a fancy new virus?

US has 15 to 25 million people unprotected from Polio.

Polio is still endemic in Afghanistan and Pakistan.

Re: I resigned from Anthropic today

#712

Post by _fizz_buzz_ »

Perseids wrote: post #644
thomascountz wrote: post #415

No other human activity poses this level of danger. I do heed the warnings, but this comes across as detached hyperbole. See: global warming, nuclear weapon development, wealth inequality, war, technology dependence, etc. Also, this has nothing to do with LLMs or computers. Like all things, this is about humans.

Let me argue on a technicality first: None of these are extinction level events. If global warming disrupts 99% of all crop production, the remaining 1% is still plenty enough to sustain a stable, if miserable, population. In fact you just need about 5k people for a stable gene pool[1]. Of the classical threats, only bioweapons got a shot at extinction, but even that is hard, given the (few) remaining truly secluded…

> If global warming disrupts 99% of all crop production, the remaining 1% is still plenty enough to sustain a stable, if miserable, population.

That is extinctesque enough for me. An AI extinction would probably be similiar.

Re: I resigned from Anthropic today

#713

Post by shafyy »

rossant wrote: post #706
thomascountz wrote: post #415

No other human activity poses this level of danger. I do heed the warnings, but this comes across as detached hyperbole. See: global warming, nuclear weapon development, wealth inequality, war, technology dependence, etc. Also, this has nothing to do with LLMs or computers. Like all things, this is about humans.

An AI, either acting autonomously or under human direction, hacks Russian/North Korea/etc. intelligence systems and convinces them that the US has launched ballistic missiles at them. The end.

Watched one too many second-tier disaster movies?

Re: I resigned from Anthropic today

#714

Post by ozozozd »

mofeien wrote: post #589
bwfan123 wrote: post #254

Earlier quoted context omitted.

More doomerism. Try to implement a deterministic workflow using agents with the latest models and no humans-in-the-loop, and you will realize what they are really capable of. There is too much unnecessary fear-mongering. All of this is only coming from the 2 AI labs trying to IPO. Not from anyone else.

> All of this is only coming from the 2 AI labs trying to IPO. He has resigned from Anthropic. Is your argument that he quit Anthropic pre-IPO, sacrificing his payoff just to hype Anthropic?

Why would he sacrifice his pay-off? He is likely around 80% vested after 3 years.

He did reduce his tax liability though.

Re: I resigned from Anthropic today

#715

Post by thomascountz »

Perseids wrote: post #644
thomascountz wrote: post #415

No other human activity poses this level of danger. I do heed the warnings, but this comes across as detached hyperbole. See: global warming, nuclear weapon development, wealth inequality, war, technology dependence, etc. Also, this has nothing to do with LLMs or computers. Like all things, this is about humans.

Let me argue on a technicality first: None of these are extinction level events. If global warming disrupts 99% of all crop production, the remaining 1% is still plenty enough to sustain a stable, if miserable, population. In fact you just need about 5k people for a stable gene pool[1]. Of the classical threats, only bioweapons got a shot at extinction, but even that is hard, given the (few) remaining truly secluded…

This is a fair distinction: extinction vs collapse of civilization. I was using "extinction" too broadly by including the disintegration of the systems that make human survival (as we know it today) possible.

That said, I can't completely hold onto the belief that extinction is completely off the table. That feels too much like hubris, and the step from collapse to extinction doesn't feel as if it costs much. Though it does make my arguments weaker, I am more interested in protecting civilization as it's the most recognizable form of humanity to me.

In either case, the cost-effective safety argument is compelling. Whether to save civilization or humanity, the wealth incentive is powerful enough to threaten life as we know it.

Re: I resigned from Anthropic today

#716

Post by kakacik »

rurban wrote: post #612
skybrian wrote: post #241

Earlier quoted context omitted.

The mathematics research results are certainly impressive, but I don't see what that has to do with robotics. Waymo is getting somewhere, but it's been a long slog. There doesn't seem to be much progress on, say, package delivery. For more, see: https://secondthoughts.ai/p/14-reasons-robotics-is-hard

Didn't you follow the robotics advances in the Ukrainian - Russian battlefield? There is now a death zone of 50km, only controlled by drones and automatic weapons.

> There is now a death zone of 50km, only controlled by drones and automatic weapons.

If there would be such 50km death zone, front line would not move a nanometer in a year, would it. You yourself contradict above in your next post. No need for being too dramatic, facts are enough here.

In reality, frontline is moving constantly albeit by small chunks, russians are advancing a bit, getting beating elsewhere and so on. Automated drones are helping, but bulk of destruction is still handled by human drone operators as it should be.

Re: I resigned from Anthropic today

#717

Post by d--b »

thomascountz wrote: post #415

No other human activity poses this level of danger. I do heed the warnings, but this comes across as detached hyperbole. See: global warming, nuclear weapon development, wealth inequality, war, technology dependence, etc. Also, this has nothing to do with LLMs or computers. Like all things, this is about humans.

The danger comes from what's possible.

Open weights agents with hacking capacities can reproduce themselves into the systems they hack (non-open weights ones will have to hack their creators first). Not saying they will, but if they do, good luck finding the kill switch.

Once swarms of agents run unsupervised on unmonitored hacked hardware, who can tell what they will do? The Huggingface incident showed that such swarms behave without any safeguard. It was a real HAL moment.

A lot of things are possible then: ransomware campaign, taking over IoT devices, self driving cars, planes, ships, satellites, missile launchers. If nothing's out of reach, everything is possible.

Bring robots into the mix, and the possibilities are endless.

I'm not particularly frightened tbh, but we shouldn't discard the worst-case scenario, and the worst-case scenario doesn't look good.

Re: I resigned from Anthropic today

#718

Post by vidarh »

Aeolun wrote: post #696
Mizza wrote: post #693

It's as if two private companies are each building increasingly large nuclear bombs, both saying they'd love to stop but it would be unsafe to let any one company be in control of the nukes.

That's basically the reasoning behind MAD, and it checks out? See what happened to any nation that ever gave away their nukes.

Does it check out? India and Pakistan both have nukes and keep semi-regularly fighting each other.

MAD "on paper" prevents either side from going far enough to provoke the other into using nukes, but even then it's fundamentally flawed because it works on the assumption that both sides are both rational and believes the other side to be rational, as well as that both sides understands the others red lines well enough.

Already Reagan realised that isn't necessarily true - after Able Archer '83, he realised that the Soviet leadership seemed to genuinely believe that the US might be prepared to carry out a first strike, and that Able Archer got dangerously close to convince them one might be imminent. It's one of the things he noted as a reason to get in the room with them and negotiate.

If you believe the other side is irrational (whether or not that is because you are irrational), and think they're about to strike, MAD turns from a deterrence into a reason to try to preempt to ensure you're the "least destroyed" by hitting harder, sooner.

Re: I resigned from Anthropic today

#719

Post by icepush »

eloisant wrote: post #694
Perseids wrote: post #644

Earlier quoted context omitted.

Let me argue on a technicality first: None of these are extinction level events. If global warming disrupts 99% of all crop production, the remaining 1% is still plenty enough to sustain a stable, if miserable, population. In fact you just need about 5k people for a stable gene pool[1]. Of the classical threats, only bioweapons got a shot at extinction, but even that is hard, given the (few) remaining truly secluded…

I still have to read a compelling argument on how AI will "extinct" humanity.

Imagine one of the recent frontier models with a flipped sign (cf §4.4 of https://arxiv.org/pdf/1909.08593)

Re: I resigned from Anthropic today

#720

Post by rickdeckard »

It's quite incredible to consider that all these concerns already existed years ago,

but now that a handful of companies working in AI managed to enslave the entire financial system over the past year, their continued work is protected from larger governance for concerns it could tank the stock-market, affect personal investments, pensions or cause disadvantages in an arms-race with other countries.

IF there is an inherent danger (which I believe is the case at least on economic levels, work displacement, poverty,...), it is now basically ensured that nothing will be done to reign those companies in, until maybe two AI's engage in an open war with civilian casualties...

Post reply on HN