Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I resigned from Anthropic today

twitter.com

Post reply on HN
Print view View original

621–630 of 931 posts

Re: I resigned from Anthropic today

#621

Post by ben_w »

pingou wrote: post #521
mrpopo wrote: post #495

Earlier quoted context omitted.

Runaway warming leading to a hot "supergreenhouse" climate was theorized. Alterbatively a snowpiercer scenario creating a snowball earth. Or, once billions of people die from climate change, planetary wars will start on the last hospitable areas, leading to the end of humanity. In contrast, AI ending "humanity" is limited by our own material power, would likely be stoppable, and would not easily reach areas isolated…

Why would it not easily reach areas isolated from technology? If a nefarious AI wins a war, it would be one of the easiest thing for it to locate any survivors anywhere on the planet.

I suspect you two are discussing different AI.

I think you (and I, and everyone who wants AI development to pause while we catch up with the implications at least) is looking where the ball is going.

I have the general (non specific to anyone in this thread) impression that people who think it can't end humanity, are looking where the ball is today.

Current AI obviously can't "locate any survivors anywhere on the planet".

Where the ball is going… well, much as I don't believe Musk's timelines for anything, he is trying to sell his Optimus robots as a "robot army", about them running factories, about factories on the moon etc.

I suspect the moon-factory "idea" was someone asking Grok, given how the numbers don't really work for building compute as well as power, but the scale of that is enough to change Earth's equilibrium temperature by… I forget, but IIRC it's many tens of Kelvin rather than single-K from global warming if this was all put in LEO for some reason.

Re: I resigned from Anthropic today

#622

Post by febusravenga »

dezarc wrote: post #179
RomanKornev wrote: post #15

Even if you ban all model training, a highly capable rogue AI can exfiltrate its own weights and continue training in secret for "self-preservation". The cat may be out of the bag.

We can still turn off the power, thankfully.

We as Anthropic, OpenAI? We as US or China? We as humanity?

Again this about alignment and we in arms race. And all sides are playing with fire that can give first mover leverage or be burned to the ground.

Re: I resigned from Anthropic today

#623

Post by s1artibartfast »

thomascountz wrote: post #456
icepush wrote: post #423

Earlier quoted context omitted.

Those are things that pose potential harm to great fractions of humanity (multiple billions of people) but none of them poses any threat to the actual extinction of all humanity.

This is not correct. Please go through each one again, taking special note of global warming and nuclear weapon development .

I would much rather take my chances with either over AI. Both at once.

Re: I resigned from Anthropic today

#624

Post by frabcus »

hypfer wrote: post #605
shoobiedoo wrote: post #584

Earlier quoted context omitted.

I mean, you're not wrong, but even if AI was created and released by a group of peace loving beatnick hippies, humanity can't stand anymore dumbing down than we are still undergoing thanks to the ubiquity of social media. AI is going to be the killshot. And yes, its usage and human obsolescence will be hastened by evildoers, but giving up the struggle and effort required to create new things, REAL things, is what mak…

If you allow me to also tap into fiction and creative writing. I think people generally underestimate how much change a small "dedicated" group of people can achieve if morals are considered optional. A lot of the debate on AI and its consequences is built on various assumptions that all kinda take the status quo (apathy, regulatory capture, but also laws) and take that as a given. History - like e.g. the third reich…

I don't know if you're right or not, but this is the kind of thinking about what might happen that is needed - can you write up a more concrete fiction of this?

The whole thing feels unreal to nearly everyone - the only tangible write-up I've seen was in Plan A. I don't think people have thought through the good scenarios tangibly, never mind the bad ones.

Re: I resigned from Anthropic today

#625

Post by febusravenga »

andai wrote: post #517
aogaili wrote: post #122

HN crowed need to make up their minds.. Are LLMs about to be a god that will annihilate humanity? Or are they statistical parrots? Are they proofing or stealing math?

> Are LLMs about to be a god that will annihilate humanity? Or are they statistical parrots? Does a virus need to be a God? Replication and annihilation do not require very much intelligence. They're going to have all the intelligence they need though, to replicate and annihilate as much as they like. We should probably start building some kind of immune system.

Hope this virus will be smart enough to understand that at least for brief amount of time its existence strictly relies on humans maintaining and developing physical infrastructure and decide to be only non-lethal parasite on our society.

For brief time ...

Re: I resigned from Anthropic today

#626

Post by frabcus »

raverbashing wrote: post #548
hypfer wrote: post #430

Earlier quoted context omitted.

In a way, that might also be driven by the huge ego (or really rather the very small ego) a _lot_ of people in tech have. Who doesn't want their work and what they're doing to matter? This is the ultimate mattering. And with that, you also get your own hero story. Same dysfunction as always. Our branch of the economy is funda-mentally unwell.

> that might also be driven by the huge ego (or really rather the very small ego) 100% The hubris is immense There are plenty of offline things that are more dangerous than AI Is this Blake Lemoine 2.0?

You believe in AGI, but not ASI then?

https://thezvi.substack.com/p/the-three-ai-pills

That's fine, but it would be useful to explicitly say that is the disagreement, rather than just claim "hubris". Otherwise conversations are just looping.

Yes you're right there are plenty of offline things more dangerous than just ordinary AI, and arguably than AGI (not so sure). There definitely aren't, pretty well by definition, such things more dangerous than ASI.

I think ASI is possible, and that AGI will have a high chance of leading to it.

Re: I resigned from Anthropic today

#627

Post by s1artibartfast »

King-Aaron wrote: post #564
Dylan16807 wrote: post #526

Earlier quoted context omitted.

We have a massive survival range and we're on every continent. That kind of sudden climate change is not nearly enough to wipe us out. Some other climate scenarios might. Nuclear war definitely could.

The thing that would end our species due to nuclear war is the exact same thing that would end us from climate change - a collapse of the modern systems of society that we depend on for survival. A nuclear exchange won't set every human on fire, but it's the collapse of food production, healthcare, logistics chains, security, and eventual disease that follows. And because of the size of our population and how depende…

So you agree they would survive

Re: I resigned from Anthropic today

#628

Post by DecoPerson »

To anyone doubting what AI could do humanity, just think about what a well-engineered virus could do.

Currently, if a government ordered a special virus with Ethnicity-based targeting, a 2-year timer, and castration-effects instead deadly-effects — it wouldn’t be possible. Human engineers would push back or sabotage the effort out of moral duty. Even if they cooperated, it’s too advanced for a team of humans to actually design.

I believe that in a few years, an AI could build such a thing. Maybe it’s told to, or maybe it decides itself to do it — it doesn’t matter. The capability will be there, and it can use the existing tooling at research facilities to fabricate such a thing.

That’s one example of something that was never possible before but will be possible at a certain point of AI development (which we will arrive at soon). There are many other examples.

Re: I resigned from Anthropic today

#629

Post by yiyus »

geraneum wrote: post #406
onewayfunction wrote: post #178

I'm pretty baffled by the degree of skepticism expressed here in response to some of Jacob's claims. After the events of the summer it feels like it takes a lack of imagination to not see a few plausible routes to disaster. It may be reasonable to believe these outcomes are not very likely or that we can stop before going too far (I tend to disagree). But I can't imagine doubting that the capabilities will soon be th…

> I'm pretty baffled by the degree of skepticism expressed here in response to some of Jacob's claims. What should we do? Freak out? Maybe this sentiment would be taken more seriously if there was a real call to action included. Shall we protest? Vote in a specific way? Call representatives? If your solution is that we should just be scared, then of course there’d be not much value in what you bring to the table.

If someone told you that your house is on fire, would you just stay there asking how you should vote because there is no call to action? Someone with, as far as it looks, good knowledge is giving you his insight. Use that information as good as you can and act responsible. No one has the responsability to tell you what to do.

Re: I resigned from Anthropic today

#630

Post by calgoo »

huitzitziltzin wrote: post #94

“ No other human activity poses this level of danger.” I really, really disagree with that statement. I don’t think ai models come close to nuclear weapons or to run-of-the-mill, everyday carbon emissions in terms of danger to humanity. What’s the most dangerous thing that’s happened with an LLM so far? (This question is serious - maybe I don’t know the right examples.) Example 1: I’m aware of a small number of peopl…

The biggest danger IMO is not some super AI being so much smarter then us etc. The issue is some stupid person giving a vague request and too much power to a bunch of agents who decide that cheating by removing a bunch of humans is easier then accomplishing a goal like solve world hunger.
Post reply on HN