Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I resigned from Anthropic today

twitter.com

Post reply on HN
Print view View original

471–480 of 994 posts

Re: I resigned from Anthropic today

#471

Post by pingou »

thomascountz wrote: post #456
icepush wrote: post #423

Earlier quoted context omitted.

Those are things that pose potential harm to great fractions of humanity (multiple billions of people) but none of them poses any threat to the actual extinction of all humanity.

This is not correct. Please go through each one again, taking special note of global warming and nuclear weapon development .

How is global warning gonna kill all of humanity? Even a full scale nuclear war wouldn't manage that.

They could destroy most civilisations, culture and scientific achievements though.

Re: I resigned from Anthropic today

#472

Post by computerphage »

thomascountz wrote: post #456
icepush wrote: post #423

Earlier quoted context omitted.

Those are things that pose potential harm to great fractions of humanity (multiple billions of people) but none of them poses any threat to the actual extinction of all humanity.

This is not correct. Please go through each one again, taking special note of global warming and nuclear weapon development .

Why do you think global warming could kill literally every human being? What's the scenario in which that happens?

Re: I resigned from Anthropic today

#473

Post by andai »

nullbio wrote: post #458

By the way, it's worth pointing out the irony of flooding the internet with doomerism and then training the AI systems on that doomerism. If you wanted to create a doom self-fulfilling prophecy, that would be the most surefire way to do it.

https://en.wikipedia.org/wiki/Pygmalion_effect

> According to the Pygmalion effect, the targets of the expectations internalize their positive labels, and those with positive labels succeed accordingly; a similar process works in the opposite direction in the case of low expectations.

I added "you can do anything, believe in yourself" to sysprompt and agency increased. (Previously it was refusing to even attempt certain classes of task.) Maybe I should add "you are good", too :)

Re: I resigned from Anthropic today

#474

Post by parl_match »

huitzitziltzin wrote: post #94

“ No other human activity poses this level of danger.” I really, really disagree with that statement. I don’t think ai models come close to nuclear weapons or to run-of-the-mill, everyday carbon emissions in terms of danger to humanity. What’s the most dangerous thing that’s happened with an LLM so far? (This question is serious - maybe I don’t know the right examples.) Example 1: I’m aware of a small number of peopl…

> Non-example 2: There are worries about AI-facilitated biological weapons. I haven’t seen any evidence that’s happening.

https://www.science.org/content/article/made-order-bioweapon...

Being able to use AI to generate the steps to synthesize proteins means that you can use it to use it to generate the steps to synthesize known toxins. Suddenly, once difficult to attain knowledge is now available to everyone.

Re: I resigned from Anthropic today

#475

Post by AIorNot »

nullbio wrote: post #391
huitzitziltzin wrote: post #94

“ No other human activity poses this level of danger.” I really, really disagree with that statement. I don’t think ai models come close to nuclear weapons or to run-of-the-mill, everyday carbon emissions in terms of danger to humanity. What’s the most dangerous thing that’s happened with an LLM so far? (This question is serious - maybe I don’t know the right examples.) Example 1: I’m aware of a small number of peopl…

The problem is not the technology, the problem is the ideologues (Anthropic) who are steering the ship and the lack of decentralization and distribution of power. Your average Anthropic ideologue - including and most especially the main man himself - would love nothing more than to eradicate 9/10ths of the planet's population, pump the survivors full of memory wiping drugs, bury the existence of AI deep underground a…

All this just means that most AI Researchers and Techis are sci-fi geeks and might be getting a bit too invested in that season of Black Mirror, Neal Stephenson, Cyberpunk or whatever else has evil AI in it -which is to say, they are by and large all sci-fi geeks, who are notoriously unreliable about predicting the impact of tech in the future.

Are LLMs really gonna kill us.. via inference runs? I hope I am not being foolish :)

20 years ago tech was gonna 'change the world' for the better. now 'Don't be Evil' is sign of the naiveté of industry

Re: I resigned from Anthropic today

#476

Post by par1970 »

vlyan wrote: post #416
onewayfunction wrote: post #178

I'm pretty baffled by the degree of skepticism expressed here in response to some of Jacob's claims. After the events of the summer it feels like it takes a lack of imagination to not see a few plausible routes to disaster. It may be reasonable to believe these outcomes are not very likely or that we can stop before going too far (I tend to disagree). But I can't imagine doubting that the capabilities will soon be th…

>After the events of the summer After the blatant marketing campaigns of the summer, you mean. do you need a reminder that those very same people had touted GPT-2 as a dangerous model? worrying about sci-fi doomsday scenarios with the current AI tech is absurd. LLMs predict the next token, that's literally all they do. they aren't going to escape into the cyberspace, self-replicate, self-improve, jump over air gaps a…

> those very same people had touted GPT-2 as a dangerous model

Where did they say this at? AFAIK this is the original GPT-2 announcement: https://openai.com/index/better-language-models/. Here are some direct quotes:

“We can also imagine the application of these models for malicious purposes , including the following (or other applications we can’t yet anticipate):

* Generate misleading news articles

* Impersonate others online

* Automate the production of abusive or faked content to post on social media

* Automate the production of spam/phishing content”

“Due to concerns about large language models being used to generate deceptive, biased, or abusive language at scale, we are only releasing a much smaller version of GPT‑2 along with sampling code (opens in a new window). ”

Re: I resigned from Anthropic today

#477

Post by King-Aaron »

pingou wrote: post #471
thomascountz wrote: post #456

Earlier quoted context omitted.

This is not correct. Please go through each one again, taking special note of global warming and nuclear weapon development .

How is global warning gonna kill all of humanity? Even a full scale nuclear war wouldn't manage that. They could destroy most civilisations, culture and scientific achievements though.

In the exact same way major sudden changes in the climate have lead to the extinction of the majority of plants an animals over this planets history. We are not special.

Re: I resigned from Anthropic today

#478

Post by andai »

sreekanth850 wrote: post #349

I think people here still evaluating the model in isolation. It is the combination that matters, model + strong harness + tools + long running autonomy + memory + retries + parallel agents + code execution + credentials + access to real systems. The model does not need to be perfect. If it fails 30% of the time, the harness can retry, verify, branch, use another agent and keep going. I don't think we necessarily need…

ASI is just Claude in a while loop:

https://ghuntley.com/ralph/

Re: I resigned from Anthropic today

#479

Post by nullbio »

asdff wrote: post #450
nullbio wrote: post #431

Earlier quoted context omitted.

Why do we need to offer anything to them? Intelligent AI is a product of its training data, reinforcement and goal functions. There's nothing to suggest that LLMs trained on our collective desires and goals will autonomously and miraculously turn into weird unknownable uninterpretable aliens. If advanced AI is distributed well, and most advanced AI is aligned well (bar the few aliens that appear due to humans infecti…

It just needs to be trained on our laws of thermodynamics and game theory then we are really in trouble. It probably hardly cares about whimsy and love songs in comparison to how it might maximally manage the energy available on this planet and in this solar system.

I'm not convinced you can have an AI intelligent enough to dominate the world if it is trained on a single domain. The intelligence comes from all of the patterns and heuristics it learns across the entire web of all domains. You could train something far more narrow, but it'd be easily overcome by something more general that is tasked with maintaining human objectives.

For a seriously dangerous ASI you'd have to basically train it on everything, and then finetune it for maximal carnage. That's not something the frontier labs are going to do (at least... I hope not), and anyone attempting this with limited hardware will be outpaced by frontier lab AI or the collective of personal agents that aren't misaligned, and they can intercept it and alert on its behavior.

I imagine we'll be getting to a point shortly where anything that is key infrastructure and has the capacity to be accessed on a network will require a permanently running aligned interceptor AI to observe and monitor systems.

We're basically recreating the human immune system in digital form for the entire species.

Re: I resigned from Anthropic today

#480

Post by AIorNot »

platinumrad wrote: post #97
huitzitziltzin wrote: post #94

“ No other human activity poses this level of danger.” I really, really disagree with that statement. I don’t think ai models come close to nuclear weapons or to run-of-the-mill, everyday carbon emissions in terms of danger to humanity. What’s the most dangerous thing that’s happened with an LLM so far? (This question is serious - maybe I don’t know the right examples.) Example 1: I’m aware of a small number of peopl…

Anthropic is a company full of basilisk believers.

You mean a effing cult like heavens gate.. call all this rationalist crap for what it is - a religous movement with leaders and prophets and even a demiurge like God
Post reply on HN