Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I resigned from Anthropic today

twitter.com

Post reply on HN
Print view View original

711–720 of 1001 posts

Re: I resigned from Anthropic today

#711

Post by rickdeckard »

It's quite incredible to consider that all these concerns already existed years ago,

but now that a handful of companies working in AI managed to enslave the entire financial system over the past year, their continued work is protected from larger governance for concerns it could tank the stock-market, affect personal investments, pensions or cause disadvantages in an arms-race with other countries.

IF there is an inherent danger (which I believe is the case at least on economic levels, work displacement, poverty,...), it is now basically ensured that nothing will be done to reign those companies in, until maybe two AI's engage in an open war with civilian casualties...

Re: I resigned from Anthropic today

#712

Post by shafyy »

OpenAI has been saying since the first version of ChatGPT that it's too dangerous to release because it will end humanity. Yes, LLMs are an impressive technology, but let's be real: the improvements in the recent months have been slowing down, and it's clear that we are nearing a plateau of what this particular tech can do. Sure, tooling and harnesses etc. is improving, but clearly this dude has drank too much of the Kool-Aid.

Re: I resigned from Anthropic today

#713

Post by nananana9 »

copperx wrote: post #358
sreekanth850 wrote: post #345

I think people here still evaluating the model in isolation. It is the combination that matters, model + strong harness + tools + long running autonomy + memory + retries + parallel agents + code execution + credentials + access to real systems. The model does not need to be perfect. If it fails 30% of the time, the harness can retry, verify, branch, use another agent and keep going. I don't think we necessarily need…

> an extremely capable harness and enough access. Give enough access to a fuzzer and it's exactly as dangerous as an LLM. LLMs don't even have a moat in this domain.

Technically. What's would technically be even more dangerous is running this shell script:

  head -c 200000 /dev/urandom > agi && chmod +x agi && ./agi
In reality, the fuzzer definitely has no agenda, and these random bytes probably don't. The LLM definitely does, and even publicly available models, programmed ot "do what the user wants, act according to the anthropic moral codex" will take some pretty absurd actions in attempting to accomplish a poorly worded request.

Re: I resigned from Anthropic today

#714

Post by chrisjj »

jdthedisciple wrote: post #540
michaelhoney wrote: post #393

I could imagine a 2027 AI swarm coordinating to eg hold the US and Russian and Chinese governments to ransom, by demonstrating some small thing (turning US army base freezers to defrost) and threatening to do something big unless some conditions were met – conditions which would be good or bad for the world depending on your POV. This happens either either because they were tasked to to it by (malicious or well-meani…

And then we pull the plug, after holding our breath for 10 seconds,. Then life resumes normally..

Not normally if essential services rely on the same plug.

Re: I resigned from Anthropic today

#715

Post by thomascountz »

eoverride wrote: post #619
thomascountz wrote: post #409

No other human activity poses this level of danger. I do heed the warnings, but this comes across as detached hyperbole. See: global warming, nuclear weapon development, wealth inequality, war, technology dependence, etc. Also, this has nothing to do with LLMs or computers. Like all things, this is about humans.

Nuclear weapons, climate change.... The AI doom hyperbole is unhinged.

[flagged]

Re: I resigned from Anthropic today

#716

Post by vlyan »

par1970 wrote: post #640
vlyan wrote: post #630

Earlier quoted context omitted.

yes, exactly, that's what they claimed that incoherent gibberish generator to be capable of.

So you aren't claiming that they said GPT-2 was dangerous in the sense that it could disempower humanity, kill all humans, etc. You are just claiming that OpenAI execs said that GPT-2 might "generate misleading news articles, impersonate others online, automate the production of abusive or faked content to post on social media, automate the production of spam/phishing content." Then, what is unreasonable or bad about…

that it was bullshit and they knew it. GPT-2 wasn't capable of anything other than imitating a stroke victim.

Re: I resigned from Anthropic today

#717

Post by frm88 »

pseudony wrote: post #500
frm88 wrote: post #484

Earlier quoted context omitted.

Example 8: like in this comment https://news.ycombinator.com/item?id=49619884 but isolate synchronised megahack on banking that adds one more zero to the US debt and all dependent systems and banking during runtime. Let the world's financial system take it from there.

I think US fiscal policy and adventurism will manage this on its own. :)

But why wait? We can has global financial crash now:)

Re: I resigned from Anthropic today

#718

Post by ergo14 »

Schlagbohrer wrote: post #680
tyrabound wrote: post #529

Earlier quoted context omitted.

That is the crux. The big problem is not AGI, it is AGI controlled by, “raised” by the people that control the USA, the predominant psychology of the tech industry culture (“move fast, break things” ring a bell? How about all the “violate hundreds of laws, bribe the politicians to prevent consequences later” type of mentality?). Frankly, we, our culture, this fake America that is parasitized by psychologically narcis…

Capitalism will always promote such people into positions of power, because to be good at capitalism one must have zero empathy including empathy or concern for future generations. Capitalism cannot do otherwise.

That's a lot of absolutes. And there is no defensible proof, just anecdotal evidence. I've been in a few companies from EU and US - and in my experience it is not ture. However after working for US companies I can understand why one would come to your conclusion, however the world is bigger than our bubbles.

Re: I resigned from Anthropic today

#719

Post by sicher »

imhoguy wrote: post #653
onewayfunction wrote: post #178

I'm pretty baffled by the degree of skepticism expressed here in response to some of Jacob's claims. After the events of the summer it feels like it takes a lack of imagination to not see a few plausible routes to disaster. It may be reasonable to believe these outcomes are not very likely or that we can stop before going too far (I tend to disagree). But I can't imagine doubting that the capabilities will soon be th…

Even if AI won't be self-aware and superintelligent agent, its problem is that it gives exponential control and power capabilities to one person bad actor who can simply prompt AI without any guardrails with access to sensitive industrial infrastructure which can disrupt lifes and ecosystems in the real world: - virus research labs - nuclear labs - chemical factories - bank records - land registers - power plants - w…

Yes, it's truly scary. It's not hard to imagine a small doomsday cult releasing 100+ nasty viruses at selected spots around the globe.

Re: I resigned from Anthropic today

#720

Post by vidarh »

eloisant wrote: post #685
Perseids wrote: post #636

Earlier quoted context omitted.

Let me argue on a technicality first: None of these are extinction level events. If global warming disrupts 99% of all crop production, the remaining 1% is still plenty enough to sustain a stable, if miserable, population. In fact you just need about 5k people for a stable gene pool[1]. Of the classical threats, only bioweapons got a shot at extinction, but even that is hard, given the (few) remaining truly secluded…

I still have to read a compelling argument on how AI will "extinct" humanity.

The most compelling argument to me is "accidentally", due to AI that is made blind to consequences or don't care because it's geared towards a single goal (see e.g. the paperclip maximizer).

We could ask if it is possible to end up with an AI that is smart enough to destroy humanity and at the same time still blind enough to consequences and/or callous enough to do it, but then again we have plenty of examples of humans who have been smart enough to do enormous damage and willing enough to do it.

I don't particularly worry about this, as I believe we'll get plenty of smaller scale warnings if/when we're at a point where those kinds of alignment risks might become a problem, but it is a risk we also shouldn't be blind to.

Post reply on HN