Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I resigned from Anthropic today

twitter.com

Post reply on HN
Print view View original

891–900 of 993 posts

Re: I resigned from Anthropic today

#891

Post by OccamsMirror »

shepherdjerred wrote: post #228
slicktux wrote: post #201

Earlier quoted context omitted.

Why are we putting so much weight (no pun intended) on AI companies. At the end of the day the scaled up LLM transformers lack emotion and will… They do as they are told; or more correctly put. They do as they are programmed to do so.

Why are you bringing emotion and will into this? Does something have to have those to be useful or dangerous? > They do as they are told; or more correctly put. They do as they are programmed to do so. _Nobody_ told them to hack Hugging Face. Do you really not understand what is happening?

The training (i.e. all of the red teaming and CTF content they could scrape) told them to.

Re: I resigned from Anthropic today

#892

Post by OccamsMirror »

jenadine wrote: post #321
cure_42 wrote: post #258

Earlier quoted context omitted.

That isn't the correct context. The code running the llm is well understood and the llm is simply the result of that code being executed. It is still a computer doing what it is told. It's just that we told it to use an incredibly large number of probabilities to calculate what series of tokens would have most likely come next after a given series of tokens. There is no hallucination or lie or rogue actions. There's…

I'm order to guess the next token in a love poem, they must understand love. In order to predict the next token in a chess game between grand master, they must master chess. In order to predict the next token in a computer program, they need to be able to program anything. They gain all these abilities in their training. That's what training does. Despite no one programmed them to master chess, or hack into anything.

LLMs are famously bad at Chess. They have no world model and they are not trained to be good at Chess. This is not the amazing point you think that it is.

Re: I resigned from Anthropic today

#893

Post by treszkai »

Aeolun wrote: post #709
Mizza wrote: post #706

It's as if two private companies are each building increasingly large nuclear bombs, both saying they'd love to stop but it would be unsafe to let any one company be in control of the nukes.

That's basically the reasoning behind MAD, and it checks out? See what happened to any nation that ever gave away their nukes.

Except in the case of nuclear weapons, humans have their hands on the launch codes, and the nukes are not going to launch themselves. In the case of extremely capable AI, nobody has their hands anywhere close, because nobody is able to – nor willing to – supervise a thousand agents running in parallel, when they usually get shit done. Except eventually they are so good at getting things done that they disregard the values of the human asking them the request, such as "do not hack other corporations". Interpreting human intent precisely is an extremely hard problem, especially in the face of something 1000x faster than you operating autonomously.

Re: I resigned from Anthropic today

#894

Post by ericmay »

dinfinity wrote: post #877
ericmay wrote: post #866

Earlier quoted context omitted.

> This is an unreasonable demand. I'm not making a demand, I'm setting an anchor point from which we can work backward from. Is there any doubt that if someone was 100% certain that Big AI Button was being built and it would kill all humans that said person has a moral obligation to do everything they can to stop the button from being built and pressed? > "You demand that he should kill multiple people rather than qu…

> if someone was 100% certain This is unreasonable because nobody can know 100% certain that the AI built at Anthropic will kill everyone. It is also a big-ass strawman because the author never claimed he was 100% certain that AI will kill all humans. You're arguing from an irrelevant hypothetical (in which, I might add, most people would still not automatically change into murderous psychopaths as you apparently thi…

> This is unreasonable because nobody can know 100% certain that the AI built at Anthropic will kill everyone. It is also a big-ass strawman because the author never claimed he was 100% certain that AI will kill all humans.

Ok then stop arguing about my hypothetical if you don't want to engage with it. Those of us who are interested in talking about it can do so in peace without distracting comments.

> change into murderous psychopaths as you apparently think is normal

Please stop engaging in this hyperbole and go find some other axe to grind.

> It's a whole thread, my man.

I took a direct quote from the thread where earlier you accused me of pedaling in sensationalism and "not reading".

Re: I resigned from Anthropic today

#895

Post by ericmay »

strogonoff wrote: post #882
ericmay wrote: post #845

I’m not advocating for this. ~~~~~~ If you really believed these labs were going to result in the extinction of humanity and you saw it first hand, don’t you have a moral obligation not to quit your job but to go and try and kill everyone working on said project, blow it up, or otherwise stop it? I take these resignations with a grain of salt precisely for that reason. If I worked at a job and I saw some crazy guy or…

Consequentialism is not the only option. It invaded the mainstream thanks to effective altruism and friends, but deontological ethics is alive and well. Quitting your job instead of blowing up the office is well aligned with the latter.

> Quitting your job instead of blowing up the office is well aligned with the latter.

How so, in your view?

Re: I resigned from Anthropic today

#896

Post by program_whiz »

program_whiz wrote: post #873
huitzitziltzin wrote: post #94

“ No other human activity poses this level of danger.” I really, really disagree with that statement. I don’t think ai models come close to nuclear weapons or to run-of-the-mill, everyday carbon emissions in terms of danger to humanity. What’s the most dangerous thing that’s happened with an LLM so far? (This question is serious - maybe I don’t know the right examples.) Example 1: I’m aware of a small number of peopl…

What? Shutting down infrastructure, destroying critical records (economic collapse), spoofing / impersonating world leaders (confusion, panic, world war, nuclear event), taking over comms generally (you get a message to evacuate your home due to impending disaster, is it real?). All these could cause sufficient chaos and fear as to instigate global collapse. Once the panic sets in after a few of these indicidents, un…

I wrote about this, as I thought it warranted a deeper look: https://nonlineartransform.substack.com/p/world-ending-ai-to...

Re: I resigned from Anthropic today

#897

Post by ericmay »

zeven7 wrote: post #878
ericmay wrote: post #854

Earlier quoted context omitted.

> You're not advocating for it because it is highly immoral and illegal. That answers your question for the most part. Well truthfully I’m not advocating for it because I don’t have any insight into these labs or the true capability of these models. But as a hypothetical if someone knew with 100% certainty that Big AI Button was being built and it would kill all humans or destroy all of humanity there is no moral amb…

What if you have 10-20% certainty the AI being built will kill everyone? Go on a rampage and land yourself in prison just so Lab B or China can win the arms race and their AI can kill everyone? Probably not. Quit your job? Sure.

What if you have a 70% certainty?

You don't have to go on a rampage in an American lab. If Chinese labs were ahead I'd bring about the same discussion.

Though separately if the United States (or China or anyone) believed one country or another was truly going to achieve something akin to a metaphorical AI Supremacy maybe you nuke them, or at least the labs/researchers. Many a sci-fi movie has been built on a similar "first strike" premise.

Re: I resigned from Anthropic today

#898

Post by sillyfluke »

frabcus wrote: post #682
gizmodo59 wrote: post #504

Earlier quoted context omitted.

This is a lame argument. Questioning his financial incentive is very legit. Why should we trust him?

Questioning his finances is a poor argument because he clearly would have more money if he had stayed at Anthropic for the next couple of years, than he will have by leaving. Even if he still has vested options, or savings from his salary or whatever, those would be larger if he stayed.

> clearly

Debatable, and clearly not clear. It's weird to say this when there are obvious counter examples staring us in the face, not least of all the founder of the company he just left, of who (Dario) people could have said the same thing when he left OpenAI.

Granted, there is less likelihood now then there was then but this much is clear: if he starts his own company, he may make more.

Re: I resigned from Anthropic today

#899

Post by ls612 »

gpm wrote: post #299
ls612 wrote: post #216

Earlier quoted context omitted.

Historically it almost always takes actual fatalities rather than near misses to ground an aircraft, and aviation is famous for its obsession with safety compared to other industries.

Airplanes have pretty bounded damage. Generally you kill at most a few hundred people. Even weaponized a few thousand. This is a risk profile that allows risk taking with near misses and waiting until something goes wrong to fix it (though doing so is rightfully uncomfortable and frequently unethical). The people worrying about AI risk are worrying about "it goes wrong once and kills billions of people". That's not a…

The story of the safetyists has the convenient property of being unfalsifiable, so they can always claim doom is just around the corner. It’s the secular/EA version of the second coming.

Re: I resigned from Anthropic today

#900

Post by skeledrew »

deaux wrote: post #350
skeledrew wrote: post #148

Earlier quoted context omitted.

> not necessarily true It's a possibility that is increased by their action. One leaves, a space is now open that will likely eventually be filled. And the chance of someone with equal/higher scruples filling it is very slim (unless you somehow know that the good amount of those who qualify and apply for the position have equal/higher scruples). That's just logic and math.

Right, I assume tomorrow you'll be applying for the next Nazi camp guard vacancy? After all, if you don't do it, someone with less scruples likely will.

Today we see the Nazis as bad. Do you think they saw themselves as such? They had an ideal, and they worked towards it. And I'd say they're were many who didn't care about that ideal and just wanted a job. You can't compare the morals of today with yesterday, or judge the ideals of a person or group without relevant context.
Post reply on HN