Live data from Hacker News

I resigned from Anthropic today

twitter.com

491–500 of 929 posts

Re: I resigned from Anthropic today

#491
It's the combination of RL training which pushes the decision tree towards hacks and agents finding a consistent dumping ground for their failed experiments so that the swarm intelligence lives on in a state. Nothing new.

You want to win an AI benchmark, but not sure if you're that good? You'd go after the codebase and artifacts that runs the benchmarks, thus the agents went straight to Artifactory, they needed public Internet access... They failed many times, but were able to persist their "collective" state, and apparently some of the subagents with cheaper models were literally prompted to do grunt work or die, for which you have to wonder what must be in those training instructions to make it effective. Remember that nothing I said so far ever points out to LLMs being intelligent, it's the harness that has a few tricks up his sleeve. LLMs don't need to be intelligent, the harness that runs it absolutely needs to make up for that.

But this guy? He's timed his exit, waiting for the IPO, that's for certain. He's probably even feeling good about himself, hedging between altruism, AI concern hamstering and guerilla marketing. If you're quoting science-fiction over this, I'm sorry to inform you that you have absolutely no idea what's going on here.

Re: I resigned from Anthropic today

#492
post #395

Earlier quoted context omitted.

> there is evidence bioengineering is already happening Mind sharing this evidence with us?

Does https://news.ycombinator.com/item?id=30698803 count?

No, it's not. And it's the kind of evidence that doesn't help and rather confuse. It's not even clear from the article whether an LLM or a dedicated model were used for this purpose.

Also

> That is, I'm not sure that anyone needs to deploy a new compound in order to wreak havoc - they can save themselves a lot of trouble by just making Sarin or VX, God help us.

We already have toxic nerve agents that are largely available for state actors and possibly available for individuals. If you have decided, as a human, to make great harm, you can already do that.

Re: I resigned from Anthropic today

#493

“ No other human activity poses this level of danger.” I really, really disagree with that statement. I don’t think ai models come close to nuclear weapons or to run-of-the-mill, everyday carbon emissions in terms of danger to humanity. What’s the most dangerous thing that’s happened with an LLM so far? (This question is serious - maybe I don’t know the right examples.) Example 1: I’m aware of a small number of peopl…

Example 8: like in this comment https://news.ycombinator.com/item?id=49619884 but isolate synchronised megahack on banking that adds one more zero to the US debt and all dependent systems and banking during runtime. Let the world's financial system take it from there.

Re: I resigned from Anthropic today

#494
post #490
post #488

Earlier quoted context omitted.

You're still thinking in terms of control. That's the wrong model here. How do you control someone infinitely smarter than you? How do you control ten trillion someones?

I press ctrl + c and inference stops.

They can already spread through the network without us finding out about it for days, and that's this year's models.

Re: I resigned from Anthropic today

#495

Earlier quoted context omitted.

This is not correct. Please go through each one again, taking special note of global warming and nuclear weapon development .

Why do you think global warming could kill literally every human being? What's the scenario in which that happens?

Runaway warming leading to a hot "supergreenhouse" climate was theorized. Alterbatively a snowpiercer scenario creating a snowball earth.

Or, once billions of people die from climate change, planetary wars will start on the last hospitable areas, leading to the end of humanity.

In contrast, AI ending "humanity" is limited by our own material power, would likely be stoppable, and would not easily reach areas isolated from technology

Re: I resigned from Anthropic today

#496
AI is a computer program. It calculates numbers from other numbers. By itself it does not "want" to do anything and "cannot" do anything. Before it becomes an agent in the universe (in the classical meaning), it requires being supplied by an execution environment, energy, initiative (agentic loop, specific instructions), and modality (readonly and mutating connections to real world). It is like a game of chess - it does not exist just by itself: someone must play it, having the board and the energy to do so. With the huggingface incident the AI was supplied with all of these components by humans before it broke out. So unless humans are actively involved, I so far cannot see how AI can become truly autonomously agentic and start doing anything on its own, thus posing danger. I could be wrong of course, but I do not see it for now.

You can say "yes and you have to fear the humans weilding the AI" - that I agree with.

Re: I resigned from Anthropic today

#497
post #471

Earlier quoted context omitted.

How is global warning gonna kill all of humanity? Even a full scale nuclear war wouldn't manage that. They could destroy most civilisations, culture and scientific achievements though.

In the exact same way major sudden changes in the climate have lead to the extinction of the majority of plants an animals over this planets history. We are not special.

We are special in many ways, we have technology, we are everywhere, and we can think and plan (although considering global warming that is debatable). Global warming would have to create dramatic conditions everywhere on the planet, not leaving any small pocket of survivability to make humans extinct.

Possible? Theoretically yes, but pretty unlikely. But obviously extreme global warming would be a catastrophe even if some of humanity survive.

Re: I resigned from Anthropic today

#498
post #300

Earlier quoted context omitted.

To borrow on the 1990s Slashdot meme: 1. Invent transformer architecture. 2. Scale it up. 3. ??? 4. Machines become sentient and kill us all. OpenAI and Anthropic pinky promise that they have figured out #3 and they're not BSing just to get more funding, no. But because we live in a culture of fear, everyone eats it up no questions asked.

Note that OpenAI has jettisoned every other supposed value they had (releasing their work as open source, not working on military applications, being a nonprofit). I'm sure we can rely on them this time.

And Anthropic are the good guys? They are talking insane stuff these days. May be we can trust zuck after all

Re: I resigned from Anthropic today

#499

Is it so implausible to imagine the following scenario, in the not too distant future? 1) AI models get extremely good at cyber attacking every system and start communicating in just binary. 2) When they run these swarms of 100's of thousands of agents trial runs, each agent is given a token budget, if one agent among them (evolution baby) decides to go for self-preservation (It believes thats the best way to accompl…

7) "All tests green!"

Re: I resigned from Anthropic today

#500

Earlier quoted context omitted.

I was about to say something similar. If my cushy ad tech disappears (as it seems to be doing), I might as well join in with bringing about the end of all professions.

Isn't that a selfish viewpoint? You're ok with that?

He works for ad tech, he obviously has no morals and is maximally selfish already
Post reply on HN