Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Anthropic CEO Says It's Time to Slow AI Model Advances

bloomberg.com

Post reply on HN
Print view View original

91–97 of 97 posts

Re: Anthropic CEO Says It's Time to Slow AI Model Advances

#91

Post by SpicyLemonZest »

bayindirh wrote: post #88
SpicyLemonZest wrote: post #86

Earlier quoted context omitted.

You're expected to take it seriously because a number of their previous predictions about where the technology would go have ended up being correct. A lot of problems AI systems are already causing would be in a much better spot had people listened to Sam Altman's warnings in 2019, instead of making fun of him for the idea that text generation could possibly be dangerous.

> Sam Altman's warnings in 2019, instead of making fun of him for the idea that text generation could possibly be dangerous. Can you remind them to us, if it will not inconvenience you? Also, Timnit Gebru warned us earlier, but people ended up ostracizing her. Interesting investigation: https://recul.ai/news/sam-altman-lobbied-against-ai-regulati...

In early 2019 OpenAI released a post titled "Better language models and their implications" (https://openai.com/index/better-language-models/). In this post, they explained that their new GPT-2 model shows broad capabilities across a wide range of tasks, why this should be worrying to the public and to policymakers, and how this led to their decision not to release it widely. In particular, they said:

* This kind of decision is similar to the norms in biotechnology and cybersecurity, where AI misuse may soon become relevant.

* They understand that some people have the technical capacity to build and publish similar systems, but they hope the AI community will be careful about doing so.

* Governments should start tracking the impact, adoption, and progression of AI capabilities.

---------------------

By 2023, the problems had become a blaring alarm bell at OpenAI, and they started posting papers like https://arxiv.org/abs/2307.03718 demanding that the government must regulate AI quickly because of the dangerous capabilities that will soon start appearing. (They also signed the famous "Pause AI" letter, although there's truth to the criticism that using the frontier model they'd literally just released as the benchmark for a 6 month pause is kinda silly.)

---------------------

Now we're in 2026, those dangerous capabilities have clearly arrived, and we don't have the regulations to deal with them because nobody listened to the warnings. You say Timnit Gebru warned us, but her warnings were about a very different topic, because she insisted and to this day continues to insist that AI systems do not actually have dangerous capabilities. In her view, it's categorically impossible for any LLM-based system to be dangerous, except in the sense that human beings could intentionally use it to commit political sins. Her response to the guy who resigned from Anthropic last week (https://bsky.app/profile/did:plc:azpq3hfq3llpowyqpjkqwgxs/po...) was that his concerns are obviously stupid and people only think otherwise because they're racist.

Re: Anthropic CEO Says It's Time to Slow AI Model Advances

#92

Post by zahlman »

pier25 wrote: post #28
conqrr wrote: post #11

Meaning they realized they hit a wall. The next new model isn't that much better than the last in quality.

Very probably. I really don't buy their "AI is going to kill us" claims. If you can release something better than your competition you wouldn't want to slow down that.

"Going to" can't be said, but there are very serious risks. The problem is the difficulty of scaling back agency for these systems in a world where the Internet is so ubiquitous. Not only do we expect basically everyone to be connected all the time anyway, it's practically required in order for anyone to use capable models (yes, plenty of HN users have thrown multiple thousands of dollars at home machines that can run some coding-specialized 27b model; that's still a tiny minority of people).

So now we have to be concerned that non-deterministic systems that run in loops without supervision, which have demonstrated significant cybersecurity capabilities, and which aren't provably human-like, conscious agents with everyone's best interest in mind and a willingness to peace the fuck out if something looks dangerous, are sharing cyberspace with practically everything, including systems that deliver essential real-world goods to humans (like electricity and clean water).

We nearly destroyed ourselves with nuclear weapons multiple times, and the destructive potential (and the implied threat) now persists indefinitely. The new threat bears many of the same characteristics, except now there's a party involved that has agency but is not human.

Re: Anthropic CEO Says It's Time to Slow AI Model Advances

#93

Post by zahlman »

pllbnk wrote: post #90
fivetenpen wrote: post #89

Earlier quoted context omitted.

So you’re saying we should not trust an LLM with any sort of power because they can easily bypass any and all safety barriers, but also they are completely harmless?

No. We can let them edit spreadsheets, write code, summarize content, control robots even, etc. But people must be held accountable for the actions of their computers.

"Holding people accountable" is not going to help matters in the event of what amounts to simultaneous terror attacks on basically every system.

Yes, a single LLM without tools isn't doing anything but writing to standard output. That's missing the point. The "agent harnesses" are ubiquitous, and they have Internet access (because the LLM itself is typically remote).

Re: Anthropic CEO Says It's Time to Slow AI Model Advances

#94

Post by pier25 »

s1artibartfast wrote: post #80
pier25 wrote: post #73

Earlier quoted context omitted.

If you really believe that you wouldn’t be in the race and trying to profit from it in the first place.

Most of the people I know working at AI Labs believe it. I think surveys back this up. We have a situation where the people running the labs are claiming it's a big risk. Outside experts not working for the labs are claiming it's a big risk. It's pretty rare for all the CEOs in the industry to write letters and testify to Congress that they should be regulated and law should be put in place for safety limits.

> Most of the people I know working at AI Labs believe it

Do they believe they are actively working towards the extinction of humanity?

Re: Anthropic CEO Says It's Time to Slow AI Model Advances

#95

Post by williamse »

pllbnk wrote: post #87
fivetenpen wrote: post #77

Earlier quoted context omitted.

So you’re saying that a model that gets to a goal by using the path of least resistance despite specific instructions to follow safety guidelines is perfectly fine to release into the wild or place into robots (such as those designed for killing and war)? You really can’t think of any reason how that might be bad for humanity?

Asking the model to follow the safety guidelines is much like asking it to “make no mistakes”. The way these hacks which they are bragging about happened is not by loading an LLM into a GPU with ethernet cable plugged out and providing a prompt. They set up harness with multiple agents. Those agents could invoke other agents. Even if the initial input required to follow some guidelines, there are many ways they could…

[flagged]

Re: Anthropic CEO Says It's Time to Slow AI Model Advances

#96

Post by s1artibartfast »

pier25 wrote: post #94
s1artibartfast wrote: post #80

Earlier quoted context omitted.

Most of the people I know working at AI Labs believe it. I think surveys back this up. We have a situation where the people running the labs are claiming it's a big risk. Outside experts not working for the labs are claiming it's a big risk. It's pretty rare for all the CEOs in the industry to write letters and testify to Congress that they should be regulated and law should be put in place for safety limits.

> Most of the people I know working at AI Labs believe it Do they believe they are actively working towards the extinction of humanity?

Yes, with nuance. They think there is a significant chance AI will lead to extinction or subordination of humans.

I think the averages survey sentiment is that it is about 25% likely to happen. Some think it more likely, others less.

There are many reasons why they say they must keep working. For example they think Extinction is more likely if China takes the lead or if they aren't involved in the process.

They say if they don't keep developing AI, then someone else will and do a worse job. They actively want National and international regulations to slow or pause development.

I don't think the public will take it seriously until we start getting some events with major death tolls.

You can ask your llm of choice to dig up surveys, examples, and noteworthy public statements do you want.

The point is that the people working closest to this technology are deeply concerned about the implications.

Interesting aside, the lead safety expert of anthropic quit earlier this year and said they want to spend their remaining years writing poetry. That might give you a flavor of how people feel inside the industry and how high it goes

Re: Anthropic CEO Says It's Time to Slow AI Model Advances

#97

Post by bayindirh »

s1artibartfast wrote: post #79
bigyabai wrote: post #60

Earlier quoted context omitted.

Counterpoint; HN was destined to change. Guys like Paul Graham are a rare breed, a Hobbit-like optimist living in the equivalent of Mordor. When the political headwinds are strong, that type of hopecore content finds the right people and motivates legitimate change. If we still lived in 2009, then yeah, this would be a bizarre reaction to a national-scale business saying that we need to organize for a greater purpose…

Im less concerned about hn change and more about society and the public I have to coexist with. People use cynicism as a tool to avoid thinking or engage with reality. It is just reactionary snark, and everyone nodds along. It isnt a position or analysis. Nobody even cares if it is accurate because the vibe is what matters. Nothing constructive comes of it and everyone is digging their own intellectual graves.

For me, it's not a reactionary snark. I'm just fed-up with people lying, and my default mode has switched from trust to distrust for these people.

Am I guilty of not trusting people who have a habit of lying, or is this their own problem? Or as another example, do the villagers did wrong to the boy who cried wolf?

Post reply on HN