Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Anthropic drops flagship safety pledge

time.com

Post reply on HN
Print view View original

601–610 of 716 posts

Re: Anthropic drops flagship safety pledge

#601

Post by kseniamorph »

> The policy change is separate and unrelated to Anthropic’s discussions with the Pentagon, according to a source familiar with the matter.

ok lol what a coincidence.

but setting aside the conspiracy. the article actually spells out the real reason pretty directly: Anthropic hoped their original safety policy would spark a "race to the top" across the industry. it didn't. everyone else just ignored it and kept moving. at some point holding the line unilaterally just means you're losing ground for nothing.

Re: Anthropic drops flagship safety pledge

#602

Post by overgard »

austinjp wrote: post #576
cmrdporcupine wrote: post #496

Earlier quoted context omitted.

We all made fun of Blake Lemoine and others for spending too many late nights up chatting with (ridiculously primitive by this year's standards) LLM chat bots and deciding they were sentient and trapped. But frankly I feel like the founders of Anthropic and others are victim of the same hallucination. LLMs are amazing tools. They play back & generate what we prompt them to play back, and more. Anybody who mistakes th…

I always feel this argument misses a point. SkyNet may still be a long way off, but autonomous killer drones are here. That is a bad situation my dudes. Every step on the journey towards SkyNet is worse than the preceding step. Let's not split hairs about which step we're on: it's getting worse, and we should stop that.

Using LLMs for weapons is a grave misunderstanding of what LLMs are actually good for. These are things that should NEVER be in charge of life or death decisions.

Re: Anthropic drops flagship safety pledge

#603

Post by whodidntante »

santadays wrote: post #550
observationist wrote: post #474

Earlier quoted context omitted.

Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an 'intelligence explosion,' and the intelligence of man would be left far behind. Thus the first ultraintelligent m…

Intelligence seems to boil down to an approximation of reality. The only scientific output is prediction. If we want to know what happens next just wait. If we want to predict what will happen next we build a model. Models only model a subset of reality and therefore can only predict a subset of what will happen. Llms are useful because they are trained to predict human knowledge, token by token. Intelligence has to…

With AI, as we currently understand it, we may have stumbled upon being able to replicate a part of the layer of our brain that provides the "reason" in humans., and a very specific type of "reason" a that.

All life has intelligence. Anyone who has spent a lot of time with animals, especially a lot of time with a specific animal, knows that they have a sense of self, that they are intelligent, that they have unique personalities, that they enjoy being alive, that they form bonds, that they have desires and wants, that they can be happy, excited, scared, sad. They can react with anger, surprise, gentleness, compassion. They are conscious, like us.

Humans seem to have this extra layer that I will loosely call "reasoning", which has given us an advantage over all other species, and has given some of us an advantage over the majority of the rest of us.

It is truly a scary thing that AI has only this "reasoning", and none of the other characteristics that all animals have.

Kurt Vonnegut's Galapagos and Peter Watts Blindsight have different, but very interesting takes on this concept. One postulates that our reasoning, our "big brains" is going to be our downfall, while the other postulates that reasoning is what will drive evolution and that everything else just causes inefficiencies and will cause our downfall.

Re: Anthropic drops flagship safety pledge

#604

Post by mcv »

> The announcement is surprising, because Anthropic has described itself as the AI company with a “soul.”

I can't help but think about how Google once had "Don't be evil" as their motto.

But the thing with for-profit companies is that when push comes to shove, they will always serve the love of money. I'm just surprised that in an industry churning through trillions, their price is $200 million.

Re: Anthropic drops flagship safety pledge

#605

Post by overgard »

keeda wrote: post #588
honeycrispy wrote: post #465

Anthropic's CEO Dario has annoyed me to no end with his "AI will take all the jobs in 6 months" doomer speeches on every podcast he graces his presence with.

I think he's right and we should be thinking about this a lot more. Even the IMF is worried about 40 - 60% of global employment : https://www.imf.org/en/blogs/articles/2024/01/14/ai-will-tra... Focusing on Dario, his exact quote IIRC was "50% of all white collar jobs in 5 years" which is still a ways off, but to check his track record, his prediction on coding was only off by a month or so. If you revisit what he act…

> Focusing on Dario, his exact quote IIRC was "50% of all white collar jobs in 5 years" which is still a ways off, but to check his track record, his prediction on coding was only off by a month or so. If you revisit what he actually said, he didn't really say AI will replace 90% of all coders, as people widely report, he said it will be able to write 90% of all code.

Ugh, people here seem to think that all software is react webapps. There are so many technologies and languages this stuff is not very good at. Web apps are basically low hanging fruit. Dario hasn't predicted anything, and he does not have anyone's interests other than his own in mind when he makes his doomer statements.

Re: Anthropic drops flagship safety pledge

#606

Post by neya »

dibujaron wrote: post #416
neya wrote: post #407

Earlier quoted context omitted.

I was a Pro subscriber until last week. When I was chatting with Claude, it kept asking a lot of personal questions - that seemed only very very vaguely relevant to the topic. And then it struck me - all these AI companies are doing are just building detailed user models for being either targeted for advertising or to be sold off to the highest bidder. It hasn't happened yet with Anthropic, but when the bubble money…

A less cynical explanation: It's heavily trained to ask follow-up questions at the end of a response, to drive more conversation and more engagement. That's useful both for making sure you want to renew your subscription, and also probably for generating more training data for future models. That's sufficient explanation for the behavior we're seeing.

Because, Anthropic can do no wrong, correct?

Re: Anthropic drops flagship safety pledge

#607

Post by lazystar »

santadays wrote: post #550
observationist wrote: post #474

Earlier quoted context omitted.

Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an 'intelligence explosion,' and the intelligence of man would be left far behind. Thus the first ultraintelligent m…

Intelligence seems to boil down to an approximation of reality. The only scientific output is prediction. If we want to know what happens next just wait. If we want to predict what will happen next we build a model. Models only model a subset of reality and therefore can only predict a subset of what will happen. Llms are useful because they are trained to predict human knowledge, token by token. Intelligence has to…

i think theres a paradox here. intelligence needs a judge - if nothing verifies that the optimal outcome was chosen, it's too easy for the intelligence to fall into biased decisions

Re: Anthropic drops flagship safety pledge

#608

Post by observationist »

visarga wrote: post #493
observationist wrote: post #474

Earlier quoted context omitted.

Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an 'intelligence explosion,' and the intelligence of man would be left far behind. Thus the first ultraintelligent m…

John Good's quote is pretty myopic, it assumes machines make better machines based on being "ultraintelligent" instead of learning from environment-action-outcome loop. It's the difference between "compute is all you need" and "compute+explorative feedback" is all you need. As if science and engineering comes from genius brains not from careful experiments.

There's an implicit assumption there, anything a computer as intelligent as a human does will be exactly what a human would do, only faster. Or more intelligent. If the process is part of the intelligent way of doing things, like the scientific method and careful experimentation, then that's what the ultraintelligent machine will do.

There's no implication that it's going to do it all magically in its head from first principles; it's become very clear in AI that embodiment and interaction with the real world is necessary. It might be practical for a world model at sufficient levels of compute to simulate engineering processes at a sufficient level of resolution that they can do all sorts of first principles simulated physical development and problem solving "in their head", but for the most part, real ultraintelligent development will happen with real world iterations, robots, and research labs doing physical things. They'll just be far more efficient and fast than us meatsacks.

Re: Anthropic drops flagship safety pledge

#609

Post by keeda »

I don't think the risk is SkyNet. I think the real risk is some disaster through an unexpected chain of events, just like any large-scale outage.

I have not read “If Anybody Builds It, Everybody Dies” but I believe that's also its premise.

Current GenAI is extremely capable but also very weird. For instance, it is extremely smart in some areas but makes extremely elementary mistakes in others (cf the Jagged Frontier.) Research from Anthropic and OpenAI gives us surprising glimpses into what might be happening internally, and how it does not necessarily correspond to the results it produces, and all kinds of non-obvious, striking things happening behind the scenes.

Like models producing different reasoning tokens from what they are really reasoning about internally!

Or models being able to subliminally influence derivative models through opaque number sequences in training data!

Or models "flipping the evil bit" when forced to produce insecure code and going full Hitler / SkyNet!

Or the converse, where models produced insecure code if the prompt includes concepts it considers "evil" -- something that was actually caught in the wild!

We are still very far from being able to truly understand these things. They behaves like us, but don't necessarily “think” like us.

And now we’ve given them direct access to tools that can affect the real world.

Maybe we am play god: https://dresdencodak.com/2009/09/22/caveman-science-fiction/

Post reply on HN