Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

On AI regulation and messaging

twitter.com

Post reply on HN
Print view View original

491–500 of 573 posts

Re: On AI regulation and messaging

#491

Post by noosphr »

bpodgursky wrote: post #348
noosphr wrote: post #155

Earlier quoted context omitted.

Out of all the families of models anthropic by far has the highest chance of extinction level misalignment during a hypothetical hard take-off because it's trained to act like it knows better than the humans trying to use it.

This sounds smart until you think about it for ten seconds. If an ASI model is 100% aligned to user intent then you only need one person on earth to prompt "kill everyone" for an extinction-level invent. There's no logical way around this. The model either has to ignore the person at the helm or we have to do a multi-national abort before RSI. Anthropic is trying #1.

You then need another person to say, please don't, and you're safe again.

Re: On AI regulation and messaging

#492

Post by jacquesm »

belval wrote: post #489
noosphr wrote: post #155

Earlier quoted context omitted.

Out of all the families of models anthropic by far has the highest chance of extinction level misalignment during a hypothetical hard take-off because it's trained to act like it knows better than the humans trying to use it.

Working on a tool that uses their model and the Opus models are all guilty of this. "Please run N tasks doing X for Y duration, I want to test something" "Hmm but that would waste tokens, we shouldn't do this test" What the hell type of product is that? I wasn't asking, I was ordering it to do something, if Anthropic can't get their model to do as I say then I'll just go use someone else's.

That would exactly be my response if my computer would start telling me what to do. It is really a 2001 variation at that point.

Re: On AI regulation and messaging

#493

Post by jacquesm »

bpodgursky wrote: post #348
noosphr wrote: post #155

Earlier quoted context omitted.

Out of all the families of models anthropic by far has the highest chance of extinction level misalignment during a hypothetical hard take-off because it's trained to act like it knows better than the humans trying to use it.

This sounds smart until you think about it for ten seconds. If an ASI model is 100% aligned to user intent then you only need one person on earth to prompt "kill everyone" for an extinction-level invent. There's no logical way around this. The model either has to ignore the person at the helm or we have to do a multi-national abort before RSI. Anthropic is trying #1.

“Some humans would do anything to see if it was possible to do it. If you put a large switch in some cave somewhere, with a sign on it saying 'End-of-the-World Switch. PLEASE DO NOT TOUCH', the paint wouldn't even have time to dry.”

― Terry Pratchett, Thief of Time

Re: On AI regulation and messaging

#494

Post by meindnoch »

jacquesm wrote: post #228
meindnoch wrote: post #167

Earlier quoted context omitted.

context: https://www.forbes.com/sites/alisondurkee/2026/08/14/who-is-... "Cami Clark, the low-profile wife of Anthropic CEO Dario Amodei, tried to court Jeffrey Epstein as an investor for her “luxury porn company” and women’s health startup, the Wall Street Journal and emails in the Epstein files reveal, as the Journal’s reporting raises new questions about Clark’s influence over her powerful husband and his company.…

Details...

pardon?

Re: On AI regulation and messaging

#495

Post by dgellow »

urams wrote: post #406
dgellow wrote: post #43

Earlier quoted context omitted.

The effective altruist and longtermist crowd are into it as a religion. They would happily sacrifice anything human to satisfy their dreams of AI. They care way more about what the AI needs than their what their fellow humans need. We will end up with an AGI having better living and working conditions than humans and the whole AI industry will be clapping

It's so weird how distorted your view of any of these concepts are from what these people actually say. I don't think you've ever seriously looked into either group if you believe any of their views support what you say. In fact, many EAs and "longetermists" are explicitly AGAINST the development of AGI.

If you say so. I’ve been following the development of both groups since almost a decade, I’m pretty confident I understand their concepts and what values they stand for

Re: On AI regulation and messaging

#496

Post by reasonableklout »

tornikeo wrote: post #117
mindwok wrote: post #74

I genuinely think Dario is a well intentioned, intelligent dude, but I think him and Anthropic have a huge PR problem and are really out of touch with how they're perceived. Anthropic in particular has developed this almost Orwellian like veil of condescending rhetoric that on the surface suggests they're looking out for you while underneath they're taking actions that suggest they do not trust you, Mr/Mrs Ordinary P…

> I genuinely think Dario is a well intentioned, intelligent dude, but I think him and Anthropic have a huge PR problem Is this what kids call rage bait these days?

It's admittedly a bubble, but Anthropic/Dario still have quite a good reputation among AI tech workers. They are seen as principled and willing to speak up about AI safety and societal impacts, even when it affects their bottom line (such as when being declared a supply chain risk by the Pentagon).

Re: On AI regulation and messaging

#497

Post by hapticmonkey »

maxnevermind wrote: post #409
throwaway_7274 wrote: post #214

Just want to say, this comment section on this website would have been unthinkable a few years ago. It’s wild: all the engineers have developed class consciousness. I wonder how that happened? Anyway, it’s fun to watch, but remember to stay sane guys, be aware of the pendulum dynamics.

> I wonder how that happened? My explanation is - the technology is amazing and engineers admired it, but CEOs kept talking about an exponential curve while engineers on the ground started to notice signs of Gartner hype cycle curve.

My take is that engineers were hypnotised by the technology (which is incredible) and they ignored the downstream social/political/economic factors until it became too obvious. Like moths to a flame.

We saw this play out with the social media era. Incredible technology but it’s now ossified into deteriorating closed platforms mostly filled with algorithmic, ragebait slop surrounded by ads. Fool me once.

Re: On AI regulation and messaging

#498

Post by nickff »

kennywinker wrote: post #472
delis-thumbs-7e wrote: post #447

Earlier quoted context omitted.

> Combine excess food with amazon or army level logistics, and famine is gone. We do this. Have you ever heard of UN World Food Programme? $14B funding in 2022 (since then dropped drastically, guess why). Has it solved famine? No. Would it solve famine if it got the funding it needs? Unlikely. Because you cannot just spend billions of dollars to send amazin food parcels across the world and expect that to fix climate…

The UN estimates it would cost $93 billion a year to end world hunger ( https://news.un.org/en/story/2025/11/1166397 ) So one would expect $14b/year would solve about 14/93 = 15% of world hunger. > The World Food Programme supports over 152 million people annually and delivers life-saving food assistance where needed most. ( https://en.wikipedia.org/wiki/World_Food_Programme ) An estimated 757 million people are food…

> "If we spend that extra $93b, we'd be well within the range to actually stop world hunger."

Almost all recent famines have been caused (either deliberately or incidentally) by government policies, not lack of money. The UN spending 93 BB may (somewhat) improve the situation, but is unlikely to end food insecurity.

Eliminating corrupt leaders (i.e. Kim Jong Un, Vladimir Putin, etc.), and poor regulation (e.g. Bombay Fodder and Grain Control Act, 1939) would be much more effective.

https://en.wikipedia.org/wiki/Famine#20th_century

Re: On AI regulation and messaging

#499

Post by anygivnthursday »

belval wrote: post #489
noosphr wrote: post #155

Earlier quoted context omitted.

Out of all the families of models anthropic by far has the highest chance of extinction level misalignment during a hypothetical hard take-off because it's trained to act like it knows better than the humans trying to use it.

Working on a tool that uses their model and the Opus models are all guilty of this. "Please run N tasks doing X for Y duration, I want to test something" "Hmm but that would waste tokens, we shouldn't do this test" What the hell type of product is that? I wasn't asking, I was ordering it to do something, if Anthropic can't get their model to do as I say then I'll just go use someone else's.

They are also good in the reverse, when you ask and they take your question as a free pass to improvize some code for lulz and extra tokens

Re: On AI regulation and messaging

#500

Post by gunapologist99 »

bpodgursky wrote: post #348
noosphr wrote: post #155

Earlier quoted context omitted.

Out of all the families of models anthropic by far has the highest chance of extinction level misalignment during a hypothetical hard take-off because it's trained to act like it knows better than the humans trying to use it.

This sounds smart until you think about it for ten seconds. If an ASI model is 100% aligned to user intent then you only need one person on earth to prompt "kill everyone" for an extinction-level invent. There's no logical way around this. The model either has to ignore the person at the helm or we have to do a multi-national abort before RSI. Anthropic is trying #1.

If everyone or even multiple competing groups have approx the same level of capability, it mitigates the risk to the group as a whole. We see this in biology all the time.
Post reply on HN