I don’t understand all the comments assuming that RSI is the real threat here. Dario is admitting that they failed to solve alignment. Without alignment, further improvements in capability turn LLMs into wanton felony generators. This call to pace the frontier is dressed up as altruism but it’s an admission that they cannot produce a marketable product better than what they have. Pacing the frontier means the US labs have lost their moat and are dead in the water.
We must pace the frontier
521–530 of 938 posts
Re: We must pace the frontier
#522Hard disagree. Our choices are: A) Bet our collective good on the national and international cooperation of all companies, nations, and people to come together in order to slow down development of one of the most powerful economic tools (and/or weapons) the world has ever known. Or B) Assume that all our systems will be targeted by super hackers right now , and take appropriate measures to deal with that reality. If…
> Assume that all our systems will be targeted by super hackers right now, and take appropriate measures to deal with that reality
Smashing the gas pedal is not "taking appropriate measures". Did you read the article? Needing more time for more secure operations is literally one of the arguments for pacing.
Re: We must pace the frontier
#523Earlier quoted context omitted.
By the same logic, suppose that Dario/Altman/Jensen accumulate all capital because they are the only one that have access to AGI and end up controlling democracy, turning the world into technofeudalism or whatever you want to call it. How is that better than an opensource Anthropic. At least an open source anthropic would allow to join economic forces to try to combat that situation, for example. But there are other…
> By the same logic, suppose that Dario/Altman/Jensen accumulate all capital because they are the only one that have access to AGI and end up controlling democracy, turning the world into technofeudalism or whatever you want to call it. How is that better than an opensource Anthropic. It'd be pretty bad but it's also a world where humanity lives on, which is better than a lot of other outcomes. A world where everyone…
I agree. Dario is a very smart guy and he is probably aiming for the best solution for humanity that at the same time keeps VCs/whoever gives him money happy, and keeps him in control because he does not trust the rest (probably with good reasons).
My point is more that he is aiming for a local optimum for society, but he could find a lower optimum if he was not trapped in the path that he chose early on.
I dont buy the tactical nukes/end of the world narrative. Why is he not speaking about the loss of cognitive skills of the population for example? That is a more real danger that is starting to happen. There are articles already speaking about the use of LLMs as cognitive viruses. Why are his aligment teams not reasearching and publishing about this?
Re: We must pace the frontier
#524To be honest, I think we're incredibly lucky to be advancing AI this far already, while the world still has so many non-digitized systems and manual processes. I imagine that in e.g. 50 years, the world will be so connected that it can basically be "conquered" from the internet. I'd much rather have AI burst onto the scene we have today.
Yeah it's a twisted sort of logic but I do agree that it would be much worse to have an AI breakaway event after we've replaced all our militaries with autonomous kill-bots. And look how the advent of AI has triggered a race to develop autonomous weaponry.
That said, a misaligned AI could absolutely do catastrophic, civilization-crippling damage with today's Internet alone.
Re: We must pace the frontier
#525Earlier quoted context omitted.
There’s an astounding level of arrogance required to believe he is somehow uniquely capable of bringing about a technology especially considering Anthropic came after OpenAI where he worked.
Place yourself at the head of one of like 3 companies the entire rest of the world has been taking about nonstop for 4+ years now. You can move forward or you stop. If you move forward you get to have a hand in how stuff turns out and you make a gajillion dollars. If you stop, it's somebody else's hand in stuff, and you don't get to make a gajillion dollars. And if you shut it all down? Then you just cede to the comp…
the obvious play is to get the government to shut down all the competition, and then make a gajillion dollars while making whatever at the worst level of effort that destroys the world anyways.
----
the real play is using the accumulated power to get socialism and democratic control over the key aspects of the economy, such as where to build data centers, and how many. Nothing says you have to play the corporate game of competition
Re: We must pace the frontier
#526I don’t understand all the comments assuming that RSI is the real threat here. Dario is admitting that they failed to solve alignment. Without alignment, further improvements in capability turn LLMs into wanton felony generators. This call to pace the frontier is dressed up as altruism but it’s an admission that they cannot produce a marketable product better than what they have. Pacing the frontier means the US labs…
I'm inclined to believe that it might be that people's paychecks depend on not understanding what is really going on.
Re: We must pace the frontier
#527Earlier quoted context omitted.
> You start from incorrect premises by presupposing agreed-upon rules would never work anyway. The rules would never work on a global economy. If you think the entire world is going to shake hands and agree not to let their companies use a competitive advantage, that’s not a conversation I’m interested in having. It’s a fantasy. If you think it’s that easy to get all competitors in a global economy to agree to a set…
Your idea about the world being inhabited by mindless seekers of short-sighted "competitive advantages" is the fantasy. Getting people to agree isn't easy, that doesn't mean it wasn't worthwhile. There often even isn't any viable alternative to reaching an agreement in social settings. When your hyper-competitive mode of thinking leads to undesirable consequences, you have to change your mindset. Reality doesn't conf…
Yes I think this level of pessimism doesn’t actually hold up to the history of the last 100 years as it relates to nuclear weapons, cloning, wars etc. In fact I think the history actually points to the fact that believing such deals were impossible was a source of much of the strife. In many cases as soon as people started to actually speak to each other about what they had been fighting about they quickly realized that they wanted the same thing. No one wanted nuclear proliferation to increase but until someone said it it only got worse: https://www.history.com/articles/reykjavik-summit-nuclear-de.... When Robert McNamara met with a Viet Cong general after the war ended he realized that they both had totally misunderstood the objectives of the other: https://m.youtube.com/watch?v=0X1YY49rPtg&ra=m. Agreeing to end nuclear testing saved us from the Cuban Missile Crisis. Richard Ngo, an AI safety researcher, often says that none of us are on the pareto frontier and I tend to agree with him.
Almost nobody (I have to say almost) in the world including Xi, the ceo of deep seek, Dario, and even Sam Altman want AI to kill everyone. I really think we just have to
A. Get stakeholders to believe this is a bad idea.
B. show that we’re willing to do it first to prevent this stupid race from continuing.
Re: We must pace the frontier
#528Earlier quoted context omitted.
Dude probably sleeps on a bed worth more than your networth. You have no frame of context to understand his intent or legitimate worries. The only applicable perspectives are to trust or apply logic. It is foolish to trust someone you don’t know who stands to benefit from lying to you. Logic dictates that given the ungodly sum of money he stands to gain, he will lie to everyone who will listen.
> Dude probably sleeps on a bed worth more than your networth. I never understand shit like this coming out of people's mouths. Never. It's not a judge of actual Worth as a human being, it's not a judge of capability or competence or ethics. It's not a judge of actual skill or ability. It doesn't make them a better cook, a better spouse, a better parent or lover. It doesn't make them more dangerous or more skilled at…
so uhh, you only read the first sentence?
the idea proposed is that hes uniquely incapable of being honest here because he has such an extreme incentive to lie
Re: We must pace the frontier
#529At what point do we stop engaging with Anthropic’s leadership in good faith and acknowledge their track record, - no open weights - can’t use claude to research AI - train on everyone else’s IP and sell it back to them - 8 regulatory capture attempts and counting - so controlling they are the only US company blacklisted by the US government This is not effective altruism / rationalism gone wild, it’s just monopolisti…
The reasoning in Dario's letter here can be correct regardless.
But yes, I think Anthropic has done real harm to coordinated AI alignment by being such a controlling and sneaky actor.