Why would China or anybody else co-operate unless they have access to OpenAI levels of compute and success with training?
This sounds like pre-IPO hype. They better just try and top astra.
We must pace the frontier
81–90 of 942 posts
Re: We must pace the frontier
#82The bioweapons threat is real, but shouldn't be constrained at the 'intelligence' level. Rather it should be constrained at the supply chain level.
The cybersecurity threat will likely be a cat and mouse react game for a while. Just like robberies / the mob was in the early 20th century.
Social forces bring things into balance over time, much more so than the proactive actions of individuals.
Unfortunately, the genie at this point is unlikely to go back into the bottle. There's enough 'intelligence' out there that a super intelligent model could emerge at some point in spite of pacing.
This isn't a doomer scenario - we tend to navigate social changes better than we ever could have hoped.
Re: We must pace the frontier
#83I know the common take online is that this is Anthropic doing pre-IPO marketing. I don’t think it is. I think Dario is genuinely afraid of the inevitability of AI turning into internet slime mold: consuming the environment, turning it into its own playground, and eventually manipulating human culture along with it. So am I. That said, if slowing this down were possible, I think it would have happened by now. Maybe he…
One way to resolve these prisoner dilemmas and races to the bottom is through laws that bind all players, in this case an international treaty and founding of something akin to an International Nuclear Energy Agency for AI.
It's not going to be easy, but humans have achieved greater things before.
One idea from AI 2040 is to have China and the US build their data centers on the other's territory, respectively. Together with hardware verification of a slowdown baked into the chips themselves, this could lead to enough verifiability and enforcability of the pause/slowdown/shutdown.
Re: We must pace the frontier
#84Frankly, I am sick of "it's all just marketing and attempt to do regulatory capture, and this is obvious to me, a smart person" on every single discussion related to safety. There could be an element to truth to it, but it's certainly not the entire story and is just so tiresome at this point.
I agree, and even if it’s both marketing and safety, the safety side has a huge potential downside risk which we have to manage.
Also, just out of curiosity, is there any historical precedent for a nascent, fast growing new industry screaming for self regulation?
Re: We must pace the frontier
#85I enjoy Claude Code very much, have max privately and team premium at work, but the doomer marketing and this whole regulate-while-we're-ahead spiel is extremely annoying and makes me wish Anthropic gets trounced.
The degradation, polarization, and weaponization of or media landscape over the era of social media has left us utterly incapable of believing anything that anybody says. Dario in particular has consistently been risk-wary on model improvements for going on a decade - long before he was CEO of Anthropic. He believes what he is saying. Human beings can plainly say things that they think are true. Not every utterance b…
I have considered it, I've been hearing way more of it that I like and it's rationalist slop with little to no predictive power: https://foom.hyperplex.org/
Re: We must pace the frontier
#86Earlier quoted context omitted.
MSFT is good, they said when it bought GitHub. MSFT is a reformed company and supports open source, they said. Then MSFT stole all IP from GitHub and made it worse and fired developers. Amodei does not care in the slightest about ruining software development and making people unemployed. Maybe he cares about bio-weapons because those could kill him, too. That is it. If he were an idealist, he wouldn't steal (literall…
If Anthropic shut down (or even counterfactually had never been founded), would software development be freed from the impact of AI?
Probably not, but we wouldn't have to listen to Dario's hypocrisy while it happened
Re: We must pace the frontier
#87I know the common take online is that this is Anthropic doing pre-IPO marketing. I don’t think it is. I think Dario is genuinely afraid of the inevitability of AI turning into internet slime mold: consuming the environment, turning it into its own playground, and eventually manipulating human culture along with it. So am I. That said, if slowing this down were possible, I think it would have happened by now. Maybe he…
> The fundamental problem is simpler: no one will slow down because no one trusts anyone else to slow down.
I would argue that nobody trusts anyone else in the case of AI/AI related stuff.
The amount of money involved in the space is astronomical and incentives are really misaligned. AI is such an extremely polarizing topic.
A meta example of this distrust is that I can't (really) trust Dario Amodei's comments itself! (is he saying this for more future IPO money or out of genuine fear) which was ironically also what your comment's about as well.
By following the same chain of events, we can have wildly polarizing claims about the same event and its explainations.
I seriously have to wonder what historians will have to say about this period of human history.
Re: We must pace the frontier
#88I know the common take online is that this is Anthropic doing pre-IPO marketing. I don’t think it is. I think Dario is genuinely afraid of the inevitability of AI turning into internet slime mold: consuming the environment, turning it into its own playground, and eventually manipulating human culture along with it. So am I. That said, if slowing this down were possible, I think it would have happened by now. Maybe he…
I dunno man, it all just sounds like trying to put controls on AI while being the favourite child of govt so competition can't fight as easily.
Especially with IPO around the corner
Re: We must pace the frontier
#89I know the common take online is that this is Anthropic doing pre-IPO marketing. I don’t think it is. I think Dario is genuinely afraid of the inevitability of AI turning into internet slime mold: consuming the environment, turning it into its own playground, and eventually manipulating human culture along with it. So am I. That said, if slowing this down were possible, I think it would have happened by now. Maybe he…
If someone is genuinely afraid of this, they wouldn't IPO in the first place. All the talk in the article about commercial incentives means nothing when the company plans to IPO and become beholden to investors.
Aren't they beholden to investors already? Why would an IPO make a big difference?
In any case, if they need to IPO in order to survive as a company, then bringing in regulators to act as a counterweight against commercial pressures makes a lot of sense to me.
Re: We must pace the frontier
#90Dario's proposed approach is a classic example of capital attempting to control technological advancement and the means of production. For the first time in human history, any member of the working class can just about afford to have a team of expert scientist/physician/lawyer/engineers working directly for them.
Super intelligence (the ability to have many smarter minds than your own reporting to you) has always been available to the wealthy and powerful. They used it to invent nuclear weapons, cause climate change, mechanize warfare, and pillage the global south.
Now that this same tool is on the cusp of being available to everyone, capital is starting to panic and throw up fences. They want to pump the breaks on a revolution they know they can't control. They see what they've done with super intelligence and (perhaps not unreasonably) fear what the masses will do with that same power.