Right as open source models catch up to frontier closed source ones for 1/10th (or less) of the cost suddenly it's time to hit the brakes! Funny how that works.
We must pace the frontier
231–240 of 942 posts
Re: We must pace the frontier
#232Earlier quoted context omitted.
If someone is genuinely afraid of this, they wouldn't IPO in the first place. All the talk in the article about commercial incentives means nothing when the company plans to IPO and become beholden to investors.
It's almost like the people at Anthropic and other AI labs are slaves to a misaligned reward function that encourages optimization of a metric (profit) over human values, even at what they claim is the risk of extinction. We built the paperclip maximizer, and it is capitalism.
> are slaves to a misaligned reward function that encourages optimization of a metric (profit) over human values, even at what they claim is the risk of extinction
This was already the case even before the "frontier AI" age. The big question is, will these glaring AI arms-race threats be obvious to enough people to rethink the underlying systemic flaw driving it all?
Not holding my breath on that one.
Re: We must pace the frontier
#233Right as open source models catch up to frontier closed source ones for 1/10th (or less) of the cost suddenly it's time to hit the brakes! Funny how that works. https://x.com/BasedTorba/status/2098795920720547916
Couldn't be more obvious.
Re: We must pace the frontier
#234Earlier quoted context omitted.
The more I read about everything that has been written regarding AI regulation since the OAI/HF incident, and the more it reminds me of the nuclear arms race (although the potential consequences would possibly be very different). Surely, we managed to negotiate a Non-Proliferation Treaty with most of the countries of the globe. Cannot we take inspiration from that for AI?
I also think the nuclear arms race is a good comparison. I think in hindsight, we've gotten extremely lucky with how the development of nuclear weaponry went, in ways we probably won't with AI. 1) Right after nuclear bombs were first developed, they were used in an extremely public way to kill 200 000 people. This was enough of a cold shower that everyone was agreeable to a global non-proliferation treaty. With AI, w…
> Right after nuclear bombs were first developed, they were used in an extremely public way to kill 200 000 people. This was enough of a cold shower that everyone was agreeable to a global non-proliferation treaty.
Not really. Before NPT you missed a small gap in there of 20 years where the USA and USSR built 10's of thousands of nuclear weapons and ICBMs.
And actually, public perception at the time saw Nuclear Weapons extremely favorably, as the bombs dropped on Hiroshima and Nagasaki ended the deadliest war in human history that killed 50-60 million people, most of which died excruciating deaths in trench warfare, fire bombing, chemical warfare, starvation and so on.
Re: We must pace the frontier
#235Earlier quoted context omitted.
The bioweapons argument is one of the favorites used by these con artists and dreamed up by their PR team. Except Bioweapons already existed before LLMs, Adversarial governments already have them, they are already easy to make. You could use the same bullshit argument for why we need to ban libraries, books, or require a license to buy an internet connection.
An undergrad bio student with some lab experience can do some effing terrifying things with about $20k in equipment and time and access to some papers and a library. AI is not needed, but it might help accelerate the research. Not going to go into it but I studied biology. It’s all out there. It’s easier than you think. It hasn’t happened yet because… nobody has done it. That’s the answer. There is no policeable phys…
I love watching NileRed/NileBlue on youtube - crazy chemist that does a lot of insane stuff. While it's obvious that he's very smart and probably way above the average, his education is still nothing that probably millions of people don't have:
> Bachelor of Science degree in biochemistry with a minor in pharmacology
Watching him explain things has made me realize that knowing how to manufacture a very dangerous thing probably requires attending some classes and knowing how to read a paper. And the way he just casually orders dangerous materials makes me feel like there are just online stores with 2-day shipping after you upload your ID or something.
It also made me think that lack of specialized education would get me nowhere if I wanted to replicate whatever he's doing, even if an LLM guided me step-by-step, because I'd probably do something stupid (or AI would miss a crucial instruction/hallucinate) and kill myself first.
So my opinion on this is that people who could pose any danger were already posing it before LLMs and LLMs won't materially change that.
Re: We must pace the frontier
#236I know the common take online is that this is Anthropic doing pre-IPO marketing. I don’t think it is. I think Dario is genuinely afraid of the inevitability of AI turning into internet slime mold: consuming the environment, turning it into its own playground, and eventually manipulating human culture along with it. So am I. That said, if slowing this down were possible, I think it would have happened by now. Maybe he…
You have to be living under a rock to believe a word this pathological liar says.
Re: We must pace the frontier
#237Earlier quoted context omitted.
This seems highly optimistic to me. What's to say current AI won't be highly power-concentrating by default? The best models are owned by a few companies, and displacement of knowledge-workers mainly seems to benefit the capital class. On-device / edge computing makes sense in a few very limited scenarios. And economically, price or watt/token (or watt/task completed) might always be better in large data centers. No…
> The best models are owned by a few companies In terms of cost per task, the open weight Chinese models are winning by a long shot. So it depends on what you mean by the best model.
That’s only one metric. It doesn’t matter how cheap it is, if it continually fails to solve a problem.
Yes they’re getting better. No, I don’t think this will be a panacea. If only for the fact that this is China (more specifically the CCP) after all - they’ve got their own plan, and altruism is not part of it.
Re: We must pace the frontier
#238We can cure Cancer. No wait nevermind, actually slow us down. If we are actually close to ASI then no one in their right mind would say slow down
They are not going to "slow down" anyway. When has tech ever "slowed down"? The only way to "slow down" progress is by implementing anticompetitive measures, which would benefit those in lead.
Re: We must pace the frontier
#239I know the common take online is that this is Anthropic doing pre-IPO marketing. I don’t think it is. I think Dario is genuinely afraid of the inevitability of AI turning into internet slime mold: consuming the environment, turning it into its own playground, and eventually manipulating human culture along with it. So am I. That said, if slowing this down were possible, I think it would have happened by now. Maybe he…
I dunno man, it all just sounds like trying to put controls on AI while being the favourite child of govt so competition can't fight as easily. Especially with IPO around the corner
And getting to choose his own "embedded evaluator" org that has deep ties to everyone in the doomer media campaign.
It's all so obvious.
Re: We must pace the frontier
#240Earlier quoted context omitted.
If someone is genuinely afraid of this, they wouldn't IPO in the first place. All the talk in the article about commercial incentives means nothing when the company plans to IPO and become beholden to investors.
>> ...and become beholden to investors Anthropic is structured as a Public Benefit Corporation with a strong charter. In addition, founders will have super-voting shares and so it won't be possible to push them out. Therefore the whole "beholden to investors" thing is not a concern.
Don't underestimate the ability of the courts and the state to pull the rug out from under the legal system. Speaking as an outsider from Canada, it looks very much like all bets are off in terms of respecting checks and balances in the United States and it's very realistically possible that unless there is a big upset and turn around the next couple of years any traces of democracy in the US will be a farcical nod to what the founders built as a way to paper over the abuses.
I really hope I am wrong as my perspective is not anti-American, it's specifically anti-corruption and pro-democracy.