Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We must pace the frontier

darioamodei.com

Post reply on HN
Print view View original

341–350 of 943 posts

Re: We must pace the frontier

#341

Post by youoy »

stratos123 wrote: post #276
youoy wrote: post #263

Earlier quoted context omitted.

AI technology is now only audited by people who think like them. The ones that dont either dont join the company, or leave after a short stint as we have seen. That creates an information or feedback bubble which is not healthy nor productive.

Okay, but suppose that Hypothetical Opensource Anthropic trains a model that turns out to be very dangerous, and releases it. Suppose that the public investigates and, not being limited by an information bubble, correctly notices that it's very dangerous. What then? The model's already released, there's effectively no way to prevent it from being used. Whatever the risks of its release were, they will now materialize…

By the same logic, suppose that Dario/Altman/Jensen accumulate all capital because they are the only one that have access to AGI and end up controlling democracy, turning the world into technofeudalism or whatever you want to call it. How is that better than an opensource Anthropic. At least an open source anthropic would allow to join economic forces to try to combat that situation, for example. But there are other more clear, less distopian benefits.

Re: We must pace the frontier

#342

Post by glub »

TheSisb2 wrote: post #311
glub wrote: post #302

Dario, Sam, and Elon are all on the same page on this. So it's either they truly think AI is going to kill us all, or there's some other motives at play here. I don't think these people could possibly agree on the color of the sky, so what could the other possible motives be, based on what we know? OpenAI / Anthropic models have largely stopped advancing - that's not a good look when you're pre-IPO and Chinese models…

> OpenAI / Anthropic models have largely stopped advancing - that's not a good look when you're pre-IPO and Chinese models are catching up. This is obviously untrue… do you use any of them?

Anthropic could serve Opus 4.5 from a year ago under opus:latest and most heavy users would probably have no idea. Some of them would probably even prefer it.

Yes, I do use them, quite heavily. The only difference at this point is in benchmarks that can't be trusted (see: artificial analysis on Astra), or in the way models communicate.

Most gains are now from RL, which for some reason is hyperfocused on improving cyber capabilities, and harnesses. Raw intelligence gains of base models is absolutely slowing down.

Re: We must pace the frontier

#343

Post by figassis »

"greatly accelerate economic growth rates" - I always wonder about this goal. I understand increasing economic efficiency. But what do we as a species gain from global economy growth.

When we say the world's GDP grew by X%, what is the point of that growth? Inflation? If the economy grows as a direct effect of humanity growing, I get that. If the economy becomes more efficient and we can not get more with less, I also get that.

But what is the point of just growth, if not simply to show others that I am growing faster than you? At which point this is a meaningless number. What am I missing?

Re: We must pace the frontier

#344

Post by stratos123 »

cubic_earth wrote: post #336
stratos123 wrote: post #190

Earlier quoted context omitted.

It's not possible for models to replicate across consumer computers (without a major advance in distributed computing, at least), but that doesn't mean they can't replicate at all. There are services that'll rent you GPU pods by the hour with zero oversight, so even today, if a model can get access to some money and exfiltrate its weights, it can rent a bunch of GPU pods and run itself there. (It's not going to be tr…

There are very lightweight models out there. Not everyone in an army is a general. The swarm could be 100,000s of thousands of tiny models, and they could manage conventional botnet computers, and they could all take direction and guidance from a handful of frontier generals

Maybe, but I don't think tiny models can be harnessed as intelligence. As in, if you have one rogue Mythos overseeing the swarm, it only produces 1 Mythos's worth of useful thoughts no matter how many gemma4:e4bs it consists of. And that removes the most dangerous part of AI-controlled botnets, which is a blowup in available inference compute - the Mythos general might as well replace the tiny models with ordinary worms and have it just be an ordinary botnet.

Re: We must pace the frontier

#345

Post by youoy »

0xDEAFBEAD wrote: post #293
youoy wrote: post #263

Earlier quoted context omitted.

AI technology is now only audited by people who think like them. The ones that dont either dont join the company, or leave after a short stint as we have seen. That creates an information or feedback bubble which is not healthy nor productive.

Why don't you create your own auditing org to fix this? Or at the very least, publish a detailed critique of what you believe existing auditing orgs are missing.

Will they give my auditing org access to their proprietary confidential secrets? Or will that happen only if they know i agree enough with them so that i am not a risk?

Re: We must pace the frontier

#346

Post by throwaway81523 »

Who is "we" and while pacing is good, I'm way more uncomfortable with the frontier being controlled by a few companies that managed to predatorially gobble all the world's human-created work as training data, before everything got throttled to stop that scraping. We need open training and open models.

Re: We must pace the frontier

#347

Post by chis »

figassis wrote: post #343

"greatly accelerate economic growth rates" - I always wonder about this goal. I understand increasing economic efficiency. But what do we as a species gain from global economy growth. When we say the world's GDP grew by X%, what is the point of that growth? Inflation? If the economy grows as a direct effect of humanity growing, I get that. If the economy becomes more efficient and we can not get more with less, I als…

GDP growth and inflation are different things. If inflation-adjusted GDP goes up, that means the average person is able to buy more things they want. Compare living in the US vs India today for the median citizen

Re: We must pace the frontier

#348

Post by rdm_blackhole »

> Therefore any agreement must either have ironclad verifiability, or must be limited enough that defection would not be militarily existential.

How does he propose that such a thing will work? Are we going to have US AI inspectors in China and vice versa?

Why would China agree to such thing in the first place since this is a winner takes all situation.

If the US slows down or stops altogether, China can continue to work on their AI and overtake the US, if China accelerates and the US keeps its current pace then it can overtake the US.

The only way out is on the contrary for the US to actually go faster and increase its lead so that China is always 6 to 12 months behind and/or reach AGI/ASI first at which point China will most likely develop its own not long after.

This would actually be the best outcome, just like the MAD doctrine contained the spread and usage of nuclear weapons, having two superpowers with AGI would ensure that they can't be used to arm anyone. Unless they escape their sandbox but that is highly theoretical.

Regarding these Embedded Evaluators who are supposed to be neutral: - who controls them? - who has oversight on their decisions? - can their decisions be challenged by the public or a government? - who has the final say whether a model is "compliant" enough? - what does "alignment" mean in this context and who decides if a model is aligned enough?

Re: We must pace the frontier

#349

Post by m12k »

To be honest, I think we're incredibly lucky to be advancing AI this far already, while the world still has so many non-digitized systems and manual processes. I imagine that in e.g. 50 years, the world will be so connected that it can basically be "conquered" from the internet. I'd much rather have AI burst onto the scene we have today.

Re: We must pace the frontier

#350

Post by logicchains »

figassis wrote: post #343

"greatly accelerate economic growth rates" - I always wonder about this goal. I understand increasing economic efficiency. But what do we as a species gain from global economy growth. When we say the world's GDP grew by X%, what is the point of that growth? Inflation? If the economy grows as a direct effect of humanity growing, I get that. If the economy becomes more efficient and we can not get more with less, I als…

Low growth means young people are miserable and have worse standards of living than their parents, as we're seeing in most of western Europe, where GDP per capita hasn't meaningfully increased for decades.
Post reply on HN