Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We must pace the frontier

darioamodei.com

Post reply on HN
Print view View original

401–410 of 937 posts

Re: We must pace the frontier

#401

Post by esafak »

politician wrote: post #394
esafak wrote: post #382

Earlier quoted context omitted.

This makes no sense. The hardware is only going to get cheaper. In a decade, everybody could be running an ASI on their phones. What we need to do is to make unaligned AIs illegal and monitor for them, like we do with nuclear weapons. And make aligned AI strong enough to counter it, for deterrence and defense.

I downvoted you because I think the OPs model is far simpler to implement and suffers from fewer conflicts of interest. If we go down the regulation of alignment route, we'll have to ask experts to create those regulations and monitoring regimes. And which experts will the government ask? Oh, right, the parties that stand to benefit the most from regulation: OpenAI and Anthropic! When "the experts" make certification…

> And which experts will the government ask? Oh, right, the parties that stand to benefit the most from regulation: OpenAI and Anthropic!

You've never heard of academia, NGOs, and intergovernmental organizations? Sure, the AI labs will have a say but not all of it. You don't ask the fox to guard the henhouse.

Re: We must pace the frontier

#402

Post by fwn »

stratos123 wrote: post #174
iloveoof wrote: post #75

Distillation is a great thing for consumers. It improves competition and reduces the massive moats that OpenAI and Anthropic have in compute that would otherwise lead them to be duopolists. It’s also only fair that AIs trained on humanity’s wealth of knowledge for Pennie’s allow competition to train on humanity’s wealth of knowledge at market cost.

Distillation is only a great thing for consumers as long as you ignore all AI risks, which are what this post is about. If you don't, you have to weight greater access to better open-source models against greater exposure to risks caused by these models existing. Everything hinges on how major you think the risks will be.

The biggest AI risk, by far, is the concentration of power in a few companies, located in the US.

There is a lot of fear marketing about our text generators turning into Terminator. But other than the centralization of power, such fears are largely fiction. (Actual fiction, stuff like ai2027.)

And the labs know it: If Anthropic or OpenAI believed in their own narrative of being on the brink of world dominating superintelligence, they absolutely would not plan to IPO rn.

The slowdown narrative is probably just a hedge, or a face-saving way to lower expectations in case they can not keep improving at the same speed until they actually IPO.

Re: We must pace the frontier

#403

Post by manquer »

enraged_camel wrote: post #182
Jcampuzano2 wrote: post #63

Earlier quoted context omitted.

If someone is genuinely afraid of this, they wouldn't IPO in the first place. All the talk in the article about commercial incentives means nothing when the company plans to IPO and become beholden to investors.

>> ...and become beholden to investors Anthropic is structured as a Public Benefit Corporation with a strong charter. In addition, founders will have super-voting shares and so it won't be possible to push them out. Therefore the whole "beholden to investors" thing is not a concern.

As long you are burning more cash than you bring in, you are beholden to investors - whether it is retail, VCs, banks or a government giving you a bailout it is still someone signing you a check.

Golden shares, vetos, PBC, charter are all paper tigers , they only matter if/when the firm is self-sustaining business with no outside capital needed, the alternative to not listening to investors till then is crash and burn.

After that point, you will have to listen to the paying customers (sometimes but not always they are also users ) as they are ones now funding your organization.

Bottom line you are always listening to someone.

Re: We must pace the frontier

#404

Post by csomar »

armcat wrote: post #288

It's interesting that the default thinking is that no one on the planet can be trusted except a privileged few. Event Karpathy has gone this way: https://x.com/karpathy/status/2098811935114551617 You can always open source and follow the example from Linux and all the amazing things that came out of the open source community. This is the only way to reach true equilibrium globally, where for every misalignment you ha…

Embedded evaluator will be a great side/consulting-gig for Karpathy and the likes. How do you think these people will be picked?

Re: We must pace the frontier

#405

Post by HarHarVeryFunny »

Chance-Device wrote: post #306

I like the idea of pacing the frontier, but while we’re talking about restrictions, I like restrictions of another sort more. Namely, restricting the use of AI in corporate environments so that it does not destroy the economy as it advances. The chance of getting broad agreement on “pacing” is fairly low, meaning that all of this likely won’t happen and the race will continue. However, even in the unlikely event that…

> Namely, restricting the use of AI in corporate environments so that it does not destroy the economy as it advances.

Indeed, and the same logic applies as to why allowing companies to replace too many worker's by H1B holders is a bad idea (which equally applies to allowing companies to replace domestic jobs by offshoring).

If you don't hire people, who pay taxes, and take what's left to buy stuff and keep the economy going, then how DOES the economy keep going?

I guess Amodei sees us all on UBI, aka food stamps, so at least the farmers will be selling something, but is OpenAI going to be accepting food stamps to pay for ChatGPT subscriptions? Tesla better be selling it's robots real cheap if it is planning on selling them to people that only have UBI as an income.

Re: We must pace the frontier

#406

Post by Razengan »

dickersnoodle wrote: post #376
Razengan wrote: post #148

Earlier quoted context omitted.

> genuinely afraid of the inevitability of AI turning into I'm going to get pitchforked on this bandwagon, but is really no one here genuinely -excited- about AI? of it turning into an evolution of sentience, or it turning out to be our first contact with alien intelligence? "durrr it's just matrix multiplications" mfer so is your brain. What humans should be doing is overhauling archaic social institutions to keep u…

>"durrr it's just matrix multiplications" mfer so is your brain. Tell me you don't know anything about neuroscience without telling me you don't know anything about neuroscience.

The point was that everything can be oversimplified down to dismiss any emergent properties

"It's just chemicals"

Like how some morons try to downplay the capacity of pain and emotions in animals: "It's just self-preservation"

"Play is just training for hunting, they're not really having 'fun'" and so on.

Re: We must pace the frontier

#407

Post by stratos123 »

armcat wrote: post #288

It's interesting that the default thinking is that no one on the planet can be trusted except a privileged few. Event Karpathy has gone this way: https://x.com/karpathy/status/2098811935114551617 You can always open source and follow the example from Linux and all the amazing things that came out of the open source community. This is the only way to reach true equilibrium globally, where for every misalignment you ha…

This only works for some technologies - those where everyone having access to it doesn't cause a tragedy of the commons. I love open source too and yet that doesn't make me like the idea of being murdered by a misaligned model. Nor, for that matter, of being infected by a bioweapon made by a different disgruntled open-source enjoyer, nor of living in a world where anyone can hack anyone.

Re: We must pace the frontier

#408

Post by try-working »

Fable and Astra are what we currently call frontier models, but to be more specific they are generalist models, built in pursuit of AGI. The strategy is to have one single model that does everything, whether it's writing code or doing research, etc. Fable is a single, massively sized models that is intended to do specialist work across every domain.

The issue with this is first of all that it is the contradiction of a generalist doing specialist work, and that contradiction creates the present situation with model profiles that ensure that these models will rarely be chosen in a pool of models like V4.1 Flash that can now do GPT 5.4-level work.

We are seeing this reflected in the market where companies and individual developers are moving away from frontier models toward models with better cost profiles. In a sense, the market is killing Anthropic's dreams of AGI.

And there we have it: the reason I say that Anthropic is no longer a frontier lab is because after this July, we have proof that their strategy and the models they put out do not match what us, the users and the market, needs and doesn't fit the work we need models to do. As a result, Anthropic's market share is dropping rapidly, and how can you be a frontier lab when you're losing every day, for months, without end in sight?

https://x.com/trydotworks/status/2098618997230985375

Re: We must pace the frontier

#409

Post by dismalaf »

This is corporate speak for "LLMs have hit a wall". I mean, it's been obvious for a bit, lately nearly all gains have been from harnesses (or whatever you want to call all the non-LLM bits that make up a chatbot or agent).

If Anthropic and OpenAI were still seeing exponential or even linear gains from scaling they'd be doing it because the rewards to reaching AGI or SGI before everyone else are basically infinite. If both are talking about slowing down it means there's no known path to AGI so they're both going to push the safety angle as an excuse to slow down training new models and take profit.

Re: We must pace the frontier

#410

Post by glub »

GeneralMayhem wrote: post #266
glub wrote: post #247

Earlier quoted context omitted.

So what's the fuss about then? Is the idea that a Chinese company will release a model that will have no safeguards? For what purpose? Basic safeguards are all that's required, and they've been there in every usable model since GPT-2, including Chinese models that are supposedly "unsafe". Or are we saying that some lunatics will start training their own models, spin up a GPU cluster, run some abliteration workflow, o…

Yes, that is exactly Dario's concern. Either one of the US labs or one of the Chinese ones will eventually release something with insufficient safety controls for its power level because it gives them slightly better user retention (look how much complaining there is about current frontier models, especially Fable, rejecting requests). Regulation or consortium is how you avoid the prisoner's dilemma.

As long as user provides inputs and LLMs stay LLMs, you can waltz through any guardrail. Fable is the extreme case, but it's not that hard if you know what you're doing and know how LLMs and their guardrails work.

Am I saying that guardrails don't work? No, they probably stop a lot of insane people trying insane things. But you don't need Fable-level guardrails to do that. You probably don't even need to do anything during pretraining, or RL, or classification to make sure model refuses to compy with "hack me a bank" or "make me a chemical weapon".

All models will automatically have guardrails just as a result of training on data that gives them intelligence. You have to actually train it to be malicious to produce something what Dario calls "insufficient guardrails".

No guardrail is going to stop a determined person with sufficient intelligence. It only has to stop ones with insufficient one, and even basic guardrail that are just by-product of training is going to achieve that.

Post reply on HN