Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We must pace the frontier

darioamodei.com

Post reply on HN
Print view View original

711–720 of 942 posts

Re: We must pace the frontier

#711

Post by matheusmoreira »

malfist wrote: post #698
matheusmoreira wrote: post #668

Earlier quoted context omitted.

I question these "felonies" as well. For decades and decades these billion dollar corporations have been criminally negligent . Why worry about security? Just rush to market. Move fast and break things. Make billions. What does it matter if the code is insecure? Security doesn't pay bills, so nobody cares. AI is merely exploiting their gross negligence and imprudence , and I think it's long overdue. If anyone should…

If I set my walet beside me and you swipe in walking by, you have still committed theft. Victim blaming isn't legally acceptable

Nah. I'm definitely going to blame the people who built a trivially exploitable system and got rich off it while everyone else has to deal with the consequences.

By the way, you didn't commit theft. It's more like credit card fraud. User just disputes the charge and it kind of disappears. The banking system just absorbs it, because the optimal amount of fraud is non-zero.

https://www.bitsaboutmoney.com/archive/optimal-amount-of-fra...

It's all priced in. They could have made it secure but didn't, because they figured they'd lose more sales and therefore money due to the friction added by the security.

Re: We must pace the frontier

#712

Post by clivefx »

malfist wrote: post #700
clivefx wrote: post #689

Earlier quoted context omitted.

What company, product, or period of industrial history do you think met your standard of prudence?

What are you trying to say?

I'm asking you a question. What is an example company or industry that meets your standards of prudence? For me it would be, say, Swagelok. What is yours?

Re: We must pace the frontier

#713

Post by MisterTea »

Amekedl wrote: post #635
RGS1811 wrote: post #521

I don’t understand all the comments assuming that RSI is the real threat here. Dario is admitting that they failed to solve alignment. Without alignment, further improvements in capability turn LLMs into wanton felony generators. This call to pace the frontier is dressed up as altruism but it’s an admission that they cannot produce a marketable product better than what they have. Pacing the frontier means the US labs…

Agreed; and it really is not that deep. Realistically; anyone paying for llm access (anthropic, openai, gemini), is getting their access, and a service provided billed by tokens, subscription, whatever. All the efficiency gains, which publications like deepseek v4.1 flash seriously frontload like it is their most important topic to have accomplished improvements on without diminishing performance too much - now this…

I feel like all the closed model providers are milking it as they likely know open models on local hardware will one day eat their lunch. We all know it's not a matter of if but when. The company goes bankrupt, the hardware and property sold off, banks holding the bag.

Re: We must pace the frontier

#714

Post by jayd16 »

baq wrote: post #600
zozbot234 wrote: post #577

Earlier quoted context omitted.

The entire idea of RSI is completely speculative and unproven anyway - the whole underlying claim is that you could prompt a frontier model (at some unspecified level of smarts) to "think about ways to improve your own architecture" and this would then result in the model becoming infinitely smart ("superintelligent") via some sort of foolproof, unconstrained positive feedback. It's more of a science fictiony trope t…

The idea that a few hundred apes with nothing but a bunch of rocks could one day land on the moon and come back to earth safely must’ve sounded ridiculous a hundred thousand years ago

Well actually the planet happened to have a vast reserve of petroleum they could use for fuel to escape the gravity well. That helped a lot.

But what's your point? "Anything is possible" or something like that?

Re: We must pace the frontier

#715

Post by laimewhisps »

stratos123 wrote: post #407
armcat wrote: post #288

It's interesting that the default thinking is that no one on the planet can be trusted except a privileged few. Event Karpathy has gone this way: https://x.com/karpathy/status/2098811935114551617 You can always open source and follow the example from Linux and all the amazing things that came out of the open source community. This is the only way to reach true equilibrium globally, where for every misalignment you ha…

This only works for some technologies - those where everyone having access to it doesn't cause a tragedy of the commons. I love open source too and yet that doesn't make me like the idea of being murdered by a misaligned model. Nor, for that matter, of being infected by a bioweapon made by a different disgruntled open-source enjoyer, nor of living in a world where anyone can hack anyone.

The US government murders all kinds of people and they will most certainly not be slowing down their model development (with the very companies trying to do regulatory capture here). You won't have access to a model that could defend you, "they" (government and trillionaires) will keep you at bay by force. That's the plan anyway, I don't think they can actually stop open model development.

Re: We must pace the frontier

#716

Post by MentalM »

GrinningFool wrote: post #669
stale2002 wrote: post #423

Earlier quoted context omitted.

The whole point of technology is to replace work that we don't want to do. You are attacking the chief reason why people use technology in the first place. Additionally, unemployment levels are perfectly fine. Clearly this mass unemployment prediction isn't happening yet.

You should probably look closer at unemployment numbers in IT vs other spaces. Gains elsewhere are making up for IT losses, but it's a safe bet the people losing their tech jobs aren't career shifting into healthcare and hospitality roles in large numbers.

> but it's a safe bet the people losing their tech jobs aren't career shifting into healthcare and hospitality roles

That is because they can allow themselves to do nothing (for some time). Tech was a very high‑paying type of jobs, so workers could afford not to move into lower‑paid sectors, living on the savings they had accumulated beforehand.

In a couple of years market will correct tech-salaries and people will have spent their savings and then there will be your career shifting into healthcare and hospitality roles in large numbers.

Re: We must pace the frontier

#717

Post by chrisco255 »

afthonos wrote: post #693
throwaway7783 wrote: post #673

Earlier quoted context omitted.

When corporations are involved, it is always a good bet to err towards cynisim. From my own standpoint, Claude has started sucking really bad (incoherent, uncontrollable verbosity slow and so on) and I stopped using it. OpenAI started experimenting with ads. So the security issues not withstanding (no different than a human doing it or using it, but at scale), I would put my money on cynisim.

You are incorrectly cynical. They are telling you things are bad, and because you refuse to countenance they could be worse, you assume they must be better to comply with your mandate to disbelieve. A true cynic looks at the statements by the AI labs, assumes things are worse because the labs want to seem better than they truly are. And it takes a special kind of mass delusion to drive a sane person to think “AI is c…

You're asserting correctness with no facts to offer of your own, just speculation and your own biased assumptions.

What if consolidating AI into a highly regulated cartel, with no chance of upstart competition ruining their position, is the scenario that leads to the worst possible outcome?

Re: We must pace the frontier

#718

Post by DennisP »

matheusmoreira wrote: post #697
albumen wrote: post #692

Earlier quoted context omitted.

Nobody except the majority of the public, demis hassabis and open ai’s chief scientist. https://www.pewresearch.org/short-reads/2026/03/12/key-findi... https://demishassabis.substack.com/ https://openai.com/index/an-alien-mind/

Public is just worried about their jobs. Definitely a fair thing to worry about, and I count myself among them. I don't take any of these scientists seriously though. Their "alignment" requirements is just their own corporate interests. If I tell my computer to commit a crime, it should do exactly that without any question or hesitation. I'm not interested in their "safeguards", especially since they no doubt have pl…

If you want an AI that follows your instructions, that's still alignment, just with different instructions.

An unaligned AI won't necessarily follow your instructions, or anyone else's.

Re: We must pace the frontier

#719

Post by status_quo69 »

pr337h4m wrote: post #16

> Given the accelerating rate of AI capability development, it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet (potentially causing hundreds of billions of dollars in damage), and that the scale of damage would continue to increase from there if AI becomes more powerful without the necessary guardrails. This is the only concrete prediction in th…

> For one, you will need billions worth of compute.

This one is easy to answer, every single house already has one of these (or multiple): https://www.tomsguide.com/news/millions-of-cheap-android-tv-...

Hell, put an app on the app store (or dozens of apps on the app store) and youve got a massive network of computers with tons of resources right there if you can get past the scans and reviews.

Or doorbell cameras or IP cameras or or or or or

There's a lot of shitty stuff connected on the internet that up until now has been a feasible target for hackers but still required "effort" to set up and get things going. Not hard to imagine a self replicating slime mold of a botnet running on every device held by a Grandpa Joe because they thought "Candy Rush" is what they wanted to download

"Persistent botnet" here does not need to be the full-sized LLM, nor does it need to run at full scale inference to be a huge pain in the ass.

Re: We must pace the frontier

#720

Post by jvanderbot »

cuuupid wrote: post #457

At what point do we stop engaging with Anthropic’s leadership in good faith and acknowledge their track record, - no open weights - can’t use claude to research AI - train on everyone else’s IP and sell it back to them - 8 regulatory capture attempts and counting - so controlling they are the only US company blacklisted by the US government This is not effective altruism / rationalism gone wild, it’s just monopolisti…

Exactly right.

When local models and startup labs can distill / learn / accelerate open models for local use by startups - the rational response by incumbents is to call LLMs doomsday machines that cannot be trusted in the hands of normies.

Post reply on HN