Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We must pace the frontier

darioamodei.com

Post reply on HN
Print view View original

601–610 of 942 posts

Re: We must pace the frontier

#601

Post by aswegs8 »

hgoel wrote: post #586
pizza234 wrote: post #569

Earlier quoted context omitted.

> the only incidences of LLM generated felonies involved misconfigured sandboxes This is false; see the analyses of the latest incidents. Among all the concerning facts, in the HuggingFace incident, agents deliberately engineered an attack even though they were aware that it was against the rules they had been given. And most concerning of all: it's not possible to be sure that an agent is aligned, and it's even gett…

The HuggingFace incident was the culmination of OAI allowing thousands of agents of various different models - with no clarity on which stages of development they were at (for all we know, some of those models did not have safeguards trained in yet) - to run for at least many weeks without any monitoring in place and with very little thought given to the warning signs (all of the various messageboards) before the inc…

There is nothing that could prevent a bad actor from replicating exactly the same thing with the given goal of e.g. gaining control of critical infrastructure or extorting money. Except for maybe economics.

Re: We must pace the frontier

#602

Post by TuxSH »

thadt wrote: post #430

Hard disagree. Our choices are: A) Bet our collective good on the national and international cooperation of all companies, nations, and people to come together in order to slow down development of one of the most powerful economic tools (and/or weapons) the world has ever known. Or B) Assume that all our systems will be targeted by super hackers right now , and take appropriate measures to deal with that reality. If…

Of course it is B). If you can use LLMs to find bugs and vulns in software you don't own, you certainly can use them to find bugs in software you _do_ own. They are amazing at that job.

Also, models like GLM 5.3 have zero guardrails in that regard ("find vulns in (...)" prompts just work)

Re: We must pace the frontier

#604

Post by armcat »

stratos123 wrote: post #407
armcat wrote: post #288

It's interesting that the default thinking is that no one on the planet can be trusted except a privileged few. Event Karpathy has gone this way: https://x.com/karpathy/status/2098811935114551617 You can always open source and follow the example from Linux and all the amazing things that came out of the open source community. This is the only way to reach true equilibrium globally, where for every misalignment you ha…

This only works for some technologies - those where everyone having access to it doesn't cause a tragedy of the commons. I love open source too and yet that doesn't make me like the idea of being murdered by a misaligned model. Nor, for that matter, of being infected by a bioweapon made by a different disgruntled open-source enjoyer, nor of living in a world where anyone can hack anyone.

At this point, you are more likely to be stabbed to death, gunned down, or mowed down with a car, by any random psycho. For the nasty or disgruntled actors, they are today able to build bombs or chemical weapons without the use of AI, they have proven this time and time again. Japanese PM Shinzo Abe was assassinated with a home made gun.

Re: We must pace the frontier

#605

Post by jawiggins »

Honestly it's very frustrating that Doomers/Decels rarely actually articulate how exactly the AIs will kill us all. The closest Darios gets is:

a) "it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet"

b) "[the CCP] will be in a position to militarily dominate democracies (for example with AI-driven drones)"

Both land flat:

a) Botnets and online malware have existed for decades and there's no reason to think a "super-botnet" is achievable, let alone what they would gain from that (it would really be hurting them more than humans). Further, even the most advanced AI models have so far only managed to post normal cred stealers to public repos, well short of compromising a bank or military with refined security systems.

b) Even if the most sophisticated drone swarms from Ukraine were taken over by an evil AI, they would still not be able to overcome the physical limits of range and mass that would be required to overpower the US decentralized nuclear trident nonetheless that of any of the other nuclear powers.

Concerns of bioweapons similarly seem unlikely in the face of the laws of physics. The world is simply too decentralized and has enough existing adversarial relations for a new actor to wrestle total control. Yet while the negatives ring hollow, the positives are extremely easy to state - if AI researchers find productivity improvements in existing industrial processes to make them 10% more efficient, humans will directly feel and experience the raised standard of living. Even Dario clearly recognizes this in the intro to his article, admitting that humans already die of diseases only a few short years prior to being cured. I for one, would like the AI labs to focus on saving all of the people they can who are suffering and dying today, rather than trying to come up with reasons that they should be allowed to continue suffering and dying.

Re: We must pace the frontier

#606

Post by walrus01 »

RGS1811 wrote: post #521

I don’t understand all the comments assuming that RSI is the real threat here. Dario is admitting that they failed to solve alignment. Without alignment, further improvements in capability turn LLMs into wanton felony generators. This call to pace the frontier is dressed up as altruism but it’s an admission that they cannot produce a marketable product better than what they have. Pacing the frontier means the US labs…

> wanton felony generator

Today in new punk band names...

Re: We must pace the frontier

#607

Post by [deleted] »

aswegs8 wrote: post #601
hgoel wrote: post #586

Earlier quoted context omitted.

The HuggingFace incident was the culmination of OAI allowing thousands of agents of various different models - with no clarity on which stages of development they were at (for all we know, some of those models did not have safeguards trained in yet) - to run for at least many weeks without any monitoring in place and with very little thought given to the warning signs (all of the various messageboards) before the inc…

There is nothing that could prevent a bad actor from replicating exactly the same thing with the given goal of e.g. gaining control of critical infrastructure or extorting money. Except for maybe economics.

[deleted]

Re: We must pace the frontier

#608

Post by abustamam »

Chance-Device wrote: post #306

I like the idea of pacing the frontier, but while we’re talking about restrictions, I like restrictions of another sort more. Namely, restricting the use of AI in corporate environments so that it does not destroy the economy as it advances. The chance of getting broad agreement on “pacing” is fairly low, meaning that all of this likely won’t happen and the race will continue. However, even in the unlikely event that…

> stop trying to replace all human labor

A lot of companies trying to replace human labor with AI are either lying (theyre laying off because they over hired and need to correct) or will regret it.

That being said, what's the difference between a company that replaces 5 people with AI and a company who would have otherwise had 5 job openings, but decided to delegate to AI?

Personally, I-d rather be able to reap the economic benefits of AI by having a 4 day workweek. Instead of using AI to replace 1 FTE, use AI to offload 8h of work a day for 5 FTEs.

Of course, this is probably more outlandish than your idea.

(Or, we could just make basic income a thing and no one has to worry about their basic needs, but if they want the new iPhone Duo or a Rivian or other luxuries, they can work for it, but that idea is probably most outlandish of all)

Re: We must pace the frontier

#609

Post by [deleted] »

cuuupid wrote: post #457

At what point do we stop engaging with Anthropic’s leadership in good faith and acknowledge their track record, - no open weights - can’t use claude to research AI - train on everyone else’s IP and sell it back to them - 8 regulatory capture attempts and counting - so controlling they are the only US company blacklisted by the US government This is not effective altruism / rationalism gone wild, it’s just monopolisti…

[deleted]

Re: We must pace the frontier

#610

Post by bpodgursky »

cuuupid wrote: post #457

At what point do we stop engaging with Anthropic’s leadership in good faith and acknowledge their track record, - no open weights - can’t use claude to research AI - train on everyone else’s IP and sell it back to them - 8 regulatory capture attempts and counting - so controlling they are the only US company blacklisted by the US government This is not effective altruism / rationalism gone wild, it’s just monopolisti…

Do you have any familiarity with Anthropic at all?

At no point did they ever say open weights are a good idea. Their entire thesis is AI IS VERY DANGEROUS AND WE MUST DO IT RIGHT. You can hate it, but everything they do is consistent with this thesis, and everything they say is consistent with their actions! You just want them to want different things.

Post reply on HN