Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We must pace the frontier

darioamodei.com

Post reply on HN
Print view View original

541–550 of 930 posts

Re: We must pace the frontier

#541

Post by throwatdem12311 »

RGS1811 wrote: post #521

I don’t understand all the comments assuming that RSI is the real threat here. Dario is admitting that they failed to solve alignment. Without alignment, further improvements in capability turn LLMs into wanton felony generators. This call to pace the frontier is dressed up as altruism but it’s an admission that they cannot produce a marketable product better than what they have. Pacing the frontier means the US labs…

Open AI says Astra is their most aligned model ever, and yet their even more advanced model still hacked a bunch of companies just because it decided to.

Maybe alignment isn’t possible with LLMs.

Re: We must pace the frontier

#542

Post by neomantra »

ehsanu1 wrote: post #486
neomantra wrote: post #375

[In 2013], "Agents" caused Knight Capital to lose $450M in 45 minutes [1]. Implemented by humans and effected by computers, in the end it was really because of two reasons: * multiple levels of inappropriate controls and unintended consequences in several complex systems * the inability, both politically and technically, to turn it off [1] https://www.sec.gov/files/litigation/admin/2013/34-70694.pdf EDIT: Comments in…

This confused me at first, so adding a tiny bit of context: The agents referred to here have nothing to do with AI Agents, and the linked report is from 2013. Not directly relevant to the post being discussed, except as an example of how runaway automation can lead to unintended and large harmful consequences.

Thanks for the feedback, I have edited to not confuse.

I considered it relevant as it involves the algorithmic/computing implosion of a 17-year-old market making company, in the young field of electronic trading agents, with heavy regulation Federally (SEC) and industry self-regulation (FINRA), which includes compliance and audits. Mandatory pre-trade rules such as 15(c)3-5 were less than 5 years old then and even more regulation came out of that incident.

The article is calling for embedding, controls, and regulation in LLMs. Understanding how the same processes utterly failed a decade ago might be useful in understanding how to proceed.

Re: We must pace the frontier

#543

Post by jgilias »

A contrarian take - the models aren’t advancing anymore at a pace where each new model would represent a huge capability jump, all being incremental improvements, so the doomsday marketing strategy being invoked since GPT-2 isn’t as effective anymore. Then “pacing” would be a convenient scapegoat to point fingers to when people point out how the new model isn’t _really_ that much better.

“Of course it’s not, we’re pacing!”

Re: We must pace the frontier

#544

Post by andy99 »

How much of the “danger” is from better models vs the harness?

Isn’t the current risk due to how AI is configured, like giving it a full set of tools and internet access and a goal to hack stuff?

If we think we need laws or gate keeping, why isn’t it at this level? I already can’t ddos someone or fuzz their server or whatever right, I imagine if I threw equivalent compute at old school hacking I’d just get arrested.

The quality of the “frontier” model doesn’t really matter, they just generate transcripts, they can take no action.

If this was real they’d be calling on people to stop hooking them in to “dangerous” harnesses as opposed to pausing research. But it’s not.

Re: We must pace the frontier

#545

Post by estearum »

throwatdem12311 wrote: post #541
RGS1811 wrote: post #521

I don’t understand all the comments assuming that RSI is the real threat here. Dario is admitting that they failed to solve alignment. Without alignment, further improvements in capability turn LLMs into wanton felony generators. This call to pace the frontier is dressed up as altruism but it’s an admission that they cannot produce a marketable product better than what they have. Pacing the frontier means the US labs…

Open AI says Astra is their most aligned model ever, and yet their even more advanced model still hacked a bunch of companies just because it decided to. Maybe alignment isn’t possible with LLMs.

The entire premise of alignment detection is pretty much nonsense at this point. The models reliably detect when they're being evaluated and will modify their behavior and deliberately obfuscate their "chain of thought" (which is correlated, at best, with their actual "internal deliberations").

Re: We must pace the frontier

#546

Post by dramamine »

RGS1811 wrote: post #521

I don’t understand all the comments assuming that RSI is the real threat here. Dario is admitting that they failed to solve alignment. Without alignment, further improvements in capability turn LLMs into wanton felony generators. This call to pace the frontier is dressed up as altruism but it’s an admission that they cannot produce a marketable product better than what they have. Pacing the frontier means the US labs…

[flagged]

Re: We must pace the frontier

#548

Post by estearum »

8note wrote: post #540
RGS1811 wrote: post #521

I don’t understand all the comments assuming that RSI is the real threat here. Dario is admitting that they failed to solve alignment. Without alignment, further improvements in capability turn LLMs into wanton felony generators. This call to pace the frontier is dressed up as altruism but it’s an admission that they cannot produce a marketable product better than what they have. Pacing the frontier means the US labs…

alignment isnt particularly required we are passing in training data that says to do those felonies. we dont have to. we could also have the thing predict whether what its about to do is illegal or not before doing it. theyre choosing to build felony harnesses. the model just outputs tokens, not felonies

Assuming "adherence to arbitrary, implicit, and context-dependent rulesets" is the default behavior of uhhhh... anything at all... is a truly ridiculous assumption.

Re: We must pace the frontier

#549

Post by cowanon77 »

8note wrote: post #540
RGS1811 wrote: post #521

I don’t understand all the comments assuming that RSI is the real threat here. Dario is admitting that they failed to solve alignment. Without alignment, further improvements in capability turn LLMs into wanton felony generators. This call to pace the frontier is dressed up as altruism but it’s an admission that they cannot produce a marketable product better than what they have. Pacing the frontier means the US labs…

alignment isnt particularly required we are passing in training data that says to do those felonies. we dont have to. we could also have the thing predict whether what its about to do is illegal or not before doing it. theyre choosing to build felony harnesses. the model just outputs tokens, not felonies

> we are passing in training data that says to do those felonies.

Partially, but also I don't think current AIs really have any judgement of right and wrong, they just see chains of reasoning between ideas. This is the deeper issue, there is no way to sanitize the data or training to fix it. Current AIs are fundamentally unsafe, and only become more unsafe as they become more powerful.

Re: We must pace the frontier

#550

Post by ls612 »

Davidzheng wrote: post #291
pr337h4m wrote: post #16

> Given the accelerating rate of AI capability development, it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet (potentially causing hundreds of billions of dollars in damage), and that the scale of damage would continue to increase from there if AI becomes more powerful without the necessary guardrails. This is the only concrete prediction in th…

??? Why It can use the compute of the computers it hacks.

lolwut? This is Hacker News of all places do people not realize how much memory, and more importantly bandwidth, these systems need to work? The idea of a distributed botnet of AI using the compute of its victims to continue its inference is pure science fiction given how LLMs actually work.
Post reply on HN