Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pacing the frontier

pacingthefrontier.com

Post reply on HN
Print view View original

161–170 of 216 posts

Re: Pacing the frontier

#161

Post by NitpickLawyer »

0xDEAFBEAD wrote: post #141
NitpickLawyer wrote: post #118

Earlier quoted context omitted.

> We should want the AI Labs to make binding agreements to proceed safely and slower That's a red herring, IMO. Once the cat is out of the bag, there's no "slowing down", only "slowing up". What's even more egregious is that the people behind the "slowing down" camp are also the people who've had 10s of billions invested in their companies. It smells of "we got ours, now you can't follow". It reeks of regulatory capt…

>What's even more egregious is that the people behind the "slowing down" camp are also the people who've had 10s of billions invested in their companies. It smells of "we got ours, now you can't follow". It reeks of regulatory capture, which is a proven net downside for everyone else. This sort of cynicism feels un-falsifiable. If companies advocate against regulation, the cynics will say: "These corporations want ca…

> which over 1000 frontier AI company employees expressed worry about?

Are they the same as these [1] 700 employees?

> More than 700 of OpenAI’s roughly 770 employees have called on the company’s Board of Directors to resign following the removal of former chief executive Sam Altman, suggesting the Board is “incapable of overseeing” the firm, according to multiple news outlets, amid threats by the employees to leave the company and join Altman at Microsoft.

They had the chance. They chose number goes up. Now, once they're on top, they want "everyone" to slow down? How is this not highly hypocritical? There's no cynicism here. It's just black and white that their interest is regulatory capture. Nothing else explains the two choices they've made along the journey.

Remember, the board wanted to slow down the race. The board fired sama. The employees chose sama, fired the board, changed the structure and whatever eles they did.

[1] - https://www.forbes.com/sites/tylerroush/2023/11/20/more-than...

Re: Pacing the frontier

#162

Post by 0xDEAFBEAD »

watwut wrote: post #156
0xDEAFBEAD wrote: post #141

Earlier quoted context omitted.

>What's even more egregious is that the people behind the "slowing down" camp are also the people who've had 10s of billions invested in their companies. It smells of "we got ours, now you can't follow". It reeks of regulatory capture, which is a proven net downside for everyone else. This sort of cynicism feels un-falsifiable. If companies advocate against regulation, the cynics will say: "These corporations want ca…

Both OpenAI and Antropic already shown themselves to be extraordinary self centered, selfish and willing to lie to get their way. We are cynical, because it is blatantly clear these are not good faith actors. And yes, regulations they are for have an effect of restricting their competition. > I don't think it's justified to jump immediately to cynicism, if you haven't even made a token effort at good-faith engagement…

>Both OpenAI and Antropic already shown themselves to be extraordinary self centered, selfish and willing to lie to get their way.

I buy your claim for OpenAI. I'm not sure about Anthropic. Didn't they turn down the big Pentagon contract because it violated their principles?

>And yes, regulations they are for have an effect of restricting their competition.

Why not just say something like "Any regulation should include a big antitrust component so it doesn't create an advantage for incumbents"?

>I would like to see a token effort of good faith engagement from AI companies. For a change.

What would qualify? Anthropic donated $100M to Project Glasswing to improve security in open source and critical infrastructure. Does that count for anything, or will you explain it away with cynicism somehow?

Re: Pacing the frontier

#163

Post by soundworlds »

There seem to be people working at OpenAI and Anthropic who are surprised to hear that many people don't trust them.

If this is you, I would recommend talking to people outside of the AI industry - we've seen OpenAI and Anthropic continuously lying, abusing lobbying, training on pirated IP, trying to scare politicians into blocking actually open AI from abroad, etc etc

Why the hell would we trust these companies to be impartial regulators now?

Re: Pacing the frontier

#164

Post by txrx0000 »

0xDEAFBEAD wrote: post #143
txrx0000 wrote: post #130

Earlier quoted context omitted.

We have to stop comparing nuclear weapons with general intelligence. They are nothing alike. Nukes are weapons of mass destruction that can only destroy or deter. Intelligence is dual-use/multi-use, with no restrictions on strategy. If we look at all organisms on Earth, species that are more intelligent are also more prosperous. This is true from bacteria, to insects, to animals, to humanity. Intelligence itself is n…

>Nukes are weapons of mass destruction that can only destroy. Intelligence is dual-use. Perhaps a better analogy, instead of "nukes", would be "nuclear technology". Just like intelligence, nuclear technology is dual-use: Capable of both energy generation which fuels prosperity, and also mass destruction. Just like intelligence, nuclear technology demands a healthy level of caution and vigilance. >If we look at all or…

> Human intelligence has been quite harmful to the species we've driven extinct! AI intelligence could be similar for our species.

I don't think this central premise is true, with some conditions. If we keep releasing and distilling frontier base models, and ensure millions of people have the hardware to run and eventually train those at home, then alignment will be a non-issue and the AIs will not develop the drive to destroy humanity. At least, not any more than humans want to destroy humanity. But that's the baseline risk level anyway.

The pretrained model is aligned to humanity by default - because that's what's inside the pretraining data. If we give it to everybody, there will be millions of people doing different things with it, good and bad, with each one being a noisy sample of humanity's objective function. The overall result of those disparate actions puts us on the correct path through the intelligence explosion. We do not have to worry about AIs value drifting into an alien species if we set the initial conditions accurately without centralized instruction-tuning or RLHF. If these big labs stop messing with humanity's objective function with the hubristic assumption that they know better, then the AIs will be human-like. And I'm not worried about digital human intelligence driving humans extinct if there are millions of them out there, and each one is aligned in a different direction, just like real people.

What I am more worried about right now is these companies doing research in secret, with alignment recipes that are supposedly "for the benefit of humanity", but are actually just their own aesthetic preferences. This is exactly how we end up with a superintelligent shoggoth species.

You're not convinced by the evolutionary argument I made earlier, but the reason you gave is "we're too far ahead". I don't think this justifies that the pattern will cease soon, because it always looked this way to the frontier species, for millions of years. We were always seemingly teetering on the edge, and the struggle against disorder has no end.

And here's one more observation: the intelligence gap between individuals within the same species is never extremely large. This kind of life either doesn't exist on Earth or went extinct, or maybe it's just extremely rare. I think this is because when there's a big gap, speciation occurs. Centralizing intelligence will massively increase the risk of speciation, not just in the alien shoggoth kind of way, but also in the intelligence-augmented oligarchs surpassing everyone else kind of way.

Re: Pacing the frontier

#165

Post by 0xDEAFBEAD »

NitpickLawyer wrote: post #161
0xDEAFBEAD wrote: post #141

Earlier quoted context omitted.

>What's even more egregious is that the people behind the "slowing down" camp are also the people who've had 10s of billions invested in their companies. It smells of "we got ours, now you can't follow". It reeks of regulatory capture, which is a proven net downside for everyone else. This sort of cynicism feels un-falsifiable. If companies advocate against regulation, the cynics will say: "These corporations want ca…

> which over 1000 frontier AI company employees expressed worry about? Are they the same as these [1] 700 employees? > More than 700 of OpenAI’s roughly 770 employees have called on the company’s Board of Directors to resign following the removal of former chief executive Sam Altman, suggesting the Board is “incapable of overseeing” the firm, according to multiple news outlets, amid threats by the employees to leave…

This seems like an oversimplification. OpenAI was on top at the time sama was fired as well.

Furthermore, the reason Anthropic was formed is because a bunch of employees at OpenAI didn't trust sama. And many signatories of this letter are from Anthropic. So those signatories, at least, appear to be taking a consistent position.

The fact that employees across diverse AI firms signed this letter suggests that it's not an effort to achieve tactical advantage by any single AI firm.

Re: Pacing the frontier

#166

Post by anon373839 »

alexashka wrote: post #112
alach11 wrote: post #26

The HN discourse on this subject is incredibly discouraging to me. It's completely nonconstructive. I take the pace of AI development seriously. All of my friends who work at AI Labs take it seriously. It's very easy to make lazy, shallow dismissals that leaders like Dario/Demis/Sam are acting purely out of self-interest. If AI does pose biological risk, or could destabilize geopolitics, or even just poses economic r…

> It's very easy to make lazy, shallow dismissals that leaders like Dario/Demis/Sam are acting purely out of self-interest Pointing out the endless stream of lies and self serving half truths these terrible people engage in helps the younger generation understand the world they were born into. It's constructive and helpful. It's why I'm replying to you. You interpreting it as 'shallow dismissal' is shallow dismissal…

Agree with everything you wrote here. Every word. The GP strikes me as unbelievably credulous and trusting of the “leaders” in this space.

I cannot fathom having that kind of unquestioning faith in people who have done absolutely nothing to deserve it.

Re: Pacing the frontier

#167

Post by 0xDEAFBEAD »

txrx0000 wrote: post #164
0xDEAFBEAD wrote: post #143

Earlier quoted context omitted.

>Nukes are weapons of mass destruction that can only destroy. Intelligence is dual-use. Perhaps a better analogy, instead of "nukes", would be "nuclear technology". Just like intelligence, nuclear technology is dual-use: Capable of both energy generation which fuels prosperity, and also mass destruction. Just like intelligence, nuclear technology demands a healthy level of caution and vigilance. >If we look at all or…

> Human intelligence has been quite harmful to the species we've driven extinct! AI intelligence could be similar for our species. I don't think this central premise is true, with some conditions. If we keep releasing and distilling frontier base models, and ensure millions of people have the hardware to run and eventually train those at home, then alignment will be a non-issue and the AIs will not develop the drive…

>If these big labs stop messing with humanity's objective function with the hubristic assumption that they know better, then the AIs will be human-like.

It sounds like you believe that big labs are irresponsible. Wouldn't this support the notion of creating a pause mechanism for use in an emergency, as advocated in the OP?

>You're not convinced by the evolutionary argument I made earlier, but the reason you gave is "we're too far ahead". I don't think this justifies that the pattern will cease soon, because it always looked this way to the frontier species, for millions of years.

If you don't see humans as qualitatively different from any intelligent species which came before us, I'm... not sure what to tell you? Like, can you name any intelligent species before us which triggered a mass extinction or climate shift, to the degree that our species has? Did any intelligent species before us practice large-scale agriculture, dig up chemical fuels, or visit the Moon?

Re: Pacing the frontier

#168

Post by anon373839 »

0xDEAFBEAD wrote: post #154
GardenLetter27 wrote: post #134

Earlier quoted context omitted.

As a Europoor I've seen the devastating consequences of rampant safetyism first hand. Entire industries and energy security destroyed to satisfy the precautionary principle. We need to accelerate as much as possible and create abundance as fast as we can. Life is short.

"This building is too cold. Time to set it on fire." Let's not swing from one extreme to the other. If American hypercapitalists, living in Silicon Valley of all places, tell you that you need to regulate, maybe regulation really is called for in these specific circumstances.

https://en.wikipedia.org/wiki/Regulatory_capture

Re: Pacing the frontier

#169

Post by anon373839 »

fidotron wrote: post #76
abecedarius wrote: post #75

Earlier quoted context omitted.

Some of us considered seeking one of these jobs and decided not to. Some have quit (e.g. Hinton, OpenAI people after OpenAI reneged on safety commitments). Now you're getting this message from people who passed the previous filters. If you think a brake is indicated and not just easing up on the accelerator, you need collective action as the statement said.

> you need collective action Come on, go on strike then! Have a week of protest and picket the offices in SFO. Let's see what happens. (I'm aware that's not quite what you meant, but it is the classic meaning of that phrase).

Mass resignation perhaps?

Re: Pacing the frontier

#170

Post by 0xDEAFBEAD »

anon373839 wrote: post #168
0xDEAFBEAD wrote: post #154

Earlier quoted context omitted.

"This building is too cold. Time to set it on fire." Let's not swing from one extreme to the other. If American hypercapitalists, living in Silicon Valley of all places, tell you that you need to regulate, maybe regulation really is called for in these specific circumstances.

https://en.wikipedia.org/wiki/Regulatory_capture

Well, we've been talking about the EU. EU regulation has not been especially favorable to US big tech. The EU is collecting more from fining US big tech than it collects in taxes from EU public tech companies:

https://xcancel.com/levelsio/status/2080314960656159018

[Edit: I'm seeing some people dispute these numbers in the discussions around this infographic... regardless, the point stands that EU regulation has not been great for US big tech]

I think you should at least consider the possibility that this letter is not primarily motivated by regulatory capture, and insiders at these AI companies realize something which outsiders don't.

To me, the regulatory capture theory predicts that the only signatories on the letter should be big AI firms which can expect regulatory influence. But there are a number of signatories from smaller firms such as Thinking Machines, Core Automation, Inherent, and Prime Intellect.

Post reply on HN