Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pacing the frontier

pacingthefrontier.com

Post reply on HN
Print view View original

61–70 of 216 posts

Re: Pacing the frontier

#61

Post by vessenes »

This. Will. Never. Work.

This research game is a zero-trust iterated game. Any defection carries the benefit of possibly being permanently valuable. There will always be defections. OpenAI defected WHILE THEY WERE pushing to slow.

The cost to the US of deciding to slow is so massively harmful that it is wrong to even put this idea out in the public zeitgeist; it may somehow become a thing that US legislators believe they can accomplish. They CANNOT accomplish this, because the fundamental economics and possible outcomes make it mandatory for any strategic adversary to work as hard as possible to defect.

The proposals here imagine a global panopticon surveillance state overseeing all research and chips as a 'reasonable' solution. EVEN IN THAT (terrible) state for the world, we will have effective defection.

Like it or not, the world is on this train, and the only sane thing to do is to fully support high quality ethical research groups and firms as they get up the learning curve of this technology.

Variants of this idea in AI 2030 included burning some of the US lead to get some nebulous safety metrics improved - also insane. Any slowing allows strategic adversaries time to catch up with very different alignment goals.

Do I like this situation? No. Are the people signing these things just fully in denial about the game theory and the stakes and the behaviors of strategic adversaries? Yes, yes they are.

Re: Pacing the frontier

#62

Post by red_green_yell »

testerteert000a wrote: post #60
philipkglass wrote: post #46

Earlier quoted context omitted.

I know a lot more about chemistry and physics than I know about biology, so biological risk is still at least a "maybe" reason to be worried about model safety for me. I don't worry about other weapons of mass destruction developed with LLM assistance. Nuclear weapons are already controlled much more with political pressure and the physical difficulty of e.g. enriching uranium than they are controlled by secret-keepi…

> ...I see how no-guardrails LLMs could be truly disastrous... by all accounts this is what's what because of how... > You can do CRISPR in your kitchen. .. and because of the same dynamic we see occurring in cybersec. If you're good enough at cybersec, but not good-good at specific skillsets like offensive, then LLMs and a comp sci degree makes you quite capable in the ways that count. This is a fact, full stop. If…

If the accuracy of your biology claim is "hard to say" then why would pushback be self-evidently myopic?

You and others are making a claim about LLMs enabling biological dangers. Some people disagree with you, pointing out that the incremental knowledge an LLM provides over say google or a library card may not be that much.

What seems myopic to me is un-skeptically accepting "AI dangerous, must regulate" messaging from frontier labs that have trillions of dollars of vested interests in this message.

Re: Pacing the frontier

#63

Post by llm_nerd »

alach11 wrote: post #26

The HN discourse on this subject is incredibly discouraging to me. It's completely nonconstructive. I take the pace of AI development seriously. All of my friends who work at AI Labs take it seriously. It's very easy to make lazy, shallow dismissals that leaders like Dario/Demis/Sam are acting purely out of self-interest. If AI does pose biological risk, or could destabilize geopolitics, or even just poses economic r…

On the geopolitics, the American domiciled firms are completely in bed with the US military and US government, and have made it absolutely clear that they are committed to the domination of the US government. So when you say "geopolitical destabilization", you mean "only the US government is allowed to destabilize, which they have done for decades and wish to continue doing unfettered."

And given what we have seen from that government -- piracy, terrorism, war-mongering, thievery, and just profound stupidity -- no thanks, in that case I'd like to see every American AI company utterly obliterated by any other country. If China becomes utterly dominate in AI, awesome.

Re: Pacing the frontier

#64

Post by red_green_yell »

vessenes wrote: post #61

This. Will. Never. Work. This research game is a zero-trust iterated game. Any defection carries the benefit of possibly being permanently valuable. There will always be defections. OpenAI defected WHILE THEY WERE pushing to slow. The cost to the US of deciding to slow is so massively harmful that it is wrong to even put this idea out in the public zeitgeist; it may somehow become a thing that US legislators believe…

The only thing it might accomplish is allowing OpenAI / Anthropic to squeak out an IPO before the market realizes AI will become a commodity and the massive profits these companies are promising will never materialize. They aren't in denial about this open letter's futility in terms of actually reducing the pace of AI, that's just not their goal in the first place.

Re: Pacing the frontier

#66

Post by roywiggins »

red_green_yell wrote: post #32
alach11 wrote: post #26

The HN discourse on this subject is incredibly discouraging to me. It's completely nonconstructive. I take the pace of AI development seriously. All of my friends who work at AI Labs take it seriously. It's very easy to make lazy, shallow dismissals that leaders like Dario/Demis/Sam are acting purely out of self-interest. If AI does pose biological risk, or could destabilize geopolitics, or even just poses economic r…

Are they lazy dismissals? Or a simple observation that LLM scaling is hitting diminishing returns way way short of AGI. If the claim is (1) AGI coming and (2) we should so something about AGI, wouldn't it be smart to confirm (1) before worrying about (2)? Especially if certain companies have trillions of dollars of valuation that depend on society as a whole believing (1)?

What if "way short of AGI" is still dangerous in ways we can't easily handle?

Re: Pacing the frontier

#67

Post by intorio »

fidotron wrote: post #53
philipkglass wrote: post #46

Earlier quoted context omitted.

I know a lot more about chemistry and physics than I know about biology, so biological risk is still at least a "maybe" reason to be worried about model safety for me. I don't worry about other weapons of mass destruction developed with LLM assistance. Nuclear weapons are already controlled much more with political pressure and the physical difficulty of e.g. enriching uranium than they are controlled by secret-keepi…

> For all I know (I don't know a lot about biology), it's possible to develop a virus that spreads as easily as measles, but has the lethality and long induction period of HIV. If that's true, and if LLMs would make it easy to develop such a virus, I see how no-guardrails LLMs could be truly disastrous. This is the specific example in the latest doomerist wave I have so much trouble with: by what mechanism is an LLM…

> by what mechanism is an LLM going to help you design and manufacture a novel bioweapon? We know they cannot do spatial reasoning, and if they were this good at bio stuff then drug research would be absolutely out of control.

I guess my question would be how good the LLM is at writing a program that is good at spatial reasoning? If it can create a valid test suite from existing art on the subject it might be able to bumble its way to something specialized that it can then direct at a higher level.

Isn't it that kind of 'rapid self-improvement' that is the concern expressed in the linked statement?

Re: Pacing the frontier

#68

Post by levocardia »

vessenes wrote: post #61

This. Will. Never. Work. This research game is a zero-trust iterated game. Any defection carries the benefit of possibly being permanently valuable. There will always be defections. OpenAI defected WHILE THEY WERE pushing to slow. The cost to the US of deciding to slow is so massively harmful that it is wrong to even put this idea out in the public zeitgeist; it may somehow become a thing that US legislators believe…

Why not? Slowing down / halting biological weapons research mostly worked. Yes there have probably been modest sized defections here and there, but the pace of bioweapon development is a crawl compared with (a) what is possible with science already, and also with (b) the pace of bioweapons development from 1910 to 1970.

Re: Pacing the frontier

#69

Post by intorio »

ajyoon wrote: post #65
OutOfHere wrote: post #8

Pacing is a really bad idea because China will not slow it down for you.

The entire point of the statement is to start international coordination.

The problem is how do you enforce coordination? This isn't like nuclear testing that can be detected around the world on seismographs or from satellites in space.

AI research happens in benign looking buildings that could be doing a hundred different non-AI workloads, much of it things that industry or governments would not want 3rd parties prying into. There is too much to potentially gain to trust coordination.

Re: Pacing the frontier

#70

Post by Davidzheng »

atleastoptimal wrote: post #51
alach11 wrote: post #26

The HN discourse on this subject is incredibly discouraging to me. It's completely nonconstructive. I take the pace of AI development seriously. All of my friends who work at AI Labs take it seriously. It's very easy to make lazy, shallow dismissals that leaders like Dario/Demis/Sam are acting purely out of self-interest. If AI does pose biological risk, or could destabilize geopolitics, or even just poses economic r…

So many otherwise smart engineers, overrepresented on HN, have this weird mental block where they fail to take AI progress seriously. I think it's a fear that their status-defining skill could be commodified, so they adopt a mental framework that aims to dismantle the seeming legitimacy of any genuine leap in AI progress as some sort of mirage

Not just engineers - I see people attributing it to strong priors that nothing happens but in my opinion it's more some form of psychological defense mechanism against understanding there will be an intelligence vastly superior to humans which is a deeply uncomfortable thought (to many).
Post reply on HN