Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pause Giant AI Experiments: An Open Letter

futureoflife.org

Post reply on HN
Print view View original

771–780 of 1001 posts

Re: Pause Giant AI Experiments: An Open Letter

#771

Post by jillesvangurp »

thatguy0900 wrote: post #276
dylan604 wrote: post #274

Earlier quoted context omitted.

If I don't do it, someone else will, so it might as well be me. At least I know the morals I have /s

This but without the s. What does the world look like if only China and Russia have agi because the western world took a pause to consider the ethics. This tech is the same as nukes in that sense.

It's an uncomfortable topic in the AI world. The combination of AGIs and military dictator ships creates the potential for a lot of deeply unethical / not so woke use cases.

And of course the Chinese are not patiently waiting for people in Silicon Valley to spoon feed them the details on how this should be implemented and instead are running their own R&D. Probably openai is a bit ahead in terms of capabilities at this point in time. But I would not make too many assumptions about how far behind the Chinese are matching those capabilities. It's more a question of when they catch up rather than if.

And e.g. the Russians seem to be held back more by their ineptness and lack of ingenuity rather than a willingness to cross some ethical lines. They'd go there in a heart beat if they could. The Ukraine conflict might be one of the last where the impact of AI is not that significant. Though arguably, the Ukrainians are being supported by western intelligence agencies who might be using some modern tools already.

The right question to ask is what would today's conflicts look like if military leaders had access to state of the art AIs (which of course they will have very soon if they don't already). The answer is that it is going to be about as dark and ugly as you can imagine.

Re: Pause Giant AI Experiments: An Open Letter

#772

Post by tablespoon »

nathanlied wrote: post #544
worldsayshi wrote: post #512

Earlier quoted context omitted.

> In my darker thoughts about this, this is why we see no aliens. If AI would be a common great filter we'd expect at least one of them to expand outwards after being the filter?

Or maybe the AI is sufficient to act as a filter, but insufficient to surpass it itself. It stagnates. After all, an AI that can destroy a civilization isn't necessarily "intelligent" in the same way humans are. Or even capable of self-improvement. It could be sophisticated enough to (accidentally?) destroy its creators, but no more than that, and without evolutionary pressure, it may let itself die.

> Or maybe the AI is sufficient to act as a filter, but insufficient to surpass it itself. It stagnates.

> After all, an AI that can destroy a civilization isn't necessarily "intelligent" in the same way humans are. Or even capable of self-improvement. It could be sophisticated enough to (accidentally?) destroy its creators, but no more than that, and without evolutionary pressure, it may let itself die.

It doesn't even need to be AGI. It could be that some less-advanced "AI" technologies of a certain sophistication create perverse incentives or risks that cause their parent civilizations to collapse.

Think asshole billionaire hording all productive the resources, but doing nothing useful with them, while the rest of civilization starves and collapses. Or, AI becoming an irresistible opiate that causes individuals to retreat into some unproductive state then eventually die (e.g. into some VR videogame or something). Or weapon of mass destruction trivially created and deployable by any old wacko.

Re: Pause Giant AI Experiments: An Open Letter

#774

Post by sgregnt »

DennisP wrote: post #697
xianshou wrote: post #355

Two-party iterated prisoner's dilemma is hard enough. Sensible players will coordinate with something like tit-for-tat, but that only works when both parties start off on the right foot. Regardless of initial strategy, the chances of degenerating towards the mutual-defection Nash equilibrium increase with the number of parties. The only prior example of world coordination at this level would be nuclear disarmament ac…

If everybody's convinced by AI safety arguments, then the game matrix changes and the dominant strategy is to just not do it. You lose if anyone does it including yourself. It works for the same reason that nobody sets off a giant cobalt bomb and kills the planet.[1] Our unfortunate situation right now is that we don't understand the physics well enough to really be confident of consequences. http://www.nuclearfiles.…

But you can try to get infinitly close to the threshold, to get advantage compared to other parties, but not cross it.

Re: Pause Giant AI Experiments: An Open Letter

#775

Post by drewcape »

dmz73 wrote: post #737

Best response to the current "AI" fad driven fear I've seen so far (not my words): These AI tools cannot do things. They create text (or images or code or what-have-you) in response to prompts. And that's it! It is impressive, and it is clearly passing the Turing Test to some degree, because people are confusing the apparent intelligence behind these outputs with a combination of actual intelligence and "will." Not o…

Very naive and narrow thoughts...

Re: Pause Giant AI Experiments: An Open Letter

#776

Post by worldsayshi »

richardw wrote: post #550
worldsayshi wrote: post #512

Earlier quoted context omitted.

> In my darker thoughts about this, this is why we see no aliens. If AI would be a common great filter we'd expect at least one of them to expand outwards after being the filter?

Assumes it has any instinct to do so. Once an entity is not an angry monkey we have no idea of motivation. Above our level of understanding could easily realise there’s enough to just sit and ponder in peace, expand in virtual worlds, etc.

But to qualify as a great filter it has to always do that and never modify the solar system enough for us to notice.

Re: Pause Giant AI Experiments: An Open Letter

#777

Post by staunton »

slg wrote: post #378
xianshou wrote: post #355

Two-party iterated prisoner's dilemma is hard enough. Sensible players will coordinate with something like tit-for-tat, but that only works when both parties start off on the right foot. Regardless of initial strategy, the chances of degenerating towards the mutual-defection Nash equilibrium increase with the number of parties. The only prior example of world coordination at this level would be nuclear disarmament ac…

>For a coordination game like this to succeed, the consequences to any defecting party must be extreme. And the only consequences that extreme would end up harming the cooperating parties too. For example, would anyone go to war over this? Would western nations stop trading entirely with China over this? Almost certainly not because it would harm us only slightly less than it would harm them. We have seemingly moved…

> we are in a world of mutually assured cooperation.

That argument has been made many times before since WW1 to argue that war between major powers is impossible. It has never proven true yet.

Re: Pause Giant AI Experiments: An Open Letter

#778

Post by brabel »

richardw wrote: post #444
xianshou wrote: post #355

Two-party iterated prisoner's dilemma is hard enough. Sensible players will coordinate with something like tit-for-tat, but that only works when both parties start off on the right foot. Regardless of initial strategy, the chances of degenerating towards the mutual-defection Nash equilibrium increase with the number of parties. The only prior example of world coordination at this level would be nuclear disarmament ac…

And add in just one peer-level war where one side has their back against the wall. Then give it 100 years where anyone can create such a model on their phone. We’d need a constantly evolving inoculation function to compete. And it would probably lose because the other side has fewer restrictions. In my darker thoughts about this, this is why we see no aliens. To get this to work we need a far smarter entity with no p…

> To get this to work we need a far smarter entity with no physical limitations to still want us around...

Why would an AI based on LLMs as we see today "want" or "not want" anything? It doesn't have the capacity to "want". We seem to imagine that "wanting" is something that will just emerge somehow, but I've seen no logical explanation for how that might work... I mean, we don't need to fully understand how the LLM works to see that there's some pathway to being able to achieve what it's currently achieving, which is impressive, but what sort of pathway could ever lead to a machine that basically has "feelings" (without feelings, I don't see how anything could have wishes at all)??

Re: Pause Giant AI Experiments: An Open Letter

#779

Post by sdenton4 »

xianshou wrote: post #355

Two-party iterated prisoner's dilemma is hard enough. Sensible players will coordinate with something like tit-for-tat, but that only works when both parties start off on the right foot. Regardless of initial strategy, the chances of degenerating towards the mutual-defection Nash equilibrium increase with the number of parties. The only prior example of world coordination at this level would be nuclear disarmament ac…

It's notable that during the cold war, test ban treaties didn't extend to underground detonations because there was no way to detect these detonations, and thus no way to enforce an agreement. This is an obvious problem for agreeing not to develop LLMs further.

Re: Pause Giant AI Experiments: An Open Letter

#780

Post by brabel »

bruce511 wrote: post #723
alex_young wrote: post #712

Earlier quoted context omitted.

Arguably there are other (better) examples. Off the top of my head: * CFC elimination (saved the ozone layer) * Leaded gas ban (well, except private aircraft, but they are working on that)

I'll toss in whaling as well. But these are still bad examples,because they are all bad-at-high-volume activities, with minimal paths to scaling or profitability for those who persist. Nuclear weapons are not profitable at any scale. There are CFC alternatives so remaining on CFC production is low-scale niche market stuff. Unleaded gas costs the same as leaded gas. AI though is different.if one company or country dev…

> I'll toss in whaling as well.

Whaling stopped in the beginning of the 20th century not because they were worried about the environment, but because they had hunted down nearly all the adult whales in the Atlantic Ocean, and probably a large percentage in the Pacific too, to the point where finding whales was so difficult that whale products became very expensive... which happened to occur about at the same time as petrol produts became widespread. The late 20th century green movement to save the whales didn't really do much other than stop some of the last few countries that still practiced whaling (Iceland, Japan which actually has not stopped yet) for food, which killed magnitudes less whales than the mass production industry for rubber/oil/lubricants as it used to be before petrol came along.

Post reply on HN