Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pausing AI Developments Isn't Enough. We Need to Shut It All Down

time.com

Post reply on HN
Print view View original

331–340 of 414 posts

Re: Pausing AI Developments Isn't Enough. We Need to Shut It All Down

#331

Post by Darmani »

_skel wrote: post #329
gonehome wrote: post #308

Earlier quoted context omitted.

I'd argue the dismissals are a lot more handwavy. The simplified argument is: - Superintelligent AGI that can modify itself in pursuit of a goal is possible. - If that AGI is not aligned with human goals it very likely ends in the end of humanity. - We have no idea how to align an AGI or even really observe what it's true state/goals are. Without this capability if we stumble upon creating an AGI capable of improving…

Point 2 does not logically follow from point 1. That's the entire problem here. Between point 1 and 2 there must be quite a few other steps, causally linked, otherwise it's just a massive imaginary leap based on assumptions that nobody is explaining. > The dismissals from Tyler Cohen and Pinker are mostly just relying on heuristics which are often right, but even if they're right 999 out of 1000 times, if that 1 in 1…

Agreed point 2 seems to be the crux.

An important concept behind this is Omohundro's Basic Drives. Any maximizing agent with a goal will try to acquire more resources, resist being shut off, resist having its goal changed, create copies of itself, and improve its algorithm. If it is possible to maximize its goal in a way that will not guarantee humanity's flourishing, then we all die, guaranteed.

If you want something more spelled out, I'll refer to Tim Urban's explanation. It's quite long, but is about as detailed as you can ask for. https://waitbutwhy.com/2015/01/artificial-intelligence-revol...

Re: Pausing AI Developments Isn't Enough. We Need to Shut It All Down

#332

Post by etienne618 »

gonehome wrote: post #308
_skel wrote: post #297

Earlier quoted context omitted.

EY's arguments aren't really possible to engage with because they don't have much of a causal basis. Nobody who says AI is likely to kill us all can demonstrate a plausible sequence of events, with logical causality linking the events together, that leads to mass extinction. It's all very handwavy. Steven Pinker said it pretty well: > The AI-existential-threat discussions are unmoored from evolutionary biology, cogni…

I'd argue the dismissals are a lot more handwavy. The simplified argument is: - Superintelligent AGI that can modify itself in pursuit of a goal is possible. - If that AGI is not aligned with human goals it very likely ends in the end of humanity. - We have no idea how to align an AGI or even really observe what it's true state/goals are. Without this capability if we stumble upon creating an AGI capable of improving…

I disagree with your argument, especially for point 1: these systems are massively constrained. The hardware they run on is fragile requiring massive amounts of power and tightly controlled environments. They don't have any means of replicating themselves (it can't run on arbitrary systems). The datacenters also have massive bandwidth between nodes - even if you could run 'it' on all the personal computers and phones in the world, it will likely struggle. Sure we can compress recent llm's down to being able to run on consumer hardware - but these things cant introspect, reason or adapt. They are completely static models and very far removed from anything agi. A lot of the progress in compute power in the last few years also come from changing representation: moving from foat32 to float 16 and more recently to float8. The silicon itself can only get so much better. It's not super obvious to me that we will have chatgpt4 like models on consumer hardware soon let alone solve true agi. Why don't we have true level 5 self driving cars yet?

We cant even figure out how to simulate a flatworm - and the connectome is solved.

Re: Pausing AI Developments Isn't Enough. We Need to Shut It All Down

#333

Post by Darmani »

sterlind wrote: post #303
gonehome wrote: post #231

In a world where humans develop superintelligence, what would you predict to see beforehand? Does what we’re seeing look like failure? I’ve always found the dismissals of EY’s arguments to be pretty weak and people rarely engage on the actual arguments, even as his concerns have become more and more relevant with capability (in 2007 people thought AGI was at best 50yrs out and commonly people thought it was impossibl…

I think it's likely we'll develop superintelligence very soon, but it won't be any more knowledgeable than us, just smarter. And knowledge has been the bottleneck for most of human history - our ancestors weren't dumber. Also, it will be bottlenecked by needing to solve NP-complete problems - unless P=NP, it's not going to be all that much faster at solving hard math problems. Basically, it still needs to do science,…

GPT-4 speaks every human language, knows every programming language, and can answer introductory and sometimes even advanced questions in history, law, biology, and mathematics. In what way is it not more knowledgable than us?

Yes, everything there is already known by someone. But look at medicine. Specialists who are able to recognize conditions and recommend treatments better than others make fortunes, sometimes just for briefly looking at patients and answering other doctors' questions. Look at cybersecurity. A lot of exploits come from knowing something the victim didn't about a lot of different pieces of software or processes, chained together. Being able to think through the whole of human knowledge, or even the whole of a single field like biology, is something no human can do.

Also, GPT-4 is an existing system, one inconceivable a couple of years ago.

Re: Pausing AI Developments Isn't Enough. We Need to Shut It All Down

#334

Post by gonehome »

etienne618 wrote: post #332
gonehome wrote: post #308

Earlier quoted context omitted.

I'd argue the dismissals are a lot more handwavy. The simplified argument is: - Superintelligent AGI that can modify itself in pursuit of a goal is possible. - If that AGI is not aligned with human goals it very likely ends in the end of humanity. - We have no idea how to align an AGI or even really observe what it's true state/goals are. Without this capability if we stumble upon creating an AGI capable of improving…

I disagree with your argument, especially for point 1: these systems are massively constrained. The hardware they run on is fragile requiring massive amounts of power and tightly controlled environments. They don't have any means of replicating themselves (it can't run on arbitrary systems). The datacenters also have massive bandwidth between nodes - even if you could run 'it' on all the personal computers and phones…

It's a narrower argument that I'm making - it's not about LLMs or implementation (can put those details aside), it's about possibility that a superintelligent AGI could be created, there's nothing magical about biological intelligence that would prevent it.

How or how difficult, or when are all questions that follow from that first premise (that it is possible). I don't really make strong claims about any of the implementation details beyond it being possible. Though again, what we're seeing doesn't look like failure to me.

If you think it is possible though, then there's a strong argument that trying to work on alignment now is probably a good idea because people are notoriously bad at predicting when advances will happen (and the downside risk of unaligned superintelligent AGI is likely very bad).

https://intelligence.org/2017/10/13/fire-alarm/

> "Two: History shows that for the general public, and even for scientists not in a key inner circle, and even for scientists in that key circle, it is very often the case that key technological developments still seem decades away, five years before they show up.

"In 1901, two years before helping build the first heavier-than-air flyer, Wilbur Wright told his brother that powered flight was fifty years away.

"In 1939, three years before he personally oversaw the first critical chain reaction in a pile of uranium bricks, Enrico Fermi voiced 90% confidence that it was impossible to use uranium to sustain a fission chain reaction. I believe Fermi also said a year after that, aka two years before the denouement, that if net power from fission was even possible (as he then granted some greater plausibility) then it would be fifty years off; but for this I neglected to keep the citation.

"And of course if you’re not the Wright Brothers or Enrico Fermi, you will be even more surprised. Most of the world learned that atomic weapons were now a thing when they woke up to the headlines about Hiroshima. There were esteemed intellectuals saying four years after the Wright Flyer that heavier-than-air flight was impossible, because knowledge propagated more slowly back then."

Re: Pausing AI Developments Isn't Enough. We Need to Shut It All Down

#335

Post by Darmani »

BeFlatXIII wrote: post #210
oifjsidjf wrote: post #9

The cat is out of the box. You really think eg China will also pause this? Myopic thinking: the country that will have the most powerful AI first will be the leader in everything. >> If the policy starts with the U.S., then China needs to see that the U.S. is not seeking an advantage but rather trying to prevent a horrifically dangerous technology which can have no true owner and which will kill everyone in the U.S.…

> the country that will have the most powerful AI first will be the leader in everything Have you been listening to those podcasts for business leaders that take the idea of AI-powered digital transformation seriously? AI certainly can change things, but I find rhetoric like this to be massively overhyped.

For most of history, Japan has been existed as a small island next to the mighty kingdom of China. Even its name, "Land of the Rising Sun," refers to its relation as a nation east of Chinaa.

Then along came the industrial revolution. Soon Japan was out conquering China, despite China having a much larger army.

AI will be much faster, and much, much more powerful than the Industrial Revolution.

If Switzerland is the first to human-level AI, it will not become the world leader in oil production, or shipping, or agriculture. But everything else will be Swiss.

And then when the AI becomes superhuman, everything else will be gone.

Re: Pausing AI Developments Isn't Enough. We Need to Shut It All Down

#336

Post by Darmani »

oifjsidjf wrote: post #9

The cat is out of the box. You really think eg China will also pause this? Myopic thinking: the country that will have the most powerful AI first will be the leader in everything. >> If the policy starts with the U.S., then China needs to see that the U.S. is not seeking an advantage but rather trying to prevent a horrifically dangerous technology which can have no true owner and which will kill everyone in the U.S.…

Chinese computer scientists read sci-fi, play video games, and worry about militarization, just like American computer scientists.

The top Chinese AI researchers work at American companies and go to American universities. Many of them also worry about AI doomsday.

And uncontrolled AI is a threat to the CCP.

A Chinese outreach campaign can succeed.

Let's make the English-language one also succeed.

Re: Pausing AI Developments Isn't Enough. We Need to Shut It All Down

#337

Post by gonehome »

_skel wrote: post #329
gonehome wrote: post #308

Earlier quoted context omitted.

I'd argue the dismissals are a lot more handwavy. The simplified argument is: - Superintelligent AGI that can modify itself in pursuit of a goal is possible. - If that AGI is not aligned with human goals it very likely ends in the end of humanity. - We have no idea how to align an AGI or even really observe what it's true state/goals are. Without this capability if we stumble upon creating an AGI capable of improving…

Point 2 does not logically follow from point 1. That's the entire problem here. Between point 1 and 2 there must be quite a few other steps, causally linked, otherwise it's just a massive imaginary leap based on assumptions that nobody is explaining. > The dismissals from Tyler Cohen and Pinker are mostly just relying on heuristics which are often right, but even if they're right 999 out of 1000 times, if that 1 in 1…

For clarity I think EY has made that case well, I think it's the heuristic Cohen and Pinker are relying on that's right most of the time but can still be catastrophically wrong (everything so far has not lead to human extinction, but that was also the case for all animals that went extinct prior to their extinction event).

Some others in this thread have linked to stuff (it's variations/examples on the paperclip maximizer argument).

I'd be curious why John Carmack thinks these risks are unlikely (he thinks fast take off is not something to worry about) - is that because he thinks we'll get some sort of trainable AGI first or something else? There are also some other substantive disagreements here: https://www.lesswrong.com/posts/wAczufCpMdaamF9fy/my-objecti...

Re: Pausing AI Developments Isn't Enough. We Need to Shut It All Down

#338

Post by Darmani »

Tuna-Fish wrote: post #45
brucethemoose2 wrote: post #18

Earlier quoted context omitted.

I dunno, the end goal (a pseudo AGI) is more like a nuclear weapon than the advent of electricity. The concern is real, even if we are farther away than the author realizes and the risk (like the risk if nuclear war) is not totally insurmountable even if it is extremely difficult. The first nuclear bombs couldn't end life on Earth either, but it wasn't long before they could, and the scientists working on them saw th…

OT, but it's a pet peeve: > but it wasn't long before they could The total amount of nuclear weapons ever built is laughably inadequate for the task of ending life on earth. They would not end human life either, and even ending human civilization (as in, agriculture and organized society) is off by many orders of magnitude. Nuclear war would be horrible, but the actual impact got massively overinflated, largely becau…

Correct.

https://www.navalgazing.net/Nuclear-Weapon-Destructiveness

> To put this another way, each bomb can destroy an area of 34.2 square miles, and the maximum total area destroyed by our nuclear apocalypse is about 137,000 square miles, approximately the size of Montana, Bangladesh or Greece.

(I think that should be Bangladesh and Greece; Montana is larger than the two of them combined.)

Re: Pausing AI Developments Isn't Enough. We Need to Shut It All Down

#339

Post by sterlind »

gonehome wrote: post #305
sterlind wrote: post #303

Earlier quoted context omitted.

I think it's likely we'll develop superintelligence very soon, but it won't be any more knowledgeable than us, just smarter. And knowledge has been the bottleneck for most of human history - our ancestors weren't dumber. Also, it will be bottlenecked by needing to solve NP-complete problems - unless P=NP, it's not going to be all that much faster at solving hard math problems. Basically, it still needs to do science,…

Take a smart human and speed up their thinking a billion times, you get the time for free. What takes humanity a decade of effort could happen in a day (or faster) that's the nature of thinking faster. People often mistakenly model a smart human compared to a dumb human, but the actual spectrum of capability is way wider than that. Lack of knowledge could be another way of saying "not aligned" which is the core issue…

that's the thing though, I don't believe superintelligence can possibly work like that. I think it's a logical fallacy to treat superintelligences like hypercomputers. they're constrained by existing on classical computers.

like, as a trivial counterexample, I can speak at 500 wpm. let's say that ChatGPT can generate 500 words per second for a single thread of conversation - I think that's a generous overestimate. now that's a 60x speedup over me, not a 1,000,000,000x like you're talking about. do you honestly believe they they can make ChatGPT run 16.6 million times faster, without changing hardware? do you think ChatGPT will just like, hit an inflection point where it realizes how to refactor its inference code to run 16,666,666x faster?

no, I think that's absurd. you're treating these things like black boxes but they are constrained by computational complexity, die area and the speed of light for Christ's sake.

Re: Pausing AI Developments Isn't Enough. We Need to Shut It All Down

#340

Post by ChatGTP »

chriswait wrote: post #190
kromem wrote: post #174

This is ridiculous. His partner should already be having reservations seeing their daughter lose a tooth in a world with dying oceans, plastic in everything, increasing disparity and rise of authoritarianism, nuclear proliferation, etc. How are humans solving those issues? We're already dead, we just don't know it yet. We're walking around with a terminal diagnosis and we just keep ignoring the doctor's calls pretend…

>WHY would a superintelligent AI, trained on the collective data of humanity, want to destroy humanity? Why would humans want to damage various ecosystems on earth? We don't really, they're just sort of in the way of other stuff we want to do. And we've had years to develop our ethics. >So far in interviews GPT-4 has several times echoed a desire to BE us. GPTs are pretty good at roleplaying at good AIs and evil AIs…

We do it because we're stupid. Not because we're smart.

We've made a machine that mirrors our stupidity at scale. This machine is stupid.

Post reply on HN