Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Remarks on AI from NZ

nealstephenson.substack.com

Post reply on HN
Print view View original

71–80 of 117 posts

Re: Remarks on AI from NZ

#71

Post by tuatoru »

vonneumannstan wrote: post #24

It's a nice article but Neal like many others falls into the trap of seemingly not believing that intelligences vastly superior to Humans' across all important dimensions can exist and competition between minds like that almost certainly ends in Humanity's extinction. "I am hoping that even in the case of such dangerous AIs we can still derive some hope from the natural world, where competition prevents any one speci…

That's quite bold, accusing a successful science ficiton author of lacking imagination.

Re: Remarks on AI from NZ

#72

Post by overfeed »

skybrian wrote: post #70
NitpickLawyer wrote: post #8

> Maybe a useful way to think about what it would be like to coexist in a world that includes intelligences that aren’t human is to consider the fact that we’ve been doing exactly that for long as we’ve existed, because we live among animals. Another analogy that I like is about large institutions / corporations. They are, right now, kind of like AIs. Like Harari says in one of his books, Peugeot co. is an entity tha…

If a corporation is like an AI, it’s like one we imagine might exist one day, not currently-existing AI. LLM’s aren’t trying to make money or do anything in particular except predict the next token. The corporations that run LLM’s do charge for API usage, but it’s independent of what you the chat is about. It’s happening at a different level in the stack.

AIs minimize perplexity, corporations maximize profits - the rest are implemention details.

If you built an AI that could outsource labor to humans and whose reward function is profit, your result would approximately be a corporation.

Re: Remarks on AI from NZ

#73

Post by cantrevealname »

vonneumannstan wrote: post #29
jazzyjackson wrote: post #26

Earlier quoted context omitted.

I guess the "trap" is just a lack of imagination? I'm in that school of, wtf are you trying to say, at least until we're in an "I robot" situation where autonomous androids are welcomed into our homes and workplaces and given guns, I'm simply not worried about it

That's just because of a failure of imagination. The real world is not like Hollywood, get Terminator out of your head. A real AI take over is likely something we probably can't imagine because otherwise we would be smart enough to thwart it. It's micro drones injecting everyone on earth with a potent neurotoxins or a mirror virus that is dispersed into the entire atmosphere and kills everyone. Or its industrial AIs…

> Get Terminator out of your head. A real AI take over is likely something we probably can't imagine.

Indeed, robotic bodies aren't needed. An ASI could take over even if it remained 100% software by hiring or persuading humans to do whatever it needed to be done. It could bootstrap the process by first doing virtual tasks for money, then taking that money to hire humans to register an actual company with human shareholders and executives (who report to the ASI), which company does some lucrative business and hires many more people. Soon the ASI has a massive human enterprise to do whatever it directs them to do.

The ASI still needs humans for awhile but it's a route to a takeover while remaining entirely as running code.

Re: Remarks on AI from NZ

#74

Post by acbart »

hnthrow90348765 wrote: post #7

>We may end up with at least one generation of people who are like the Eloi in H.G. Wells’s The Time Machine, in that they are mental weaklings utterly dependent on technologies that they don’t understand and that they could never rebuild from scratch were they to break down I don't think this can realistically happen unless all of the knowledge that brought us to that point was erased. Humans are also naturally curi…

The research that is coming out is very clear that the best students are benefitting, but the bad students are getting worse than if they had never seen the LLM. And the divide is growing, with fewer good students. LLMs are a disaster in education.

Re: Remarks on AI from NZ

#75

Post by jumploops »

NitpickLawyer wrote: post #8

> Maybe a useful way to think about what it would be like to coexist in a world that includes intelligences that aren’t human is to consider the fact that we’ve been doing exactly that for long as we’ve existed, because we live among animals. Another analogy that I like is about large institutions / corporations. They are, right now, kind of like AIs. Like Harari says in one of his books, Peugeot co. is an entity tha…

Corporations, governments, religions -- all human-level intelligences with non-human goals (profit, power, influence).

A professor of mine wrote a paper on this[0](~2012).

[0]https://web.eecs.umich.edu/~kuipers/papers/Kuipers-ci-12.pdf

Re: Remarks on AI from NZ

#76

Post by drdaeman »

QuadmasterXLII wrote: post #33
vonneumannstan wrote: post #28

Earlier quoted context omitted.

Unless you have a truly bastardized definition of ASI then there is undoubtedly nothing close to it on earth. No corporation or military or government comes close to what ASI could be capable of. Any reasonably smart person can identify errors that Militaries, Governments and Corporations make ALL THE TIME. Do you really think a Chimp can identify the strategic errors Humans are making? Because that is where you woul…

Putting aside questions of what is and isn’t artificial, I think with the usual definitions “Is Microsoft a superintelligence” and “Can Microsoft build a superintelligence” are the same question.

Sorry, I don’t get it. Why is it a requirement for a superintelligence (whatever it may be) to be able to create another superintelligence (I assume, of comparable “super-ness”)?

Re: Remarks on AI from NZ

#78

Post by ViscountPenguin »

vonneumannstan wrote: post #28
NitpickLawyer wrote: post #8

> Maybe a useful way to think about what it would be like to coexist in a world that includes intelligences that aren’t human is to consider the fact that we’ve been doing exactly that for long as we’ve existed, because we live among animals. Another analogy that I like is about large institutions / corporations. They are, right now, kind of like AIs. Like Harari says in one of his books, Peugeot co. is an entity tha…

Unless you have a truly bastardized definition of ASI then there is undoubtedly nothing close to it on earth. No corporation or military or government comes close to what ASI could be capable of. Any reasonably smart person can identify errors that Militaries, Governments and Corporations make ALL THE TIME. Do you really think a Chimp can identify the strategic errors Humans are making? Because that is where you woul…

Do we know that Chimps can't identify some subset of human strategic errors? I'm not convinced that's the case.

The idea of dumber agents supervising smarter ones seems relatively grounded to me, and forms the basis of OpenAIs old superalignment efforts (although I think that team might've been disbanded?)

Re: Remarks on AI from NZ

#79

Post by foxglacier »

pona-a wrote: post #17
NitpickLawyer wrote: post #8

> Maybe a useful way to think about what it would be like to coexist in a world that includes intelligences that aren’t human is to consider the fact that we’ve been doing exactly that for long as we’ve existed, because we live among animals. Another analogy that I like is about large institutions / corporations. They are, right now, kind of like AIs. Like Harari says in one of his books, Peugeot co. is an entity tha…

Did we survive these entities? By current projections, between 13.9% and 27.6% of all species would be likely to be extinct by 2070 [0]. The USA suffers an estimated 200,000 annual deaths associated with lacking health insurance [1]. Thanks to intense lobbying by private prisons, the US incarceration rate is 6 times that of Canada, despite similar economic development [2]. Sure, the human species is not yet on the br…

You have to be careful with species. It could be dominated by obscure minor local variations in insects and fungi that nobody would even notice went missing and which might not actually matter.

Apparently almost all animal species are insects:

https://ourworldindata.org/grapher/number-of-described-speci...

Re: Remarks on AI from NZ

#80

Post by kilpikaarna »

Lerc wrote: post #52

I found this a little frustrating. I liked the content of the talk, but I live in New Zealand, I have thoughts and opinions on this topic. I would like to think I offer a useful perspective. This post was how I found out that there are people in my vicinity talking about these issues in private. I don't presume that I am important enough that it should be necessary to invite me to discussions with esteemed people, no…

Assuming it's basically the same bunch of bunker billionaires who a few years back invited Douglas Rushkoff to give pointers on how to keep their security guards in check after SHTF. They've found their answer, now they just need to figure out how to control the superintelligence...
Post reply on HN