Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Superintelligence: The Idea That Eats Smart People (2016)

idlewords.com

Post reply on HN
Print view View original

101–110 of 253 posts

Re: Superintelligence: The Idea That Eats Smart People (2016)

#101

Post by neaden »

notahacker wrote: post #65
aqfamnzc wrote: post #49

Earlier quoted context omitted.

Wow, seems surprisingly balanced. I would guess that if it was 50% longer distance than humans would win reliably, and 50% shorter would allow the horses to win?

Think it's more to do with hilly, wooded courses unfavourable to horses and a requirement that the horse's heart rate remain below a certain threshold when inspected by a vet in the middle and end of a course, whereas the leading humans are maxing out their efforts I don't think the Mongol cavalry would lose races to humans over any distance of steppe

Yeah I think on say, a proper road the horse would win at any distance.

Re: Superintelligence: The Idea That Eats Smart People (2016)

#102

Post by goatlover »

smaudet wrote: post #64
stared wrote: post #53

I like this essay, and have very often referred to it when someone talks about AGI. There is a common narrative bias to look at AGI as the Abrahamic God, if not explicitly, then just by saying that it is omniscient, omnipotent, immortal - and will judge us for our deeds. It is tempting for anyone raised in the West, and immersed in Judeo-Christian culture. And for anyone, in general, as it offers an epic narration of…

For all its talk of inoculation, this is a terribly written essay. They do not make a point, nor even arguments, instead, opting to ramble in hopes that you forget whatever it was you were thinking. The issue is simple. Just like us (who are arguably complex, look at what we're building over here, this AI computer stuff!), entities have simple core needs (like food, water, power, etc.). An infinitely smart AGI has th…

Infinities aren't a physical reality. Resource are always limited, physics is limited at the Planck scale. You can only do so much compute in a finite volume of space, and there will only be so much energy available.

As for simple needs, humans also have complex ones around social interactions and the need for mental stimulation.

Re: Superintelligence: The Idea That Eats Smart People (2016)

#104

Post by goatlover »

ACCount37 wrote: post #70
blamestross wrote: post #6

AI Superintelligence doesn't scare me for the same reasons "grey goo" doesn't scare me. We are awash in self-replicating machines. The biosphere is already a grey-goo apocalypse. Any new competitors have a serious moat to cross to out compete any existing self-replicators. We are awash in intelligent agents. Our society (and meta society) is full of superhuman agents already. There is a huge moat for any new intellig…

"Intelligent agents" we have around run on a metabolic budget of 25W and a hardware platform the size of a melon. Human intelligence doesn't scale upwards well. Individual humans only get this smart, and there are gains from getting multiple humans to work together - but the more of them you add, the larger is your communication and coordination overhead. In no small part because humans are self-interested agents tha…

> AI can scale intensively and extensively. AI can be scaled up by upping the compute budgets. AI can be replicated and copied indefinitely. AI doesn't have the innate human "I don't live to work, I work to live" overhead. AI can outclass human intelligence by a long shot.

These are claims about future AI, not actual facts. Part of the counter argument is the world will already be awash in AIs institutions and individuals make use of. An ASI would arise in a world that is already full of formidable intelligences that provide a check on what it can do. This is what happened with the evolution of replicators/life. No species was able to fully dominate the biosphere because there are too many other capable replicators, and there are always tradeoffs in capabilities.

We imagine the possibility of an unrestrained god-like ASI ruling the solar system. But it's just that, an imagination backed by the assumption that self-recursive improvement leads there. Problem is, the real world never turns out to be that simple.

It's probably the case that alien ASI replicators aren't devouring the universe either because of various restraints.

Re: Superintelligence: The Idea That Eats Smart People (2016)

#105

Post by projektfu »

stephencanon wrote: post #57
blaze33 wrote: post #52

Earlier quoted context omitted.

The Man vs Horse marathon is interesting but in a real race we have no actual chance of winning. There are horse endurance races where the winner arrived in 7,5 hours after 160km[1]. That's a sub 2-hours marathon almost 4 times in a row (not to mention with a guy on your back). [1] https://eatnstays.com/uaes-almazrouei-wins-almutadil-cup-at-...

“in a real race we have no actual chance of winning” is an absolutely wild thing to say in response to a link to a real race in which the human has won the last few years in a row.

I think it implies that the best competitors are not participating.

Re: Superintelligence: The Idea That Eats Smart People (2016)

#106

Post by CGMthrowaway »

> Premise 2: The brain is an ordinary configuration of matter, albeit an extraordinarily complicated one. If we knew enough, and had the technology, we could exactly copy its structure and emulate its behavior with electronic components.... The mind arises out of ordinary physics.... If you are very religious, you might believe that a brain is not possible without a soul. But for most of us, this is an easy premise to accept.

This is the premise I rejected immediately and, if you agree with me, it takes down the whole house of cards. Let me explain. The rationale has nothing to do with "quantum shenanigans."

I have been called religious but will readily concede that of course a physical brain is possible without a soul. What is impossible is to replicate a soul with purely physical matter. Therefore we may understand that "superintelligence" is possible, and maybe inevitable on the long thread of time, but - crucially - it will never be able to approach that supernatural element present in us (the spark of the godhead) and therefore never be able to replace humanity.

In that sense it is like any other natural disaster that threatens to make us extinct, but it is not some "superhuman" nor anything close.

Re: Superintelligence: The Idea That Eats Smart People (2016)

#107

Post by stared »

CamperBob2 wrote: post #91
stared wrote: post #55

Earlier quoted context omitted.

SOMA is existentialism 2.0 (wrote about it here https://p.migdal.pl/blog/2019/06/games-in-which-you-walk-and... ). To concepts you menton, I would add grey goo and x-risk.

Neat. Not sure if your site is a gold mine inside a rabbit hole, or a rabbit hole inside a gold mine, but I really dig both the aesthetic and the content.

It is lovely to hear these words!

Re: Superintelligence: The Idea That Eats Smart People (2016)

#108

Post by dan-robertson »

throwuxiytayq wrote: post #36
dmos62 wrote: post #25

> The AI is all-powerful and gives you what you ask for, but interprets everything in a super-literal way that you end up regretting. I like imagining similar discourse when a more basic tool was invented: "A hammer is like a genie, it's all powerful, but, when you hit something with it, it interprets that super-literally, and it hits it."

Isn't this a misinterpretation of what everyone in the AI safety space is worried about, though? I think the idea is that having an AI that interprets everything in a super-literal way would probably be catastrophic, but we can't even build that . It would be a nice world-ending problem to have.

The super literal interpretation ideas were much more common in the past when LLMs didn’t exist. Now we have models that are generally pretty good at picking up on nuance and understanding what you mean but also often quite bad at execution, which is roughly the opposite of that idea. I think reward hacking is perhaps the closest we see llms get to literal/malicious interpretations of instructions.

Re: Superintelligence: The Idea That Eats Smart People (2016)

#109

Post by thoughtpeddler »

dang wrote: post #77

Related. Others? Superintelligence: The Idea That Eats Smart People - https://news.ycombinator.com/item?id=34257025 - Jan 2023 (1 comment) Superintelligence: The Idea That Eats Smart People (2016) - https://news.ycombinator.com/item?id=18499973 - Nov 2018 (248 comments) Superintelligence: The Idea That Eats Smart People - https://news.ycombinator.com/item?id=13240811 - Dec 2016 (580 comments) Maciej Ceglowski – Super…

Thanks dang for compiling this. I suspect the Nov 2018 resurgence was due to Google publishing BERT [0] around that time? The release of OpenAI’s GPT-1 [1] was earlier that year in June, so unlikely that. Of course Jan 2023 needs no explanation… And now in 2026 things are at a fever pitch.

Interesting to trace these 10yr old AI posts from then to the present moment. The other one with a similar vintage would be “Should AI Be Open” [2] from Dec 2015, which is fascinating to juxtapose against the recent public battles.

[0] “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding“: https://arxiv.org/abs/1810.04805

[1] “Improving Language Understanding by Generative Pre-Training”: https://cdn.openai.com/research-covers/language-unsupervised...

[2] “Should AI Be Open?” | Slate Star Codex: https://slatestarcodex.com/2015/12/17/should-ai-be-open/

Re: Superintelligence: The Idea That Eats Smart People (2016)

#110

Post by altruios »

CGMthrowaway wrote: post #106

> Premise 2: The brain is an ordinary configuration of matter, albeit an extraordinarily complicated one. If we knew enough, and had the technology, we could exactly copy its structure and emulate its behavior with electronic components.... The mind arises out of ordinary physics.... If you are very religious, you might believe that a brain is not possible without a soul. But for most of us, this is an easy premise t…

What do even mean by 'soul'?

What do you mean by 'supernatural' - and (assuming your definition is the standard one of 'not detectable by any measurement') by what mechanism that could possibly affect physical matter? (the onus is on you to prove the positive claim that there exists the supernatural or soul to begin with, which there is currently no evidence for).

The concept is self defeating by its own definition, either it is physical in some capacity (and therefore can be measured and replicated through yet unknown means) or it is not (and therefore indistinguishable from not being there at all).

Feeling that there 'must be' a soul is not enough to prove that it exists.

The feeling of experience is not enough to prove that experience is in anyway supernatural.

> What is impossible is to replicate a soul with purely physical matter.

What? Why? Where is the proof of this?

First and foremost, I'll give you an in. There is a difference between material, and processes like waves, waves I would argue are non-physical things manifested in physical material: you might want to start there.

But all roads lead to Rome from that line of thinking too, so you might need to come up with something far more clever.

Post reply on HN