Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Superintelligence: The Idea That Eats Smart People (2016)

idlewords.com

Post reply on HN
Print view View original

131–140 of 253 posts

Re: Superintelligence: The Idea That Eats Smart People (2016)

#131

Post by slopinthebag »

willis936 wrote: post #5

Some of the themes remind me of themes mentioned in this matrix analysis. Specifically I am reminded of the Dune concept of control: "you control what you can destroy" and then asking "do you control your refrigerator?". Sure, you can turn it off but then your food would rot and you might starve. So in a real sense humans have not controlled machines for a long time but have been co evolving in symbiosis. Sure, it's…

> Sure, you can turn it off but then your food would rot and you might starve.

If I live in a world where I can afford a freezer with food in it, it's practically guaranteed I can destroy my fridge without starving to death after. Heck, even if I was completely broke I could destroy my fridge and would have a pretty good (+99.999%) chance of not dying in the next year from starvation.

I get I'm nitpicking your point a bit, but I actually think most of our machines we could destroy and still be fine. We'd need to make sacrifices to our quality of life of course..

Re: Superintelligence: The Idea That Eats Smart People (2016)

#132

Post by aidenn0 »

ctoth wrote: post #54
cute_boi wrote: post #12

Earlier quoted context omitted.

Yep. That said, unlike cheetahs, there’s plenty of evidence of leopards attacking humans. And these days, it’s the leopards, the closed-AI types and misanthropes -- telling everyone, “AI will take your job and there’s nothing you can do about it.”

I keep seeing this and I want to speak to it. When Dario and others say things like "this is happening and we should probably figure out what to do about it" what ends up happening is people hear "this is happening," see that the person warning them is the person doing the thing, and then short-circuit. "Why can't you just stop then?" Dario's point, and the point of the people actually trying to solve the problem, is…

I don't think we can stop it, but the people saying "this is happening and we should probably figure out what to do about it" are doing their best to accelerate it, which gives us less time to figure out what to do about it.

Re: Superintelligence: The Idea That Eats Smart People (2016)

#133

Post by tim333 »

alex-reyss wrote: post #4

The main problem of the hard takeoff theory is not the abstract nature of the scenario but rather the fact that it makes the same mistake as the unconstrained optimization paradigm, it takes intelligence to be an unconstrained optimization process. In fact, if we consider the strongest version of the safety argument for AI, namely one in which the danger is not coming from robots but rather from a disembodied AI cont…

I figure the main problem with a hard takeoff is lack of hardware capacity. The large labs are trying to improve as fast as they can but you can only do so many training runs. If you say to Claude Code try making a hundred improved versions of yourself, train them test them and then have the best three do that again, it's not going to go very well because you'll max out the compute which probably could have been used more efficiently by human researchers.

Re: Superintelligence: The Idea That Eats Smart People (2016)

#134

Post by fragmede »

aidenn0 wrote: post #132
ctoth wrote: post #54

Earlier quoted context omitted.

I keep seeing this and I want to speak to it. When Dario and others say things like "this is happening and we should probably figure out what to do about it" what ends up happening is people hear "this is happening," see that the person warning them is the person doing the thing, and then short-circuit. "Why can't you just stop then?" Dario's point, and the point of the people actually trying to solve the problem, is…

I don't think we can stop it, but the people saying "this is happening and we should probably figure out what to do about it" are doing their best to accelerate it, which gives us less time to figure out what to do about it.

The people saying this is happening are competing with each other for resources, so there's no way for one of them to hold back without losing out to the others. We see this with people dropping Claude subscriptions in favor of ChatGPT because codex 5.5 > Claude Opus 4.6/7/8. Anthropic is losing money by not releasing Mythos.

Re: Superintelligence: The Idea That Eats Smart People (2016)

#135

Post by slopinthebag »

altruios wrote: post #110
CGMthrowaway wrote: post #106

> Premise 2: The brain is an ordinary configuration of matter, albeit an extraordinarily complicated one. If we knew enough, and had the technology, we could exactly copy its structure and emulate its behavior with electronic components.... The mind arises out of ordinary physics.... If you are very religious, you might believe that a brain is not possible without a soul. But for most of us, this is an easy premise t…

What do even mean by 'soul'? What do you mean by 'supernatural' - and (assuming your definition is the standard one of 'not detectable by any measurement') by what mechanism that could possibly affect physical matter? (the onus is on you to prove the positive claim that there exists the supernatural or soul to begin with, which there is currently no evidence for). The concept is self defeating by its own definition,…

Not OP but I'll take a shot since I have somewhat similar sounding views. (I assume OP is talking about consciousness and it's origins when they use the term "soul").

> What do you mean by 'supernatural'

I would just say something outside our current capacity for understanding. How does that quote go...something like "sufficiently advanced technology is indistinguishable from magic". "Not detectable by any measurement" isn't right because we clearly detect it in some way since we are discussing it now.

> Feeling that there 'must be' a soul is not enough to prove that it exists.

We don't have any proof that consciousness is part of the brain and is produced by it either. We also can't even prove other people are conscious besides ourselves. In this domain the idea of "proof" becomes less relevant.

In a simulation of a storm, does anything get wet? In a simulation of a mind, is there a real conscious? A real soul? Or just a simulation of one?

My guess is our brains act as a receiver for some "field" of consciousness. Of course it's just a guess, same as yours or anybody else's conceptions of consciousness and the spiritual world.

Re: Superintelligence: The Idea That Eats Smart People (2016)

#136

Post by wizzwizz4 »

dan-robertson wrote: post #108
throwuxiytayq wrote: post #36

Earlier quoted context omitted.

Isn't this a misinterpretation of what everyone in the AI safety space is worried about, though? I think the idea is that having an AI that interprets everything in a super-literal way would probably be catastrophic, but we can't even build that . It would be a nice world-ending problem to have.

The super literal interpretation ideas were much more common in the past when LLMs didn’t exist. Now we have models that are generally pretty good at picking up on nuance and understanding what you mean but also often quite bad at execution, which is roughly the opposite of that idea. I think reward hacking is perhaps the closest we see llms get to literal/malicious interpretations of instructions.

LLMs are neither of those. They're quite good at pretending they understand what you mean, but they don't. That's why they can't execute: they're mimicking the form, not the substance, and then we see the form and anthropomorphise them in our minds.

Re: Superintelligence: The Idea That Eats Smart People (2016)

#137

Post by 8note »

neaden wrote: post #101
notahacker wrote: post #65

Earlier quoted context omitted.

Think it's more to do with hilly, wooded courses unfavourable to horses and a requirement that the horse's heart rate remain below a certain threshold when inspected by a vet in the middle and end of a course, whereas the leading humans are maxing out their efforts I don't think the Mongol cavalry would lose races to humans over any distance of steppe

Yeah I think on say, a proper road the horse would win at any distance.

the road adds some infrastructure - i think at that point the person should get a bike?

Re: Superintelligence: The Idea That Eats Smart People (2016)

#138

Post by ACCount37 »

garaetjjte wrote: post #100
ACCount37 wrote: post #70

Earlier quoted context omitted.

"Intelligent agents" we have around run on a metabolic budget of 25W and a hardware platform the size of a melon. Human intelligence doesn't scale upwards well. Individual humans only get this smart, and there are gains from getting multiple humans to work together - but the more of them you add, the larger is your communication and coordination overhead. In no small part because humans are self-interested agents tha…

>the more of them you add, the larger is your communication and coordination overhead. In no small part because humans are self-interested agents that simply aren't designed to compose their capabilities seamlessly. What proves that AI doesn't have the same limitations? There's only so much computation you can do in given space, and all communication is limited by universal speed limit.

What? Humans are made of sloppy wet meat. Brains are nowhere near brushing against the physical limits of computation, speed of causality or others, in any way, fashion or form. You need to put a lot of intelligent design on the table before you even start getting close to those walls.

Which doesn't bode well for the future of human intelligence. Computing hardware gets better at what it does generation to generation, but no one is about to release Human Brain 2.0 any time soon. Human mind is not a fast-moving target.

Principal-agent problem isn't a physical law. It's a limitation that AIs don't have to suffer from. Humans have to delegate to other humans - but for AI, "principal" and "agent" might just be the same exact system instanced twice.

Re: Superintelligence: The Idea That Eats Smart People (2016)

#139

Post by JumpCrisscross »

smaudet wrote: post #64
stared wrote: post #53

I like this essay, and have very often referred to it when someone talks about AGI. There is a common narrative bias to look at AGI as the Abrahamic God, if not explicitly, then just by saying that it is omniscient, omnipotent, immortal - and will judge us for our deeds. It is tempting for anyone raised in the West, and immersed in Judeo-Christian culture. And for anyone, in general, as it offers an epic narration of…

For all its talk of inoculation, this is a terribly written essay. They do not make a point, nor even arguments, instead, opting to ramble in hopes that you forget whatever it was you were thinking. The issue is simple. Just like us (who are arguably complex, look at what we're building over here, this AI computer stuff!), entities have simple core needs (like food, water, power, etc.). An infinitely smart AGI has th…

> We're already seeing the effect in the computing sector on e.g. chips, there's no reason to think this trend won't continue...

That effect is underwritten by economic demand and moderated by economic costs. There are more reasons to expect the trend to asymptote than somehow turn into an infinite process.

Re: Superintelligence: The Idea That Eats Smart People (2016)

#140

Post by willis936 »

JumpCrisscross wrote: post #126
wasabi991011 wrote: post #116

Earlier quoted context omitted.

Why do you think it's a bad definition? Not for fridges, I think that was a bad example. But it seems accurate at the level of geopolitics, where e.g. Iran shows it controls Hormuz by closing it with mines and other weaponry.

> Why do you think it's a bad definition? It presumes a sentient, rational counterparty. Being able to shoot a horse isn't the same as being able to ride it.

It only requires agency from the party assuming control. There are other definitions of control in contexts other than power dynamics that are important. Like a PLL can control an oscillator frequency without the philosophical question of agency needing to be applied. That is a closer definition of control for riding a horse. You control the horse when you pull the reigns. You also control the horse when you decide when it is put down. Two different controls.
Post reply on HN