Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Superintelligence: The Idea That Eats Smart People (2016)

idlewords.com

Post reply on HN
Print view View original

191–200 of 253 posts

Re: Superintelligence: The Idea That Eats Smart People (2016)

#191

Post by wizzwizz4 »

Timwi wrote: post #147
wizzwizz4 wrote: post #136

Earlier quoted context omitted.

LLMs are neither of those. They're quite good at pretending they understand what you mean, but they don't. That's why they can't execute: they're mimicking the form, not the substance, and then we see the form and anthropomorphise them in our minds.

That's a lot of assertions with no real argument to back it up.

I've repeated the argument over and over since the GPT-2 days, when I derived it theoretically by inspecting the architecture of the model. I am now fatigued, and enough other people have taken up similar arguments – some developed half-way to a mathematical proof – that I no longer feel the obligation to keep repeating myself.

Re: Superintelligence: The Idea That Eats Smart People (2016)

#192

Post by quirkot »

willis936 wrote: post #5

Some of the themes remind me of themes mentioned in this matrix analysis. Specifically I am reminded of the Dune concept of control: "you control what you can destroy" and then asking "do you control your refrigerator?". Sure, you can turn it off but then your food would rot and you might starve. So in a real sense humans have not controlled machines for a long time but have been co evolving in symbiosis. Sure, it's…

Gotta keep in mind that "you control what you can destroy" was expressed as the crux of a single-planet guerrilla group's maximalist negotiation strategy with an interstellar government, where the destruction was of the ability to travel between the stars. It's not a coherent theory of control outside of the razor thin application of "I'm willing to accept pain as long it hurts you much more." It is not a control that builds stability and infrastructure, it is the control of base extortion

Re: Superintelligence: The Idea That Eats Smart People (2016)

#193

Post by fontain »

stephencanon wrote: post #57
blaze33 wrote: post #52

Earlier quoted context omitted.

The Man vs Horse marathon is interesting but in a real race we have no actual chance of winning. There are horse endurance races where the winner arrived in 7,5 hours after 160km[1]. That's a sub 2-hours marathon almost 4 times in a row (not to mention with a guy on your back). [1] https://eatnstays.com/uaes-almazrouei-wins-almutadil-cup-at-...

“in a real race we have no actual chance of winning” is an absolutely wild thing to say in response to a link to a real race in which the human has won the last few years in a row.

The race is pretty much designed to be a difficult for horses as possible to give humans a chance. Except for the parts that are extremely difficult for horses, horses steamroll the human competitors.

https://youtu.be/FY9Ee6-CIFM?t=342

Re: Superintelligence: The Idea That Eats Smart People (2016)

#194

Post by blamestross »

ACCount37 wrote: post #70
blamestross wrote: post #6

AI Superintelligence doesn't scare me for the same reasons "grey goo" doesn't scare me. We are awash in self-replicating machines. The biosphere is already a grey-goo apocalypse. Any new competitors have a serious moat to cross to out compete any existing self-replicators. We are awash in intelligent agents. Our society (and meta society) is full of superhuman agents already. There is a huge moat for any new intellig…

"Intelligent agents" we have around run on a metabolic budget of 25W and a hardware platform the size of a melon. Human intelligence doesn't scale upwards well. Individual humans only get this smart, and there are gains from getting multiple humans to work together - but the more of them you add, the larger is your communication and coordination overhead. In no small part because humans are self-interested agents tha…

As a distributed systems engineer, we are a LONG way from "magical scalable ai".

The bottleneck for a developing AI is experience. Yes we need compute, but we need data to compute on.

We have bypassed that limit by starting with literally every scrap of human generated prose that ever existed. I expect an explosion of expansion when visual and world models hit critical mass to properly leverage new experiences. But even then, engaging with reality is the bottleneck.

I can build you a very efficient scalable online map-reduce-like that runs inference on new corpus. We already made that. It took hardware getting large enough to fit the corpus in memory, instead of "scaling" it with networks for it to be viable. The latency of the network passing around partial solutions was WAY too high.

Computers don't scale forever. They are made of hot metals. The limits are heat, material, and the speed of light, but those are very real limits, that don't offer more than a constant multiplier of advantage over meat.

AIs might get smarter than us, arguably, like many other meat and paper based super-human intelligences around us, they already are. But it doesn't scale forever. It will hit limits, fairly quickly, of compute and experience to integrate into it's overfit model.

Re: Superintelligence: The Idea That Eats Smart People (2016)

#195

Post by antonvs »

mbgerring wrote: post #63
ctoth wrote: post #54

Earlier quoted context omitted.

I keep seeing this and I want to speak to it. When Dario and others say things like "this is happening and we should probably figure out what to do about it" what ends up happening is people hear "this is happening," see that the person warning them is the person doing the thing, and then short-circuit. "Why can't you just stop then?" Dario's point, and the point of the people actually trying to solve the problem, is…

Physics allows this, and actually taking advantage of it requires billions of dollars of unprecedented infrastructure buildout that is already destabilizing the power grid . The only reason that infrastructure buildout is happening at all is the ideological capture of a small handful of obscenely wealthy people, who are fueling this buildout by spreading the extreme paranoia you’re echoing here. I do not understand w…

> There is nothing inevitable about tying up all of this productive capital in the pursuit of AI.

In the US capitalist context, it's certainly inevitable, because AI is the biggest and most attractive source of profit and power out there right now. In that context, the broad strokes of what's happening currently, including the financial bubble, are predictable and inevitable.

What are the concrete steps which would allow us to "easily stop this train"? And why haven't we used steps like that to stop other cases where obscenely wealthy people have screwed everyone else over to increase their wealth? Is public control of the means of production involved, perhaps? If so, your definition of "easily" and mine are incompatible.

Re: Superintelligence: The Idea That Eats Smart People (2016)

#196

Post by ACCount37 »

blamestross wrote: post #194
ACCount37 wrote: post #70

Earlier quoted context omitted.

"Intelligent agents" we have around run on a metabolic budget of 25W and a hardware platform the size of a melon. Human intelligence doesn't scale upwards well. Individual humans only get this smart, and there are gains from getting multiple humans to work together - but the more of them you add, the larger is your communication and coordination overhead. In no small part because humans are self-interested agents tha…

As a distributed systems engineer, we are a LONG way from "magical scalable ai". The bottleneck for a developing AI is experience. Yes we need compute, but we need data to compute on. We have bypassed that limit by starting with literally every scrap of human generated prose that ever existed. I expect an explosion of expansion when visual and world models hit critical mass to properly leverage new experiences. But e…

Nah. Physical limits of computation are far enough away that the "constant multiplier of advantage" would have to be measured in OOMs. "Computers can be, at most, 1e11 times more powerful than brains" is not the saving grace you want it to be.

And, so far, the results of "visual data for improving general intelligence" runs were nothing but disappointments.

I think vision is just a piss poor modality to learn intelligence from? Very low value, per bit and per token both. You only ever want to tap it if you need your AI to operate based on visual data at deployment time. Otherwise, even "experience" is best gathered in text RLVR rollouts.

The secret of human sample efficiency isn't that visual data is somehow better for learning intelligence. It just isn't. Human "training data" is a hundred kinds of awful - humans are just good at scavenging it for all its worth. Evolution has tuned that very well.

Which means: AIs can get good at it too. It's not a wall - it's a skill issue.

Re: Superintelligence: The Idea That Eats Smart People (2016)

#197

Post by dmos62 »

wizzwizz4 wrote: post #191
Timwi wrote: post #147

Earlier quoted context omitted.

That's a lot of assertions with no real argument to back it up.

I've repeated the argument over and over since the GPT-2 days, when I derived it theoretically by inspecting the architecture of the model. I am now fatigued, and enough other people have taken up similar arguments – some developed half-way to a mathematical proof – that I no longer feel the obligation to keep repeating myself.

You could post a link.

Re: Superintelligence: The Idea That Eats Smart People (2016)

#198

Post by CGMthrowaway »

My_Name wrote: post #174

I note for premise 2 that the people who doubt the premise are the very stupid, but more importantly the very smart. The former is no challenge to the premise, but the latter? That is a different story. EDIT : For S&G I asked Claude about it. It replied : The talk groups Penrose with the religious doubter, as if the two objections were the same species and could be dispatched by the same gesture: most of us find this…

Idk whether I fall into the former or latter bucket, but "smarter minds see further, judge better" and "there is a soul that taps into a cosmic godhead" are one and the same statement to me

Re: Superintelligence: The Idea That Eats Smart People (2016)

#199

Post by famouswaffles »

fragmede wrote: post #188
famouswaffles wrote: post #95

Earlier quoted context omitted.

Yeah the whole thing is strange. He does such a good job in the first bit outlining all the reasons for caution that I was intrigued to see what he had to say against it, but it's just one bad argument after another. Even Hawking's cat is bad. Make some money and pay someone to get the cat in the box.

Paying someone to physically force the cat in the carrier is missing the point. Compared to the cat, Hawking is a super intelligence. But all that intelligence isn't able to get him to speak cat and give the cat a reason to get inside the carrier of its own volition.

Who said anything about physically forcing the cat in the box? The point of Hawkings cat is is the lack of physical embodiment for AI. So get someone else to do it. Millions of people put cats in boxes without any force just fine. Or do you classify it as 'force' because it went beyond speaking? You can't actually speak to cats (unlike humans) so acting like anything more than words is brute force is strange.

Who was Hitler to most people ? Just a persuasive powerful voice on the radio, or words on a paper. So not even two way communication and yet he inspired armies that killed millions. Hawkings wouldn't even need to pay anyone. There would be plenty of people willing to get the cat in the box for Stephen Hawkings.

Re: Superintelligence: The Idea That Eats Smart People (2016)

#200

Post by smaudet »

goatlover wrote: post #102
smaudet wrote: post #64

Earlier quoted context omitted.

For all its talk of inoculation, this is a terribly written essay. They do not make a point, nor even arguments, instead, opting to ramble in hopes that you forget whatever it was you were thinking. The issue is simple. Just like us (who are arguably complex, look at what we're building over here, this AI computer stuff!), entities have simple core needs (like food, water, power, etc.). An infinitely smart AGI has th…

Infinities aren't a physical reality. Resource are always limited, physics is limited at the Planck scale. You can only do so much compute in a finite volume of space, and there will only be so much energy available. As for simple needs, humans also have complex ones around social interactions and the need for mental stimulation.

Infinity is....subtle.

They are more characterized by how they grow and when they stop than by their "physical reality". Proof of this is in that different infinities exist - characterized precisely by how fast they grow, i.e., one "infinity" is "larger" than the other.

My point is, getting hung up on "infinity" as being unrealistic is not the point. It is the tool with which to understand how thing behave. The same as any calculus problem - you take the limit to the infinity to understand how the function behaves.

Post reply on HN