Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Superintelligence: The Idea That Eats Smart People (2016)

idlewords.com

Post reply on HN
Print view View original

141–150 of 253 posts

Re: Superintelligence: The Idea That Eats Smart People (2016)

#141

Post by JumpCrisscross »

willis936 wrote: post #140
JumpCrisscross wrote: post #126

Earlier quoted context omitted.

> Why do you think it's a bad definition? It presumes a sentient, rational counterparty. Being able to shoot a horse isn't the same as being able to ride it.

It only requires agency from the party assuming control. There are other definitions of control in contexts other than power dynamics that are important. Like a PLL can control an oscillator frequency without the philosophical question of agency needing to be applied. That is a closer definition of control for riding a horse. You control the horse when you pull the reigns. You also control the horse when you decide w…

> only requires agency from the party assuming control

It's a political concept. It requires agency from the actor recognising the threat. We're pretty close to being able to hurl a giant rock at Mars. That doesn't by a long shot mean we "control" it.

If there were a human settlement on it, on the other hand, being able to credibly threaten Armageddon does give the thrower control.

Re: Superintelligence: The Idea That Eats Smart People (2016)

#142

Post by Timwi »

stoneman24 wrote: post #67
foobiekr wrote: post #38

Earlier quoted context omitted.

There's also the story "Go, Go, Go, Said the Bird" by Sonya Dorman in the first Dangerous Visions which is like this, in a way.

There’s also “the ruum” by Arthur Porges[0]. We got as part of English class in high school, a long time ago. Try not to read the Wikipedia as it might spoil the short story, there’s the pdf available on the web somewhere [0] https://en.wikipedia.org/wiki/The_Ruum

I read it too in school. I thought it was super obscure!

Re: Superintelligence: The Idea That Eats Smart People (2016)

#143

Post by cevn »

ACCount37 wrote: post #83
cevn wrote: post #81

Earlier quoted context omitted.

What's 1.05 x 0? Even if you run it 1000 times, it's still 0.

Are you trying to say that the capability of modern LLMs is zero? Because if so, I'm pretty sure any frontier LLM is better at evaluating AI capabilities than you are.

Claims of AGI imply that LLM's have intelligence. They don't, they are fancy probability machines. They don't THINK the way we do, they just do 200 matrix multiplications until their training data is massaged into what you need. They don't dream, they don't remember what you tell them. Even if you write one sentence, 'attention' means they will ignore half of what you say and key in on the wrong thing. This just happened to me today on a frontier model.

I'm not saying that AGI is impossible, but the focus on LLM's is probably not the right approach. I don't think we will ever make it until we understand the human mind better.

Re: Superintelligence: The Idea That Eats Smart People (2016)

#144

Post by tim333 »

stared wrote: post #53

I like this essay, and have very often referred to it when someone talks about AGI. There is a common narrative bias to look at AGI as the Abrahamic God, if not explicitly, then just by saying that it is omniscient, omnipotent, immortal - and will judge us for our deeds. It is tempting for anyone raised in the West, and immersed in Judeo-Christian culture. And for anyone, in general, as it offers an epic narration of…

I've only heard the God narrative from skeptics as in look at those idiots thinking they are building God. I think most people who believe in AGI arriving see it more as something like a chess computer but as well as beating us at chess it'll do other thinking too. A souped up chess computer isn't God.

Re: Superintelligence: The Idea That Eats Smart People (2016)

#145

Post by throwawayffffas »

topkai22 wrote: post #26
forinti wrote: post #15

Earlier quoted context omitted.

There's a story about some Kenyans outrunning a Cheetah in 6km. It had been killing their livestock, so they decided to go after it. Cheetahs are very fast, but humans have way more endurance.

I think the term is persistence hunting ( https://en.wikipedia.org/wiki/Persistence_hunting ). There was an intriguing blog post years ago (which sadly I can’t find) about how terrifying a fictional portrayal of persistence hunting would be- grinding down prey through exhaustion, thinking you’ve escaped but being slowly tracked down.

That is how wolves hunt. Their system includes switching leads to maintain pressure on the prey while maintaining the packs endurance.

Re: Superintelligence: The Idea That Eats Smart People (2016)

#146

Post by throwawayffffas »

tptacek wrote: post #93
topkai22 wrote: post #26

Earlier quoted context omitted.

I think the term is persistence hunting ( https://en.wikipedia.org/wiki/Persistence_hunting ). There was an intriguing blog post years ago (which sadly I can’t find) about how terrifying a fictional portrayal of persistence hunting would be- grinding down prey through exhaustion, thinking you’ve escaped but being slowly tracked down.

https://en.wikipedia.org/wiki/It_Follows

The fact that no one in that movie considered placing a treadmill at a strategic point vexed me to no end.

Re: Superintelligence: The Idea That Eats Smart People (2016)

#147

Post by Timwi »

wizzwizz4 wrote: post #136
dan-robertson wrote: post #108

Earlier quoted context omitted.

The super literal interpretation ideas were much more common in the past when LLMs didn’t exist. Now we have models that are generally pretty good at picking up on nuance and understanding what you mean but also often quite bad at execution, which is roughly the opposite of that idea. I think reward hacking is perhaps the closest we see llms get to literal/malicious interpretations of instructions.

LLMs are neither of those. They're quite good at pretending they understand what you mean, but they don't. That's why they can't execute: they're mimicking the form, not the substance, and then we see the form and anthropomorphise them in our minds.

That's a lot of assertions with no real argument to back it up.

Re: Superintelligence: The Idea That Eats Smart People (2016)

#148

Post by ACCount37 »

cevn wrote: post #143
ACCount37 wrote: post #83

Earlier quoted context omitted.

Are you trying to say that the capability of modern LLMs is zero? Because if so, I'm pretty sure any frontier LLM is better at evaluating AI capabilities than you are.

Claims of AGI imply that LLM's have intelligence. They don't, they are fancy probability machines. They don't THINK the way we do, they just do 200 matrix multiplications until their training data is massaged into what you need. They don't dream, they don't remember what you tell them. Even if you write one sentence, 'attention' means they will ignore half of what you say and key in on the wrong thing. This just happ…

Do you think your brain doesn't do a type of gradient descent, trying to fit its little predictive algorithms to its senses? Do you think you aren't a fancy probability machine with overinflated self-esteem?

An average LLM of today has better reading comprehension than an average human, and the gap only grows release to release.

"Understand the human mind" turned out to be a distractor. The bitter lesson won: you can take a "good enough" AI architecture, burn a shitton of data into it with an unholy amount of training compute, and get halfway to AGI - no "understand the brain" required. LLMs are so fried in imitation learning on human-generated data they even inherit humanlike failure modes.

Re: Superintelligence: The Idea That Eats Smart People (2016)

#150

Post by loeg »

drfloyd51 wrote: post #89
loeg wrote: post #80

Earlier quoted context omitted.

The thing about fiction is, it's fiction. The author can write whatever outcome they want. It doesn't necessarily tell us anything about the real world.

But it does give us an idea to chew over. And we can determine how “real world” that idea is. Some of us can even see how to bend the idea a little and make a reasonable version of it “real”.

Yeah, that's the most charitable possible interpretation. But a fiction writer could just as easily write a utopias as a dystopia, and one isn't any more "real" than the other. It's just fiction.
Post reply on HN