Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI 2040 and the cult of intelligence

geohot.github.io

Post reply on HN
Print view View original

91–100 of 280 posts

Re: AI 2040 and the cult of intelligence

#91

Post by fasterik »

johnfn wrote: post #60
Planktonne wrote: post #17

> I used to be one of these people. I read Yudkowsky and was like, OMG recursive self improvement hard takeoff AI is coming. I really think we need to stop giving credence to people who have 1) Been consistently wrong with all their predictions 2) Demonstrated an endless spiteful cynicism Some of these people are very talented in their fields, sure. But malevolent and incorrect should be disqualifying when they talk…

The guy was saying that AI was going to be a serious problem decades before anyone else was even considering it as a remote possibility. How is that in any way "consistently wrong with all their predictions"?

>The guy was saying that AI was going to be a serious problem decades before anyone else was even considering it as a remote possibility

That's not accurate at all. AI research has been around since the 1950s and pioneers of the field identified risks early on, including Alan Turing, Norbert Wiener, and I. J. Good.

The problem with Yudkowsky is that he lays out elaborate doomsday scenarios with extreme confidence, except none of it is grounded in realistic physical constraints, timelines, or empirical data. It's all divined a priori from Yudkowsky's ad hoc "rationalist" principles.

Re: AI 2040 and the cult of intelligence

#92

Post by piker »

johnfn wrote: post #60
Planktonne wrote: post #17

> I used to be one of these people. I read Yudkowsky and was like, OMG recursive self improvement hard takeoff AI is coming. I really think we need to stop giving credence to people who have 1) Been consistently wrong with all their predictions 2) Demonstrated an endless spiteful cynicism Some of these people are very talented in their fields, sure. But malevolent and incorrect should be disqualifying when they talk…

The guy was saying that AI was going to be a serious problem decades before anyone else was even considering it as a remote possibility. How is that in any way "consistently wrong with all their predictions"?

Has AI yet been a serious problem?

Re: AI 2040 and the cult of intelligence

#93

Post by bloaf »

charcircuit wrote: post #28
medler wrote: post #16

Strange way to structure an argument. He seems to be arguing in favor of local models, but most of his examples are reasons local models are bad.

Local models and alignment are orthogonal concepts.

I don't think this is entirely true, insofar as non-local models inherently have a "man cannot serve two masters" problem. The end user and the org running the AI may have different goals. That means even if AI alignment were to be in some sense "solved" for local models, you wouldn't be guaranteed an aligned AI using a non-local model because the "owners" of the non-local model may have deliberately misaligned the AI in their favor and against your interests.

Re: AI 2040 and the cult of intelligence

#95

Post by tccole »

ben_w wrote: post #67
mlinsey wrote: post #51

I have some sympathy to geohot's view when it comes to pure informational chatbots. It's a first amendment issue, I'm allowed to write and read books that are useful to getting away with crimes, etc. This obviously doesn't work at all when the agents start doing real things in the real world, though. "Hey AI, I don't like my neighbor, find an exploit in the firmware for his car and make the cruise control malfunction…

> It's a first amendment issue, I'm allowed to write and read books that are useful to getting away with crimes, etc. I'm neither American nor a lawyer. Is "conspiracy" protected under the first amendment? If you discuss a crime with someone to learn about it, does that count as "conspiracy"?

Yes conspiracy is covered, now if you act it out… that’s a different story.

Re: AI 2040 and the cult of intelligence

#96

Post by [deleted] »

skybrian wrote: post #52
gilrain wrote: post #46

Earlier quoted context omitted.

> Seems like being directionally right early that AI would be a Big Deal and scary should count for something? Only if it was an uncommon prediction; otherwise, it’s just evidence of common sense.

It's common sense now . It hasn't always been common sense that AI alignment is an important problem.

[deleted]

Re: AI 2040 and the cult of intelligence

#97

Post by 2snakes »

fasterik wrote: post #91
johnfn wrote: post #60

Earlier quoted context omitted.

The guy was saying that AI was going to be a serious problem decades before anyone else was even considering it as a remote possibility. How is that in any way "consistently wrong with all their predictions"?

>The guy was saying that AI was going to be a serious problem decades before anyone else was even considering it as a remote possibility That's not accurate at all. AI research has been around since the 1950s and pioneers of the field identified risks early on, including Alan Turing, Norbert Wiener, and I. J. Good. The problem with Yudkowsky is that he lays out elaborate doomsday scenarios with extreme confidence, ex…

Yah, another way of putting this is theory-ladenness of hyperreality of personal world model (rationality vs empiricism).

Re: AI 2040 and the cult of intelligence

#98

Post by AnthonyMouse »

mlinsey wrote: post #51

I have some sympathy to geohot's view when it comes to pure informational chatbots. It's a first amendment issue, I'm allowed to write and read books that are useful to getting away with crimes, etc. This obviously doesn't work at all when the agents start doing real things in the real world, though. "Hey AI, I don't like my neighbor, find an exploit in the firmware for his car and make the cruise control malfunction…

> "Hey AI, I don't like my neighbor, find an exploit in the firmware for his car and make the cruise control malfunction and crash him next time he gets on the highway". This is committing a crime, not just talking about theoretical crimes. The AI can and should refuse it.

The obvious problem being that the user doesn't have any need to provide the context that they're trying to commit a crime and can just ask how to do something without providing a reason or making one up.

At which point you'd have the model trying to impute a reason and often getting it dangerously wrong, e.g. refusing to disclose a vulnerability when the user is actually the defender who needs to patch/mitigate it.

Re: AI 2040 and the cult of intelligence

#99

Post by emp17344 »

luciana1u wrote: post #27

by 2040 the cult of intelligence will have its own schisms. the AGI maximalists will accuse the ASI accelerationists of heresy, and both sides will use AI to generate increasingly unhinged theological arguments.

This is already happening. R/accelerate broke away from r/singularity because the latter wasn’t zealous enough.

Re: AI 2040 and the cult of intelligence

#100

Post by Barrin92 »

drc500free wrote: post #32
Planktonne wrote: post #17

> I used to be one of these people. I read Yudkowsky and was like, OMG recursive self improvement hard takeoff AI is coming. I really think we need to stop giving credence to people who have 1) Been consistently wrong with all their predictions 2) Demonstrated an endless spiteful cynicism Some of these people are very talented in their fields, sure. But malevolent and incorrect should be disqualifying when they talk…

If you replace "AI" with "Adonai" in EY's framing, it reduces to biblical parables from his childhood about arguing with God to negotiate a new covenant. He's clearly a bright guy, but a lot of his work seems to be trying to reconcile Old Testament narrative patterns with atheism, and simply slotting an omnipotent AI in as a replacement.

Charles Stross has a very pithy version of this too, Singularity mania as 'duck-typed Evangelicalism'. It's really telling that if you go to China, Japan or even Europe there's virtually nothing of this. I don't think there's genuine atheists in America, as soon as they lose institutional religion they project the same kind of patterns onto tech or politics.

Brian Johnson is also an interesting case of this with his longevity / immortality obsession and his Mormon background who have this whole thing about genealogy and eternal families etc.

Post reply on HN