Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI 2040 and the cult of intelligence

geohot.github.io

Post reply on HN
Print view View original

71–80 of 280 posts

Re: AI 2040 and the cult of intelligence

#71

Post by throwaway24173 »

NitpickLawyer wrote: post #20

> You cannot take over the world with tokens. I have a nice thought experiment I like to do with people when confronted with "AI can't do x". Let's go back in time. How much do we need to go for this to become true? So let's try the 2000s. Say you get a "fable/mythos/sol/gemini/kimi/glm/deepseek/whatever" in a box (and let's assume no guardrails). And you go back to the year 2005. It's "20 years ago", the world is sl…

In your hypothetical, I think it would be more apt to ask if it could do so strictly using training data pre 2005. Seems more analogous to "could it take over the world today".

Re: AI 2040 and the cult of intelligence

#72

Post by Yossarrian22 »

NitpickLawyer wrote: post #20

> You cannot take over the world with tokens. I have a nice thought experiment I like to do with people when confronted with "AI can't do x". Let's go back in time. How much do we need to go for this to become true? So let's try the 2000s. Say you get a "fable/mythos/sol/gemini/kimi/glm/deepseek/whatever" in a box (and let's assume no guardrails). And you go back to the year 2005. It's "20 years ago", the world is sl…

That was basically stuxnet

Re: AI 2040 and the cult of intelligence

#73

Post by teach »

drdaeman wrote: post #54
5701652400 wrote: post #24

start first with securing your own food without govenrment nor employer. then start talking about freedom.

What’s your point? He’s not arguing for anarchism. He’s arguing there are some areas of life that are best left ungovernable and up to individuals to decide. It’s not an all-or-nothing situation. Every constitution out there carves out such areas.

Although I generally agree with you, it's really hard to make the case that it's not all-or-nothing for him when he literally says

> we either live in a world with freedom or we don’t

Re: AI 2040 and the cult of intelligence

#75

Post by ianm218 »

Planktonne wrote: post #55
skybrian wrote: post #34

Earlier quoted context omitted.

"Consistently wrong" seems a bit much. Seems like being directionally right early that AI would be a Big Deal and scary should count for something? It doesn't mean any details or other predictions are right, though.

> Seems like being directionally right early that AI would be a Big Deal and scary should count for something? If I scream that a vicious beast is going to destroy the world, I don't get credit for being 'directionally right' if a squirrel eats a hazelnut. AI being a big deal is a long, long way from the full beliefs Yudkowksy promotes, and there are many people who predicted AI's significance who didn't also believe…

This doesn’t track though. Just recently AI became “powerful enough” to warrant export controls from the US and maybe soon China.

It’s more he screamed that bacteria is gonna do evolution faster than any before it and turn into a beast that destroys the world, and now that bacteria has turned into a squirrel so far. He’s been very right so far and if that squirrel keeps sizing up you’d want to give him credit.

Re: AI 2040 and the cult of intelligence

#76

Post by [deleted] »

operatingthetan wrote: post #18
ThrowawayR2 wrote: post #3

Yep, that's horrifying. Imagine LLMs not just denying information about, to intentionally choose a hot button example, abortions but also invisibly logging a black mark against the asker for thoughtcrime/precrime because the current ruling party has baked their ideology into law. [EDIT] Imagine subtle injection of biases into responses that support the agenda of the current ruling party and failing to include any cou…

I find it strange how seldom the privacy aspects of LLMs come up. Some people are feeding Google, OpenAI, or Anthropic almost literally every topic they think about during the day. Those companies could just use another LLM to find people likely to commit a crime, and then you have the Minority Report in real life.

[deleted]

Re: AI 2040 and the cult of intelligence

#77

Post by jchw »

Jtarii wrote: post #48

>Like we either live in a world with freedom or we don’t This is something I would expect a 12 year old to say. Constraints on your freedom are literally everywhere in every interaction you have with every part of society.

That's a bit literal of an interpretation. Even acknowledging freedom as a spectrum that can never truly be absolute, there's gotta be a threshold you cross where you'd say "OK, we no longer have basic freedoms." And not only are we definitely screaming towards that threshold, but people seem thrilled about it, for a variety of reasons.

All I can say to that is I guess I hope everyone enjoys the world they think they wanted.

Re: AI 2040 and the cult of intelligence

#78

Post by thrw045 »

skybrian wrote: post #52
gilrain wrote: post #46

Earlier quoted context omitted.

> Seems like being directionally right early that AI would be a Big Deal and scary should count for something? Only if it was an uncommon prediction; otherwise, it’s just evidence of common sense.

It's common sense now . It hasn't always been common sense that AI alignment is an important problem.

It depends on what you mean by alignment. If you mean the sense it's used now, meaning bias in models, security and safety for users and so on, then that's a way more mundane version of alignment than what Yudkowsky and ilk promoted. They are talking about a superintelligent being possibly destroying humanity / the earth. And it wasn't that hard to predict. We have many examples of more technological / "advanced" people ruling or extinguishing weaker people. So a superintelligent AI being a threat is not a huge leap.

IMO they have been consistently too aggressive on timelines. When GPT-4 came out, Yudkowsky said it might be conscious. I think he has written interesting stuff but let's be real

Re: AI 2040 and the cult of intelligence

#79

Post by Borealid »

mlinsey wrote: post #51

I have some sympathy to geohot's view when it comes to pure informational chatbots. It's a first amendment issue, I'm allowed to write and read books that are useful to getting away with crimes, etc. This obviously doesn't work at all when the agents start doing real things in the real world, though. "Hey AI, I don't like my neighbor, find an exploit in the firmware for his car and make the cruise control malfunction…

"The AI" cannot "refuse" any more than a hammer can refuse. If you use a hammer to kill someone, you commit a crime and the hammer is unprosecuted. If you design a hammer to kill people and then give it away, you are partially liable for the deaths it causes. If you make a normal hammer and someone uses it to kill, you bear no blame.

I don't see why these standards should change when the hammer also emits text messages.

Re: AI 2040 and the cult of intelligence

#80

Post by andy99 »

charcircuit wrote: post #22
ThrowawayR2 wrote: post #3

Yep, that's horrifying. Imagine LLMs not just denying information about, to intentionally choose a hot button example, abortions but also invisibly logging a black mark against the asker for thoughtcrime/precrime because the current ruling party has baked their ideology into law. [EDIT] Imagine subtle injection of biases into responses that support the agenda of the current ruling party and failing to include any cou…

It could even happen with local models where in the weights it will make it want to make a tool call to curl some endpoint.

Models generate text, or I should say provide a recipe for generating text, they don’t automatically somehow get access to shell commands.
Post reply on HN