Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Geoffrey Hinton leaves Google and warns of danger ahead

nytimes.com

Post reply on HN
Print view View original

671–680 of 1001 posts

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#671

Post by mesebrec »

zamnos wrote: post #665
munificent wrote: post #279

Earlier quoted context omitted.

Note that in that analogy, the caterpillar is dissolved during the process.

Fun fact: Scientists have determined that a moth or butterfly may not remember being a caterpillar, it can remember experiences it learned as a caterpillar. Hence I'm not sure dissolved is quite the right word to use because the nervous system stays with the creature during there process. The most accurate word is metamorphosis, since that's the word we gave for that process. The other detail is that at the end of th…

I think dissolved is the correct term since the caterpillar turns into goo, which turns into the butterfly. Given how the metamorphosis works, it's still an open question how it's able to retain memories. A brain floating in a bath of goo?

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#672

Post by throwaway2037 »

This article reads like Bill Joy's WIRED article "Why The Future Doesn't Need Us", published in year 2000.

Ref: https://en.wikipedia.org/wiki/Why_The_Future_Doesn%27t_Need_...

The New York Times and The Atlantic love publishing these long form, doom-and-gloom, click bait articles. They usually share the same message: "It's never been worse." I'm sure they are great for revenue generation (adverts, subscriptions, etc.).

Edit:

Just look at this quote:

    Dr. Hinton’s journey from A.I. groundbreaker to doomsayer marks a remarkable moment for the technology industry at perhaps its most important inflection point in decades.
"remarkable moment" and "perhaps its most important inflection point in decades". The overreach on the second phrase is an excellent example of absurdism. If I had a dollar for every time I see those phrases in these doom-and-gloom articles, I would be rich.

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#673

Post by biztos »

usgroup wrote: post #666

Could anyone frame -- in fairly plain words -- what would be the mechanism by which LLMs become generally "smarter than humans" in the "and humans can't control them" sense? Has there been some advance in self-learning or self-training? Is there some way to make them independent of human data and human curation of said data? And so on.

I’m not an AI expert but as I see it:

1. LLMs are already doing much more complex and useful things than most people thought possible even in the foreseeable future.

2. They are also showing emergent behaviors that their own creators can’t explain nor really control.

3. People and corporations and governments everywhere are trying whatever they can think of to accelerate this.

4. Therefore it makes sense to worry about newly powerful systems with scary emergent behaviors precisely because we do not know the mechanism.

Maybe it’s all an overreaction and ChatGPT 5 will be the end of the line, but I doubt it. There’s just too much disruption, profit, and havoc possible, humans will find a way to make it better/worse.

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#674

Post by willbudd »

efficientsticks wrote: post #485
hn_throwaway_99 wrote: post #153

Earlier quoted context omitted.

Trying to be diplomatic, but this is such an unnecessary snarky, useless response. Google obviously did go slow with their rollout of AI, to the point where most of the world criticized them to no end for "being caught flat footed" on AI (myself included, so mea culpa). I don't necessarily think they did it "right", and I think the way they set up their "Ethical AI" team was doomed to fail, but at least they did clea…

Google went slow not due to ethics but because running neural inference is a lot more expensive than serving SERP data from cache.

You honestly suggesting the inventors of the TPU bailed because they couldn't foot the compute bill?

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#675

Post by sholladay »

usgroup wrote: post #666

Could anyone frame -- in fairly plain words -- what would be the mechanism by which LLMs become generally "smarter than humans" in the "and humans can't control them" sense? Has there been some advance in self-learning or self-training? Is there some way to make them independent of human data and human curation of said data? And so on.

I am not convinced that an AI has to be smarter than humans for us to lose control of it. I would argue that it simply needs to be capable of meaningful actions without human input and it needs to be opaque, as in it operates as a black box.

Both of those characteristics apply to some degree to Auto-GPT, even though it does try to explain what it is doing. Surely ChaosGPT would omit the truth or lie about its actions. How do we know it didn’t mine some Bitcoin and self-replicate to the cloud already, unbeknownst to its own creator? That is well within its capabilities and it doesn’t need to be superhuman intelligent or self-aware to do so.

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#676

Post by jacquesm »

mlajtos wrote: post #64
archerx wrote: post #36

There is part of me that thinks that this A.I. fear-mongering is some kind of tactic by Google to get everybody to pause training their A.I.s so they can secretly catch up in the background. If I was to do some quick game theory in my mind this would be the result. Imagine being Google, leading the way in A.I. for years, create the frameworks (tensorflow), create custom hardware for A.I. (TPUs), fund a ton of researc…

You are partially right — OpenAI is way ahead of everybody else. Even though OpenAI team is thinking and doing everything for safe deployment of (baby) AGI, public and experts don’t think this should be effort lead by single company. So Google naturaly wants to be the counterweight. (Ironic that OpenAI was supposed to be counterweight, not vice versa.) However, when you want to catch up somebody, you cheat. And cheat…

> Even though OpenAI team is thinking and doing everything for safe deployment of (baby) AGI

That claim needs more proof than is available to me at the moment.

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#677

Post by mschuster91 »

danShumway wrote: post #525

Another article about fears of AGI. As a reminder, there is not a single LLM on the market today that is not vulnerable to prompt injection, and nobody has demonstrated a fully reliable method to guard against it. And by and large, companies don't really seem to care. Google recently launched a cloud offering that uses a LLM to analyze untrusted code. It's vulnerable to prompt injection through that code. Microsoft B…

> As a reminder, there is not a single LLM on the market today that is not vulnerable to prompt injection, and nobody has demonstrated a fully reliable method to guard against it. And by and large, companies don't really seem to care.

Why should they? What can one gain from knowing the prompt, other than maybe bypass safeguards and make it sound like Tay after 4chan had a whole day to play with it - but even that, only valid for the current session and not for any other user?

The real value in any AI service is the quality of the training data and the amount of compute time invested into training it, and the resulting weights can't be leaked.

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#678

Post by jacquesm »

janalsncm wrote: post #434
neel8986 wrote: post #151

Earlier quoted context omitted.

Really? Hinton dont need openAI to be relevant. He literally invented back propagation. He sticked to deep learning through 1990s and 2000s when almost all major scientist abandoned it. He was using neural networks for language model in 2007-08 when no one knew what it was. Again the deep learning in 2010s started when his students created AlexNet by coding deep learning in GPU. Chief Scientist of OpenAI Ilya Sutskev…

I’m not convinced that inventing back propagation gives one the authority to opine on more general technological/social trends. Frankly, many of the most important questions are difficult or impossible to know. In the case of neural networks, Hinton himself would never have become as famous were it not for one of those trends (the cost of GPU compute and the breakthrough of using GPUs for training) which was difficul…

To say Hinton is just lucky is short-changing both the work he did, the environment he did it in and utterly ignores the amount of pressure to abandon the work he was doing because it was considered to be a dead end by just about everybody else until it suddenly wasn't.

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#679

Post by TeMPOraL »

danShumway wrote: post #576
TeMPOraL wrote: post #529

Earlier quoted context omitted.

There is one system, also widely-deployed, other than LLMs, that's well-known to be vulnerable to prompt injection: humans . Prompt injection isn't something you can solve . Security people are sometimes pushing things beyond sense or reason, but even they won't be able to fix that one - not without overhauling our understanding of fundamental reality in the process. The distinction between "code" and "data", between…

I have multiple objections: - LLMs aren't just more gullable humans, they're gullable in novel ways. Injection attacks that wouldn't work on a human work on LLMs. - LLMs are scalable in a way that human beings aren't. Additionally, because of how LLMs are deployed (as multiple clean sessions to mitigate regression issues) there are defenses that help for humans that can't be used for LLMs. - Finally and most importan…

I generally agree with the observations behind your objections, however my point is slightly different:

> When GPT-7 or whatever comes along and it has comparable defenses to a human and it can be trained like a human to resist domain-specific attacks, then we can compare the security between the two. But that's not where we are, and articles like this give people the impression that prompt injection is less serious and harder to pull off than it actually is.

My point is that talking about "prompt injection" is bad framing from the start, because it makes people think that "prompt injection" is some vulnerability class that can be patched, case by case, until it no longer is present. It's not like "SQL injection", which is a result of doing dumb things like gluing strings together without minding for the code/data difference that actually exists in formal constructs like SQL and programming languages, and just needs to be respected. You can't fix "prompt injection" by prepared statements, or by generally not doing dumb things like working in plaintext-space with things that should be worked with in AST-space.

"Prompt injection" will always happen, because you can't fundamentally separate trusted from untrusted input for LLMs, any more than you can in humans - successful attack is always a matter of making the "prompt" complex and clever enough. So we can't talk in terms of "solving" "prompt injection" - the discussion needs to be about how to live with it, the way we've learned to live with each other, built systems that mitigate the inherent exploitability of every human.

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#680

Post by mise_en_place »

skepticATX wrote: post #229
drcode wrote: post #223

Earlier quoted context omitted.

I think GPT4 can converse on any subject at all as well as a (let's say) 80 IQ human. On some subjects it can converse much better. That feels fundamentally different than a calculator.

GPT-4 is absolutely more generally knowledgeable than any individual person. Individual humans can still easily beat it when it comes to knowledge of individual subjects. Let’s not conflate knowledge with intelligence though. GPT-4 simply isn’t intelligent.

Great take. But I think when autonomous agents become good enough, intelligence is certainly possible. Especially when those agents start to interact with the real world.
Post reply on HN