Geoffrey Hinton leaves Google and warns of danger ahead • Hacker News Board

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#721

Post by ballenf » Tue, May 02, 2023, 12:54 PM UTC

kromem wrote: post #719

danShumway wrote: post #525

Another article about fears of AGI. As a reminder, there is not a single LLM on the market today that is not vulnerable to prompt injection, and nobody has demonstrated a fully reliable method to guard against it. And by and large, companies don't really seem to care. Google recently launched a cloud offering that uses a LLM to analyze untrusted code. It's vulnerable to prompt injection through that code. Microsoft B…

It's trivial to fix prompt injection. You simply add another 'reviewer' layer that does a clarification task on the input and response to detect it. The problem is this over doubles the cost of implementation to prevent something no one actually cares about fixing for a chatbot. "Oh no, the user got it to say silly things to themselves." This isn't impacting other people's experiences or any critical infrastructure.…

What makes that layer immune from a meta-injection (an injection crafted for that layer)?

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#722

Post by FarMcKon » Tue, May 02, 2023, 1:02 PM UTC

whynotmaybe wrote: post #702

mise_en_place wrote: post #694

It's important to have a discussion about AI safety, and the ethics surrounding LLMs. But I'm really tired of all this sensationalism. It completely muddies the waters; it almost seems intentional at this point.

Did we have some ethics discussion when the lightbulb was invented? Or when the car was invented? No, and if we did, we couldn't have foreshadowed all the positive and negative impacts. My grandfather always told me that back in the days, "smart" people said that no train should be allowed to go faster than 50km/h because the heart would explode. Nobody here can say that he wasn't impressed by ChatGPT. How could we e…

> Did we have some ethics discussion when the lightbulb was invented? Or when the car was invented?

Yes, yes we did. We debated photos, voice recording, radio waves, TV shows. Every innovation spawns years of Ethics discussion.

Weird to see someone on this site, presumably in tech, who hasn't been taught or learned that yet.

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#723

Post by williamtrask » Tue, May 02, 2023, 1:02 PM UTC

kromem wrote: post #715

mise_en_place wrote: post #694

It's important to have a discussion about AI safety, and the ethics surrounding LLMs. But I'm really tired of all this sensationalism. It completely muddies the waters; it almost seems intentional at this point.

It is. And it gets clicks. But what I rarely see discussed is the opportunity costs in not having this progress at as fast a pace as possible. The pie chart of existential threats for humanity definitely has rouge AI on it. But that's a slice amidst many human driven threats ranging from nuclear war to oceans dying. What there's not very many human driven slices of pie for is realistic solutions to these issues. On t…

Fwiw most of the academic labs working on AI safety also work on these threats. Future of Humanity Institute, Center for the Study of Existential Risk, etc.

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#724

Post by mensetmanusman » Tue, May 02, 2023, 1:05 PM UTC

mise_en_place wrote: post #694

It's important to have a discussion about AI safety, and the ethics surrounding LLMs. But I'm really tired of all this sensationalism. It completely muddies the waters; it almost seems intentional at this point.

AI will obviously kill many aspects of the internet. There will be no test to distinguish human versus AI to post BS content at a pace 10^9 times faster than humans can filter it out.

One bad actor and our existing communication network dies.

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#725

Post by ipython » Tue, May 02, 2023, 1:10 PM UTC

whynotmaybe wrote: post #702

mise_en_place wrote: post #694

It's important to have a discussion about AI safety, and the ethics surrounding LLMs. But I'm really tired of all this sensationalism. It completely muddies the waters; it almost seems intentional at this point.

Did we have some ethics discussion when the lightbulb was invented? Or when the car was invented? No, and if we did, we couldn't have foreshadowed all the positive and negative impacts. My grandfather always told me that back in the days, "smart" people said that no train should be allowed to go faster than 50km/h because the heart would explode. Nobody here can say that he wasn't impressed by ChatGPT. How could we e…

Heck. Plato and Socrates questioned the value of the written word, arguing that it would impair the shared memory of events and the only way to impart knowledge is by speaking face to face.

https://fs.blog/an-old-argument-against-writing/

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#726

Post by Retric » Tue, May 02, 2023, 1:11 PM UTC

mensetmanusman wrote: post #724

mise_en_place wrote: post #694

It's important to have a discussion about AI safety, and the ethics surrounding LLMs. But I'm really tired of all this sensationalism. It completely muddies the waters; it almost seems intentional at this point.

AI will obviously kill many aspects of the internet. There will be no test to distinguish human versus AI to post BS content at a pace 10^9 times faster than humans can filter it out. One bad actor and our existing communication network dies.

Physically showing up somewhere is a test for people vs AI.

Longer term there’s an opportunity for anonymity services to verify someone is a unique human without giving out which specific person they are.

Worst case there’s falling back on real world ID’s.

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#727

Post by __MatrixMan__ » Tue, May 02, 2023, 1:12 PM UTC

mensetmanusman wrote: post #724

mise_en_place wrote: post #694

It's important to have a discussion about AI safety, and the ethics surrounding LLMs. But I'm really tired of all this sensationalism. It completely muddies the waters; it almost seems intentional at this point.

AI will obviously kill many aspects of the internet. There will be no test to distinguish human versus AI to post BS content at a pace 10^9 times faster than humans can filter it out. One bad actor and our existing communication network dies.

It's the Web that's at risk, the Internet will be fine.

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#728

Post by myrmidon » Tue, May 02, 2023, 1:22 PM UTC

kromem wrote: post #715

mise_en_place wrote: post #694

It's important to have a discussion about AI safety, and the ethics surrounding LLMs. But I'm really tired of all this sensationalism. It completely muddies the waters; it almost seems intentional at this point.

It is. And it gets clicks. But what I rarely see discussed is the opportunity costs in not having this progress at as fast a pace as possible. The pie chart of existential threats for humanity definitely has rouge AI on it. But that's a slice amidst many human driven threats ranging from nuclear war to oceans dying. What there's not very many human driven slices of pie for is realistic solutions to these issues. On t…

Do you honestly believe that climate change, environmental pollution or nuclear war could eradicate our species?

Because I strongly disagree on that-- civilization collapse I'm willing to believe, but I don't really see how any of these could really eradicate our whole species. That seems unreasonably alarmist to me, but I'm quite open to change my view if you want to back up your outlook.

I honestly believe that the risk from AI to our species over the next century or so is MUCH higher than basically ANYTHING else, by orders of magnitude.

But the only other "viable" ways to wipe humans to me appear:

- Big asteroid impact (historically unlikely)

- Completely runaway greenhouse effect (extremely unlikely according to current science)

- Specifically designed pathogen (big maybe?)

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#729

Post by neel8986 » Tue, May 02, 2023, 1:24 PM UTC

ecocentrik wrote: post #146

archerx wrote: post #36

There is part of me that thinks that this A.I. fear-mongering is some kind of tactic by Google to get everybody to pause training their A.I.s so they can secretly catch up in the background. If I was to do some quick game theory in my mind this would be the result. Imagine being Google, leading the way in A.I. for years, create the frameworks (tensorflow), create custom hardware for A.I. (TPUs), fund a ton of researc…

"You release Bard and it is an embarrassing disaster, a critical fail leading to an almost 50% reduction of Google's stock price" This didn't happen so maybe you need to reexamine your entire premise.

LOL Google stock price is more that what it was before ChatGPT's release. Search engine market share hasn't changed by a even 1% neither did profit from search. Every day HN's hyperbole is increasing.

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#730

Post by TeMPOraL » Tue, May 02, 2023, 1:26 PM UTC

killerstorm wrote: post #686

TeMPOraL wrote: post #529

Earlier quoted context omitted.

There is one system, also widely-deployed, other than LLMs, that's well-known to be vulnerable to prompt injection: humans . Prompt injection isn't something you can solve . Security people are sometimes pushing things beyond sense or reason, but even they won't be able to fix that one - not without overhauling our understanding of fundamental reality in the process. The distinction between "code" and "data", between…

> Prompt injection isn't something you can solve. Eh? People only tried few half-assed techniques for less than a year, and you're saying we are out of ideas now? Prompt injections are a thing because the bulk of training happens in a self-supervised fashion and there's no separation between "control" and "data" planes there. There is no law of nature saying that you cannot obtain better quality data. Note that the i…

> Eh? People only tried few half-assed techniques for less than a year, and you're saying we are out of ideas now?

I'm saying it because it's a fundamental limitation. It's not about lack of training data - it's that, from the POV of a LLM, "system" input, user input, and their own output reflected back at them, are indistinguishable. They all get mixed together and pushed through a single channel.

Sure, you can add funny prefixes, like "System prompt", or play with things like ChatML, but the LLM is literally unable to tell the difference between that, and a "user prompt" that contains the literal words "System prompt" in it, or "system\n". No matter how hard you pre-prompt the system to ignore user-provided instructions, the user can override it by prompting the model harder. Or trick it into self-prompting through its own output. Or both.

Inside a transformer model, there is only one runtime. There is no one eval() for owner-provided code, and another one in a sandbox for user-provided code. There is only one eval(), and one stream of tokens, and all tokens are created equal. At this level, there is no such thing as "system data", "assistant data", "user data". There is only a stream of tokens that slice off areas in the latent space.

There isn't a way to fix it while retaining the general-purpose architecture. And there's definitely no way of fixing it from inside - no amount of good training data can cover for the fact that user input and system input are indistinguishable as a category.

(And no, doing silly things like setting the "evil bit" on every token coming from the user won't do anything other than double the amount of tokens your model needs to distinguish, while diminishing its capacity. It definitely won't prevent users being able to work around the "evil bit". This should be self-evident, but I can try and explain it if it isn't.)