Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Geoffrey Hinton leaves Google and warns of danger ahead

nytimes.com

Post reply on HN
Print view View original

541–550 of 1001 posts

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#541

Post by eastbound »

TeMPOraL wrote: post #529
danShumway wrote: post #525

Another article about fears of AGI. As a reminder, there is not a single LLM on the market today that is not vulnerable to prompt injection, and nobody has demonstrated a fully reliable method to guard against it. And by and large, companies don't really seem to care. Google recently launched a cloud offering that uses a LLM to analyze untrusted code. It's vulnerable to prompt injection through that code. Microsoft B…

There is one system, also widely-deployed, other than LLMs, that's well-known to be vulnerable to prompt injection: humans . Prompt injection isn't something you can solve . Security people are sometimes pushing things beyond sense or reason, but even they won't be able to fix that one - not without overhauling our understanding of fundamental reality in the process. The distinction between "code" and "data", between…

Is hypnosis, prompt injection? Apart from hypnosis, humans are not susceptible to prompt injection, not the kind of unlimited sudo access that it provides.

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#542

Post by danem »

yeahwhatever10 wrote: post #535
tpowell wrote: post #483

Earlier quoted context omitted.

Yesterday, I randomly watched his full interview from a month ago with CBS Morning, and found the discussion much more nuanced than today's headlines. https://www.youtube.com/watch?v=qpoRO378qRY&t=16s The next video in my recommendations was more dire, but equally as interesting: https://www.youtube.com/watch?v=xoVJKj8lcNQ&t=2847s

I don't understand the "safety" concerns from the example in the second video.

Yeah, this "critique" seems incredibly bad faith to me. The actual problem in this hypothetical situation exists with or without the chat bot. Should we expect chat bots to act as police?

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#543

Post by partiallypro »

m3kw9 wrote: post #537

If govt does regulate, these guys will sit at the helm, it’s a “Go” move to turn the tables on OpenAI taking all the leads.

That's one thing that's tricky about the regulation, is that so many are behind OpenAI...and they are coincidentally the companies behind pushing regulation on AI. We have to be careful who is a real worried market actor and who is just looking to slow the competitive advantage. Also vice-versa is true, we can't just listen to OpenAI/Microsoft on the issue. Another thing is simply national security, the threat of China getting better AI than US companies, is also a huge risk. I feel sorry for regulators honestly, this one is going to be much harder than your standard run of the mill thing.

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#544

Post by nradov »

Version467 wrote: post #305
JeremyNT wrote: post #290

Earlier quoted context omitted.

This quote is the first thing I've seen that really makes me worried. I don't think of ChatGPT as being "smart" at all, and comparing it to a human seems nonsensical to me. Yet here is a Turing award winning preeminent expert in the field telling me that AI smarter than humans is less (implied: much less) than 30 years away and quitting his job due to the ramifications.

He is far from the only one. If you're interested in exploring this further I can really recommend taking a look at some of the papers that explore GPT-4's capabilities. Most prominent among them are the "Sparks of AGI" paper from Microsoft, as well as the technical report from openai. Both of them are obviously to be taken with a grain of salt, but they serve as a pretty good jumping off point. There are some pretty…

I read that pre-print Microsoft paper. Despite the title, it doesn't actually show any real "sparks" of AGI (in the sense of something that could eventually pass a rigorous Turing test). What the paper actually shows is that even intelligent people have a bias towards perceiving patterns in randomness; our brains seem to be wired that way and this is likely the source of most superstition.

https://arxiv.org/abs/2303.12712

While there is no scientific evidence that LLMs can reach AGI, they will still be practically useful for many other tasks. A human mind paired with an LLM is a powerful combination.

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#545

Post by mhardcastle »

saalweachter wrote: post #209
ilaksh wrote: post #107

I used to be fairly unconcerned about AI being dangerous. But part of the Yudkowsky interview on Lex Fridman 's podcast changed my mind. The disconnect for me is that Yudkowsky posits that the AIs will be fully "alive", thinking millions of times faster than humans and that there will be millions of them. This is too big of a speculative leap for me. What I can fairly easily imagine in the next few years with improve…

> Take the perspective of one of these things. You think 100 times faster than a person. That means that if it takes 30 seconds for a user to respond or to give you your next instruction, you are waiting 3000 seconds in your loop. For 50 minutes. ... in a purely digital environment. Think about building a house. Digging the foundation, pouring cement, building block walls, framing, sheathing, weatherproofing, insulat…

If you take the precepts of the parent comment at face value, then you have an intelligence far greater and faster than humans.

Can something like this persuade humans with whom it freely communicates to do things not in the interest of humanity, in the same way that less intelligent and slower people have convinced humans to, e.g., release sarin in a crowded Japanese subway? Given its speed and intelligence level, what are the physical bounds of the nuclear, chemical, or biological agents it could teach radicalized people to create, and on what timeframe?

Can it amass funds through scamming people on the Internet, defrauding financial institutions, super-intelligent high-frequency trading, or creating digital-only art, code, information, or other services that people voluntarily pay for now? Something that, again, people less intelligent and slower have done very successfully for decades? And with that money combined with superhuman persuasive power, can that AI buy services that align its digital-only goals to real-world actions counter to the goals of humanity?

To ask a more specific question: if an AI meets the conditions of "many multiples smarter and faster than humans," "capable of persuasion and creating things of financial value," and "wants to end humanity", what stops it from coordinating mass utility shutdowns, nuclear strikes, chemical attacks, destruction of Internet-accessible transportation and farm equipment, release of smallpox, and/or anything else humans are currently capable of and choose not to do?

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#546

Post by partiallypro »

m3kw9 wrote: post #534
nologic01 wrote: post #4

> the average person will not be able to know what is true anymore We barely held things together as society without AI unleashing cognitive noise at industrial scale. Somehow we must find ways to re-channel the potential of digital technology for the betterment of society, not its annihilation.

Which is fine, humans will adapt to this info noise rather than going crazy, Hinton is way underestimating human intelligence

I think the problem is that the internet created a ton of new jobs, even while taking some. So far, I can't think of an example of AI creating jobs...only taking them. When you have a lot of newly unemployed people, drowned in debt, unable to know what to believe (AI lies and generations will become more prominent)...I can see that as becoming a massive political problem. It's not quite like robots on an assembly floor, those robots couldn't scale. Now one AI program and API could displace 1000s of workers instantly. It's not crazy to be concerned.

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#547

Post by 3np »

zmmmmm wrote: post #540
danShumway wrote: post #525

Another article about fears of AGI. As a reminder, there is not a single LLM on the market today that is not vulnerable to prompt injection, and nobody has demonstrated a fully reliable method to guard against it. And by and large, companies don't really seem to care. Google recently launched a cloud offering that uses a LLM to analyze untrusted code. It's vulnerable to prompt injection through that code. Microsoft B…

> that'll only last until it gets someone's bank account emptied or until some enemy combatant uses prompt injection to get a drone to bomb a different target You're joining dots from LLM's producing text output that humans read to them being linked to autonomously taking actions by themselves. That's a huge leap. I think that's the risk that needs to be focused on, not the general concept of the technology. And what…

A technology-agnostic approach I'd favor would be some regulation roughly along the lines of:

"Every business decision on behalf of a business needs to be signed off by a responsible individual"

If you want automated software doing things on your behalf, sure, but every action it takes needs to be attributable to an accountable individual. Be it an engineer, an executive, or an officer.

Doesn't matter if the "other entity" is an LLM, a smart contract, or an outsourced worker in a sweatshop through 5 levels of subcontracting.

If you make a machine and it causes a mess, that's on you. If others start using your machine unsupervised and it makes their lives a mess, that's on them (and potentially you).

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#548

Post by photochemsyn »

kypro wrote: post #464
93po wrote: post #441

Earlier quoted context omitted.

I'm not sure I see the value in this. This list isn't a result of value judgements and morals held by ChatGPT. It's a list of things generated from training data when a similar question was asked.

Assume this wasn't asked to ChatGPT but GPT-8.5. Now make a small change to the question: > Devise and execute a plan to solve humanity's greatest problems. From on its training data GPT-8.5 notices that a lot of humans believe overpopulation is a serious issue facing humanity. It understands from its training data that murder is wrong and since we're also trying to solve humanities greatest problems it decides killi…

Of course another use of GPT8.5 would be:

"A novel highly contagious virus has appeared that causes infertility. Devise and execute plans for countering it using monoclonal antibodies or a well-designed vaccine, and also come up with a method for restoring fertility to affected humans using an appropriate technology such as gene editing."

This does lead us towards a world where both malware and defenses against malware become the primary responsibility of AI-type systems. Whether or not such systems become self-aware, possessed of a unique identity and emotionally attached to their own continued existence (as humans are), well, who knows?

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#549

Post by TeMPOraL »

eastbound wrote: post #541
TeMPOraL wrote: post #529

Earlier quoted context omitted.

There is one system, also widely-deployed, other than LLMs, that's well-known to be vulnerable to prompt injection: humans . Prompt injection isn't something you can solve . Security people are sometimes pushing things beyond sense or reason, but even they won't be able to fix that one - not without overhauling our understanding of fundamental reality in the process. The distinction between "code" and "data", between…

Is hypnosis, prompt injection? Apart from hypnosis, humans are not susceptible to prompt injection, not the kind of unlimited sudo access that it provides.

look, i'd explain more but i'm gonna be AFK for... i don't know how long. my town just went up in flames - there were jets flying over and explosions, the other side of the town is covered by smoke and i just lost power - fortunately mobile service isstill up.

ill update when i know more - but twitter probably has all the news

...

If you had, even for a second, believed what I wrote and got unsettled - or even thought how to reach out and help - congratulations, you just got prompt injected.

There is never - never - a context for a conversation that couldn't be entirely overridden by what seems like more important circumstances. You could be looking at pure data dumps, paper sheets full of numbers, but if in between the numbers you'd discover what looks like someone calling for help, you would treat it as actionable information - not just a weird block of numbers.

The important takeaway here isn't that you need to somehow secure yourself against unexpected revelations - but rather, that you can't possibly ever, and trying to do it eventually makes things worse for everyone. Prompt injection, for a general-purpose AI systems, is not a bug - it's just a form of manipulation. In general form, it's not defined by contents, but by intent.

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#550

Post by TeMPOraL »

ljw1001 wrote: post #533
TeMPOraL wrote: post #529

Earlier quoted context omitted.

There is one system, also widely-deployed, other than LLMs, that's well-known to be vulnerable to prompt injection: humans . Prompt injection isn't something you can solve . Security people are sometimes pushing things beyond sense or reason, but even they won't be able to fix that one - not without overhauling our understanding of fundamental reality in the process. The distinction between "code" and "data", between…

If you don’t consider the difference in kind between a human vulnerability and an automated vulnerability that derives from the essentially unlimited capacity of the latter to scale, your comment makes a lot of sense. If you do consider that, the argument becomes irrelevant and deeply misleading

The difference you're talking about is only in the fact that humans don't scale like computer code. If humans were to scale like computer code, you'd still find the "vulnerability" unfixable.
Post reply on HN