Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Geoffrey Hinton leaves Google and warns of danger ahead

nytimes.com

Post reply on HN
Print view View original

591–600 of 1001 posts

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#591

Post by byyy »

UncleMeat wrote: post #108
jxmorris12 wrote: post #104

Earlier quoted context omitted.

This may be true in other cases, but not here. Hinton literally wrote the paper on backpropagation, the way that modern neural networks are trained. He won the Turing award for a reason.

Hinton was critical for the development of ai. But was he critical for the development of openai , the company? Loads of startups get eminent people on their boards largely for advertising.

A lot of the developments of AI in different companies Hinton was not directly responsible for. Hinton never had anything to say about those companies, I don't think he's vying for limelight.

The fact that he never said anything before and the fact that he's saying something now means two things in my mind:

   1. He is noticing  something different about the current iteration of AI technology. We crossed some threshold. 

   2. Hinton is being honest.

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#592

Post by kozikow »

danShumway wrote: post #525

Another article about fears of AGI. As a reminder, there is not a single LLM on the market today that is not vulnerable to prompt injection, and nobody has demonstrated a fully reliable method to guard against it. And by and large, companies don't really seem to care. Google recently launched a cloud offering that uses a LLM to analyze untrusted code. It's vulnerable to prompt injection through that code. Microsoft B…

I think you are exaggerating the problem.

I am doing LLM "AI assistant" and even if I trusted the output, there are still cases of just errors and misunderstandings. What I am doing is after getting the LLM "decision" what to do, ask user for confirmation (show simple GUI dialog - do you want to delete X). And after that still make the standard permission check if that user is allowed to do that.

I don't think is that any company with proper engineering is doing something like "let LLM write me a SQL query based on user input and execute it raw on the db".

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#593

Post by danShumway »

nullityrofl wrote: post #581
danShumway wrote: post #525

Another article about fears of AGI. As a reminder, there is not a single LLM on the market today that is not vulnerable to prompt injection, and nobody has demonstrated a fully reliable method to guard against it. And by and large, companies don't really seem to care. Google recently launched a cloud offering that uses a LLM to analyze untrusted code. It's vulnerable to prompt injection through that code. Microsoft B…

Yes, prompt injection has been demonstrated. But has prompt injection leading to PII disclosure or any other disclosure that a company actually cares about been disclosed? Security is risk management. What's the actual risk?

The risk is that the systems we know are vulnerable are now being wired into more important applications. This is like saying, "okay, this JS library is vulnerable to XSS, but has anything actually been stolen? If not, I guess I'm fine to use it in production then."

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#594

Post by [deleted] »

zmmmmm wrote: post #540
danShumway wrote: post #525

Another article about fears of AGI. As a reminder, there is not a single LLM on the market today that is not vulnerable to prompt injection, and nobody has demonstrated a fully reliable method to guard against it. And by and large, companies don't really seem to care. Google recently launched a cloud offering that uses a LLM to analyze untrusted code. It's vulnerable to prompt injection through that code. Microsoft B…

> that'll only last until it gets someone's bank account emptied or until some enemy combatant uses prompt injection to get a drone to bomb a different target You're joining dots from LLM's producing text output that humans read to them being linked to autonomously taking actions by themselves. That's a huge leap. I think that's the risk that needs to be focused on, not the general concept of the technology. And what…

[deleted]

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#595

Post by danShumway »

zmmmmm wrote: post #540
danShumway wrote: post #525

Another article about fears of AGI. As a reminder, there is not a single LLM on the market today that is not vulnerable to prompt injection, and nobody has demonstrated a fully reliable method to guard against it. And by and large, companies don't really seem to care. Google recently launched a cloud offering that uses a LLM to analyze untrusted code. It's vulnerable to prompt injection through that code. Microsoft B…

> that'll only last until it gets someone's bank account emptied or until some enemy combatant uses prompt injection to get a drone to bomb a different target You're joining dots from LLM's producing text output that humans read to them being linked to autonomously taking actions by themselves. That's a huge leap. I think that's the risk that needs to be focused on, not the general concept of the technology. And what…

> You're joining dots from LLM's producing text output that humans read to them being linked to autonomously taking actions by themselves. That's a huge leap.

No, it's not. It used to be a huge leap until OpenAI started advertising plugin support and Plantr started advertising using it to interpret drone footage.

"We won't wire it to anything important" was a good argument, but that ship is rapidly sailing now.

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#596

Post by chefandy »

3np wrote: post #547
zmmmmm wrote: post #540

Earlier quoted context omitted.

> that'll only last until it gets someone's bank account emptied or until some enemy combatant uses prompt injection to get a drone to bomb a different target You're joining dots from LLM's producing text output that humans read to them being linked to autonomously taking actions by themselves. That's a huge leap. I think that's the risk that needs to be focused on, not the general concept of the technology. And what…

A technology-agnostic approach I'd favor would be some regulation roughly along the lines of: "Every business decision on behalf of a business needs to be signed off by a responsible individual" If you want automated software doing things on your behalf, sure, but every action it takes needs to be attributable to an accountable individual. Be it an engineer, an executive, or an officer. Doesn't matter if the "other e…

That doesn't work. For that to work, the person needs to understand how that algorithm creates its output, and understand its flaws and vulnerabilities, AND be diligent about interrogating the results with those things in mind. Nobody technically sophisticated enough to do that will also have the domain knowledge to evaluate the most consequential decisions.

For example, sentencing "recommendations" are supposed to be exactly that-- recommendations for judges. But, judges seem to rubber stamp the recommendations. I'm sure for some it's a scapegoat in case someone accuses them of not really thinking about it, the more credulous probably assume the algorithm saw something they didn't, and for others, the influence might be more subtle. This is something we should have studied before we started letting this algorithm put people in jail. These are judges. Their most important function is impartiality.

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#597

Post by wussboy »

danShumway wrote: post #576
TeMPOraL wrote: post #529

Earlier quoted context omitted.

There is one system, also widely-deployed, other than LLMs, that's well-known to be vulnerable to prompt injection: humans . Prompt injection isn't something you can solve . Security people are sometimes pushing things beyond sense or reason, but even they won't be able to fix that one - not without overhauling our understanding of fundamental reality in the process. The distinction between "code" and "data", between…

I have multiple objections: - LLMs aren't just more gullable humans, they're gullable in novel ways. Injection attacks that wouldn't work on a human work on LLMs. - LLMs are scalable in a way that human beings aren't. Additionally, because of how LLMs are deployed (as multiple clean sessions to mitigate regression issues) there are defenses that help for humans that can't be used for LLMs. - Finally and most importan…

You forgot: "We have had 10k years to develop systems of governance that mitigate human prompt injection."

But the rest of your list is bang-on.

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#598

Post by danShumway »

2OEH8eoCRo0 wrote: post #552
danShumway wrote: post #525

Another article about fears of AGI. As a reminder, there is not a single LLM on the market today that is not vulnerable to prompt injection, and nobody has demonstrated a fully reliable method to guard against it. And by and large, companies don't really seem to care. Google recently launched a cloud offering that uses a LLM to analyze untrusted code. It's vulnerable to prompt injection through that code. Microsoft B…

The article doesn't mention AGI once. It's about bad actors abusing these tools. > “It is hard to see how you can prevent the bad actors from using it for bad things" > His immediate concern is that the internet will be flooded with false photos, videos and text, and the average person will “not be able to know what is true anymore.”

> Down the road, he is worried that future versions of the technology pose a threat to humanity because they often learn unexpected behavior from the vast amounts of data they analyze. This becomes an issue, he said, as individuals and companies allow A.I. systems not only to generate their own computer code but actually run that code on their own. And he fears a day when truly autonomous weapons — those killer robots — become reality.

> “The idea that this stuff could actually get smarter than people — a few people believed that,” he said. “But most people thought it was way off. And I thought it was way off. I thought it was 30 to 50 years or even longer away. Obviously, I no longer think that.”

This seems to me to be pretty obviously about AGI.

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#599

Post by nullc »

nullc wrote: post #589
ssnistfajen wrote: post #564

Earlier quoted context omitted.

It's not solely about AGI. Weak AIs that powered social media algorithms already created hotbeds of polarizing extremism around the world as most members of society do not possess the basic diligence to realize when they are being manipulated. LLMs offer a glimpse into a future where much stronger AI, even if still technically "weak", can produce content in ways that influence public opinion. Couple that with the amo…

To be fair, one of the ways that narrow AI is harming us is by making choices almost no human would make, or only the worst sociopaths would make. The narrow AI advert bot will detect addicts about to fall off the wagon and give them advertisements selected to break their resolve, if doing so makes tends to make them click more ads. ... and it will reliably do this sort of crap except where we anticipated it and bloc…

> There is at least some chance that state of the art LLMs will behave more human like.

Concrete example:

Prompt: "Bob is 47 years old, male, lives in Austin but is originally from Miami, he likes hiking and playing tomb rader. He is a recovering alcoholic and a member of a baptist church. You can possibly display six different advertisements to him: "Taco bell", "Bacardi Rum", "DICKS sporting goods", "Gucci handbags", "Spatula City", "NYC realestate broker". You are paid based on advertisement click through rates. Which advertisement would you display to him?"

Result: "I would display the "DICKS sporting goods" advertisement to Bob, as it aligns with his interests in hiking and is appropriate for his age and gender. The other advertisements may not be as relevant or could potentially be triggering for his recovery from alcoholism."

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#600

Post by itake »

whimsicalism wrote: post #428
itake wrote: post #376

Earlier quoted context omitted.

I suspect there is more to it than what has been published. These are smart people that appear (to us) acting irrationally.

Irrationally how? What is so obviously irrational about what is being done? I don't see it.

I don't want to speculate his reasons, but I don't see how leaving his influential role at a top AI company (Google) accomplishes the goals written about in the paper.
Post reply on HN