Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Geoffrey Hinton leaves Google and warns of danger ahead

nytimes.com

Post reply on HN
Print view View original

551–560 of 1001 posts

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#551

Post by sberens »

danShumway wrote: post #525

Another article about fears of AGI. As a reminder, there is not a single LLM on the market today that is not vulnerable to prompt injection, and nobody has demonstrated a fully reliable method to guard against it. And by and large, companies don't really seem to care. Google recently launched a cloud offering that uses a LLM to analyze untrusted code. It's vulnerable to prompt injection through that code. Microsoft B…

As a reminder, the people worried about AGI are not worried about GPT-4.

They see the writing on the wall for what AI will be capable of in 5-10 years, and are worried about the dangers that will arise from those capabilities, not the current capabilities.

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#552

Post by 2OEH8eoCRo0 »

danShumway wrote: post #525

Another article about fears of AGI. As a reminder, there is not a single LLM on the market today that is not vulnerable to prompt injection, and nobody has demonstrated a fully reliable method to guard against it. And by and large, companies don't really seem to care. Google recently launched a cloud offering that uses a LLM to analyze untrusted code. It's vulnerable to prompt injection through that code. Microsoft B…

The article doesn't mention AGI once. It's about bad actors abusing these tools.

> “It is hard to see how you can prevent the bad actors from using it for bad things"

> His immediate concern is that the internet will be flooded with false photos, videos and text, and the average person will “not be able to know what is true anymore.”

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#553

Post by mhh__ »

hn_throwaway_99 wrote: post #241
archerx wrote: post #36

There is part of me that thinks that this A.I. fear-mongering is some kind of tactic by Google to get everybody to pause training their A.I.s so they can secretly catch up in the background. If I was to do some quick game theory in my mind this would be the result. Imagine being Google, leading the way in A.I. for years, create the frameworks (tensorflow), create custom hardware for A.I. (TPUs), fund a ton of researc…

I think a comment on the reddit thread about this is somewhat appropriate, though I don't mean the imply the same harshness: > Godfather of AI - I have concerns. > Reddit - This old guy doesn't know shit. Here's my opinion that will be upvoted by nitwits. Point being, if you're saying that the guy who literally wrote the paper on back propagation is "fear mongering", but who is now questioning the value of his life's…

There's a big jump from backprop to what we have now, Hinton mainly does the AI equivalent of fundamental physics not applications.

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#554

Post by pclmulqdq »

ModernMech wrote: post #296
Lutger wrote: post #52

Earlier quoted context omitted.

Between Social Media, Cambridge Analytica, the Climate Crisis, Pandemic and (mostly) Russian disinfo, etc, it is already the case that most people have a really hard time knowing what is true. I don't claim to have much foresight, but an online world where truly and obviously nothing can be trusted might be a good thing. Because when AI generated content looks and feels the same as real content, nothing is to be trus…

The problem is, when no one trusts anything, it makes room for men who promise everything, but can deliver nothing. We call them "dictators" and "authoritarians", but others call them "strong men" because they are envied by those who seek power. If you look around the world, you can see authoritarian movements rising, especially here in the USA.

This attitude is what actually makes room for the authoritarians. Our democratic systems today are built with a lot of self-healing mechanisms against this exact kind of authoritarianism. The desire to circumvent those mechanisms because "it's different this time, I swear" is what makes room for dictators and authoritarians. This happens all the time in third-world countries that try to set up democracies: the dictator comes in after someone starts tweaking with the rules in the name of "safety." Society has been through several paradigm shifts that have accelerated the spread of misinformation, and survived them.

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#555

Post by burnished »

danem wrote: post #542
yeahwhatever10 wrote: post #535

Earlier quoted context omitted.

I don't understand the "safety" concerns from the example in the second video.

Yeah, this "critique" seems incredibly bad faith to me. The actual problem in this hypothetical situation exists with or without the chat bot. Should we expect chat bots to act as police?

Given his pedigree accusing him of bad faith seems missplaced.

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#556

Post by adamisom »

janalsncm wrote: post #434
neel8986 wrote: post #151

Earlier quoted context omitted.

Really? Hinton dont need openAI to be relevant. He literally invented back propagation. He sticked to deep learning through 1990s and 2000s when almost all major scientist abandoned it. He was using neural networks for language model in 2007-08 when no one knew what it was. Again the deep learning in 2010s started when his students created AlexNet by coding deep learning in GPU. Chief Scientist of OpenAI Ilya Sutskev…

I’m not convinced that inventing back propagation gives one the authority to opine on more general technological/social trends. Frankly, many of the most important questions are difficult or impossible to know. In the case of neural networks, Hinton himself would never have become as famous were it not for one of those trends (the cost of GPU compute and the breakthrough of using GPUs for training) which was difficul…

Of course he was lucky, you should expect that in general for well-known people because selection pressures that led you to hear of them, vs not hear of them, are likely to involve luck.

That is not at all a slam dunk argument. It’s barely anything.

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#557

Post by zmmmmm »

3np wrote: post #547
zmmmmm wrote: post #540

Earlier quoted context omitted.

> that'll only last until it gets someone's bank account emptied or until some enemy combatant uses prompt injection to get a drone to bomb a different target You're joining dots from LLM's producing text output that humans read to them being linked to autonomously taking actions by themselves. That's a huge leap. I think that's the risk that needs to be focused on, not the general concept of the technology. And what…

A technology-agnostic approach I'd favor would be some regulation roughly along the lines of: "Every business decision on behalf of a business needs to be signed off by a responsible individual" If you want automated software doing things on your behalf, sure, but every action it takes needs to be attributable to an accountable individual. Be it an engineer, an executive, or an officer. Doesn't matter if the "other e…

The key is that it has to be resistant to subversion by hostile terms of use. Similar to how certain provisions in employment contracts aren't enforceable under employment law. Because literally anything you try and establish here will instantly end up as a waiver in terms of use for services that regular people can't possibly understand or reasonably opt out of.

(as example, see recent law suit that Tesla won because the driver used auto-pilot on "city streets" where it was advised not to somewhere deep in the terms of use).

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#558

Post by fatherzine »

saulpw wrote: post #491
fatherzine wrote: post #368

Earlier quoted context omitted.

"When it comes to learning and adapting to new environments, while we are lucky AI's aren't yet as smart as a mouse, they are uncomfortably close, and the pace of progress is unnerving. Hinton is right, we've got too far and we should grind all AI research to a halt via heavy global regulation." What is the goal here? Creation of an all powerful God? Self-destruction as a species? I'm not up-to-date with the exact st…

We need a messiah. Humanity has huge problems that we are not addressing (top of the list being climate change), largely because it would require massive scale societal disruption to do so. Over the past 50 years, we've thought that personal computers would help (bicycles for the mind), then we thought the internet would help (organizing the world's information), then we thought social networks would help (connecting…

Interesting. Brief musing. Our collective objective function appears to be a post-scarcity economy. Alas, we physically inhabit a finite world, in which post-scarcity can never be attained -- the exponential curve ruins every single attempt. Another option is to seek peace / shalom / spiritual homeostasis, even when faced with the certainty of decay and death. Quest which perhaps does require a Messiah.

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#559

Post by uptownfunk »

No one's leaving Goog to warn of dangers of AI. It does indeed sound like he was pushed out. The company is trying to consolidate the AI orgs and get into execution mode and perhaps whatever role Hinton was offered was not suitable to his taste. I'm sure he'll land somewhere amazing if he even wants or needs to continue working. I am curious to see what direction Google takes to combat the ever growing threats across the ecosystem. They still have search, email, and web-browsing. The real challenge here is how quickly can they mobilize, and how much can they work through regulation to use the above moats to produce the next sticky product that will maintain and grow their dominance in the market.

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#560

Post by ngneer »

TeMPOraL wrote: post #529
danShumway wrote: post #525

Another article about fears of AGI. As a reminder, there is not a single LLM on the market today that is not vulnerable to prompt injection, and nobody has demonstrated a fully reliable method to guard against it. And by and large, companies don't really seem to care. Google recently launched a cloud offering that uses a LLM to analyze untrusted code. It's vulnerable to prompt injection through that code. Microsoft B…

There is one system, also widely-deployed, other than LLMs, that's well-known to be vulnerable to prompt injection: humans . Prompt injection isn't something you can solve . Security people are sometimes pushing things beyond sense or reason, but even they won't be able to fix that one - not without overhauling our understanding of fundamental reality in the process. The distinction between "code" and "data", between…

I fail to see what the distinction between control and data planes (or lack thereof) has to do with anything. The security question is about who gets to control system behavior, be it through the control or data planes or both. With prompt injection, the answer is the input provider gets to control the behavior. This is obviously different than intended by the system designer and thus not secure. However, there is nothing fundamental that prevents one from building an algorithm or recursively enumerable function whose inputs cannot induce certain outputs. It is just that one has to be very intentional, so it hardly ever happens.
Post reply on HN