Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Geoffrey Hinton leaves Google and warns of danger ahead

nytimes.com

Post reply on HN
Print view View original

561–570 of 1001 posts

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#561

Post by crote »

TeMPOraL wrote: post #529
danShumway wrote: post #525

Another article about fears of AGI. As a reminder, there is not a single LLM on the market today that is not vulnerable to prompt injection, and nobody has demonstrated a fully reliable method to guard against it. And by and large, companies don't really seem to care. Google recently launched a cloud offering that uses a LLM to analyze untrusted code. It's vulnerable to prompt injection through that code. Microsoft B…

There is one system, also widely-deployed, other than LLMs, that's well-known to be vulnerable to prompt injection: humans . Prompt injection isn't something you can solve . Security people are sometimes pushing things beyond sense or reason, but even they won't be able to fix that one - not without overhauling our understanding of fundamental reality in the process. The distinction between "code" and "data", between…

The distinction between code and data is very real, and dates back to at least the original Harvard Architecture machine in 1944. Things like W^X and stack canaries have been around for decades too.

LLMs are trying to essentially undo this by concatenating code and user-provided data and executing it as one. From a security perspective it is just a plainly stupid idea, but I do not believe it is impossible to construct a similar system where those two are separate.

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#562

Post by anon-3988 »

danShumway wrote: post #525

Another article about fears of AGI. As a reminder, there is not a single LLM on the market today that is not vulnerable to prompt injection, and nobody has demonstrated a fully reliable method to guard against it. And by and large, companies don't really seem to care. Google recently launched a cloud offering that uses a LLM to analyze untrusted code. It's vulnerable to prompt injection through that code. Microsoft B…

Obviously you haven't read or skimmed the article because the article makes no mention of AGI. However, it is very bland and predictable. I am not sure why any technologists or pioneer would think their technology wouldn't be used for bad. You can probably replace any mention of AI, ML or NN in the article with any other invention in the past 1 billion years and it will still make sense.

What technology/inventions out there that _can't_ and _isn't_ used for bad? AGI is a red herring. Even if AGI is possible, we will soon destroy ourselves through simpler means and those are much more important concerns. It is much sexier to be talking about AGI whichever side you are on. But who wants to talk about solving the issues of the downtrodden?

> I guess the silver lining is that if you're genuinely losing sleep about GPT-4 becoming a general agent that does every job, don't worry -- that'll only last until it gets someone's bank account emptied or until some enemy combatant uses prompt injection to get a drone to bomb a different target. Unless this security problem gets solved, but none of the companies seem to care that much about security or view it as a blocker for launching whatever new product they have to try and drive up stock price or grab VC funding. So I'm not really holding my breath on that.

Have any technologists ever considered what making the lower bound of necessary intelligence higher? Have anyone in SV ever talked or known someone that can't do elementary math? And how common this is? All technological advancement have a long term cost to society. You are making the assumption that the human part is going to be completely removed. This is not true. Of course there will still be human somewhere in the mix. But there will be significantly less. Automating the whole customer service industry wouldn't make every shop void of any human. There will be only a single human, managing all the machines and spend their days looking at gigabytes of generated logs from 9 to 5. Is this a way to live? Yes, for some. But everyone?

Just think about the consequence of having all manual labor jobs getting replaced. Which is probably conceivable in the next 30 years at least. What do you think will happen to these people? Do you think they became manual labor because they wanted to or have to? Now that they can't join the manual labor force, what now? Turn their career around to looking at spreadsheets everyday? Do you seriously think everyone is capable of that? HN folks are probably on the right end of the distribution but refuses to consider the existence of the people at left end of the distribution or even the center.

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#563

Post by theptip »

danShumway wrote: post #525

Another article about fears of AGI. As a reminder, there is not a single LLM on the market today that is not vulnerable to prompt injection, and nobody has demonstrated a fully reliable method to guard against it. And by and large, companies don't really seem to care. Google recently launched a cloud offering that uses a LLM to analyze untrusted code. It's vulnerable to prompt injection through that code. Microsoft B…

I find this position hard to grok. You’re complaining about people worrying about AGI because you view the short-run implications of this tech to be quite bad. To me, a lack of prompt security in the short term bodes poorly for our safety in N generations when these systems are actually powerful. Like, sure, someone is gonna get swatted by an AI in the next year or two, and that sucks, but that is a tiny speck of dust compared to the potential disutility of unaligned powerful AI systems.

Is it that you just think P(AGI) is really low, so worrying about an unlikely future outcome bothers you when there is actual harm now?

> that'll only last until it gets someone's bank account emptied or until some enemy combatant uses prompt injection to get a drone to bomb a different target

If that’s all it would take to prevent AGI I’m sure folks would not be scared. I don’t see why these things would prevent companies/countries from chasing a potential multi-trillion (quintillion?) dollar technology though.

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#564

Post by ssnistfajen »

danShumway wrote: post #525

Another article about fears of AGI. As a reminder, there is not a single LLM on the market today that is not vulnerable to prompt injection, and nobody has demonstrated a fully reliable method to guard against it. And by and large, companies don't really seem to care. Google recently launched a cloud offering that uses a LLM to analyze untrusted code. It's vulnerable to prompt injection through that code. Microsoft B…

It's not solely about AGI. Weak AIs that powered social media algorithms already created hotbeds of polarizing extremism around the world as most members of society do not possess the basic diligence to realize when they are being manipulated. LLMs offer a glimpse into a future where much stronger AI, even if still technically "weak", can produce content in ways that influence public opinion. Couple that with the amount of white collar work eliminated/reduced through LLMs, and it's a recipe for mass social disruption that inevitably leads to unrest unless public policy decision makers act fast. The problem is there is no clear path. Not even the smartest and most rational ones know where this road is going.

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#565

Post by kovac »

maxdoop wrote: post #400
JeremyNT wrote: post #290

Earlier quoted context omitted.

This quote is the first thing I've seen that really makes me worried. I don't think of ChatGPT as being "smart" at all, and comparing it to a human seems nonsensical to me. Yet here is a Turing award winning preeminent expert in the field telling me that AI smarter than humans is less (implied: much less) than 30 years away and quitting his job due to the ramifications.

Every single retort of “these machines aren’t smart or intelligent” requires answering the question, “what is intelligence”? I struggle to see how GPT-4 is not intelligent by any definition that applies to a human.

To me real intelligence is the ability to reason coherently. Humans, even when they are wrong, work in a way coherent with their prinicples/beliefs/axioms. Consider Sheldon from Big Bang Theory who does a very convincing job as a theoretical physicist, at least to the untrained ear, merely by memorising lines. However, as soon as he is questioned on something he didn't memorise, the act falls apart in a way a real Physicist wouldn't even in a domain he doesn't specialise in. For a trained ear, though, even during the act, the inconsistencies are audible.

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#566

Post by randomguy3344 »

mimd wrote: post #450

[flagged]

So because one of your family members has a better quality of life and can live closer to a normal person the rest of us shouldn't worry at all about AI? And anyone who talks against it is a cunt then? What an intelligent argument lmao.

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#567

Post by m3kw9 »

partiallypro wrote: post #543
m3kw9 wrote: post #537

If govt does regulate, these guys will sit at the helm, it’s a “Go” move to turn the tables on OpenAI taking all the leads.

That's one thing that's tricky about the regulation, is that so many are behind OpenAI...and they are coincidentally the companies behind pushing regulation on AI. We have to be careful who is a real worried market actor and who is just looking to slow the competitive advantage. Also vice-versa is true, we can't just listen to OpenAI/Microsoft on the issue. Another thing is simply national security, the threat of Chi…

Usually regulation becomes a moat, I have doubts the type of regulation OpenAI wants to implement isn’t just regulating but make it very costly for compliance as well

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#568

Post by ssnistfajen »

JeremyNT wrote: post #290
neatze wrote: post #99

“The idea that this stuff could actually get smarter than people — a few people believed that,” said Hinton to the NYT. “But most people thought it was way off. And I thought it was way off. I thought it was 30 to 50 years or even longer away. Obviously, I no longer think that.” Calculators are smarter then humans in calculating, what does he mean by that?

This quote is the first thing I've seen that really makes me worried. I don't think of ChatGPT as being "smart" at all, and comparing it to a human seems nonsensical to me. Yet here is a Turing award winning preeminent expert in the field telling me that AI smarter than humans is less (implied: much less) than 30 years away and quitting his job due to the ramifications.

By saying "I no longer think that", it's not necessarily that he thinks ChatGPT is smart than humans. Google Search has been far more capable at indexing and retrieving information than humans for over two decades now. He's talking about AGI no longer being 30-50 years away but instead may arrive far sooner than society is ready to deal with.

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#569

Post by ethanbond »

anon-3988 wrote: post #562
danShumway wrote: post #525

Another article about fears of AGI. As a reminder, there is not a single LLM on the market today that is not vulnerable to prompt injection, and nobody has demonstrated a fully reliable method to guard against it. And by and large, companies don't really seem to care. Google recently launched a cloud offering that uses a LLM to analyze untrusted code. It's vulnerable to prompt injection through that code. Microsoft B…

Obviously you haven't read or skimmed the article because the article makes no mention of AGI. However, it is very bland and predictable. I am not sure why any technologists or pioneer would think their technology wouldn't be used for bad. You can probably replace any mention of AI, ML or NN in the article with any other invention in the past 1 billion years and it will still make sense. What technology/inventions ou…

What’s different in the downside case of AI is exactly what’s different in the upside case of AI: immense power that we have never seen before.

We now have a few “live x-risks,” each one representing a high wire we cannot fall off of even once, but nor can we just choose to step off of them safely. AI is an additional potential doom we are suspended above, and if it lives up to its positive potential it’ll also be able to produce a million more x-risks by itself (e.g. viruses).

Additional risk is always bad, and is not mitigated by “other technologies before it also carried risk.” It’s all additional, and this is the biggest addition so far (if it lives up to its positive technological promise).

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#570

Post by noduerme »

danShumway wrote: post #525

Another article about fears of AGI. As a reminder, there is not a single LLM on the market today that is not vulnerable to prompt injection, and nobody has demonstrated a fully reliable method to guard against it. And by and large, companies don't really seem to care. Google recently launched a cloud offering that uses a LLM to analyze untrusted code. It's vulnerable to prompt injection through that code. Microsoft B…

>> if you're genuinely losing sleep about GPT-4 becoming a general agent that does every job

I guess I'm one of those people, because I'm not convinced that GPT-3.5 didn't do some heavy lifting in training GPT-4... that is the take-off. The fact that there are still some data scientists or coders or "ethics committees" in the loop manifestly is not preventing AI from accelerating its own development. Unless you believe that LLMs cannot, with sufficient processing power and API links, ever under any circumstances emulate an AGI, then GPT-4 needs to be viewed seriously as a potential AGI in utero.

In any event, you make a good case that can be extended: If the companies throwing endless processing at LLMs can't even conceive of a way to prioritize injection threats thought up by humans, how would they even notice LLMs injecting each other or themselves for nefarious purposes? What then stops a rapid oppositional escalation? The whole idea of fast takeoff is that a sufficiently clever AI won't make its first move in a small way, but in a devastating single checkmate. There's no reason to think GPT-4 can't already write an infinite number of scenarios to perform this feat; if loosed to train another model itself, where is the line between that LLM evolution and AGI?

Post reply on HN