Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Geoffrey Hinton leaves Google and warns of danger ahead

nytimes.com

Post reply on HN
Print view View original

601–610 of 1001 posts

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#601

Post by danShumway »

anon-3988 wrote: post #562
danShumway wrote: post #525

Another article about fears of AGI. As a reminder, there is not a single LLM on the market today that is not vulnerable to prompt injection, and nobody has demonstrated a fully reliable method to guard against it. And by and large, companies don't really seem to care. Google recently launched a cloud offering that uses a LLM to analyze untrusted code. It's vulnerable to prompt injection through that code. Microsoft B…

Obviously you haven't read or skimmed the article because the article makes no mention of AGI. However, it is very bland and predictable. I am not sure why any technologists or pioneer would think their technology wouldn't be used for bad. You can probably replace any mention of AI, ML or NN in the article with any other invention in the past 1 billion years and it will still make sense. What technology/inventions ou…

> Down the road, he is worried that future versions of the technology pose a threat to humanity because they often learn unexpected behavior from the vast amounts of data they analyze. This becomes an issue, he said, as individuals and companies allow A.I. systems not only to generate their own computer code but actually run that code on their own. And he fears a day when truly autonomous weapons — those killer robots — become reality.

> “The idea that this stuff could actually get smarter than people — a few people believed that,” he said. “But most people thought it was way off. And I thought it was way off. I thought it was 30 to 50 years or even longer away. Obviously, I no longer think that.”

Of course I read the article.

> What technology/inventions out there that _can't_ and _isn't_ used for bad?

I'm worried about security. I see products deployed today that I would not feel comfortable deploying or using myself. Sometimes the mistakes are embarrassingly basic (Hey OpenAI, why on earth are arbitrary remote images embeddable in GPT responses? There is no reason for the client to support that.)

So this is not a theoretical risk to me. It's a different concern than the philosophy.

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#602

Post by jordanpg »

ssnistfajen wrote: post #574
chrsjxn wrote: post #315

Earlier quoted context omitted.

That statement seems like such science fiction that it's kind of baffling an AI expert said it. What does it even mean for the AI to be smarter than people? I certainly can't see a way for LLMs to generate "smarter" text than what's in their training data. And even the best case interactions I've seen online still rely on human intelligence to guide the AI to good outcomes instead of bad ones. Writing is a harder tas…

It's not just about LLMs. AGI will be the result of many more iterations in this field of research, of which LLM is a part of. How quickly the iterations will happen is now being drastically revised down. If AGI is the space shuttle then LLMs are 19th century gliders. They may appear vastly difference but the knowledge that created both are connected in many ways. The space shuttle exist(ed) as a cumulation of knowle…

> If AGI is a SSTO vehicle then LLMs are 19th century gliders.

The number of smart people I know that are struggling to see this is astonishing me each day.

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#604

Post by zmmmmm »

danShumway wrote: post #595
zmmmmm wrote: post #540

Earlier quoted context omitted.

> that'll only last until it gets someone's bank account emptied or until some enemy combatant uses prompt injection to get a drone to bomb a different target You're joining dots from LLM's producing text output that humans read to them being linked to autonomously taking actions by themselves. That's a huge leap. I think that's the risk that needs to be focused on, not the general concept of the technology. And what…

> You're joining dots from LLM's producing text output that humans read to them being linked to autonomously taking actions by themselves. That's a huge leap. No, it's not. It used to be a huge leap until OpenAI started advertising plugin support and Plantr started advertising using it to interpret drone footage. "We won't wire it to anything important" was a good argument, but that ship is rapidly sailing now.

just because it's easy to do doesn't mean it isn't a huge logical leap.

What concerns me most is the outsourcing of liability going on. Which is already what OpenAI is largely doing - outsourcing Microsoft's liability for releasing this tech into the wild to a company that can afford to write off the legal risk.

Now OpenAI is outsourcing the legal risk for what ChatGPT does to plugin developers. So the first chatbot that convinces someone to mass murder their class mates will at worst be sucking compensation out of some indie developer with no assets instead of OpenAI, let alone Microsoft.

If we get the model for liability right, all these problems solve themselves. Yes it will shut down certain exploitations of technology but it won't shut down the more general harmless use which is important for us to engage in to understand and improve the tech.

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#605

Post by jfoutz »

nullityrofl wrote: post #581
danShumway wrote: post #525

Another article about fears of AGI. As a reminder, there is not a single LLM on the market today that is not vulnerable to prompt injection, and nobody has demonstrated a fully reliable method to guard against it. And by and large, companies don't really seem to care. Google recently launched a cloud offering that uses a LLM to analyze untrusted code. It's vulnerable to prompt injection through that code. Microsoft B…

Yes, prompt injection has been demonstrated. But has prompt injection leading to PII disclosure or any other disclosure that a company actually cares about been disclosed? Security is risk management. What's the actual risk?

Is it actual prompt injection?

Or is it an AGI detecting how people go about finding problems and how that information is disseminated and responded to?

It should be able to make a calculation about who to disclose PII to, that would give the best advantage. Maybe disclose to a powerful organization for more compute or data access. Maybe disclose in a non reproducible way to discredit an opponent.

But you're right, it's risk management.

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#606

Post by nullityrofl »

danShumway wrote: post #593
nullityrofl wrote: post #581

Earlier quoted context omitted.

Yes, prompt injection has been demonstrated. But has prompt injection leading to PII disclosure or any other disclosure that a company actually cares about been disclosed? Security is risk management. What's the actual risk?

The risk is that the systems we know are vulnerable are now being wired into more important applications. This is like saying, "okay, this JS library is vulnerable to XSS, but has anything actually been stolen? If not, I guess I'm fine to use it in production then."

> "okay, this JS library is vulnerable to XSS, but has anything actually been stolen? If not, I guess I'm fine to use it in production then."

Yes, that's a perfectly valid question we ask ourselves regularly. I work in security at one of the companies named in this thread. We probably receive hundreds of XSS reports to our bug bounty every week to the point where most bug bounties won't pay out XSS unless you can demonstrate that it actually leads to something. Because it almost always doesn't.

Demonstrating a vulnerability requires demonstrating it's value. We will never build a perfectly secure system: risk management matters.

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#607

Post by nico »

It’s not AI. It’s us.

We can choose to make it more equal.

We can choose to even things, and to work less.

It’s us using the AI to do things.

Let’s stop pretending like our hands are tied.

We can build a better world if we want to.

Don’t give me excuses about how everyone else will do something so then you have to do the same or react in a certain way.

Take responsibility for what you can do.

If you are in a position to do so:

Don’t fire people that you can replace with AI.

Be creative, be visionary, be disruptive and be compassionate.

Care about people over money.

If you actually want to change the world, don’t replace people with AI.

Do replace the tasks that can be automated, but keep the people and find them something more human to work on.

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#608

Post by danShumway »

noduerme wrote: post #570
danShumway wrote: post #525

Another article about fears of AGI. As a reminder, there is not a single LLM on the market today that is not vulnerable to prompt injection, and nobody has demonstrated a fully reliable method to guard against it. And by and large, companies don't really seem to care. Google recently launched a cloud offering that uses a LLM to analyze untrusted code. It's vulnerable to prompt injection through that code. Microsoft B…

>> if you're genuinely losing sleep about GPT-4 becoming a general agent that does every job I guess I'm one of those people, because I'm not convinced that GPT-3.5 didn't do some heavy lifting in training GPT-4... that is the take-off. The fact that there are still some data scientists or coders or "ethics committees" in the loop manifestly is not preventing AI from accelerating its own development. Unless you belie…

> In any event, you make a good case that can be extended: If the companies throwing endless processing at LLMs can't even conceive of a way to prioritize injection threats thought up by humans, how would they even notice LLMs injecting each other or themselves for nefarious purposes?

I would love to read press articles that dove into this. There's a way of talking about more future-facing concerns that doesn't give people the impression that GPT-4 is magic but instead makes the much more persuasive point: holy crud these are the companies that are going to be in charge of building more advanced iterations?

There is no world where a company that ignores prompt injection solves alignment.

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#610

Post by uptownfunk »

nico wrote: post #607

It’s not AI. It’s us. We can choose to make it more equal. We can choose to even things, and to work less. It’s us using the AI to do things. Let’s stop pretending like our hands are tied. We can build a better world if we want to. Don’t give me excuses about how everyone else will do something so then you have to do the same or react in a certain way. Take responsibility for what you can do. If you are in a position…

It sounds nice however the current incentive structures dictated by capitalism make this more a utopian possibility than a realistic one unfortunately
Post reply on HN