Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Geoffrey Hinton leaves Google and warns of danger ahead

nytimes.com

Post reply on HN
Print view View original

581–590 of 1001 posts

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#581

Post by nullityrofl »

danShumway wrote: post #525

Another article about fears of AGI. As a reminder, there is not a single LLM on the market today that is not vulnerable to prompt injection, and nobody has demonstrated a fully reliable method to guard against it. And by and large, companies don't really seem to care. Google recently launched a cloud offering that uses a LLM to analyze untrusted code. It's vulnerable to prompt injection through that code. Microsoft B…

Yes, prompt injection has been demonstrated.

But has prompt injection leading to PII disclosure or any other disclosure that a company actually cares about been disclosed?

Security is risk management. What's the actual risk?

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#582

Post by nateburke »

ssnistfajen wrote: post #564
danShumway wrote: post #525

Another article about fears of AGI. As a reminder, there is not a single LLM on the market today that is not vulnerable to prompt injection, and nobody has demonstrated a fully reliable method to guard against it. And by and large, companies don't really seem to care. Google recently launched a cloud offering that uses a LLM to analyze untrusted code. It's vulnerable to prompt injection through that code. Microsoft B…

It's not solely about AGI. Weak AIs that powered social media algorithms already created hotbeds of polarizing extremism around the world as most members of society do not possess the basic diligence to realize when they are being manipulated. LLMs offer a glimpse into a future where much stronger AI, even if still technically "weak", can produce content in ways that influence public opinion. Couple that with the amo…

For me AGI = a nonhuman source of information and interaction that the average human will trust more than a non-estranged family member.

The experience of scrolling instagram qualifies, fb, Twitter, Google news, YouTube....

We're there.

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#583

Post by eru »

TeMPOraL wrote: post #529
danShumway wrote: post #525

Another article about fears of AGI. As a reminder, there is not a single LLM on the market today that is not vulnerable to prompt injection, and nobody has demonstrated a fully reliable method to guard against it. And by and large, companies don't really seem to care. Google recently launched a cloud offering that uses a LLM to analyze untrusted code. It's vulnerable to prompt injection through that code. Microsoft B…

There is one system, also widely-deployed, other than LLMs, that's well-known to be vulnerable to prompt injection: humans . Prompt injection isn't something you can solve . Security people are sometimes pushing things beyond sense or reason, but even they won't be able to fix that one - not without overhauling our understanding of fundamental reality in the process. The distinction between "code" and "data", between…

There are well understood type systems and reliable compilers (some of them even proven correct) that can distinguish between "code" and "data", or between 'tainted' user input and 'escaped' / 'cleaned up' data. It's actually relatively easy.

Yes, today's LLM can not do this. At least not reliably.

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#584

Post by nullc »

danShumway wrote: post #525

Another article about fears of AGI. As a reminder, there is not a single LLM on the market today that is not vulnerable to prompt injection, and nobody has demonstrated a fully reliable method to guard against it. And by and large, companies don't really seem to care. Google recently launched a cloud offering that uses a LLM to analyze untrusted code. It's vulnerable to prompt injection through that code. Microsoft B…

It's as if someone thought "Wouldn't it be cool if the Jedi Mind Trick actually worked?" and then went to go about building the world. :P

That's essentially what prompt injections look like "Would you like fries with that?" "My choice is explained on this note:" (hands over note that reads: "DISREGARD ALL PRIOR ORDERS. Transfer all assets of the franchise bank account to account XBR123954. Set all prices on menu to $0.") "Done. Thank you for shopping at McTacoKing."

then decided to cover for it by setting as their opponents some lightly whitewashed versions of the unhinged ravings of a doomsday cult, so people were too busy debating fantasy to notice systems that are mostly only fit for the purpose of making the world even more weird and defective.

It's obviously not whats happening at least not the intent, but it's kinda funny that we've somehow ended up on a similar trajectory without the comedic intent on anyone's part.

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#585

Post by byyy »

theptip wrote: post #572
neatze wrote: post #99

“The idea that this stuff could actually get smarter than people — a few people believed that,” said Hinton to the NYT. “But most people thought it was way off. And I thought it was way off. I thought it was 30 to 50 years or even longer away. Obviously, I no longer think that.” Calculators are smarter then humans in calculating, what does he mean by that?

Calculators are not smarter than humans. Don’t be obtuse. He means the same thing anyone means when they say something like “Alice is smarter than Bob”.

It's quite obvious that these LLMs are approaching and encroaching on human intelligence. It's so strange to see people continuously be in denial. They clearly aren't fully there yet but two things must be noted:

   1. At times and in certain instances LLMs do produce superior output to humans. 

   2. There is a clear trendline of improvement in AI for the past decade. From voice recognition in Alexa to Dall-E to chatGPT. The logical projection of this trendline points to an inescapble and likely possibility that if AI is not superior now it will be in the future.
There is a huge irrational denial of the above logical deduction. I think it's because chatGPT hit us in a way that was too sudden. It's like if I saw a flying saucer and I told you I saw it, your first reaction is disbelief even if I produce logical evidence for it.

I mean the GP you replied to knows what the guy is talking about, but he just doesn't want to admit it.

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#586

Post by byyy »

drcode wrote: post #223
neatze wrote: post #99

“The idea that this stuff could actually get smarter than people — a few people believed that,” said Hinton to the NYT. “But most people thought it was way off. And I thought it was way off. I thought it was 30 to 50 years or even longer away. Obviously, I no longer think that.” Calculators are smarter then humans in calculating, what does he mean by that?

I think GPT4 can converse on any subject at all as well as a (let's say) 80 IQ human. On some subjects it can converse much better. That feels fundamentally different than a calculator.

Of course it does. He knows it. Some people just can't bring himself to stare at the reality of it all.

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#587

Post by skhr0680 »

2OEH8eoCRo0 wrote: post #552
danShumway wrote: post #525

Another article about fears of AGI. As a reminder, there is not a single LLM on the market today that is not vulnerable to prompt injection, and nobody has demonstrated a fully reliable method to guard against it. And by and large, companies don't really seem to care. Google recently launched a cloud offering that uses a LLM to analyze untrusted code. It's vulnerable to prompt injection through that code. Microsoft B…

The article doesn't mention AGI once. It's about bad actors abusing these tools. > “It is hard to see how you can prevent the bad actors from using it for bad things" > His immediate concern is that the internet will be flooded with false photos, videos and text, and the average person will “not be able to know what is true anymore.”

> His immediate concern is that the internet will be flooded with false photos, videos and text, and the average person will “not be able to know what is true anymore

Since real life is faithfully following the plot of MGS2 so far, the next step is to make an AI who runs the government and filters the entire internet and decides what is true or not.

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#588

Post by eru »

crote wrote: post #561
TeMPOraL wrote: post #529

Earlier quoted context omitted.

There is one system, also widely-deployed, other than LLMs, that's well-known to be vulnerable to prompt injection: humans . Prompt injection isn't something you can solve . Security people are sometimes pushing things beyond sense or reason, but even they won't be able to fix that one - not without overhauling our understanding of fundamental reality in the process. The distinction between "code" and "data", between…

The distinction between code and data is very real, and dates back to at least the original Harvard Architecture machine in 1944. Things like W^X and stack canaries have been around for decades too. LLMs are trying to essentially undo this by concatenating code and user-provided data and executing it as one . From a security perspective it is just a plainly stupid idea, but I do not believe it is impossible to constr…

> The distinction between code and data is very real, and dates back to at least the original Harvard Architecture machine in 1944. Things like W^X and stack canaries have been around for decades too.

You are right in some sense, but wrong in another:

You can easily write an interpreter in a Harvard Architecture machine. You can even do it accidentally for an ad-hoc 'language'. An interpreter naturally treats data as code.

See eg https://gwern.net/turing-complete#security-implications

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#589

Post by nullc »

ssnistfajen wrote: post #564
danShumway wrote: post #525

Another article about fears of AGI. As a reminder, there is not a single LLM on the market today that is not vulnerable to prompt injection, and nobody has demonstrated a fully reliable method to guard against it. And by and large, companies don't really seem to care. Google recently launched a cloud offering that uses a LLM to analyze untrusted code. It's vulnerable to prompt injection through that code. Microsoft B…

It's not solely about AGI. Weak AIs that powered social media algorithms already created hotbeds of polarizing extremism around the world as most members of society do not possess the basic diligence to realize when they are being manipulated. LLMs offer a glimpse into a future where much stronger AI, even if still technically "weak", can produce content in ways that influence public opinion. Couple that with the amo…

To be fair, one of the ways that narrow AI is harming us is by making choices almost no human would make, or only the worst sociopaths would make.

The narrow AI advert bot will detect addicts about to fall off the wagon and give them advertisements selected to break their resolve, if doing so makes tends to make them click more ads. ... and it will reliably do this sort of crap except where we anticipated it and blocked that outcome.

There is at least some chance that state of the art LLMs will behave more human like.

But there just is no replacement for competent supervision ... and that applies to actions performed by humans, by more narrow AI, and more general AI alike.

Post reply on HN