Live data from Hacker News

Geoffrey Hinton leaves Google and warns of danger ahead

nytimes.com

711–720 of 1001 posts

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#711

Another article about fears of AGI. As a reminder, there is not a single LLM on the market today that is not vulnerable to prompt injection, and nobody has demonstrated a fully reliable method to guard against it. And by and large, companies don't really seem to care. Google recently launched a cloud offering that uses a LLM to analyze untrusted code. It's vulnerable to prompt injection through that code. Microsoft B…

Should we really be talking about AGI here? LLM are interesting and may have security challenges but they're light years away from AGI.

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#712

Earlier quoted context omitted.

Yeah, it seems that driving this fact home may in fact be beneficial. Right now a lot of people still assume that everyone on the internet is truthful with good intentions. Making it very clear that this isn't true may be helpful to reset this frame of mind.

Does anyone else hate the fact that we are actively encouraging making a low trust society? That can't lead to good things...

Yes, that isn't lost on me but it seems somewhat unavoidable. I would love it if we could trust everyone we meet, but unless we can somehow get everyone to stop doing bad things then you will need to be skeptical of people that you don't have a reason to trust. I think it is important that we add strong mechanisms for trust where possible such as regulations for companies and trust via relationships but it seems that no trust is the inevitable default. People have been running phone scams for decades and it doesn't seem like stopping them from making these attempts is feasible. The most viable solution seems to be helping people identify who they are talking with so that they can avoid falling for the scam.

Maybe I am too pessimistic and we will be able to fulfill everyone's needs one day. But as the rich keep getting richer and focusing on power for themselves that doesn't seem like a direction that we are going to move towards soon.

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#713
post #167
post #6

Earlier quoted context omitted.

Cade Metz is the same hack who tried to smear Scott Alexander. This guy is the personification of journalistic malpractice.

> Cade Metz is the same hack who tried to smear Scott Alexander. This guy is the personification of journalistic malpractice. He didn't "smear" Scott Alexander. That's just the hit-job framing pushed by Alexander's fans, who were mad he didn't write a puff piece and they couldn't just make up rules on about stuff on their websites (e.g. about using people's self-disclosed real names) and have the rest of the world be…

As far as I remember, the controversy started like this: the person in question is called Scott Alexander Siskind (which he has said himself publicly in his first post on the new ACX).

In his previous work as a psychiatrist as a hospital, he went by Scott Siskind (which seems to be the name he uses with family, on identity documents etc), whereas in rationalist circles and his old blog SSC he went by Scott Alexander. He has explained why it is a problem for a psychiatrist to have a known identity beyond the usual "blank slate", that his clients can project onto. Indeed, his real name being associated with his online one led to him having to leave his former job (albeit more "by mutual agreement" rather than "fired" as I understand).

It seems that the NYT, despite having a "real names" policy, is more than willing to bend it when it is convenient to them (for example they are more than happy not to mention birth names of transgender people even if they haven't gone through a full legal name change process). But they weren't willing to do it in this case, just referring to "the rules". The cost of this was that Scott lost his job (although he now makes more on Substack than he used to as a full-time psychiatrist), and the benefit ... I really don't see what benefit there is for NYT readers to know the surname of Scott-the-psychiatrist, when the article was about Scott-the-rationalist.

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#714

Another article about fears of AGI. As a reminder, there is not a single LLM on the market today that is not vulnerable to prompt injection, and nobody has demonstrated a fully reliable method to guard against it. And by and large, companies don't really seem to care. Google recently launched a cloud offering that uses a LLM to analyze untrusted code. It's vulnerable to prompt injection through that code. Microsoft B…

I guess I'm out of the loop. What's prompt injection?

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#715

It's important to have a discussion about AI safety, and the ethics surrounding LLMs. But I'm really tired of all this sensationalism. It completely muddies the waters; it almost seems intentional at this point.

It is. And it gets clicks.

But what I rarely see discussed is the opportunity costs in not having this progress at as fast a pace as possible.

The pie chart of existential threats for humanity definitely has rouge AI on it.

But that's a slice amidst many human driven threats ranging from nuclear war to oceans dying.

What there's not very many human driven slices of pie for is realistic solutions to these issues.

On that pie chart, some sort of AGI deus ex machina is at the current rate of progress probably the only realistic hope we have.

People who have been in the AI industry for a long time are still operating on thought experiments that barely resemble the actual emergence we are watching.

So while you have old AI thought leaders going on about cold heartless AI not being able to stop itself from making paperclips because of its innate design from training, we have humans unable to stop themselves from literally making paperclips (and many other things) because of our own evolutionary brain shortcomings, even as it dooms us.

From what I've seen so far, it seems far easier to align AI to care about the lives and well-being of humans different from it than it seems to be to align most humans to care about the lives of fellow humans different from them.

The opportunity cost of leaving the world in human hands as opposed to accelerating a handoff to something better at adapting to the emerging world and leaving the quirks of its design from generations of training behind seems far more dangerous than the threat of that new intelligence in isolation.

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#716

It's important to have a discussion about AI safety, and the ethics surrounding LLMs. But I'm really tired of all this sensationalism. It completely muddies the waters; it almost seems intentional at this point.

Did we have some ethics discussion when the lightbulb was invented? Or when the car was invented? No, and if we did, we couldn't have foreshadowed all the positive and negative impacts. My grandfather always told me that back in the days, "smart" people said that no train should be allowed to go faster than 50km/h because the heart would explode. Nobody here can say that he wasn't impressed by ChatGPT. How could we e…

> that no train should be allowed to go faster than 50km/h

"There was some wonderful stuff about [railway trains] too in the U.S., that women's bodies were not designed to go at 50 miles an hour. Our uteruses would fly out of our bodies as they were accelerated to that speed."

https://www.wsj.com/articles/BL-TEB-2814

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#717

It's important to have a discussion about AI safety, and the ethics surrounding LLMs. But I'm really tired of all this sensationalism. It completely muddies the waters; it almost seems intentional at this point.

Did we have some ethics discussion when the lightbulb was invented? Or when the car was invented? No, and if we did, we couldn't have foreshadowed all the positive and negative impacts. My grandfather always told me that back in the days, "smart" people said that no train should be allowed to go faster than 50km/h because the heart would explode. Nobody here can say that he wasn't impressed by ChatGPT. How could we e…

There was a whole panic where mobile phones were thought to melt your brain. Similar with radio. Some belived TV would literally give you square eyes. Vaccines. The internet was simultaneously apocalyptic and useless.

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#718
post #223
post #99

“The idea that this stuff could actually get smarter than people — a few people believed that,” said Hinton to the NYT. “But most people thought it was way off. And I thought it was way off. I thought it was 30 to 50 years or even longer away. Obviously, I no longer think that.” Calculators are smarter then humans in calculating, what does he mean by that?

I think GPT4 can converse on any subject at all as well as a (let's say) 80 IQ human. On some subjects it can converse much better. That feels fundamentally different than a calculator.

I'm curious if you actually ever interacted with IQ 80 humans. They are definitely not on this scale.

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#719

Another article about fears of AGI. As a reminder, there is not a single LLM on the market today that is not vulnerable to prompt injection, and nobody has demonstrated a fully reliable method to guard against it. And by and large, companies don't really seem to care. Google recently launched a cloud offering that uses a LLM to analyze untrusted code. It's vulnerable to prompt injection through that code. Microsoft B…

It's trivial to fix prompt injection.

You simply add another 'reviewer' layer that does a clarification task on the input and response to detect it.

The problem is this over doubles the cost of implementation to prevent something no one actually cares about fixing for a chatbot.

"Oh no, the user got it to say silly things to themselves."

This isn't impacting other people's experiences or any critical infrastructure.

And in any applications that do, quality analysis by another GPT-4 layer will be incredibly robust, halting malicious behavior in its tracks without sophisticated reflection techniques that I'm skeptical could successfully both trick the responding AI to answer but evade the classifying AI in detecting it.

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#720

Another article about fears of AGI. As a reminder, there is not a single LLM on the market today that is not vulnerable to prompt injection, and nobody has demonstrated a fully reliable method to guard against it. And by and large, companies don't really seem to care. Google recently launched a cloud offering that uses a LLM to analyze untrusted code. It's vulnerable to prompt injection through that code. Microsoft B…

There is one system, also widely-deployed, other than LLMs, that's well-known to be vulnerable to prompt injection: humans . Prompt injection isn't something you can solve . Security people are sometimes pushing things beyond sense or reason, but even they won't be able to fix that one - not without overhauling our understanding of fundamental reality in the process. The distinction between "code" and "data", between…

>The distinction between "code" and "data"

Hello LISP :)

Post reply on HN