Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Geoffrey Hinton leaves Google and warns of danger ahead

nytimes.com

Post reply on HN
Print view View original

641–650 of 1001 posts

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#641

Post by emtel »

mrtranscendence wrote: post #304
ilaksh wrote: post #107

I used to be fairly unconcerned about AI being dangerous. But part of the Yudkowsky interview on Lex Fridman 's podcast changed my mind. The disconnect for me is that Yudkowsky posits that the AIs will be fully "alive", thinking millions of times faster than humans and that there will be millions of them. This is too big of a speculative leap for me. What I can fairly easily imagine in the next few years with improve…

Is there any indication that current methods could lead to a model that generates text as if it had an IQ of 200? These are trained on texts written by humans who are, quite overwhelmingly, much lower in IQ than 200. Where's the research on developing models that don't just produce better or faster facsimiles of broadly average-IQ text?

Think a little bit deeper about what it means to be able to predict the next token. Think about what a predictor has to do in order to do this extremely accurately across a very large corpus of text.

There is a big difference between being able to predict what a median human might write next, and being able to predict, in all cases, what the particular human author of a particular passage will write next.

Or from another angle: the human authors of training data may have made errors when writing the data. The token predictor may learn to correctly predict those errors. These are not the same thing!

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#642

Post by XorNot »

ssnistfajen wrote: post #564
danShumway wrote: post #525

Another article about fears of AGI. As a reminder, there is not a single LLM on the market today that is not vulnerable to prompt injection, and nobody has demonstrated a fully reliable method to guard against it. And by and large, companies don't really seem to care. Google recently launched a cloud offering that uses a LLM to analyze untrusted code. It's vulnerable to prompt injection through that code. Microsoft B…

It's not solely about AGI. Weak AIs that powered social media algorithms already created hotbeds of polarizing extremism around the world as most members of society do not possess the basic diligence to realize when they are being manipulated. LLMs offer a glimpse into a future where much stronger AI, even if still technically "weak", can produce content in ways that influence public opinion. Couple that with the amo…

This argument is in the same category as when people link you a 2 hour YouTube video and say "if you watch this you'll understand (unsaid: and agree with) my viewpoint!"

Which is to say, I don't think the political disputes of the United States are being driven by social media algorithms because they are exactly dividing along historic fault-lines dating back the founding of the country.

The thing about blaming social media is it excuses anyone from dealing with the content of any sides complaints or stated intentions by pretending it's not real.

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#644

Post by gdiamos »

neatze wrote: post #99

“The idea that this stuff could actually get smarter than people — a few people believed that,” said Hinton to the NYT. “But most people thought it was way off. And I thought it was way off. I thought it was 30 to 50 years or even longer away. Obviously, I no longer think that.” Calculators are smarter then humans in calculating, what does he mean by that?

These results are predicted by LLM Scaling Laws and the GPT authors knew it before they started.

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#645

Post by hammyhavoc »

ssnistfajen wrote: post #564
danShumway wrote: post #525

Another article about fears of AGI. As a reminder, there is not a single LLM on the market today that is not vulnerable to prompt injection, and nobody has demonstrated a fully reliable method to guard against it. And by and large, companies don't really seem to care. Google recently launched a cloud offering that uses a LLM to analyze untrusted code. It's vulnerable to prompt injection through that code. Microsoft B…

It's not solely about AGI. Weak AIs that powered social media algorithms already created hotbeds of polarizing extremism around the world as most members of society do not possess the basic diligence to realize when they are being manipulated. LLMs offer a glimpse into a future where much stronger AI, even if still technically "weak", can produce content in ways that influence public opinion. Couple that with the amo…

You don't even need AI to influence public opinion detrimentally.

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#646

Post by rl3 »

zackmorris wrote: post #254
greatpostman wrote: post #98

My honest take is a lot of these famous academics played almost no part in the developments at openai. But they want the limelight. They aren’t as relevant as they want to be. In many cases, they were directly wrong about how ai would develop

I don't disagree. But for me, their mistake wasn't in the algorithms or their approach or anything like that. The problem has always been, and now will likely always be, the hardware. I've written about this at length in my previous comments, but a split happened in the mid-late 1990s with the arrival of video cards like the Voodoo that set alternative computation like AI back decades. At the time, GPUs sounded like…

>What we really needed was something like the 3D chip from the Terminator II movie, ...

>... Docker takes away much of the complexity of orchestrating distributed processes.

The T-800 running on Docker: After failing to balance its minigun, it falls forward out of the office window, pancaking in the parking lot below. Roll credits.

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#647

Post by theptip »

danShumway wrote: post #617
theptip wrote: post #563

Earlier quoted context omitted.

I find this position hard to grok. You’re complaining about people worrying about AGI because you view the short-run implications of this tech to be quite bad. To me, a lack of prompt security in the short term bodes poorly for our safety in N generations when these systems are actually powerful. Like, sure, someone is gonna get swatted by an AI in the next year or two, and that sucks, but that is a tiny speck of dus…

> Is it that you just think P(AGI) is really low, so worrying about an unlikely future outcome bothers you when there is actual harm now? Having now gotten a few opportunities to really use GPT-4 in-depth, I am much more bearish about the intelligence potential of LLMs than many other people are. This is not something I lose sleep over. But I don't like to focus on that because I'm not sure it matters. I commented th…

Thanks for clarifying. I strongly agree with your paragraphs 2-5, but I draw the opposite conclusion.

Many alignment researchers don’t think that solving prompt security will be similar to hard alignment challenges, I suspect it will at least somewhat (both requiring strong interpretability). Either way, it’s clearly a necessary precursor as you say.

Most people I know that take AGI risk seriously _are_ terrified of how cavalier the companies like Microsoft are being. Nadela’s “I want everyone to know we made Google dance” line was frankly chilling.

However, where I diverge from you is your final paragraph. Until very recently, as Hinton himself said, pretty much nobody credible thought this stuff was going to happen in our lifetimes, and the EA movement was considered kooky for putting money into AGI risk.

If most people think that the worst that could happen is some AI saying racist/biased stuff, some hacks, maybe some wars - that is business as usual for humanity. It’s not going to get anyone to change what they are doing. And the justification for fixing prompt security is just like fixing IoT security; a dumpster fire that nobody cares enough about to do anything.

If people, now, discovered an asteroid hurtling towards us, you’d hope they drop petty wars and unite (at least somewhat) to save the planet. I don’t happen to put P(doom) that high, but hopefully that illustrates why I think it’s important to discuss doom now. Put differently, political movements and the Overton Window take decades to effect change; we might not have decades until AGI takes off.

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#648

Post by preordained »

evanlivingston wrote: post #465
93po wrote: post #441

Earlier quoted context omitted.

I'm not sure I see the value in this. This list isn't a result of value judgements and morals held by ChatGPT. It's a list of things generated from training data when a similar question was asked.

This comment captures exactly what I find concerning around the common line of thinking about AI. How is a value judgement or moral that _I_ hold different than the list output by ChatGPT here?

You actually identify with your values and they have personal meaning for you? You don't spit them out only when given a specific prompt, them becoming completely forgotten and irrelevant when given some other, perhaps contradictory prompt?

Re: Geoffrey Hinton leaves Google and warns of danger ahead

#650

Post by sjkoelle »

ftxbro wrote: post #239
greatpostman wrote: post #98

My honest take is a lot of these famous academics played almost no part in the developments at openai. But they want the limelight. They aren’t as relevant as they want to be. In many cases, they were directly wrong about how ai would develop

> they want the limelight Maybe, but there is another force at play here too. It's that journalists want stories about AI, so they look for the most prominent people related to AI. The ones who the readers will recognize, or the ones who have good enough credentials for the journalists to impress upon their editors and readers that these are experts. The ones being asked to share their story might be trying to grab t…

Hinton is absolutely not a pop scientist. That said, AI doomerism is Planckian
Post reply on HN