http://web.archive.org/web/20230501153311/https://www.nytime...
Geoffrey Hinton leaves Google and warns of danger ahead
661–670 of 1001 posts
Re: Geoffrey Hinton leaves Google and warns of danger ahead
#662If you put together two of his statements (below), and to be fair these could be isolated responses taken out of context or rephrased by the journalist. He seems to be saying that he thought: "autonomous killer robots" were 30 to 50 years or even longer away - but he continued working on the technology and then grew a conscience only when things came a long a little earlier than he expected. What did he think? that t…
It’s like the Manhattan project. They built it and suddenly grew a conscience when they realized the US government was about to authorize melting hundreds of thousands of Japanese civilians.
"They built it and suddenly a (small minority of them) grew a conscience..."
Edward Teller didn't lose too much sleep about the moral implications of the use of his research.
Re: Geoffrey Hinton leaves Google and warns of danger ahead
#663Another article about fears of AGI. As a reminder, there is not a single LLM on the market today that is not vulnerable to prompt injection, and nobody has demonstrated a fully reliable method to guard against it. And by and large, companies don't really seem to care. Google recently launched a cloud offering that uses a LLM to analyze untrusted code. It's vulnerable to prompt injection through that code. Microsoft B…
Couldn’t a possible solution to “prompt injection attacks” be to train/fine-tune a separate specialised model to detect them?
I personally think a lot of these problems with GPT-like models are because we are trying to train a single model to do everything. What if instead we have multiple models working together, each specialised in a different task?
E.g. With ChatGPT, OpenAI trained a single model to meet competing constraints, such as “be safe” versus “be helpful”. Maybe it would perform better with a separate model focused on safety, used to filter the inputs/outputs of the “be helpful” model?
Maybe you can never build a foolproof “prompt injection detector”. But, if you get it good enough, you then offer a bounty program for false negatives and use that to further train it. And realistic/natural false positives can be reported, human-reviewed and approved as harmless, and then fed back into the feedback loop to improve the model too. (I think contrived/unrealistic false positives, where someone is asking something innocent in a weird way just to try to get a false positive, aren’t worth responding to.)
Re: Geoffrey Hinton leaves Google and warns of danger ahead
#664Another article about fears of AGI. As a reminder, there is not a single LLM on the market today that is not vulnerable to prompt injection, and nobody has demonstrated a fully reliable method to guard against it. And by and large, companies don't really seem to care. Google recently launched a cloud offering that uses a LLM to analyze untrusted code. It's vulnerable to prompt injection through that code. Microsoft B…
There is one system, also widely-deployed, other than LLMs, that's well-known to be vulnerable to prompt injection: humans . Prompt injection isn't something you can solve . Security people are sometimes pushing things beyond sense or reason, but even they won't be able to fix that one - not without overhauling our understanding of fundamental reality in the process. The distinction between "code" and "data", between…
Humans have a trust model (however flawed) that allows them to judge whether they should follow instructions they encounter, LLMs do not.
Re: Geoffrey Hinton leaves Google and warns of danger ahead
#665This was his Tweet from several weeks ago, which I thought was insightful, both from a technical as well as socieconomic perspective when you think about data usage etc in these models - "Caterpillars extract nutrients which are then converted into butterflies. People have extracted billions of nuggets of understanding and GPT-4 is humanity's butterfly." Did he see enough in the past 6 weeks that made him change his…
Note that in that analogy, the caterpillar is dissolved during the process.
Fun fact: Scientists have determined that a moth or butterfly may not remember being a caterpillar, it can remember experiences it learned as a caterpillar. Hence I'm not sure dissolved is quite the right word to use because the nervous system stays with the creature during there process. The most accurate word is metamorphosis, since that's the word we gave for that process.
The other detail is that at the end of the process is a butterfly, so y'know, there's that.
Re: Geoffrey Hinton leaves Google and warns of danger ahead
#666Could anyone frame -- in fairly plain words -- what would be the mechanism by which LLMs become generally "smarter than humans" in the "and humans can't control them" sense?
Has there been some advance in self-learning or self-training? Is there some way to make them independent of human data and human curation of said data? And so on.
Re: Geoffrey Hinton leaves Google and warns of danger ahead
#667My honest take is a lot of these famous academics played almost no part in the developments at openai. But they want the limelight. They aren’t as relevant as they want to be. In many cases, they were directly wrong about how ai would develop
Really? Hinton dont need openAI to be relevant. He literally invented back propagation. He sticked to deep learning through 1990s and 2000s when almost all major scientist abandoned it. He was using neural networks for language model in 2007-08 when no one knew what it was. Again the deep learning in 2010s started when his students created AlexNet by coding deep learning in GPU. Chief Scientist of OpenAI Ilya Sutskev…
This sort of reminds me of Bloomberg articles wherein every time there is some "black swan" event, they go and find an analyst or economist that "got it right" and he gets to be prophet for a day: never mind that said analyst/economist may have predicted 100 of the last 3 financial crashes, they were "right" about this one.
Re: Geoffrey Hinton leaves Google and warns of danger ahead
#668Earlier quoted context omitted.
This quote is the first thing I've seen that really makes me worried. I don't think of ChatGPT as being "smart" at all, and comparing it to a human seems nonsensical to me. Yet here is a Turing award winning preeminent expert in the field telling me that AI smarter than humans is less (implied: much less) than 30 years away and quitting his job due to the ramifications.
By saying "I no longer think that", it's not necessarily that he thinks ChatGPT is smart than humans. Google Search has been far more capable at indexing and retrieving information than humans for over two decades now. He's talking about AGI no longer being 30-50 years away but instead may arrive far sooner than society is ready to deal with.
Who are you arguing against? You are just repeating the comment you replied to.
Re: Geoffrey Hinton leaves Google and warns of danger ahead
#669Another article about fears of AGI. As a reminder, there is not a single LLM on the market today that is not vulnerable to prompt injection, and nobody has demonstrated a fully reliable method to guard against it. And by and large, companies don't really seem to care. Google recently launched a cloud offering that uses a LLM to analyze untrusted code. It's vulnerable to prompt injection through that code. Microsoft B…
LLMs that can be run locally don't even have any guardrails. I tried running gpt4chan on my PC and it was outputting really horrible stuff.
Soon it won't matter what kind of guardrails OpenAI's ChatGPT has if anyone with a good GPU could run their own unrestricted LLM locally on their own machine.
Re: Geoffrey Hinton leaves Google and warns of danger ahead
#670Earlier quoted context omitted.
Society will be fine, actually AI will make things much better, just as the internet did. People have been making these kind of extreme predictions for decades and it was always wrong. The only people still upset about better communications tech are the people who pine for the days when all that was expected of respectable people was automatically trusting anyone working for the government, a university or a newspape…
Social media algorithms on "the internet" have caused wars, supported genocides, created extreme societal polarization, have led to dramatically increased suicide rates among teens, especially teen girls, and more. But I got to share baby pics with my mom. How will a far noisier information flow help? Generative AI will only help us do what we've been doing in far greater quantity. Just like calculators can only help…
None of those claims about the effects of social media hold up under inspection. They're all academic pseudo-babble. Look at the Haidt response to people pointing out his evidence of social media = suicides isn't robust; he doesn't argue the evidence actually is robust, he argues that censoring social media shouldn't require you to actually prove your case!
These ideas are all motivated narratives by people who want to control the internet to try and re-establish the world where their intuitions are the only ones allowed to be expressed.