Live data from Hacker News

John Carmack and Rich Sutton partner to accelerate development of AGI

amii.ca

171–180 of 226 posts

Re: John Carmack and Rich Sutton partner to accelerate development of AGI

#171
post #109

Earlier quoted context omitted.

> I thought that philosophy has been debunked as a scam after the likes of Sam Bankman and others. How can a philosophy be "debunked as a scam" by a fraudster being in the news? Or are you talking about something else? One can distort and misuse almost any moral philosophy, such as utilitarianism or any other. History is filled with examples. You may disagree with particular assessments, conclusions, or even the basi…

> Or are you talking about something else? I'm talking about immoral or questionable behavior today to justify a possible future. It's like going back in time to kill Hitler's grandfather to prevent Hitler's birth. You have no idea of unintended consequences, and you are intentionally causing harm now for a future completely out of your control (although you deceive yourself into thinking it is in your control by tak…

>> Or are you talking about something else?

> I'm talking about immoral or questionable behavior today to justify a possible future.

You've outlined a form of deontological ethics, seems to me. You don't subscribe to consequentialism, that fine. But hopefully you recognize there are many moral theories.

> It's like going back in time to kill Hitler's grandfather to prevent Hitler's birth.

This is a strawman. (A really loaded one at that.)

Re: John Carmack and Rich Sutton partner to accelerate development of AGI

#172

Earlier quoted context omitted.

>I wondered the other day what the result would be if you strapped a microphone on a baby and every utterance that both the baby and mic heard was fed through speech-to-text and went toward training the baby's personal LLM. I read about a paper[1] a while back, it was a rather unpleasant animal study in cats. Using two kittens, one was free to look around, but the other they immobilised in some way, and made it see w…

> They discovered the immobilised kitten's visual processing did not develop normally whereas the mobile kitten's did, suggesting that it's not just the sensory input, but it being feedback to some internal agency within the brain. > I suspect the same thing would be true for attempts to develop AGI... On the other hand, I would not be surprised if this did _not_ generalize and translate to LLMs. Biological entities…

This would be like learning everything from books vs learning by experience. (Can’t learn skiing from books. I talk from experience)

Re: John Carmack and Rich Sutton partner to accelerate development of AGI

#173
post #104

Earlier quoted context omitted.

> I think Gödel had a proof for how it’s impossible to fully describe a system from within that system. That’s the nail in the coffin for AGI. Gödels theorems are about formal axiomatic theories. To apply them to human intelligence, you'd have to prove that human intelligence springs from formal axiomatic theories. I don't think this is possible, which would mean that you can't apply the theorems. > No matter how muc…

> To apply them to human intelligence, you'd have to prove that human intelligence springs from formal axiomatic theories. It applies to everything we could say or think or measure about intelligence (and anything else). All that probably "spring from" things that don't (rely on our axioms), but they are not accessible to us, so that doesn't really help.

> It applies to everything we could say or think or measure about intelligence (and anything else).

How do you know? Gödels proof doesn't support your claim since it only applies to formal axiomatic theories. Do you have an alternative proof for his theorems also applying to all other systems?

Re: John Carmack and Rich Sutton partner to accelerate development of AGI

#175
post #63

Earlier quoted context omitted.

> I mean, living unethically today in order to affect future lives positively is not for me. Though it is an almost-inescapable conclusion of being a utilitarian and significantly smarter than average. If you can defraud millions and put that money to work saving lives and buying many more QALYs than the defrauded would have otherwise had, you have a moral obligation to do so. ...there are many ways to make utilitari…

> If you can defraud millions and put that money to work saving lives and buying many more QALYs than the defrauded would have otherwise had, you have a moral obligation to do so So defrauding millions to accrue QALYs (for whom?) is morally acceptable? > Though it is an almost-inescapable conclusion of being a utilitarian and significantly smarter than average. A modicum of spirituality or religion will quickly quash…

> So you have to add atheism to the requirement.

Yep, you're absolutely right - I forgot to specify that.

Most utilitarians don't believe in transcendence, in my experience - not believing in it is usually why they're utilitarian (trying to salvage any argument for behavioral standards other than "I like it when other people do what I like").

Re: John Carmack and Rich Sutton partner to accelerate development of AGI

#176
post #28

Earlier quoted context omitted.

Ethics/privacy aside ... I wondered the other day what the result would be if you strapped a microphone on a baby and every utterance that both the baby and mic heard was fed through speech-to-text and went toward training the baby's personal LLM . By the time the child had grown to an adult I wonder what kind of results their LLM would produce and the degree to which it might compare to what the child (now grown) wo…

I thought about this too. People say that LLM models are only saying the most common tokens that come after the previous token. And that this makes them incomparable to human intelligence. But Humans are basically long running LLMs that are retrained in real-time. We are the product of our environment.

Sometimes people use LLMs very broadly to talk about neural networks overall.

But to be clear, humans don't have emergent reasoning from language, we learn language as part of our overall reasoning. The short evidence for that being that children are capable of solving logic and spatial puzzles before they learn how to speak. Humans learn concepts like object permanence before we learn language complicated enough to describe that concept. And obviously people are capable of reasoning without learning how to write or interpret text tokens, there are plenty of illiterate people in the world who are nonetheless indisputably intelligent agents.

So ignoring other differences about how prediction works, humans are not similar to LLMs in the sense that LLMs are language models that when large enough either develop (or appear to develop depending on who you ask) reasoning capabilities. And that's not how humans work; we don't learn text tokens before we learn how to reason.

But very often when people make this claim they're trying to make a broader claim about neural networks or the role of prediction in learning in general. People might disagree or agree with the broader claim, I still think it oversimplifies how humans work, but the point is -- they're not actually saying something specific about LLMs, even though it sounds that way sometimes. It's just that the terminology gets conflated in people's heads.

We can have a debate about the similarities and differences between humans and neural networks, but I don't think anyone would seriously claim that GPT-4 in specific works the same way as a human does. I think people are using LLMs to refer to a broader category of AI research.

Re: John Carmack and Rich Sutton partner to accelerate development of AGI

#177
post #109

Earlier quoted context omitted.

> I thought that philosophy has been debunked as a scam after the likes of Sam Bankman and others. How can a philosophy be "debunked as a scam" by a fraudster being in the news? Or are you talking about something else? One can distort and misuse almost any moral philosophy, such as utilitarianism or any other. History is filled with examples. You may disagree with particular assessments, conclusions, or even the basi…

> Or are you talking about something else? I'm talking about immoral or questionable behavior today to justify a possible future. It's like going back in time to kill Hitler's grandfather to prevent Hitler's birth. You have no idea of unintended consequences, and you are intentionally causing harm now for a future completely out of your control (although you deceive yourself into thinking it is in your control by tak…

> I lived through the Cold War. I don't need this exercise because, the bottom line is, no one knows what existential risk - if any - will happen. The worst of the Cold War never happened, yet many of us thought it would. Likewise, you might choose to focus on AI as your existential risk, go through all the sacrifices and immoral behaviors that longtermism leads you to do in order to mitigate AI, and then a giant meteor collides with Earth. Oops! You chose the wrong existential risk. And meanwhile, you compromised your morals and right behavior in order to mitigate the wrong existential risk. No thanks, not for me.

Your claim seems to boil down to two things: First, people sometimes make very incorrect predictions, so you've stopped worrying about them. Second, instead, you focus on "doing the right thing" based on your ethical criteria (which, as I say in sibling comment, seems to be aligned with deontological ethics -- i.e. the ends don't justify the means.)

I'm not going to argue with the second (your choice of ethics) here. But your first point is terribly flawed; it ignores probability and decision-making under uncertainty.

Sure, you have to make assessments about what to believe. Strictly speaking, as a society, it is irrational to stop paying attention and responding to the most important things, even if any one of them might be unlikely.

Personally, you might find it necessary or useful to tune out this stuff. I get that. I'd guess this happened: you have stopped _caring_ for various reasons (perhaps information overload, sensationalization of the news, more pressing matters right in front of you) and your brain has constructed a worldview that _rationalizes_ not caring. Totally human, I get it. But this isn't reasoning nor logic.

I may not have your years of experience, but I'm no spring chicken either. Age doesn't always bring wisdom, unfortunately. People learn the wrong lessons. We've learned many. I have a keen interest in the Cold War. If you study the history, you'll recognize that our path 'through' it was no means assured. We are still living with the downstream effects of it, including huge nuclear arsenals and proxy wars.

Your comment above is naive with regards to history. It paints a simplistic and convenient narrative with the benefit of hindsight. You've ignored one key aspect of history, which is to understand it _in context_. Using hindsight exclusively isn't the only nor best way to learn from history.

Even in deontological ethics, not _everything_ is intrinsically good of bad. There is still a responsibility to act rationally in order to achieve moral ends in a moral way. To completely ignore probability is immoral.

Please, remember this: using poor logic and FUD to sully a serious book (based on some vague implied connection to a crook) is not ok.

Re: John Carmack and Rich Sutton partner to accelerate development of AGI

#178
post #63

Earlier quoted context omitted.

> I mean, living unethically today in order to affect future lives positively is not for me. Though it is an almost-inescapable conclusion of being a utilitarian and significantly smarter than average. If you can defraud millions and put that money to work saving lives and buying many more QALYs than the defrauded would have otherwise had, you have a moral obligation to do so. ...there are many ways to make utilitari…

> If you can defraud millions and put that money to work saving lives and buying many more QALYs than the defrauded would have otherwise had, you have a moral obligation to do so So defrauding millions to accrue QALYs (for whom?) is morally acceptable? > Though it is an almost-inescapable conclusion of being a utilitarian and significantly smarter than average. A modicum of spirituality or religion will quickly quash…

> So defrauding millions to accrue QALYs (for whom?) is morally acceptable?

Under most utilitarian theories of ethics, I think it's morally required, if you can show the fraud really does yield a significantly higher QALY count.

The best out I can see to argue the fraud is not required is to show that another course of action will yield even more QALYs. Then you'd be obligated you do that instead.

That, of course, comes to another fundamental flaws of utilitarianism, namely the belief that you can know what the results of your actions will be with any meaningful completeness or certitude.

You can't.

Without that it's all just excuses that justify whatever you want to do.

Re: John Carmack and Rich Sutton partner to accelerate development of AGI

#180
post #58
post #28

Earlier quoted context omitted.

I thought about this too. People say that LLM models are only saying the most common tokens that come after the previous token. And that this makes them incomparable to human intelligence. But Humans are basically long running LLMs that are retrained in real-time. We are the product of our environment.

You’re vastly oversimplifying humans. That’s like saying a physics simulation is basically an entire sub universe on your computer. It sounds true, but it’s just not. It’s a gross oversimplification I think Gödel had a proof for how it’s impossible to fully describe a system from within that system. That’s the nail in the coffin for AGI. No matter how much data we give it, no matter how big it is, it’ll never be “hum…

Nature managed to bootstrap human intelligence even though nature is not even a conscious entity, let alone a conscious entity that can describe loss functions and datasets. That seems to me like pretty solid evidence that those things are not hard requirements for creating human-level intelligence.

It sounds like maybe you're arguing that humans will never be able to conclusively determine if an artificial intelligence is equivalent to a human intelligence, on the basis of a theory that a human can't describe precisely what it is to have human-equivalent intelligence from inside the system of a human brain. Humans build things that are too complex for any one person to hold the entirety of in their conscious mind all the time, by working together, or by organizing it into simpler pieces. But even setting that tangent aside, if your theory is accurate, would you accept that an AGI could prove it was more intelligent than a human by successfully describing human intelligence and how to create an equivalent AGI?

Post reply on HN