John Carmack and Rich Sutton partner to accelerate development of AGI • Hacker News Board

Re: John Carmack and Rich Sutton partner to accelerate development of AGI

#201

Post by xpe » Sat, Oct 07, 2023, 10:10 PM UTC

TedDoesntTalk wrote: post #188

xpe wrote: post #171

Earlier quoted context omitted.

>> Or are you talking about something else? > I'm talking about immoral or questionable behavior today to justify a possible future. You've outlined a form of deontological ethics, seems to me. You don't subscribe to consequentialism, that fine. But hopefully you recognize there are many moral theories. > It's like going back in time to kill Hitler's grandfather to prevent Hitler's birth. This is a strawman. (A reall…

> You've outlined a form of deontological ethics, I had to look this up. You've given a modern label to an ancient concept sometimes known as the Golden Rule. I'm not sure why. >> It's like going back in time to kill Hitler's grandfather to prevent Hitler's birth. > This is a strawman. (A really loaded one at that.) It's not intended that way, and I'm posing it with honesty. If it's too loaded, then replace Hitler wi…

> I had to look this up.

Great! Philosophers have explored the ideas of "inherent goodness" versus "goodness because of the effects" for thousands of years. People face these decisions everyday, whether they realize it or not.

> You've given a modern label to an ancient concept sometimes known as the Golden Rule. I'm not sure why.

First, I didn't discover the idea of deontological ethics (DE). From Brittanica:

> In deontological ethics an action is considered morally good because of some characteristic of the action itself, not because the product of the action is good. Deontological ethics holds that at least some acts are morally obligatory regardless of their consequences for human welfare. Descriptive of such ethics are such expressions as “Duty for duty’s sake,” “Virtue is its own reward,” and “Let justice be done though the heavens fall.”

Second, the Golden Rule is not the same thing as DE.

Consider this:

> Like most key tenets of ethics, the golden rule shows two major sides: one promoting fairness and individual entitlement, conceived as reciprocity; the other promoting helpfulness and generosity to the end of social welfare. Both the Kantian and Utilitarian traditions focus on only one side, furthering the great distinctions in philosophical ethics—the deontology-teleology and justice-benevolence distinctions. For the general theory project, this one-sidedness is purposeful, a research tool for reductive explanation. > > - https://iep.utm.edu/goldrule/

If you want to follow the Golden Rule, you need to think about consequences! Hence probability over future scenarios.

You _really_ need to branch out more from Hacker News. Go read some philosophy!

Re: John Carmack and Rich Sutton partner to accelerate development of AGI

#202

Post by layer8 » Sat, Oct 07, 2023, 10:27 PM UTC

HanClinto wrote: post #194

layer8 wrote: post #69

Earlier quoted context omitted.

You might enjoy this short story: http://hubski.com/pub/78001 Published in 1990, by the way.

That was a gem. Thank you for sharing!

I recommend the Axiomatic anthology which contains this short story.

Re: John Carmack and Rich Sutton partner to accelerate development of AGI

#203

Post by xpe » Sat, Oct 07, 2023, 10:40 PM UTC

TedDoesntTalk wrote: post #188

xpe wrote: post #171

Earlier quoted context omitted.

>> Or are you talking about something else? > I'm talking about immoral or questionable behavior today to justify a possible future. You've outlined a form of deontological ethics, seems to me. You don't subscribe to consequentialism, that fine. But hopefully you recognize there are many moral theories. > It's like going back in time to kill Hitler's grandfather to prevent Hitler's birth. This is a strawman. (A reall…

> You've outlined a form of deontological ethics, I had to look this up. You've given a modern label to an ancient concept sometimes known as the Golden Rule. I'm not sure why. >> It's like going back in time to kill Hitler's grandfather to prevent Hitler's birth. > This is a strawman. (A really loaded one at that.) It's not intended that way, and I'm posing it with honesty. If it's too loaded, then replace Hitler wi…

> It's not intended that way, and I'm posing it with honesty.

Yes, I believe you. But still, I don't care for your framing of it; it is still too loaded.

When learning about a new concept, one owe it to the concept to give it a fair chance. It is all too easy to misrepresent an idea. (To me, when looking at your comments overall, it seems possible that this is what you are doing: you aren't fully open to really understanding the idea; you've already _judged_ and _dismissed_ it.)

Aside: To be clear, I don't think "open mindedness" is the ideal _end_ goal. Open mindedness is a _tool_ to evaluate and understand. But the _goal_ is to draw conclusions in order to act wisely.

> If it's too loaded, then replace Hitler with, I don't know, . It doesn't matter who the bad guy is; the point is: you kill an innocent person (grandfather_of_bad_guy) in order to mold the future in a way that you think will save more, other innocent lives. Correct?

To do consequentialist ethics (CE) rigorously, one has to explore all likely outcomes. In my view, the best intellectual framework for doing so would be to build and evaluate a probabilistic tree of possible outcomes over a suitable time frame. See Decision Theory [1]

The example you gave has two major problems. First, it offers a false dichotomy. Why would one think the only choice is to kill an innocent ancestor or just "let" the bad guy do all the terrible things? Second, even to a pure consequentialist who is willing to set aside the "present value of a life" (I don't even like saying this), killing a person has negative consequences on their family, friends, community, city, state, and country.

Of course there are differences between CE and DE, but in practice I don't think they are as jarring as people can make them out to be. The specially-constructed "wedge" examples get too much airtime in my opinion. It is far more important to teach people about decision theory and how it applies to not just economics and cybersecurity but also morality.

I've written enough here for now. I hope you view your questions as jumping off points for additional philosophical investigation. I have seen for myself that studying these ideas is worthwhile: personally and from studying history.

[1]: https://en.wikipedia.org/wiki/Decision_theory

Re: John Carmack and Rich Sutton partner to accelerate development of AGI

#204

Post by Auracle » Sun, Oct 08, 2023, 1:03 AM UTC

charlie0 wrote: post #113

esrauch wrote: post #107

Earlier quoted context omitted.

Humans don't put ants in zoos because the ants overconsume their localized habitat though. A naive super intelligent agi has no reason to care if global warming makes the planet 150 degrees F on average, if it's self-interested it would surely be able to be "ok" on a planet under those conditions.

Sure, but at some point it will be asked to solve those problems and if it's not properly built, could easily arrive at that conclusion from a problem solving perspective. Self interest doesn't have to come into play at all here.

If it’s tasked with solving those problems it can also be done with constraints such as “without killing humans.”

Re: John Carmack and Rich Sutton partner to accelerate development of AGI

#205

Post by FrustratedMonky » Sun, Oct 08, 2023, 1:04 AM UTC

ben_w wrote: post #170

FrustratedMonky wrote: post #38

Earlier quoted context omitted.

Not sure about everyone else, but I write one word at a time.

I seem to have two parts of my inner monologue, one comes up with complete concepts, the other puts them into words. When I started to notice this, I tried skipping the "make words" part to save time, as clearly I had not only had the thought but also was aware that I had already had the thought. This felt wrong in a way I have no words to describe as there's nothing else like that particular feeling of wrongness, th…

Sounds kind of like some Zen concepts about moving beyond language. And that language is a faulty method for communicating.

Re: John Carmack and Rich Sutton partner to accelerate development of AGI

#206

Post by FrustratedMonky » Sun, Oct 08, 2023, 1:12 AM UTC

danShumway wrote: post #176

volent wrote: post #28

Earlier quoted context omitted.

I thought about this too. People say that LLM models are only saying the most common tokens that come after the previous token. And that this makes them incomparable to human intelligence. But Humans are basically long running LLMs that are retrained in real-time. We are the product of our environment.

Sometimes people use LLMs very broadly to talk about neural networks overall. But to be clear, humans don't have emergent reasoning from language, we learn language as part of our overall reasoning. The short evidence for that being that children are capable of solving logic and spatial puzzles before they learn how to speak. Humans learn concepts like object permanence before we learn language complicated enough to…

Definitely, LLM's have gotten so much press, that many people arguing about 'AI', are thinking about LLM.

And, LLM's are not all that a human can do. Language is not everything about a human.

But there is an argument that there is part of the brain that produces language, and it has some LLM characteristics. It's just that the brain is bigger and does more than an LLM. So the brain is not an LLM.

The brain has many components. What happens when you take the problem solving of something like AlphaGo/AlphaStar, with the Vision processing in Cars or DaLLe, and the language processing in LLM. Add in hearing, touch.

It starts to look like the components of a brain.

Re: John Carmack and Rich Sutton partner to accelerate development of AGI

#207

Post by FrustratedMonky » Sun, Oct 08, 2023, 1:18 AM UTC

xwowsersx wrote: post #52

volent wrote: post #28

Earlier quoted context omitted.

I thought about this too. People say that LLM models are only saying the most common tokens that come after the previous token. And that this makes them incomparable to human intelligence. But Humans are basically long running LLMs that are retrained in real-time. We are the product of our environment.

The claim that "humans are basically long running LLMs" oversimplifies the complexity of human cognition and experience. Humans don't just process information; they experience emotions, desires, and subjective experiences that are deeply intertwined with their cognition. LLMs don't have feelings, motivations, or consciousness. Humans have inner subjective experience, self-awareness, and the ability to reflect on our…

Humans aren't LLM.s But there are more AI models than LLM's.

AlphaGO satisfies all of the requirements you listed for a human. Creativity, problem solving.

Just put it on a game loop, with more variables.

Re: John Carmack and Rich Sutton partner to accelerate development of AGI

#208

Post by FrustratedMonky » Sun, Oct 08, 2023, 1:20 AM UTC

dartos wrote: post #58

volent wrote: post #28

Earlier quoted context omitted.

I thought about this too. People say that LLM models are only saying the most common tokens that come after the previous token. And that this makes them incomparable to human intelligence. But Humans are basically long running LLMs that are retrained in real-time. We are the product of our environment.

You’re vastly oversimplifying humans. That’s like saying a physics simulation is basically an entire sub universe on your computer. It sounds true, but it’s just not. It’s a gross oversimplification I think Gödel had a proof for how it’s impossible to fully describe a system from within that system. That’s the nail in the coffin for AGI. No matter how much data we give it, no matter how big it is, it’ll never be “hum…

>"I think Gödel had a proof for how it’s impossible to fully describe a system from within that system. That’s the nail in the coffin for AGI."

Technically, this is also a nail in the coffin for humans. Humans are also inside a system.

Even back with Kant, humans have been trying to figure out how to see beyond their own systems.

Re: John Carmack and Rich Sutton partner to accelerate development of AGI

#209

Post by bch » Sun, Oct 08, 2023, 1:21 AM UTC

funnymony wrote: post #172

danielbarla wrote: post #157

Earlier quoted context omitted.

> They discovered the immobilised kitten's visual processing did not develop normally whereas the mobile kitten's did, suggesting that it's not just the sensory input, but it being feedback to some internal agency within the brain. > I suspect the same thing would be true for attempts to develop AGI... On the other hand, I would not be surprised if this did _not_ generalize and translate to LLMs. Biological entities…

This would be like learning everything from books vs learning by experience. (Can’t learn skiing from books. I talk from experience)

Clever - reminds me of:

In school, we learn the lesson then get the test. In life we get the test then learn the lesson.

Re: John Carmack and Rich Sutton partner to accelerate development of AGI

#210

Post by cactusplant7374 » Sun, Oct 08, 2023, 1:30 AM UTC

rvz wrote: post #55

cactusplant7374 wrote: post #48

Earlier quoted context omitted.

It’s AGI. So far everyone’s track record is nonexistent.

No. The parent comment said "AI Research". If you want "AGI" so badly, you certainly need to have an "AI research" background and a track record. Can't achieve your "AGI" without that. Given that Carmack has a close to zero track record in this research, it makes total sense for him to partner with someone that does.

We don't know how to achieve AGI. So screwing around with LLM might not be anything close to the process to achieve AGI.

https://aeon.co/essays/how-close-are-we-to-creating-artifici...