John Carmack and Rich Sutton partner to accelerate development of AGI

What would pain receptors be in a robot?

I ask this as I think the ethical dilemma should more about implementing suffering as a long term damaging effect of a stimuli.

When I think about pain I see it as a level that triggers a need to change something. Our brains are perceiving this level in a very uncomfortable way so we call this pain.

So I really think that without consciousness we cannot implement pain or suffering as we humans perceive it.

It doesn't seem LLMs are capable of "understanding" that they don't "know" something, so there are certainly some observable differences. But the intent of my original comment was precisely to highlight that we don't know. It could be that next token predication is somehow mathematically equivalent to what we call consciousness. That would certainly be a revelation.

Write better as is has more correct and clear syntax, sure.

But it doesn’t write better as in has new, challenging ideas. Or ways to move/relate to humans on a personal level.

GPT writes clear, concise, authoritative, boring, and generic text.

> Can you try to describe how you'd "drill down" on informal speech to transform it into such a system?

As I said, just keep asking "why?" or "what does that mean?", then repeat that with the answer. Sooner or later you hit an assumption and a shrug. I wouldn't understand Gödel's proof even if I tried to, I'm sure -- it "rings true" because it matches my own intellectual observations regardless where I turn.

> As long as those assumptions align with reality to the best of our knowledge and the end result passes all tests we can think of, what does it matter whether there might be more to know?

It matters for the question whether you can fully describe a system from within that system, that's all. But I'd argue even whether we made an effort or no effort, whether it passes all the tests we came up with or doesn't, doesn't really matter (in regards to that question) because any ground we cover won't bridge what remains an infinite distance. I still think it's good, but it's more like going for a walk each day: you always arrive where you started out, you're just getting fresh air and what other temporary benefits come with it. It beats just staying where you started out.

I can't find the quote but apparently Werner Heisenberg said something along those lines, that we basically set out to find the bed rock of reality, but more and more are just facing ourselves, that is, our instruments of measurement and ways to conceptualize things. And again, I don't know jack about quantum mechanics and don't want to call on the authority of Heisenberg and Gödel. But I hear they know their fields, right, and it matches everything I know in any area, both the ones I am bad in and the ones I am really bad in.

I'm not saying it's a problem, just that that's how it is. But thinking you know the ultimate and final truth because it passes all tests (e.g. witches sink), and thinking software is actually intelligent because it convinces you it is, when it really isn't, can be super mega dangerous. And comments how picking the statistically most likely word is "basically what our brain does" [0] etc. show an even worse possibility; where we take the shortcut of just confusing what we are creating with us, because then it's easy and now we know how we think (when we really don't, not remotely). It just generally seems backwards to start out with the goal of "AI" when we can't even describe what we're looking for, much less how to build or find it. Having no more than "we'll know it when we have it", plus eagerness to claim we have it, is a recipe for at least a lot of circus, if not disaster.

[0] And that's on HN. Now ponder, for example, the average opinion of HN on say, whether banks should limit your passwords in all sorts of weird way that imply they're not hashing them, and how much worse the "real world" is. In this case, even the people at the forefront are so keen to move fast and break things, so the "real world" is pretty much doomed I'd say.

>> Your claim seems to boil down to two things: First, people sometimes make very incorrect predictions, so you've stopped worrying about them.

> Didn't say I stopped worrying about them: global warming worries me. Ubiquitous microplastics worry me. Heavy metals in our food supply & PFAS in our water worries me.

On what basis do you worry about them? Why worry about some more than others? (I know my answers, but I'm interested in how you do it.)

Do you factor in the probability of their occurrence? I'd expect you do, this is sort of inevitable for people, though people aren't typically very well calibrated in probabilities, especially on the upper (near 100%) and lower ends (near 0%).

I'm trying to understand what I perceive as your pushback against moral reasoning that involves probabilities over future scenarios.

Here is my rough overall take at this point: given your ethical foundations (DE or close to it) and life experience (hard to know, often not driven by logic and reason -- such is the human condition) you bristle against consequentialism and longtermism in particular. I get it, but I don't think this is a good place to end up, even when I factor in your core beliefs. I 'chalk it up' to the high cost of mental reorgnization. My hypothesis is this: the 'wedge' examples that people use against consequentialism have had an outsized effect on how you view it. I don't detect any antipathy towards logic or probabilistic reasoning in general, but I also don't perceive a relentless drive towards it.