To create a super-intelligent machine, start with an equation

It's interesting that, while the system can learn lots of different games, you still have to give it the reward function special for each different game. This may seem obvious, and not much of a limitation- after all, lots of things that we think of as intelligent to varying degrees (different human beings, as well as members of other species) have wildly different ideas of what constitutes "reward", so that can't be an inherent part of the definition of intelligence.

But you don't have to explicitly tell a human what the reward function for Pac-Man is. Show a human the game and they'll figure it out. Which makes me wonder if, while there is some room for variability in reward functions, there might be some basic underlying reward computation that is inherent in intelligence. I can't find the link just now, but I read an article a few months ago (might've even been here on HN) about a system that demonstrated the appearance of intelligent behavior by trying to minimize entropy and maximize its possible choices for as long as possible within a given world model.

Is AIXI capable of recognizing itself? If an AIXI agent were controlling a robot body, would it realize that the body's actions correlate with its own intentions? Would it pass the mirror test?

> How many times do we need to re-define the same concept? Artificial Intelligence, Human Level Artificial Intelligence, Artificial General Intelligence, Strong Artificial Intelligence etc....

Every time the name gets co-opted to mean something else. These days "artificial intelligence" == "machine learning and natural language processing" which is most definitely not what TFA is about.

1. Do we have to engineer how much it's being rewarded in each game?

2. What happens in the infinite case? With pacman, there are always a finite number of choices at each state.

3. If it doesn't have to be told the rules, then how does it make decisions? In theory, it may be able to learn how to play jeopardy, but it may be way too inefficient in practice. Humans don't even start with a blank state.

4. I don't think I ever consciously apply philosophical principles like ockham's razor when I problem solve or learn. It makes me a little uncomfortable that we're starting with a philosophy, rather than having the system discover things itself. I would be ok with it if there was some parallel between ockham's razor and physics (not the methodology of science).

Thanks, that sheds a bit more light on it.

To me it still sounds like "I solved the problem because I formalized it." The formalization is probably novel (since he got his PhD for it) but somehow I find it difficult to believe noone else had a similar idea before (it definitely sounds familiar to me having learned about reinforcement learning, minimum description length, etc. before). I'll have to read up on it.