Reinforcement Learning: An Introduction (2018) [pdf]

For instance, a machine would operate via optimal control in regimes that are known and characterized by a model, but if it ever gets into a new unmodeled situation, it can use RL to figure stuff out and find a way to proceed suboptimally (subject to safety constraints, etc.).

An illustrative example is Roomba. Roomba is probably based on some form of RL, and it does a decent job. But suppose we have a map of the room that Roomba can use -- this would let it plot the optimal path. However suppose the map of the room is incomplete. Roomba can still operate near optimally within the mapped area, but will have to learn the environment outside the map. Or if the layout of the room has changed since the map was created (new furniture), Roomba's RL can kick in.

I would say though that from my experience, computational cost is rarely the issue with model-based control, because there are various attacks ranging from model simplification (surrogate models, piecewise-affine multi-models i.e. switching between many simpler local models, etc) to precomputing the optimal control law [1] to embedding the model in silicon. Also, some optimal models/control laws can actually parallelized fairly easily (MLD models are expressed mixed-integer programs which can be solved in performant ways using parallel algorithms, with some provisos). This is a well-trodden space with a tremendous amount of industry-driven research behind it.

Most of these methods come under the Model Predictive Control (MPC) umbrella which has been studied extensively over 3 decades [2]. The paradigm is extremely simple: (1) given a model of how output y responds to input u, predict over the next n time periods the values of u's needed to optimize an objective function. (2) Implement ONLY the first u. (3) Read the sensor value for y (actual y in real world). (4) Update your model with the difference between actual y and predicted y, move the prediction window forward, and repeat (feedback). When this is applied recursively, you obtain approximately optimal control on real-life systems even in the presence of model-reality mismatch, noise and bounded uncertainty.

If you think about it, this is the paradigm behind many planning strategies -- forecast, take a small action, get feedback, try again. The difference though is that MPC is a strategy with a substantial amount of mathematical theory (including stability analysis, reachability, controllability, etc.), software, and industrial practice behind it.

[1] Explicit MPC http://divf.eng.cam.ac.uk/cfes/pub/Main/Presentations/Morari...

[2] https://en.wikipedia.org/wiki/Model_predictive_control

This is also my suspicion. :) But to ignore optimal control altogether makes me suspect many AI researchers aren't familiar with the body of research, and many who've managed a cursory read of Wikipedia may believe that the state of the art in optimal control are LQRs and LQGs, when it's really MPC (which can be thought of as a generalization of LQRs).

Also, MPC is a model-type and optimization-algorithm agnostic paradigm, so there's plenty of ways to combine models/algorithms within its broad framework -- this is partly how many MPC researchers come up with new papers :). I think AI researchers should take a look at it in complement with RL for the problems they're trying to solve.

Thanks for the link to the paper -- I will take a look.

In RL, the goal is to try to find a function that produces actions that optimize the expected reward of some reward function. Model-based RL methods typically try to extract a function for 'representing' the environment and employ techniques to optimize action selection over that 'representation' (replace the word 'representation' with the word 'model'). Model-free RL methods instead try to directly learn to predict which actions to take without extracting a representation. A good paper describing deep q-learning -- a commonly cited model-free method that was one of the earliest to employ deep-learning for a reinforcement learning task [1].

I think it's worth clarifying -- RL algorithms as a whole are more akin to search than to control algorithms. RL algorithms can be used to solve some control problems -- but that is not all they are used for unless you take an extremely broad view about what constitutes a 'control problem' ... I don't think it would be common to model playing 'go' as a control problem for example -- nor would I consider learning how to play all atari games ever created given only image frames and the current score and no other pre-supplied knowledge to be a control problem ...?

(I'm talking way passed my familiarity now) -- That said, optimal control theory intersects with RL quite a bit in the foundations -- Q-Learning techniques (a foundational family of methods in RL) have proofs that show under what conditions they will converge on the optimal policy -- I believe this mathematics to be quite similar to the mathematics used in optimal control theory...

[1] Deep Q-Networks https://storage.googleapis.com/deepmind-media/dqn/DQNNatureP...