Reinforcement Learning – Bandit Problems
oneraynyday.github.io
Reinforcement Learning – Bandit Problems
1–10 of 41 posts
Re: Reinforcement Learning – Bandit Problems
#2nice blog but you should include citations/references even if it is very informal
Re: Reinforcement Learning – Bandit Problems
#3I was heavily into reinforcement learning around the turn of the century, and at the time, "Reinforcement Learning - An introduction" (Barto and Sutton) https://mitpress.mit.edu/books/reinforcement-learning was an absolute goldmine for me getting started. I think parts of it are online somewhere including all their pseudocode and solutions.
Re: Reinforcement Learning – Bandit Problems
#4I'm surprised there was no octopus to represent the bandits.
In all seriousness, this post makes sense to me, as someone who does RL research. However, the intuition behind the concepts could be communicated more clearly. I would reason that this piece is less accessible to those who have much less knowledge of RL/bandits. Given that it's an introduction, I presume that's your intended reader, though perhaps writing can also be for your own edification. Who's your audience?
Re: Reinforcement Learning – Bandit Problems
#5I was heavily into reinforcement learning around the turn of the century, and at the time, "Reinforcement Learning - An introduction" (Barto and Sutton) https://mitpress.mit.edu/books/reinforcement-learning was an absolute goldmine for me getting started. I think parts of it are online somewhere including all their pseudocode and solutions. https://mitpress.mit.edu/books/reinforcement-learning
It's a fantastic book! The authors have been working on a second edition for a few years, and I think it's finally finished. A draft is generously available here: http://incompleteideas.net/book/the-book-2nd.html
Re: Reinforcement Learning – Bandit Problems
#6At the beginning of this post the given definition of value doesn't seem correct to me, because I think it should be the expected value of the sum of all the rewards from t to infinity and not of R_t, but I could be wrong. Citing D. Silver: The agent's job is to maximise cumulative rewards (highlight cumulative). I would suggest David Silver course for reinforcement learning, http://www0.cs.ucl.ac.uk/staff/d.silver/web/Teaching.html, another interesting post is https://jeremykun.com/tag/bandit-learning/ and the following (5 related post comparing implementations of the algorithms in python).
I know I should not discourage people, but you should read the post Deep reinforcement learning doesn't work yet before drinking all the cool aid: https://www.alexirpan.com/2018/02/14/rl-hard.html
Edited many times.
Re: Reinforcement Learning – Bandit Problems
#7As someone who is doing his bachelor thesis on Reinforcement Learning this is some useful information.
OT (but not really) question: does anyone here use Reinforcement Learning techniques at work? For the thesis I am working on black-box optimization of 2 variable functions with Reinforcement Learning (and comparing it with Bayesian Optimization techniques).
As someone else suggested the Sutton & Barto book is really great knowledge but I would also like to suggest these lessons (https://www.youtube.com/watch?v=2pWv7GOvuf0) by David Silver (who worked on AlphaGo)
Re: Reinforcement Learning – Bandit Problems
#8At the beginning of this post the given definition of value doesn't seem correct to me, because I think it should be the expected value of the sum of all the rewards from t to infinity and not of R_t, but I could be wrong. Citing D. Silver: The agent's job is to maximise cumulative rewards (highlight cumulative). I would suggest David Silver course for reinforcement learning, http://www0.cs.ucl.ac.uk/staff/d.silver/w…
You are right, it should be the expected value for the sum of all future rewards.
At the same time, given that we're talking about bandits, it doesn't really matter, since there is no state. Thus, the summatory over time you'd like to see doesn't change the relative ordering of the actions: the expected value in your definition is simply the expected reward for the action multiplied by the number of times you expect to play. So it doesn't really change anything.
Re: Reinforcement Learning – Bandit Problems
#9I was heavily into reinforcement learning around the turn of the century, and at the time, "Reinforcement Learning - An introduction" (Barto and Sutton) https://mitpress.mit.edu/books/reinforcement-learning was an absolute goldmine for me getting started. I think parts of it are online somewhere including all their pseudocode and solutions. https://mitpress.mit.edu/books/reinforcement-learning
I was wondering how you were over 120 years old there, for a moment, then I realized turn of the century doesn't mean that century any more.
Re: Reinforcement Learning – Bandit Problems
#10I was heavily into reinforcement learning around the turn of the century, and at the time, "Reinforcement Learning - An introduction" (Barto and Sutton) https://mitpress.mit.edu/books/reinforcement-learning was an absolute goldmine for me getting started. I think parts of it are online somewhere including all their pseudocode and solutions. https://mitpress.mit.edu/books/reinforcement-learning
The complete first edition can be found here: http://incompleteideas.net/book/ebook/the-book.html
If you're interested in some well documented C++ implementations of the algorithms shown in the book, feel free to check out https://github.com/Svalorzen/AI-Toolbox. I started the project because when I was first reading the book I had no reference implementation to compare the book to, and personally I learn better with practical examples, so maybe it can help you too.