Spinning Up in Deep RL
blog.openai.com
Spinning Up in Deep RL
1–10 of 20 posts
Re: Spinning Up in Deep RL
#2This developed-and-maintained package is a good approach towards furthering RL development; as the writeups state, the biggest problem in RL is subtle bugs from an implementation which don't cause an error but tank learning performance. (+ loggers/utils to help debug things)
Granted, a lot of RL thought pieces/examples on places like Medium.com take an existing RL implementation without many tweaks, run it on a new task, and see what happens. A better RL library might make this workflow more prevalent; hence why it's very important for researchers to make their pipelines transparent.
Re: Spinning Up in Deep RL
#3This developed-and-maintained package is a good approach towards furthering RL development; as the writeups state, the biggest problem in RL is subtle bugs from an implementation which don't cause an error but tank learning performance. (+ loggers/utils to help debug things) Granted, a lot of RL thought pieces/examples on places like Medium.com take an existing RL implementation without many tweaks, run it on a new t…
[deleted]
Re: Spinning Up in Deep RL
#4This developed-and-maintained package is a good approach towards furthering RL development; as the writeups state, the biggest problem in RL is subtle bugs from an implementation which don't cause an error but tank learning performance. (+ loggers/utils to help debug things) Granted, a lot of RL thought pieces/examples on places like Medium.com take an existing RL implementation without many tweaks, run it on a new t…
I've made some effort to provide a set of similar high-quality implementations available in PyTorch: https://blog.millionintegrals.com/vel-pytorch-meets-baseline...
In my opinion PyTorch code is easier to understand and debug for newcomers. Code is definitely lacking in documentation, but whenever there was a tradeoff between clarity and modularity in the end I've chosen modularity. Ideally I would like others to be able to take bits and pieces and incorporate into their projects to speed up time to delivery of their ideas.
Re: Spinning Up in Deep RL
#5whenever I see high quality submissions I bookmark it and promise myself to come back and spend time learning it.
this time...I promise myself its different
Re: Spinning Up in Deep RL
#6This is awesome and I hope will allow more people to experiment with algorithms, instead of only re-applying OpenAI's baselines. Baselines are great, but are very hard (for me, at least) to tinker with.
It helps me to understand something new if I can controllably break it. In other words, I progress by predicting the edge-conditions when something shouldn't work - and then testing if algorithm indeed experienced expected type of failure. Transparent algorithm implementation is key for this.
One thing, which I immediately checked in the spinningup-repo is if it uses TF Eager. And it doesn't. @OpenAI what's your reasoning for that?
Re: Spinning Up in Deep RL
#7This is awesome and I hope will allow more people to experiment with algorithms, instead of only re-applying OpenAI's baselines. Baselines are great, but are very hard (for me, at least) to tinker with. It helps me to understand something new if I can controllably break it. In other words, I progress by predicting the edge-conditions when something shouldn't work - and then testing if algorithm indeed experienced exp…
Hi! Primary developer for Spinning Up here. The code for this was developed mostly in June and July this year, and Eager still felt relatively new to me. I wanted to wait for Eager to stabilize and hit maturity before investing in it. I also wanted to see how TF would change on the road to TF 2.0, since that could change the picture even more.
At the six month review in 2019, we'll evaluate whether it makes sense to rewrite the implementation examples for TF 2.0. I'll speculate that the answer will be "yes, it does." Since Eager execution will be a central feature of TF 2.0, the (probable) revamp for Spinning Up will include it.
Good luck with your experiments! And please let us know about your experience with Spinning Up---we want to make sure it fulfills the mission of helping anyone learn about deep RL, and user feedback is vital for that.
Re: Spinning Up in Deep RL
#8This developed-and-maintained package is a good approach towards furthering RL development; as the writeups state, the biggest problem in RL is subtle bugs from an implementation which don't cause an error but tank learning performance. (+ loggers/utils to help debug things) Granted, a lot of RL thought pieces/examples on places like Medium.com take an existing RL implementation without many tweaks, run it on a new t…
I've made some effort to provide a set of similar high-quality implementations available in PyTorch: https://blog.millionintegrals.com/vel-pytorch-meets-baseline... In my opinion PyTorch code is easier to understand and debug for newcomers. Code is definitely lacking in documentation, but whenever there was a tradeoff between clarity and modularity in the end I've chosen modularity. Ideally I would like others to be…
+1 on that, that's a great project.
PyTorch with its explicit state that can be easily examined by hand in PyCharm debugger will be way easier for people coming into the field.
Re: Spinning Up in Deep RL
#9This is awesome and I hope will allow more people to experiment with algorithms, instead of only re-applying OpenAI's baselines. Baselines are great, but are very hard (for me, at least) to tinker with. It helps me to understand something new if I can controllably break it. In other words, I progress by predicting the edge-conditions when something shouldn't work - and then testing if algorithm indeed experienced exp…
Hi! Primary developer for Spinning Up here. The code for this was developed mostly in June and July this year, and Eager still felt relatively new to me. I wanted to wait for Eager to stabilize and hit maturity before investing in it. I also wanted to see how TF would change on the road to TF 2.0, since that could change the picture even more. At the six month review in 2019, we'll evaluate whether it makes sense to…
Thank you for sharing your thought process!
Re: Spinning Up in Deep RL
#10Discovered two small issues in the doc. Where can I send feedback?