Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Fast Deep Reinforcement Learning Course

courses.dibya.online

Post reply on HN
Print view View original

31–40 of 41 posts

Re: Show HN: Fast Deep Reinforcement Learning Course

#31

Post by pciexpgpu »

gh1 wrote: post #30
pciexpgpu wrote: post #27

Earlier quoted context omitted.

Thank you for doing this! I haven't looked deeply enough, but does this course use a higher-level 'package' such as OpenAI Gym or teach at a lower-level? (Is lower-level stuff even possible...)

I think the levels (high, low etc.) are relevant for the Deep RL algorithm, not the environment. The lower level version of OpenAI Gym canned environments would be custom Gym environments. I don't see much reason to go any lower than that. The situation looks different for Deep RL algorithm. You can implement them from scratch yourself using Tensorflow or any other similar library. Otherwise, you could just use a hig…

Great, yep, that is good to know.

Re: Show HN: Fast Deep Reinforcement Learning Course

#32

Post by gh1 »

InefficientRed wrote: post #29
gh1 wrote: post #10

Earlier quoted context omitted.

My experience matches yours. Recently, I was trying to solve an optimization problem using Deep RL. As usual, I had to run many experiments over several days using various tricks and hyperparameters. Finally, it turned out something related to the symmetry of the action space made a huge difference in learning. Anyhow, the experimentation stage requires a certain discipline and feels tedious at times. But the moment…

Have you ever encountered a situation where RL solved a (IRL "people paid me non-research-grant money for this") problem for you faster than classical controls engineering and/or planning? I have not.

Depends on what you mean by faster. Do you mean "time to solution" or "time to inference"? I think there are also more factors to take into consideration when considering the merit of the method e.g. performance, robustness, ability to handle non-linearity, ability to solve the full online problem etc.

When all these factors are taken into account, I have encountered situations where Deep RL performed better.

There are also very public examples of this e.g. Google's data center cooling [0] and competitive sailing [1].

[0] https://www.technologyreview.com/2018/08/17/140987/google-ju... [1] https://www.mckinsey.com/business-functions/mckinsey-digital...

Re: Show HN: Fast Deep Reinforcement Learning Course

#33

Post by InefficientRed »

gh1 wrote: post #32
InefficientRed wrote: post #29

Earlier quoted context omitted.

Have you ever encountered a situation where RL solved a (IRL "people paid me non-research-grant money for this") problem for you faster than classical controls engineering and/or planning? I have not.

Depends on what you mean by faster. Do you mean "time to solution" or "time to inference"? I think there are also more factors to take into consideration when considering the merit of the method e.g. performance, robustness, ability to handle non-linearity, ability to solve the full online problem etc. When all these factors are taken into account, I have encountered situations where Deep RL performed better. There a…

> Do you mean "time to solution" or "time to inference"?

I meant time to a real solution that works well enough to put into a product.

> There are also very public examples of this e.g. Google's data center cooling [0] and competitive sailing [1].

DeepMind really needed DRL wins on real problems.

McKinsy has a strong incentive to be able to say "we know all about the AI RL magic" (and all the better that it's in the context of an oligarchy's entry in a Rich Person Sport... such C-suite/investor class cred!)

In both cases, DRL was used because it was the right tool for the job. But, in both cases, proving DRL can be useful was the job! Go is a better example, but of course wasn't solving a real problem.

If you throw enough engineering time and compute at DRL, it can usually work well enough. (There is a real benefit to "just hack at it long enough" over "know the right bits of control theory".)

Re: Show HN: Fast Deep Reinforcement Learning Course

#34

Post by mywaifuismeta »

This looks nicely done, but for anyone interested I'd like to mention that these courses aren't something that can replace learning the fundamental concepts and theories behind ML/RL, for which there exist excellent books and courses that focus more on math and theory. I would go there.

These courses teach you how to call a library and use an API. You get nearly the same thing from just looking at the docs. Please don't say you "know RL" after this.

Re: Show HN: Fast Deep Reinforcement Learning Course

#35

Post by sul_tasto »

mywaifuismeta wrote: post #34

This looks nicely done, but for anyone interested I'd like to mention that these courses aren't something that can replace learning the fundamental concepts and theories behind ML/RL, for which there exist excellent books and courses that focus more on math and theory. I would go there. These courses teach you how to call a library and use an API. You get nearly the same thing from just looking at the docs. Please do…

Any references you have would be greatly appreciated.

Re: Show HN: Fast Deep Reinforcement Learning Course

#36

Post by mywaifuismeta »

sul_tasto wrote: post #35
mywaifuismeta wrote: post #34

This looks nicely done, but for anyone interested I'd like to mention that these courses aren't something that can replace learning the fundamental concepts and theories behind ML/RL, for which there exist excellent books and courses that focus more on math and theory. I would go there. These courses teach you how to call a library and use an API. You get nearly the same thing from just looking at the docs. Please do…

Any references you have would be greatly appreciated.

IMO the best intro book is Sutton's [1], it's extremely accessible (little math background needed) and covers all basic concepts. Work through David Silver's course (search youtube, it overlaps heavily with the book above) and then you are ready for something more advanced like [2] and you can start reading and implementing research papers.

[1] http://incompleteideas.net/book/the-book.html

[2] http://rail.eecs.berkeley.edu/deeprlcourse/

Re: Show HN: Fast Deep Reinforcement Learning Course

#37

Post by khalilravanna »

(Related but kind of off-topic)

I’m a software engineer (non-ML) currently working at big tech company that does ML and has a fair amount of open roles in ML and I’ve wondered is ML the sort of thing you could jump into a team and learn on the job? Or do you really need to take some courses, read some books, or even get a degree?

I got a CS/Math bachelors but it’s been nigh on a decade and my higher level math is rusty. Curious on people’s thoughts here.

Re: Show HN: Fast Deep Reinforcement Learning Course

#38

Post by leaflets2 »

khalilravanna wrote: post #37

(Related but kind of off-topic) I’m a software engineer (non-ML) currently working at big tech company that does ML and has a fair amount of open roles in ML and I’ve wondered is ML the sort of thing you could jump into a team and learn on the job? Or do you really need to take some courses, read some books, or even get a degree? I got a CS/Math bachelors but it’s been nigh on a decade and my higher level math is rus…

If it's neutral networks: I think to some extent you can self study and learn on the job, yes. You can learn how to construct and use neutral networks, and tweak and improve the structure and results, without understanding the maths underneath.

You'd need to (self) study & learn how to train a network, eg course or book or articles?

Hmm isn't that was this HN post is about :-) the course: https://courses.dibya.online/p/fastdeeprl, 4 hours it says, self study

I think often the most challenging part isn't the ML, but to gather training data and clean and prepare it so the ML has sth to learn from

Re: Show HN: Fast Deep Reinforcement Learning Course

#39

Post by gh1 »

mywaifuismeta wrote: post #34

This looks nicely done, but for anyone interested I'd like to mention that these courses aren't something that can replace learning the fundamental concepts and theories behind ML/RL, for which there exist excellent books and courses that focus more on math and theory. I would go there. These courses teach you how to call a library and use an API. You get nearly the same thing from just looking at the docs. Please do…

I think the idea is not to replace learning the theory or math, but rather to just postpone it. Learning the practical aspects of an engineering discipline can provide the necessary motivation to study the theory/math, and this is the oft ignored factor. I also think there is some benefit in learning a topic using the tool that you will actually use in production (if that is possible without adding unnecessary complexity in the syllabus).

I personally learned DRL from David Silver's course and Sutton & Burto back in the days. They were the only good resources around and I liked them very much. But I think that with the advent of high-level frameworks in DRL, there are better learning paths.

I do intend to teach the theory/math in a later installment of this series, but I wanted to do it by showing students how to implement the various classes of algorithms e.g. Q-learning (DQN/Rainbow), policy gradients (PPO) and model-based (AlphaZero) using RLlib. This would kill two birds with one stone: you can simultaneously pick up the theory/math and the lower level API of the tool that you will be using in the future anyway.

Re: Show HN: Fast Deep Reinforcement Learning Course

#40

Post by sul_tasto »

mywaifuismeta wrote: post #36
sul_tasto wrote: post #35

Earlier quoted context omitted.

Any references you have would be greatly appreciated.

IMO the best intro book is Sutton's [1], it's extremely accessible (little math background needed) and covers all basic concepts. Work through David Silver's course (search youtube, it overlaps heavily with the book above) and then you are ready for something more advanced like [2] and you can start reading and implementing research papers. [1] http://incompleteideas.net/book/the-book.html [2] http://rail.eecs.berkel…

this is exactly what i was hoping for- thank you!
Post reply on HN