Solving Path of Exile Item Crafting with Reinforcement Learning
1–10 of 20 posts
Re: Solving Path of Exile Item Crafting with Reinforcement Learning
#2Wow. Absolutely incredible. I have a truly shameful amount of time playing PoE, crafting by jumping between craftofexile, poedb, spreadsheets, etc. I was always curious if something like this was possible and you’ve gone and done it. Fantastic read as well, thank you for sharing.
Re: Solving Path of Exile Item Crafting with Reinforcement Learning
#3Wow. Absolutely incredible. I have a truly shameful amount of time playing PoE, crafting by jumping between craftofexile, poedb, spreadsheets, etc. I was always curious if something like this was possible and you’ve gone and done it. Fantastic read as well, thank you for sharing.
Thank you Exile :) I'm glad you enjoyed the read!
Re: Solving Path of Exile Item Crafting with Reinforcement Learning
#4Great read, I followed it the best I could with my vast poe knowledge and zero knowledge about the techniques you used. I got the gist of it I think, well done.
Love that poe attracts nerds from many domains and people come up with amazing stuff while impatiently waiting for the next season :)
Stay sane, Exile.
Re: Solving Path of Exile Item Crafting with Reinforcement Learning
#5Chess also has a huge search space and allows cycles, still, wouldn't a better approach be refining a scoring function and then doing a deep search and suggest an action and min maxes your odds of victory?
Re: Solving Path of Exile Item Crafting with Reinforcement Learning
#6Chess also has a huge search space and allows cycles, still, wouldn't a better approach be refining a scoring function and then doing a deep search and suggest an action and min maxes your odds of victory?
I think there are some key differences to cycles in Chess. In Chess, the cycles come from your (or your opponents) actions. With minimax you would max (or min) over these actions, so if you pruned away the cycles at some depth your result is not incorrect because the min or max does not change. But here the cycles come from randomness in the game of which chess has none, and you need to take an expectation over the cycles. So you can't just prune them away without changing the result.
A scoring function could definitely help guide the exploration and/or prune the tree, but only at the action nodes, not the environment randomness nodes. Rolling out the full tree more than 1-2 levels would be infeasible because of the randomness in the environment. When you take an action, the randomness can transport you into an exponential number of states, so you have a huge branching factor that is much larger than chess. I think in chess you have a factor of 40ish? Here it's more like ~1000 or ~10,000 depending on the item.
I also wouldn't know how to design a scoring function for this. If you do something simple to take the number of missing modifiers you will end up stuck in bad states. Maybe there is something really clever here that you can do, but I don't know what it is.
If you have an idea how to make tree search work for this I'd love to try it.
Re: Solving Path of Exile Item Crafting with Reinforcement Learning
#7Chess also has a huge search space and allows cycles, still, wouldn't a better approach be refining a scoring function and then doing a deep search and suggest an action and min maxes your odds of victory?
I think there are some key differences to cycles in Chess. In Chess, the cycles come from your (or your opponents) actions. With minimax you would max (or min) over these actions, so if you pruned away the cycles at some depth your result is not incorrect because the min or max does not change. But here the cycles come from randomness in the game of which chess has none, and you need to take an expectation over the c…
Just a remark about the cycles not changing the min-max.
That's not correct. Identical positions do not always have the same value.
Sometimes, a reaching the same position again will allow a player to claim a draw (3 times repetition). Sometimes a repeated position will bring them closer to some other draw mechanism (fe 50 moves without a capture or pawn move).
Re: Solving Path of Exile Item Crafting with Reinforcement Learning
#8Chess also has a huge search space and allows cycles, still, wouldn't a better approach be refining a scoring function and then doing a deep search and suggest an action and min maxes your odds of victory?
I think there are some key differences to cycles in Chess. In Chess, the cycles come from your (or your opponents) actions. With minimax you would max (or min) over these actions, so if you pruned away the cycles at some depth your result is not incorrect because the min or max does not change. But here the cycles come from randomness in the game of which chess has none, and you need to take an expectation over the c…
However, reading this, It seems like a good fit to try a genetic algorithm. I know, GAs are regarded as a complete backwater in AI these days, but nature has shown that they work in these kind of situations.
Re: Solving Path of Exile Item Crafting with Reinforcement Learning
#9Wow. Absolutely incredible. I have a truly shameful amount of time playing PoE, crafting by jumping between craftofexile, poedb, spreadsheets, etc. I was always curious if something like this was possible and you’ve gone and done it. Fantastic read as well, thank you for sharing.
Thank you Exile :) I'm glad you enjoyed the read!
Is the code available by chance?
See you on the shore next Friday :)
Re: Solving Path of Exile Item Crafting with Reinforcement Learning
#10> does something like the following pseudocode:
The following code is Python. Hilarious!