Live data from Hacker News

Solving Path of Exile Item Crafting with Reinforcement Learning

dennybritz.com

11–20 of 20 posts

Re: Solving Path of Exile Item Crafting with Reinforcement Learning

#11
post #9

Earlier quoted context omitted.

Thank you Exile :) I'm glad you enjoyed the read!

Is the code available by chance? See you on the shore next Friday :)

I do want to release the code, but it may take a couple more weeks. It's currently a mess of research code that's mixed with some other stuff I was working on so I would need to do a cleanup first. See you on the boat!

Re: Solving Path of Exile Item Crafting with Reinforcement Learning

#12
post #10

> does something like the following pseudocode: The following code is Python. Hilarious!

I used to not like Python, but i gave it another try thanks to Sublime Text's plugin api, and i now love it, it allows you to be very creative very fast, perfect for prototyping

Pseudo code -> working example, same thing lol

Re: Solving Path of Exile Item Crafting with Reinforcement Learning

#13
post #9

Earlier quoted context omitted.

Is the code available by chance? See you on the shore next Friday :)

I do want to release the code, but it may take a couple more weeks. It's currently a mess of research code that's mixed with some other stuff I was working on so I would need to do a cleanup first. See you on the boat!

Wait a couple weeks until I'm done with the league before you destroy the economy.

Re: Solving Path of Exile Item Crafting with Reinforcement Learning

#14

Chess also has a huge search space and allows cycles, still, wouldn't a better approach be refining a scoring function and then doing a deep search and suggest an action and min maxes your odds of victory?

I think there are some key differences to cycles in Chess. In Chess, the cycles come from your (or your opponents) actions. With minimax you would max (or min) over these actions, so if you pruned away the cycles at some depth your result is not incorrect because the min or max does not change. But here the cycles come from randomness in the game of which chess has none, and you need to take an expectation over the c…

I think the heuristic approach is not as bad as you think. Human crafters, and the current crafting techniques we have are pretty efficient. You don't need to explore the full graph of random states because the mod weights themselves are a representation of those states. As a human I can't think of any crafting item (alt and chaos spamming aside) where the probability isn't a simple to understand number. You think you can use something like modweight x currency cost (x some time modifier for acquisition and use of currency) for scoring. This is how craftofexile does it.

Re: Solving Path of Exile Item Crafting with Reinforcement Learning

#15

Earlier quoted context omitted.

I think there are some key differences to cycles in Chess. In Chess, the cycles come from your (or your opponents) actions. With minimax you would max (or min) over these actions, so if you pruned away the cycles at some depth your result is not incorrect because the min or max does not change. But here the cycles come from randomness in the game of which chess has none, and you need to take an expectation over the c…

I think the heuristic approach is not as bad as you think. Human crafters, and the current crafting techniques we have are pretty efficient. You don't need to explore the full graph of random states because the mod weights themselves are a representation of those states. As a human I can't think of any crafting item (alt and chaos spamming aside) where the probability isn't a simple to understand number. You think yo…

The more I think about this the more I feel you went really overkill on this. The complexity is several orderers of magnitude lower than what you claim. Like try crafting stuff on the emulator and see how easily you can narrow it down to usually 2 or 3 choices at most steps. There should be really easy heuristics to invalidate most branches. I've also used the COE simulator a ton and never had to model something that's on the order of 50+ states even.

Re: Solving Path of Exile Item Crafting with Reinforcement Learning

#16
This is awesome! I don’t really play PoE anymore but was addicted to the game back in open beta/v1 launch. I still have my chaos orb key chain I got from my donor pack, unfortunately I lost my shirt somewhere. It’s to this day imo the best ARPG out there. I love seeing RL applied to this game because it is seriously complex! Nicely done and would love to see the full code at some point!

Re: Solving Path of Exile Item Crafting with Reinforcement Learning

#17
It’s often associated in PoE that friction is what usually drives profits. Using a recent meta example, Graveyard crafting was extremely tedious while simultaneously being extremely lucrative. I am quite interested then to see what the implications on the economy are if adopted mainstream by the masses. People nowadays would rather pay 3x the prices of the craft rather than learning how to do it themselves, but if there was a convenient calculator that could derive the optimal algorithm I suspect we will see a different behavior emerge.

Re: Solving Path of Exile Item Crafting with Reinforcement Learning

#18
post #17

It’s often associated in PoE that friction is what usually drives profits. Using a recent meta example, Graveyard crafting was extremely tedious while simultaneously being extremely lucrative. I am quite interested then to see what the implications on the economy are if adopted mainstream by the masses. People nowadays would rather pay 3x the prices of the craft rather than learning how to do it themselves, but if th…

I think most experienced players are already pretty optimal at crafting. There might be some rare case where it might save you a couple div at best. It's usually obvious by design how to achieve something. Of course there is a market because some part of the playerbase can't do this (which might become lower) and some part of the playerbase still sees it as time efficient to just buy it precrafted. Likely there's less incentive to craft in bulk, large bulk currency trades become more expensive, making the crafted items more expensive for the unkowing new player who doesn't use this tool. Personally, I don't think it would change the economy that much.

Re: Solving Path of Exile Item Crafting with Reinforcement Learning

#19
post #17

It’s often associated in PoE that friction is what usually drives profits. Using a recent meta example, Graveyard crafting was extremely tedious while simultaneously being extremely lucrative. I am quite interested then to see what the implications on the economy are if adopted mainstream by the masses. People nowadays would rather pay 3x the prices of the craft rather than learning how to do it themselves, but if th…

I think most experienced players are already pretty optimal at crafting. There might be some rare case where it might save you a couple div at best. It's usually obvious by design how to achieve something. Of course there is a market because some part of the playerbase can't do this (which might become lower) and some part of the playerbase still sees it as time efficient to just buy it precrafted. Likely there's les…

Agreed, and I think my point around adoption by the masses was made a little too lightly - as you mentioned there will always be a part of the player base who prefer buying and playing the game rather than spending time crafting (regardless of how trivial the craft is).

Re: Solving Path of Exile Item Crafting with Reinforcement Learning

#20
post #17

It’s often associated in PoE that friction is what usually drives profits. Using a recent meta example, Graveyard crafting was extremely tedious while simultaneously being extremely lucrative. I am quite interested then to see what the implications on the economy are if adopted mainstream by the masses. People nowadays would rather pay 3x the prices of the craft rather than learning how to do it themselves, but if th…

It gets more interesting if you optimise for time instead of cost. If it takes you 15 minutes to craft something using 80c of currency, but you can instead run maps and average 300c an hour, then you still might prefer paying 100c for the guaranteed outcome (buying it already made) than going for "average" cost of 80c and having to craft it yourself.

Do it again, but optimise for fun instead of time and we've come full-circle :)

Post reply on HN