Having lived in Japan for two years, what I'd add to this is to spend a lot of time walking, if physically able. For example, in my opinion the best way to explore Tokyo is to simply leave your hotel in the morning, wander around all day, and stop anywhere and everywhere that looks interesting. The excellent mass transit system means that it won't be a problem to get back. In more rural parts of Japan, there are sign…
Tokyo is absurdly gigantic, I would not suggest randomly walking it. Also the subway system can be challenging, and the staff don't speak English, so if you get stuck they likely won't be able to help. Another random thing about the train system: there are women-only cars. I didn't realize this until I had already boarded and (as a man) I was absolutely mortified.
As for walking, you don't need to walk _everywhere_ but it really is cool to spend a few hours like OP said and just walk around your hotels neighborhood or other places.
I usually stay in Higashi Shinjuku when I visit, and if I hadn't done that I wouldn't have known about this really quiet residential area that leads to a nice quiet park. Or the little Korea town neighborhood that's about a twenty minute walk away.