Earlier quoted context omitted.
I went to Japan in the 90s. The amount of leftover Nazi symbols (carved everywhere, marked on city maps where old depots were, etc) and casual use of Nazi references in literature was shocking.
You sure those weren't actually Manji? They mark where temples are. https://www.bbc.com/news/world-asia-35349619
Any inhabitant of the temples will explain how german visitors would (infrequently) vandalize the temples. Not the intentional decoration (which is obvious) but the crude carvings into random wooden walls. I'm a little surprised that you can no longer find images of these carving after a few minutes of searching on the web. That's unfortunate, but I stand by my observations.