Japanese is likely to have been a hybrid language, somewhat similar with many European languages that had both a substrate and a superstrate, e.g. a Romance language like French had a Celtic substrate and a Germanic superstrate.
However, in the case of such European languages the 3 combined languages were not radically different, but they belonged to the same great language family, only to different branches. For Japanese, its sources have come from completely unrelated language families, which is the probable cause of the difficulties in determining the affinities of Japanese.
The grammar of Japanese is very similar to its Western neighbor, i.e. Korean, while its phonology is very similar to its Southern neighbor, i.e. the Austronesian languages of Ancient Taiwan and Philippines.
On the other hand, for the vocabulary of native Japanese, before it incorporated the huge amount of borrowings from Chinese, it has been more difficult to find relationships with other languages. Besides the Southern and Western influences, Japanese was also affected by a Northern influence, from people related to Ainu. As there are no old enough recorded sources about languages related to Ainu, it is possible that many of the words that do not appear to have a Southern or Western source may have come from a Northern contribution to the Japanese language.
I did not find any linguistic publication that does an adequate analysis of the relationships of Japanese with other languages. To be fair, such an analysis would require a huge amount of work, because unlike for Indo-European and Afro-Asiatic languages, where a large amount of texts have been preserved from several millennia ago, when the evolution of the languages had not changed most words so much as to make their correspondences in related languages unrecognizable, for Japanese many of the languages related to those which have contributed to the formation of Japanese have probably disappeared before leaving any written records. A credible analysis of the possible relationships of Japanese would require the compilation of a great amount of information about poorly documented languages, in order to try to reconstruct their earlier stages, where similarities with Old Japanese could be identified.
Korean has old written records, but only about as old as Japanese itself, so those are not very helpful to reconstruct the stage from many centuries before, which could have provided a component of Japanese. A language related to Korean appears to have contributed to Japanese, but only as a late superstrate that has applied a new grammar on the vocabulary inherited from the previous inhabitants of the islands. The language providing this superstrate was probably the language of the Yayoi people, who immigrated in Japan more than two thousand years ago.
For the Southern and Northern languages that could have contributed to the vocabulary and phonology of the language of Japan before the Yayoi immigration, there are extremely low chances of becoming able to reconstruct them as they were a few millennia ago, so it is unlikely that the origin of Japanese will ever be known with certainty.
Still, the fact that the languages that share features with Japanese are exactly its former neighbors in the 3 directions besides the Ocean (from before Taiwan became Chinese), is not surprising at all, but it is exactly what would be expected. What are not known are the details of what exactly each source has contributed and when did this happen.