I wish that people referring to Japanese as "the most difficult language to learn..." could add the important caveat "...among major global languages". There are thousands of languages spoken on Earth, and some of them involve complexities that would daunt speakers of English or other Standard Average European languages more than Japanese. As just two examples, I remain daunted by the morphology of Skolt Saami and Ng…
The problem with Japanese kanji vs Chinese Hanzi is not just that Japanese has two (or more) common readings for every character, it’s that they don’t follow the phonetic element to nearly the same degree as Chinese.
In Chinese you really do have a fighting Chinese of reading a word out loud that contains never by you seen before characters.
In Japanese you will regularly come across names consisting only of known to you characters and you will still have no chance of knowing how to read them out loud.
But I totally agree with your main point: Japanese is not as difficult as it’s reputation will lead would be learners to believe but it does have some road blocks (mainly the writing system) that will stand in the learners way.