Earlier quoted context omitted.
As a Japanese, I agree with you. When friends who are learning the language ask me questions, I sometimes can't even answer them. So I end up saying "that's just how we say/use it". And much respect from me. I wouldn't learn Japanese if my mother tongue is European language.
I've tried twice to learn Japanese and I consistently get stuck on kanji. The spoken language is actually really intuitive in a lot of ways, and hiragana/katakana fit easily into my brain, but kanji, with the multiple and inconsistent readings, is just mind-blowingly hard for me. And it's a shame that I keep getting stuck there as it really is a beautiful language.
On a slight side note, kids can't even write their own names in kanji. You can imagine how happy we got when we were taught kanjis that were part of our name. So you shouldn't feel bad about it, taking small steps serve you well in the long term.