In my observation of non-native speakers working & living in the US (a group to which I belong), there's also something special about number-words. If you are a non-native speaker, even if you boast near-perfect English pronunciation and fluency, and are accustomed to using English in all aspects of your life, not just for communication but for e.g. personal note-taking, and even if you haven't spoken/written your na…
I'm not sure if it's related but a similar thing happens with spelling. I guess because there's some ordinal relation to the letters in a word? But I feel whenever I need to spell my name in English it takes me about the same mental effort as giving my phone number, and I'm otherwise rather fluent in that language.