One reason why Chinese have less difficulty remembering and using these numbers than the western fellows might be the difference in pronunciation. In Chinese, all single digit number has the same structure: a consonant plus a vowel. For example, 7, in Chinese is pronounced as [ch-i:], while in English is pronounced as [ˈsɛvən]. Also, every digit takes the same time to pronounce. For example, a string "123456" is [yi:…
I think there is a cultural bias in the U.S. against remembering numbers, but that people are generally able to do so. Memorization tends to happen naturally through use: it's easier for me to remember a bank account number after making deposits over a few months than by purposefully memorizing it, for instance.