I've been told by a friend of mine that speaks Japanese that the most difficult aspect is that how you speak changes due the social setting. It's not just the syntax, you must learn how to speak in each different situation.
Here is Richard Feynman talking about the same problem:
While in Kyoto I tried to learn Japanese with a vengeance. I worked much harder at it, and got to a point where I could go around in taxis and do things. I took lessons from a Japanese man every day for an hour. One day he was teaching me the word for "see." "All right," he said. "You want to say, 'May I see your garden?' What do you say?" I made up a sentence with the word that I had just learned. "No, no!" he said. "When you say to someone, 'Would you like to see my garden? you use the first 'see.' But when you want to see someone else's garden, you must use another 'see,' which is more polite." "Would you like to glance at my lousy garden?" is essentially what you're saying in the first case, but when you want to look at the other fella's garden, you have to say something like, "May I observe your gorgeous garden?" So there's two different words you have to use. Then he gave me another one: "You go to a temple, and you want to look at the gardens..." I made up a sentence, this time with the polite "see." "No, no!" he said. "In the temple, the gardens are much more elegant. So you have to say something that would be equivalent to 'May I hang my eyes on your most exquisite gardens?" Three or four different words for one idea, because when I'm doing it, it's miserable; when you're doing it, it's elegant.