There -is- a difference of natively learning a language and learning it later on as EFL/ESL. It has nothing to do with politics, or being perfect. In fact, many ESL/EFL speakers have better grammar and vocabulary than the average native speaker because they have to focus on those things in order to learn the language.
However, a native speaker won't struggle in speech, and they'll know the various idioms used all the time in colloquial speech. Of course native speakers don't have to take the TEFL/TOEFL! They don't consciously think about grammar rules, they just employ them nearly always perfectly in speech. Of course not every native speaker is good at writing at an academic level and we should be aware of that.
This phenomenon is also not unique to English! This article and the one linked are basically following the trend of bashing one's own English; how dare native speakers speak their language like they have been their entire life? Part of learning any language is learning the cultural idiosyncrasies and idioms. Of course, native speakers should strive to make sure they're not overusing idioms with a EFL/ESL audience, but knowing "grammar" and "complex technical terms" doesn't actually mean you can speak [American/UK/Indian/Black American/etc] English.
Humans are just curious when someone has a different accent than the local area. There are various American accents. I've had that question asked a lot when I was living in Korea. Getting upset about it is entirely a personal choice.