Earlier quoted context omitted.
While it's true that people that want to understand you would understand you, even if you speak wrong, it's not something "not to worry about". Using the wrong gender or articles quickly gives a hint that you are some lowly educated factory worker living in Germany for 20 years and not caring to learn the language.
That's true but I'd say it's less about appearing lowly educated than about reinforcing your foreign-ness. A lot of Germans are to varying degrees prejudiced against foreigners, especially if you're brown or can be read as Turkish/Arabic or Eastern European. If you pass as white European you're more likely to be read as educated but still learning the language whereas otherwise your language problems are more likely…
It's the general bad practice of putting individuals into larger groups with a stigma based on visual appearance.
I think a lot of the resentment about bad German skills in my generation (80s/90s) comes from 10+ year residents and 2nd generation immigrants not capable of speaking our language correctly.