This is solved with a resume.
The short answer is "no, they won't care generally", but employers may care a little bit very early in your career and that level of caring will eventually fade to zero. In order for them to care, they first have to actually know. We can prevent that rather easily.
The secret to counteracting this is done on your resume typically (I'm a resume writer and recruiter, who coincidentally also graduated college after 6 years due to self-financing).
Listing college dates on a resume doesn't have any real rules. Early in your career you will likely list your graduation date, and perhaps even some coursework, projects, and a GPA. There is no need to list the date you started college.
At some point in your career your resume will change. You might not even list graduation date at all down the road, in order to prevent ageism. You'll just have your degree, major, and university - no other details. You also won't list your GPA at some point, and you don't even need to list it now. If they want to know, they'll ask.
If you're asked about college attendance, you should be honest and tell them how long it took and why. But usually they won't ask, unless there is something else on your resume that tips them off.