Earlier quoted context omitted.
The bad news is that if a black hole is not primordial, the minimum mass is high enough (2-4 M_sol) that it doesn't so much "pass through our system" as "we pass through its". I'm not 100% confident, but rough approximation suggests that a close approach of even 5 AU even at typical interstellar velocities might have an unplesant climate impact by increasing the eccentricity of our orbit.
Did you mean 5 AU or 5 ly? 5 AU at 2-4 M sol intuitively seems like a complete and utter havoc.
5 AU is the perfect distance to really wreck Jupiter, though.