> The right to not use my things exists everywhere, universally.
For physical rival [1] goods, yes. Not necessarily the same for intangible non-rival things (e.g. the text of a book, not the physical ink and paper). Copyright law creates a legal right of exclusive control over creative works, but to me there isn't a non-economic-related social right to exclusive control over creative works. In the US, fair use is a major limit on the legal aspect of copyright. The First Amendment's freedom of expression is the raison d'être of fair use. Most countries don't have a flexible exception similar to fair use.
> OR, I might agree it is fair use and not sue you. However, that weakens my standing on my copyright, so it's better for me to sue you
No, choosing not to sue over a copyrighted work doesn't weaken your copyright. It only weakens the specific case of changing your mind after the statute of limitations expires. The statute of limitations means that you have a time limit of some number of years (three years in the US) to sue, with the timer starting only after you become aware of an instance of alleged infringement. Copyright is not like trademark. You don't lose your copyright by failing to enforce it.
Furthermore, even though the fair use right can only be exercised as an affirmative defense in court, fair use is by definition not copyright infringement [3]:
> Importantly, the court viewed fair use not as a valid excuse for otherwise infringing conduct, but rather as consumer behavior that is not infringement in the first place. "Because 17 U.S.C. § 107[9] created a type of non-infringing use, fair use is 'authorized by the law' and a copyright holder must consider the existence of fair use before sending a takedown notification under § 512(c)."[1]
(Ignore the bracket citations that were copied over.)
[1] https://en.wikipedia.org/wiki/Rivalry_(economics)
[2] https://www.law.cornell.edu/uscode/text/17/507
[3] https://en.wikipedia.org/wiki/Lenz_v._Universal_Music_Corp.