This is one of my concerns ever since the first app asked for my contacts. You could be a mountain man and value your privacy.... but if you just know someone who uses tech you are exposed in any number of ways. In the US things like third-party doctrine have not aged well in the information age as just communicating with people can expose you in ways you can't ever control.
This is why I don't use WhatsApp despite the fact I know many people who use it. To use the app, you have to give access to your entire contacts list. Even if I'm okay with them having my information, I don't feel comfortable consenting for those in my contacts list who don't use WhatsApp.
Fun fact, that has technically been illegal for a while here in Germany. There's quite a few videos / articles on how "Using whatsapp is TECHNICALLY illegal". Of course, nobody really complains, so no legal action was ever taken against any user, but it's still a good excuse for me to say "I'm not installing that. It's illegal."