This is one of my concerns ever since the first app asked for my contacts. You could be a mountain man and value your privacy.... but if you just know someone who uses tech you are exposed in any number of ways. In the US things like third-party doctrine have not aged well in the information age as just communicating with people can expose you in ways you can't ever control.
Blackberry android phones have a feature where it notifies you when any app accessed some resource that requires OS level permissions, such as contacts, gps, etc. Whatsapp and other facebook apps, access my contact list every 3 minutes.
Even the 3 minute delay is annoying - it's common I meet someone, add them to my phone, and then want to send them a Whatsapp message and they don't appear in the list. I have to awkwardly hang around for a minute or so before being able to send them the message I wanted to send.
(for context, in most of Europe, Whatsapp is used pretty much like iMessage or SMS is used in the USA - wouldn't it be annoying not to be able to SMS a new phone number without waiting 3 minutes?)