>I refuse to own any store loyalty cards because they give the store an exact list of what I buy, tied to my identity, which is a much bigger privacy threat.
You're engaging in self delusion if you think the almost uniformly cashless payment systems you embrace in Sweden don't already cause this to be the case about you down to the minutest level: with your payments, where you make them, what you buy, when, and a very fine-grained general profile of your daily activities and movements (paying digitally in any phyiscal location obviously ties you to said location at a certain place and time). It's almost absurd that you even mention concerns about "potential" privacy invasions while claiming the system you live in is in any way a good thing, just because it's convenient. None of this is to mention the problems it could cause if some sort of major disaster occurs or a data breach somewhere in the chain causes part or all of your entire history of payments to become commodity data should anyone ever want to use it against you in some way..
Not to mention the potential future ramifications of paternalistic or authoritarian government policies being used to "guide" your buying habits and personal life through a near total capacity to access your purchase history, and then tying this to regulatory "incentives". It would be all too easy for a government to implement something like this under the right circumstances if the entire consumer economy is already cashless and run through large institutions. That you think this isn't a problem indicates a deep blindness to just how important fundamental privacy via untraceable payments on a basic level is to any society that does value a modicum of breathing space for a sincerely private life.
Also, there's a vast margin of difference between some bank or other third party guessing where you shop with cash because of your address and them knowing exactly what you bought and where by (very easily) monitoring your digital payment activities.