My concerns about a cashless society are the power it hands to a few oligopolistic credit card processors. Look at what happened to Pornhub. Regardless of whether you think Pornhub was in the wrong, it is striking that a business was cut off from global financial system by a small number of credit card companies. This was done without due process and under no countries laws. That should scare us at least a little.
edit:
Wikileaks to sue Visa, Mastercard over "financial blockade"
https://www.cbsnews.com/news/wikileaks-to-sue-visa-mastercar...