I've been thinking of a similar system, actually, but instead of having each end user pay per transaction, you have each domain put up a bond (in a digital currency) which can be forfeited if the domain is found guilty of spamming.
The one controversial part of this is deciding who is permitted to redeem the bond, i.e. who is the central arbiter of what is and isn't spam? Well, fortunately we already have an entity like that, de facto, since we have the Big 4 email providers ("Gmail, Hotmail, Yahoo, and AOL", or any other similar group that all mail servers must, in practice, comply with the demands of).
In the system I'm proposing, if any of these big providers detect a domain being involved in spam (or N out of M providers, to reduce the risk of false positives), they could redeem the bond. Of course, the bond would be structured such that the only recipient was a non-profit, like the IETF, or possibly an entity like ICANN. That way there is no financial incentive to make false positive claims.
As for how this system would be introduced, that's the easy part. All pre-existing domains would be grandfathered in, so that no current email users would be negatively affected. (Indeed, there is a moral hazard that some email providers would support this system precisely because it only affects new entrants to the market). There would simply be a flag day, after which if you register a new domain, and want to send emails from it, your domain registrar would have a check box saying "Yes, I am happy to be charged an extra $10 and have put it in a bond so that I can send email to the Big 4 providers". After sending a certain amount of legitimate email, a Big 4 provider could then send a transaction which makes the locked funds return to the original issuer of the bond.
Finally, to decide how big the bond has to be, we can look at data like this:
https://krebsonsecurity.com/2018/06/bad-men-at-work-please-d...
If spammers are going to extra effort to save a few dollars on their registration costs, then their margins must be pretty tight. It also means that domain reputation systems are working, since spammers have to keep bulk buying new domains:
http://conferences.sigcomm.org/imc/2013/papers/imc247-haoA.p...