> The law is based on the doctrine of unclean hands. Judges decide on a case by case basis, based on the facts pertaining to the case.
Judges don't scale and they are a step function, not an incentive gradient.
> An artist finds their life’s work online, and issues a takedown notice, not realizing it contained a few songs they don’t notice? No big deal.
If he had to pay a small(!) fee for mistakes in his takedown it would mean little harm to him since he protected his works with one takdown. One could even add an exemption for first-time mistakes. If he has to do it repeatedly then one would hope that he improves his search process.
Additionally you have to consider that takedown notices can be filed against middle-men targeting many different 3rd parties at once. E.g. consider URL takedowns submitted to google. Google has no dirty hands and neither do some of the mistaken targets.
Small fees would only add up if you issue bogus takedowns in bulk. They also have the advantage of not burdening the courts and providing the right incentives. The fact that the cost exists would drive people to reduce false positives which would make the penalty being required less often.
As far as fees go one could also distinguish between fair use disputes vs. wholly incorrect claims. After all in the former case the claimant has a valid case and it is just that there exists a valid defense. In the latter case the claim itself was invalid.
> A label pays a takedown notice mill, that produces completely bogus / mistargeted takedown requests on a regular basis? Scorched earth. They lose the ability to enforce any copyrights on their catalog.
That is not what is happening in reality.