I have a bike helmet but only wear it for some rides - i.e. the long-distance or fast ones. When I ride into town for shopping, I don't wear it. I hope it will always remain my personal choice.
Anyway, the arguments are:
a) Mandatory bike helmets mean you can't run a bike-share scheme because who wants to wear a much-used sweaty helmet? How can the people running the scheme sanitise them?
b) Mandatory bike helmets put people off riding. They are inconvenient to carry around with you if you cycle to get somewhere (work, meeting, lunch date, etc) rather than just for recreation. It's estimated that 30% of cyclists (in Melbourne, I think) stopped cycling when helmets became mandatory. This is hailed as a success by politicians in Australia because cycling fatalities also fell by 30%. Miracle!
c) They probably don't protect you that much - anything over 12mph or a collision with something other than the pavement, and they're not guaranteed to do anything to help you. (Specialized helmets are made to the older Snell standard, which is a bit safer; I have a Specialized.)
d) They give the impression that cycling is super-dangerous and you need all sorts of safety gear, which puts people off. In some areas, it is dangerous (I'm thinking parts of the US which seem actively cyclist-hostile) but the risks of inactivity (heart disease, diabetes, stroke, etc) outweigh the very slight risk of traumatic head injury, in many parts of Europe at least. So, get people on bikes regardless, you'll save lives as people become fitter from cycling.
e) More of a political point, but the main danger of riding a bike comes from reckless or distracted motorists. Helmets represent masking the symptoms rather than dealing with the real problem. They also shift the blame from dangerous drivers to cyclists: how many times do we see descriptions of car-on-bike collisions with the stock phrase "the cyclist was not wearing a helmet"? Victim blaming, especially in cases where the cyclist was crushed to death by a 30-ton vehicle (little bit of polystyrene won't fix that, sorry).
The main ways I protect myself when riding are visibility (always use bright lights, my back light can be seen even in bright sunlight) and safe, assertive riding (e.g. taking the lane when necessary, approaching a junction for example). So many cyclists fail badly at safe riding that I think mandatory training would be better than mandatory helmets. But really I don't support either. For me, cycling is a safe activity and a helmet is usually not needed.