Not all pedophiles are child molesters, yes.
But more importantly, not all child molesters are pedophiles. Just like any other form of rape, a significant portion of cases has more to do with power than with attraction. Sexual abuse is abuse first.
The problem with child porn (in the strict sense of "pornography involving real children") is that all child porn is child abuse. So someone had to be abused to produce it and they depict actual victims in the act of being victimised. This is why possession is illegal, just as possessing other pictures of victims being victimised without the victim's consent is illegal (in countries with real privacy laws anyway) -- with the obvious addition that children can't give consent in the first place.
Of course the (understandable) moral panic doesn't help. While it's obvious why actual child porn should be illegal by default, it's not as obvious why the same goes for non-pornographic material (unless your definition of pornographic is simply "what someone could masturbate to", in which case you might as well extend it to everything).
An intuitive argument is that the proliferation of "child porn substitutes" (e.g. drawings, "naturalist" photography, etc) increases the risk that pedophiles might act on their drives. But considering that pedophilia is neither sufficient nor necessary for child abuse, that doesn't seem as clear cut. Also the numbers seem to indicate that for "normal" pornography, proliferation actually results in a decrease of sexual assaults (i.e. the "release valve" counter-argument).
Legalizing it all is obviously wrong. Banning it all (as in those parts of the world where you might become a registered sex offender just because a shop clerk finds out you took a photograph of your own child in an entirely non-sexual setting) is obviously wrong, too. But I don't think we can have a reasonable discussion about this issue as a society before we've learned to accept that morality isn't defined by arbitrary commandments and every human is different.