So here's the deal. There are lots of people who don't understand enough about science to have a real opinion that spout off, but you also get people in the skeptic community that rail on about "chemophobia", when they know just enough to be smug. The reality is that there is a significant practical difference between natural and artificial substances. Specifically, organisms on earth have evolved to be able to tolerate substances commonly found in the natural environment. When we introduce new chemicals, organisms may be able to deal with them, or there may be negative effects. If I pick two random chemicals, one natural, and one artificial, and I'm forced to ingest one of them with no other information, I'm going to pick the natural one, because it's much more likely that ancestors of mine have seen them, and survived. Those organisms that didn't survive didn't get to become my ancestor.
Now of course this is just an indicator and not a rule, but it's a pretty good indicator if you've got nothing else to go on. If you told me that humans have been ingesting some particular chemical as food for millions of years, I'm going to feel fairly safe, whereas if you told me that some artificial chemical just passed FDA trials, I'm going to be much more skeptical. Many substances that have appeared even in full FDA drug trials to be benign had very negative consequences.
BPA is a perfect example. It has a very strong affinity for estrogen receptors. If BPA had been in the environment for millions of years, the estrogen receptor would have changed shape to not bind to BPA, but BPA is a new substance, so evolution hasn't had time to adapt to an environment with BPA in it.
Now, we get to how to rationally deal with all of this. We get a lot of benefit from artificial substances, but they can also cause harm. We have to have reasonable cost/benefit calculations. The FDA at least has something like a reasonable framework, where you go from phase I to II to III, and then postclinical surveillance, each step having a greater potential risk with more people under less supervision, and a greater potential reward, after we have data for a smaller group.
This sort of thing doesn't exist with industrial chemicals that aren't specifically designed to be ingested, and that's reasonable. If every industrial chemical had to go through FDA approval, we wouldn't have a modern world. However, even chemicals that aren't designed to be ingested do get ingested. For things like endocrine disruptors, even tiny quantities can have significant biological effects, and it's easy to ingest microgram quantities of any chemical we make millions of tons of per year. There should be a lot more scrutiny on chemicals that have endocrine effects, there should be a lot more scrutiny on chemicals that have contact with food or children are likely to put in their mouths, and there should be more scrutiny on chemicals that are produced in very large quantities. Chemicals like BPA and Phthalates hit all three.
The problem is that most of the people publicly involved in the debate are either the crazy hippie type that thinks everything should be banned, or the American Chemistry Council that thinks nothing should be restricted in any way. There is a large silent majority that wants reasonable regulation so that their boys don't have deformed penises.