I think the people that dismiss this concern are just as bad and unscientific. It's a pretty decent Bayesian prior to assume that regular exposure to synthetic organic compounds in quantities or concentrations that hominids didn't receive exposure to in our evolution are likely to be potentially problematic. This is especially true when they don't occur naturally in any organisms biochemical processes and yet are active enough to interact with many of these things. This is OBVIOUSLY true for things used as pesticides / herbicides. We have evolved a natural aversion to areas where all the plants and animals are dead and rotting. There are good reasons for this to be a really good heuristic.
I'll go so far as to say that almost any pesticide or herbicide is likely to be bad for vertebrates and invertebrates alike. This is really likely the case for perservatives as well, for what should be obvious reasons.
It's really not that crazy to assume they're probably not good as a default assumption.
Go into a hardware store and almost every chemical, solvent, paint, etc. that you encounter is not good for you. Eat a salmon and enjoy billions of plastic particles. Open almost any prepackaged food and you'll be ingesting all manner of dyes, perservatives, anti-caking agents, etc. etc. etc. that simply weren't around in your food environment during our evolution. It's a surprisingly good baseline assumption that these things aren't likely to be good for you.
If you think about the study design and epidemiologic studies, it should be clear that it's going to be very difficult to prove harm in a lot of cases for things that are only a little harmful, or only harmful in combination, or harmful only after 20 years have passed since exposure, etc. ...except that the science is VERY clear: something (or lots of things) associated with "processed food" is really bad for you.