In the last century the landscape of human health problems has pretty drastically shifted (more cancers, more obesity, more asthma, perhaps more mental illness, lower rates of maiming, less malaria and consumption deaths, and frankly this is all just deaths, which is not the only kind of health outcome we care about) so the fact that we can't suss out all the cause and effect from every factor and every outcome is neither surprising nor an argument against it being alarming any time a novel foreign substance can penetrate cell membranes and readily bioaccumulates
Like we study mechanisms for a reason. "Little things get into all your cells" isn't a specific causal argument for specific health outcomes, but it's reasonable to assume it might do something undesirable based on what we know about how cells work and other cases like it. Also, the intervention space is pretty large and some scaling back on plastic usage from the current levels would not be some crazy obvious harm either
Also, "we haven't found effects yet" is especially troublesome in a case where the contaminant in question is a cash cow for an industry that has continuously spent more money than I can even meaningfully reason about suppressing science and muddying discourse about products they sell. Sowing doubt has been an explicit strategy of the petroleum industry for greenhouse effects and it took a long time for us to even definitively know that, so now that we do, why should I assume the same companies aren't taking the same approach for this problem? Again, it feels like isolated demands for rigor