While endocrine disruption or inhibitors are very likely a factor to blame, I think we also have cultural and socioeconomic factors compounding the problem. There are so many who just don't want to deal with the current dating and casual sex culture. The rewards for time investment and effort just aren't that high compared to alternatives available to many, compounded by other factors of daily life competing for that time.
In heterosexual groups, it's challenging for many men (from my experience) who are interested in casual sex but the effort is simply too high for them to pursue so they largely just check-out (I know quite a few heterosexual males who just feel like it isn't worth their time anymore). In other groups like the LGBTQ+ arena, specifically gay men, it's not nearly as difficult to find the sexual reward many seek and as a result, you'll find casual sex seems quite high. I have quite a few friends here as well and they continue to seek and find causal sexual encounters, dates, etc. or at least they say.
I'd be very interested to see if any empirical data exists that shows casual sex rates over time, normalized, and broken into heterosexual and non-heterosexual groups. If there's a similar decline in both groups then I suspect endocrine disruption may be the real problem. If there's less correlation, I'd tend to think much is culturally driven.