Does anyone remember a meta study that showed ANY even slightly reasonable restrictive diet works because they almost by definition dramatically reduced sugar and HFCS in the diet, and if you just controlled for that, the individual types of restriction didn't matter? I can't find it on PubMed or in my bookmarks. If memory serves - high/low fat, high/low carb, high/low protein, etc - didn't matter as long as the rest…
The thing that fully convinced me over a decade ago that all effective methods for weight loss are just "calories in, calories out" underneath was the dude that lost 27lbs in 2 and a half months on a diet of mostly twinkies with occasional other junk food for variety. https://www.cnn.com/2010/HEALTH/11/08/twinkie.diet.professor... Yes, there are other variables like hormone levels (thyroid adjacent ones particularly)…
Because when people try to put it into practice in a form that's not dangerous they struggle, and when humans struggle at something they feel compelled to theorize the struggle. "There must be a reason this isn't working for me," they say, "one that doesn't ultimately reflect on my morally culpable lack of capacity to exert my will." [To be clear, I don't think people are to blame for lacking willpower, I'm just identifying the logic here.]
Looking around, they observe that some people have more success on a diet where they restrict some particular type of food X. This success comes from playing a trick on the brain, making it think that it is not in a position of deciding to eat less every day, but is being forced to eat less. Some restrictions manage to play this trick on the brain successfully for some people some of the time. People promoting diets that are not just "calories in, calories out" are hoping to take advantage of this effect. When these diets become popular, it's because the logic I identified allows people to say "it's not that I failed to force myself to eat less, it's that I didn't stop eating carbs" - or what have you.
One reason it's hard to eat less that you didn't bring up is that the body reacts to you eating less by reducing your metabolism. I don't mean this in a magical fashion where the body has a high efficiency mode or that you can somehow gain weight while eating fewer calories than you consume, but in the simple ordinary sense that people dieting tend to be tired and more lazy. Your body does not want to stay at the same level of activity while dieting, and so you lose less weight than you'd expect with a simple metabolic rate calculation. You can allow for this of course, but it means you have to cut even more.
So I don't know that many people reject the basic physics of calories in, calories out per se. Most reasonable people talking about this are more interested in why attempting a pure "eat less food" diet tends to fail, and what tricks can be employed to achieve appetite reduction. The various individualized factors can be overwhelming, especially when you consider that many people don't have access to adequate nutrition and healthcare.