Noted study in psychology fails to replicate, crumbles with evidence of fraud

Interesting point. So the original paper (2012) didn't give a table or anything, but said: "Customers who signed at the beginning on average revealed higher use (M = 26,098.4, SD = 12,253.4) than those who signed at the end [M = 23,670.6, SD = 12,621.4; F(1, 13,485) = 128.63, P Was M = 26,098.4, SD = 12,253.4 enough to infer a uniform distribution?

Yes. Find it. And when you do, you'll have something reproducible. The purpose of science is to advance the species. Given this, when that happens, this other thing(s) occur (or occur with a certain frequency)

I agree that this is a generalization problem. I just don't believe that this particular way of describing it is helpful. If you'll allow me an apocryphal example, when Newton saw the apple fall, if he had generalized to "all things from apple trees fall at the same rate", he would have been correct for a while, then ludicrously incorrect once the leaves started blowing off in Autumn. If at that point he had tried to figure out exactly what kind of tree produces this gravity thing, or which parts of trees, we wouldn't be any further along than we were before.

Generalizations work along multiple axes. With some degree of imprecision, every experiment can be a success (or failure). This is why we have the scientific method in the first place.

Now, perhaps you're making the argument that this is somehow, for lack of a better term, half-science. It's not right, but it's not wrong either. It's truthy. The logical follow-up would be: ok, where is the second half? In this and many other cases there is no second half. Papers were published, people received awards, wrote books, became famous, and lots of other folks used their work. So even if somehow this is better than nothing, in practice I'm not seeing it.

Let me continue the example...

We found that students participating in the "2hr active recess program" have meaningfully better health outcomes, psychological health outcomes, and better grades. It was tested faithfully in 25 schools over 10 years.

It expands to other places. Different countries. Different decades. Different people run it in different ways. The kids are different. In some cases, the kids are already so active that the extra activity is detrimental. In some places, it's really hot. Kids tend to read, or do more organized indoor activities.

Basically, the world is not a laboratory. You're not predicting the trajectory of a projectile, where unknowables can be quantified as a function. It's all 3 body problems all the time.

Yet, we as humans do manage to make sense of the world. We run schools, companies. We learn more about how to do those things. Formal studies are an important part of learning about how to do those things. Scientific mindset and methodologies are useful to this. It's a lot better than just philosophizing the whole thing. The type of knowledge we gain, OTOH, isn't like F=MA. It never will be.

The value of the above study does relate/imply to it containing knowledge. It's a partial knowledge of a general problem. It is not complete knowledge of a particular problem, like you would get in a lab. Neither lab experiments or wild studies tend to replicate in one another's world.

No matter how good the study, how many times it replicated, or how well it worked throughout the 80s and 90s... there's no guarantee that your program is working well or a guaranteed explanation for why it isn't.

I disagree.

You're not going to find the parameter which makes a high school nutrition program deterministic. It shouldn't be the goal.

Another issue is books written by journalists blowing up ideas they find interesting from research and the pulling things out of their ass.

Look at emotional intelligence (EQ) as an example. One of the early books on the subject written by a journalist which was based on a shaky foundation and then added a bunch more sugar. It might as well be a self-help movement which could spawn coaches, authors courses and certifications.