Cargo cult science (1974)

Hard to say. It's hard to get a YEC to come out of the closet here. Maybe I'm a little overly sensitized to YECs in particular because I spent some time on the YouTube YEC debate circuit.

But it hardly matters. What matters is that there are a lot of anti-science ideologues out there, some of them are well organized and well-funded, they are developing a collection of stock rhetoric (including that Feynman quote) which can be very persuasive, and some of that rhetoric appears regularly here. Whenever I see it I try to smack it down, but sometimes it feels like whack-a-mole.

Took until Kepler and the idea that orbits were mostly elliptical for people to throw out epicycles altogether.

Not at all, I definitely did not express myself correctly. I really have no worries that the smart kids will mostly thrive, no matter what educational methods they are subjected to. Sure, some systems will work against them, but that is unavoidable.

My gripe is, first of all, with the fact that it is impossible to criticize education and, as a consequence, it is impossible to improve it. Nobody really knows whether one or another type/method works better than any other, and yet many people pretend to know, and pretend to have scientific knowledge about it. But because teaching is such an emotional subject, any teaching seems to be good teaching, and as such, we'll never really know which one works or not.

Given that, I don't really feel the obligation to use one method or the other, and I also don't feel like the success or failure of a student is my success or my fault. Don't get me wrong, I do my absolute best to do a good job. I believe in teaching by example, I believe in connecting theory to practice, I believe in reaching out to eventual special needs of specific students, etc. However, I also believe (and make it very clear to my students) that their success is still 95% based on their own hard work and motivation, and only 5% based on whatever it is that I offer them. I simply don't know whether my teaching is good or not; I'll never know.

As such, it is irresponsible of me to try to convince them (like some kind of religious leader or a witch doctor) that by following my "methods" they will thrive, and if they haven't, it's because they didn't follow me close enough. Sure, some will pass the exams with straight A's. Some students are very good in overfitting to the institutional assessments, and will build a kind of fake safety and comfort with their results. Does that mean they will do well out there? I have no idea. Others will fail my exams miserably. Some students are bored by structured, predictable goals and need more complex challenges; others are in the wrong area; others simply don't want to study. Does that mean they won't do well out there? I have no idea.

As such, I think that the only honest thing to do is this: offer them everything you can, but make it clear to them that they'll only learn as much they work to learn. The school will give them more space and time to make errors and learn from them, in a less traumatic way than the real world. But if they don't use that, i.e. if they don't actually work hard to learn and experiment and fail and try again, there is nothing I can do to make that better. Sorry. I'm only human, and I don't pretend to have a solution to this problem just to try to justify my existence as a teacher.

As for your question: I believe there is definitely an effect, but it could be good or bad. Some teaching will give motivation for students to develop their abilities; others will kill their creativity. My point is exactly that: we don't know, and never will. We're just witch doctors trying every possible plant to treat a disease. Randomly, some will actually work, some will not, and some will make it worse. But since we cannot check it objectively, we'll never know which is which.

I don't think people's lack of understanding scientific methods is the main factor shaping the incentive structure, though it's very interesting how big a role it plays and I would love to hear more detailed arguments.

In order to change the system, there must be enough alignment on what the actual goals are and what incentives might benefit them, ways to effectively organize around it, and ways to actually enforce the new incentives. (Perhaps even more things...)

I think there are big hurdles on each of the above. First, concerning alignment on the goal, many people don't agree that faster scientific progress is actually good. You might argue that trying to teach people epistemology might change that. I neither think it would nor that it should, but one could argue about it a lot. "Indoctrinating" people starting at a young age that "science and knowledge = good" as a moral value might work (it is being done and I mostly endorse that) but that's different from teaching the scientific method.

The really important important factor here is, of course, that many individuals and organizations benefit from the current system and don't want to change it.

Concerning organization, political intersts and competition hinder it somewhat, but co-operation between scientists is quite high and increasing. Hopefully that trend will continue and not sucumb to geopolitical pressures.

Concerning enforcing better incentives, that's very hard. For example, we can't objectively and reliably evaluate someone's scientific contributions. Any metric you set up is going to be gamed, etc. I think the incentives have improved over the (very short) history of collaborative science and the trend looks positive. Take for example the trend towards open access publication and growing awareness of many issues (replication crisis, etc.).

Generally, I would say that understanding how science (the system) works and how it should work (or would, given certain changes) is very hard. Similarly, I also think really deep understanding how science (the process) should work is also hard and applying that knowledge in practice is much harder still. At some point you get diminishing returns for the efforts of teaching it. You still haven't proposed how you would promote this knowledge. I can't think of anything that seems likely to do much good but would love to hear about it.

> Remarably, nearly all the commenters here betray this very lack of knowledge by talking about it as "philosophy"

I believe epistemology and philosophy of science are philosophy. I also believe that having and improving these things is very useful and one cannot overstate their role in getting science where it is now. This is in my mind perfectly consistent with there not being a great need (way?) to teach more of it to students.

Perhaps an explanation of how you would teach it would make me realize we haven't been talking about quite the same thing?

Of course you are 100% right that those PIs are short term incentivized to keep the cheap labor coming. I think there's a decent chance that this utterly backfires on them in the next few decades, but I guess we will see. It has already started to backfire as far as obtaining postdocs goes.

Anyway, a small scale example of a better academic requirement IMO that would definitely be implementable by those making program decisions (because it already has at UCSD neurobio, perhaps elsewhere) is to change qualifying exams. In engineering programs I know quals involve grad level course work exams too so they're already a big time commitment, but in many bio programs it is simply a project proposal presentation and you get asked some questions on the background lit.

Because the project proposal is just a project you are planning to do in your lab for your PhD, it is not uncommon for it to be pre-scaffolded by the PI, not requiring original thought. At UCSD they require that you write your quals proposal/presentation on a different neurobio topic entirely from anything to do with your chosen lab. This makes it far more likely that students will put together an in depth scientific proposal themselves before embarking on executing what is often someone else's. Of course their proposal does not get implemented, it is just evaluated and given feedback by a committee of established faculty members.

However what I was getting at on a bigger level is that not all PhDs need specialized meta science skills, but some do. It doesn't have to happen overnight, but I'd like to see the process started by having some sort of branching in PhD requirements, perhaps like the quals change I described along with differing year 1 curriculums being optional. This shouldn't require many resources because at the graduate level it is completely fair game to stick some students in a room and give them some suggested references and a problem, and check back a month later. The important part is having not only the clear encouragement but also the protected time.

Oh and I think more opportunities to work with fellow students on meaningful projects is sorely needed too. PhD students from different labs mostly only help each other on a very surface level, it is rare to see students work together across labs in a way that each would be contributing authors on the other's paper. This has the opportunity to still be productive from a traditional viewpoint but also to naturally produce more thinking about a range of science.

Anyway, at a minimum I think professors can stop telling PhD students that e.g. wanting to attend a meta science reading group is a waste of time. Having that as a component of PhD education need not be compulsory or even expected, but for it to be culturally frowned upon is bizarre to me (though the selfish reasoning behind the attitude is obvious).

So the incentive structures part is again true, but incentive structures aren't going to change without an attitude shift and a willingness to diversify intellectually. And it kind of is chicken/egg - the incentive structures created a bad educational environment for some time now, and some of those people are already early career profs. They're in an even worse position to be trying to fix things because they're more ignorant of the fundamental problems to begin with. Seriously there are bad actors but there are also people that are just remarkably unaware of how to be remotely statistically rigorous for example. Either that or they are really good actors who are willing to look dumb in front of certain peers to further their long term goals, but academics don't strike me as the type to be content with playing dumb (if it's being perceived by others in any event).

Regardless, I'd be happy to start from a place of changing incentive structures if that were possible, and work from there. But at the end of the day I think this is an important factor for the incentive structures to consider. Bad training systems can create a real telephone game effect after a few generations.