Live data from Hacker News

How scientists fool themselves and how they can stop

nature.com

11–20 of 26 posts

Re: How scientists fool themselves and how they can stop

#11
post #5

A scientist's first instinct is "get more funding". This cannot help but bias their conclusions.

Unnecessarily cynical. Its a problem that science funding makes scientist's careers so precarious, but nobody goes into science for the money.

"Get more funding" isn't the difference between making a little money vs. making a lot of money. It's the difference between making a little money and not having a job.

Re: How scientists fool themselves and how they can stop

#12
post #9
post #5

A scientist's first instinct is "get more funding". This cannot help but bias their conclusions.

Do you expect to see any differences in the quality of science of those few scientist who are self-funded? (I heard a lecture once from a researcher who was on Wall Street, made millions, then switched to femtosecond lasers. He worked out of a university, and funded his own lab. So such people do exist.) If there are differences, are they measurable?

The problem is that there is massive selection bias. If you're one of the very top researchers in your field you are much more likely to A) produce world class research and B) never have much problems with funding.

Conversely many of those funding their own research are doing so because they cannot get any other funding, which may be an indicator that they aren't very good.

Re: How scientists fool themselves and how they can stop

#13

Earlier quoted context omitted.

We don't - first we have correcting mechanisms in place. Second - the self deception ability is greater the softer/fuzzier the field is.

Who measures softness/fuzziness of a field? Who makes sure that measurement isn't biased? Just because you use maths doesn't imply your conclusions and theories are as precise and clean cut as 2+2.

I don't like the use of soft/fuzzy for certain fields, but I do know what it means in this context: it's shorthand for "fields so complicated that we aren't even sure if the questions we're asking mean anything." Take physics vs. psychology as an example. The top story on HN right now is about standardizing the kilogram by answering two questions: how many silicon-28 atoms do you need to equal the mass of the current reference kilogram, and at what pount a watt balance can support the 1 kg weight. This depends on a bunch of terminology and assumptions, but we're pretty clear what those are.

In psychology on the other hand (a field I have a huge amount of respect for, incidentally), we're trying to ask questions about human behavior; everything we ask about is taken within a cultural context, predicated on assumed levels of "normal" behavior, and even assumes that humans /are/ repeatable in some sense (and a lot of psychology studies may not be repeatable at all...)

That repeatability question is a big one, I think: we're a lot more sure about our ability to prepare equivalent circumstances with silicon-28 atoms for an experiment than with people.

Re: How scientists fool themselves and how they can stop

#14
post #12
post #9

Earlier quoted context omitted.

Do you expect to see any differences in the quality of science of those few scientist who are self-funded? (I heard a lecture once from a researcher who was on Wall Street, made millions, then switched to femtosecond lasers. He worked out of a university, and funded his own lab. So such people do exist.) If there are differences, are they measurable?

The problem is that there is massive selection bias. If you're one of the very top researchers in your field you are much more likely to A) produce world class research and B) never have much problems with funding. Conversely many of those funding their own research are doing so because they cannot get any other funding, which may be an indicator that they aren't very good.

> Conversely many of those funding their own research are doing so because they cannot get any other funding, which may be an indicator that they aren't very good.

... at selling. In my experience the skill sets needed for aquiring funds and those for doing research don't have much in common.

Re: How scientists fool themselves and how they can stop

#15
post #12
post #9

Earlier quoted context omitted.

Do you expect to see any differences in the quality of science of those few scientist who are self-funded? (I heard a lecture once from a researcher who was on Wall Street, made millions, then switched to femtosecond lasers. He worked out of a university, and funded his own lab. So such people do exist.) If there are differences, are they measurable?

The problem is that there is massive selection bias. If you're one of the very top researchers in your field you are much more likely to A) produce world class research and B) never have much problems with funding. Conversely many of those funding their own research are doing so because they cannot get any other funding, which may be an indicator that they aren't very good.

> because they cannot get any other funding

Really? I seriously question that "because". If I had FU money I would definitely start my own lab and pay for research out of pocket. Science is fantastically fun if you don't have to care about funding (and by extension tenure) and only publish because it's nice to get feedback from your peers (rather than because you need X papers to get your next grant and keep your job).

Conversely, writing grants is a total bore.

A "because" I would be more willing to buy: if you made that kind of money for yourself, odds are you spent your prime years achieving that and therefore haven't dedicated the time to science that's necessary to be at the top of your field.

Re: How scientists fool themselves and how they can stop

#16

Earlier quoted context omitted.

We don't - first we have correcting mechanisms in place. Second - the self deception ability is greater the softer/fuzzier the field is.

Who measures softness/fuzziness of a field? Who makes sure that measurement isn't biased? Just because you use maths doesn't imply your conclusions and theories are as precise and clean cut as 2+2.

> Who measures softness/fuzziness of a field?

There's an absolutely unambiguous distinction between the "hard" and "soft" sciences

Social scientists hold themselves to fundamentally different standards on things like repeatability and predictive power and methodolgy than other fields. And whenever that difference can be quantified (e.g. statistical results), "different" means "lower". It's just a fact, and it's one most sociologists will freely admit, for example. You simply can't get the same sorts of results on humans that you can get on e.g. chemical reactions.

Unfortunately, soft/hard comes along with a value judgement that's not necessarily appropriate. But pretending like there isn't a clear difference between what are now referred to as the soft and hard sciences is disinegenious.

Re: How scientists fool themselves and how they can stop

#17

It's too bad that there's not a more robust discussion about the institutional incentives that drive scientists to fool themselves.

Some things in science have become "immoral" to question. In a big way evolution, including macro evolution, has become the religion of science. If you even entertain the idea that macro evolution has not been happening, you might be labeled an evil heretic. You might be labeled anti-science. You might be laughed out of your field. You could get fired from your job. You might lose all your funding. All of this could happen even if you go about your research, testing, and analysis in a very objective, scientific way.

Re: How scientists fool themselves and how they can stop

#18
post #11

Earlier quoted context omitted.

Unnecessarily cynical. Its a problem that science funding makes scientist's careers so precarious, but nobody goes into science for the money.

"Get more funding" isn't the difference between making a little money vs. making a lot of money. It's the difference between making a little money and not having a job.

Depends on the scientist's position. Tenure-track and tenured faculty in the US typically have 9 months of their salary covered by the institution employing them. Only the remaining three months need be covered by grants (or teaching).

Re: How scientists fool themselves and how they can stop

#19

It's too bad that there's not a more robust discussion about the institutional incentives that drive scientists to fool themselves.

This is a key aspect. There is a pressure (both institutional and self-imposed) to produce something original and unique, so trying to reproduce other people results is seen as a lost of time.

Either you confirm previous results, which is nice but not something you can publish or write a thesis about; or you find they are wrong and then you have to tell the authors, which is something they may appreciate or not, and be exhaustive to prove it, which is something in which you may be interested or not.

What I would like to see is that PhD students were required to reproduce some previous results as part of their thesis. This would increase the quality of scientific results and would prepare the candidates to do their own research (at difference of now, that they are put on front of a laboratory to get "interesting results" and they don't know what to do with it).

Re: How scientists fool themselves and how they can stop

#20
post #5

A scientist's first instinct is "get more funding". This cannot help but bias their conclusions.

That's not true. Funding is important, but once you have it you stop looking for more. I've been working in research for years, both in companies and universities, and I have always been involved in long-term projects, so I have never had to worry about funding.

Of course, other colleagues are not so lucky, and getting funding is an important part of their day-to-day jobs, specially once you reach higher positions. But it is a necessity, not an instinct.

Post reply on HN