AAAS: Machine learning 'causing science crisis'
1–10 of 119 posts
Re: AAAS: Machine learning 'causing science crisis'
#2Good read. It's also refreshing to see a mainstream article that talks about ML without once mentioning 'AI'.
Re: AAAS: Machine learning 'causing science crisis'
#3ML is not causing a reproducibility crisis, it just exposes one that is already there.
> If we had an additional dataset would we see the same scientific discovery or principle on the same dataset?
The same holds true for traditional science based on traditional statistics. It just seems that traditional datasets are under less scrutiny of reproducibility and are taken more easily at face value.
Re: AAAS: Machine learning 'causing science crisis'
#4Is machine learning really to blame for the reproducibility crisis? I'm not in academia, but it seemed to me that the problem was entirely present without machine learning being involed.
For example, Amgen reporting that of landmark cancer papers they reviewed, 47 of the 53 could not be replicated [1]. I would have assumed that most of them didn't involve 'machine learning'
[1] https://www.reuters.com/article/us-science-cancer/in-cancer-...
Re: AAAS: Machine learning 'causing science crisis'
#5I wonder: don't machine learning frameworks' results come with a level of confidence?
Ps: I have no experience with anything regarding ML.
Re: AAAS: Machine learning 'causing science crisis'
#6Fails to touch on the perverse incentives in academia, "publish or perish" etc. Torturing a dataset to find a p value that a journal will like (or equivalent stat measure) is better for your career than not publishing a paper that will be discredited in time. You have no incentive at all to decide "my results are unconvincing at this point, I'm not going to submit them" and every reason to write them up as a useful contribution to human understanding even if you kind of know, deep down, it really isn't. Especially if you're not senior...
Re: AAAS: Machine learning 'causing science crisis'
#7An undergrad to his supervisor in our office talking about publishing a paper: I've fixed the data, now the plots look ok. I (undergrad too) am sitting there thinking - well, you are using ML as a regression blackbox to plot a line, I can do that too w/o ML if I'm fixing the data. Supervisor: ok, that's really great. Me cringing...
I'm not hammering the ML-keyword above my work (and thus am getting considerably less academic attention), but it's nice to hear from people who made it in academia that they support my theory. 50% off the people are just showoffs throwing buzzwords and positivity around while they produce a load of sh...
Re: AAAS: Machine learning 'causing science crisis'
#8A dishonest scientist can mine a dataset for statistically significant hypotheses and for a long time no institutional protection against it was in place:
https://en.wikipedia.org/wiki/Data_dredging
Machine learning makes it easier to test great many hypothesis, but even going fully "by hand" it is very easy to deviate from what the statistical framework of hypothesis testing would demand. There are now some discussions about counter-measures, e.g. about preregistration of studies:
http://www.sciencemag.org/news/2018/09/more-and-more-scienti...
You can see this as another chapter in the long debate about the correct way to test scientific hypotheses:
https://en.wikipedia.org/wiki/Statistical_hypothesis_testing...
Re: AAAS: Machine learning 'causing science crisis'
#9One example?
Re: AAAS: Machine learning 'causing science crisis'
#10Hopefully machine learning helps with confidence and making predictions out of experiments as opposed to the limited capability of "understanding" from the way things are done now (as if an experiment with slightly higher p values are ignored or with smaller values might have hidden biases, etc).