91–100 of 119 posts
Earlier quoted context omitted.
that author’s problem was that they had been a postdoc for 23 years. That duration alone raises fatal red flags.
It's possible you're right, but that's not what people were saying IIRC.
An undergrad to his supervisor in our office talking about publishing a paper: I've fixed the data, now the plots look ok. I (undergrad too) am sitting there thinking - well, you are using ML as a regression blackbox to plot a line, I can do that too w/o ML if I'm fixing the data. Supervisor: ok, that's really great. Me cringing... I'm not hammering the ML-keyword above my work (and thus am getting considerably less…
Better get on this train because 10 years after undergraduate I can say it seems that’s all there is out there. Once you make it then maybe you don’t have to be full of sh but I don’t see people making it on merit unless they’re Albert Einstein.
Is machine learning really to blame for the reproducibility crisis? I'm not in academia, but it seemed to me that the problem was entirely present without machine learning being involed. For example, Amgen reporting that of landmark cancer papers they reviewed, 47 of the 53 could not be replicated [1]. I would have assumed that most of them didn't involve 'machine learning' [1] https://www.reuters.com/article/us-scie…
Of course the problem is compounded if we also fail to propose a causal mechanism and you don't try to validate it — something I see data scientists doing all too often since we seldom employ anything like Design Of Experiment practices, and the data we're working with is very rarely created by us.
IMHO, the RC is a reminder to scientists that to confirm a hypothesis you need to pass more than one test, and a reminder to data scientists that we must test using more than one model.
Earlier quoted context omitted.
Machine learning trivializes p-hacking. Take a database of random datums. Pick e.g. 3 input datums at random and map them against one manually chosen output datum. Run the machine learning system and observe the error rate. If it decreases below some value 'p' you now have a [most likely completely spurious] correlation. Spin up an explanation for it - the more sensationalized the better. Claim that the process was d…
No one uses or needs ML for overfitting 4 variables. You can do that with regular statistics just fine. And how you interpret ML results is just as fraught with error as any statistical argument—just because the technique gives you some result doesn’t mean it’s explanatory, that is science 101.
True, but an ML routine can try an approximately infinitely greater number of models. If I'm using x,y and z to predict w, and I've tried all the linear terms, all the squared terms, all the interactions, all the log terms, and I start throwing in other things, my readers will, rightly, raise an eyebrow. Maybe there's some discontinuity to exploit, but if so, I'll explain it -- a policy change for people age 65 or older, say, or a market that exists in one state and not an adjacent one.
The ML, by contrast, can invent the most absurdly jagged multivariate functions imaginable, and we typically* don't even know that it's doing so, let alone why.
*as others have written here, we can actually investigate the how (not the why) by inspecting the algorithm -- but the number of papers that do is much smaller than the number that don't.
Earlier quoted context omitted.
The problem really is: there is no alternative (which I know of). Having a number of papers in well known journals is everything we have to gauge the quality of people that look for a life time position. It’s sad but true
No, the number of papers is an okay metric, thr problem is that journals don't like to publish honest negative results
In the long term, it seems like journals that publish lots of falsified papers should be punished, and journals that don't (e.g. because of a judge-upon-pre-registration policy) should crowd them out.
> ... developing the next generation of machine learning and statistical techniques that can ... also report how uncertain their results are and their likely reproducibility.
So she's actually using machine learning to access systematic uncertainties, i.e. to get better, more reproducible research. Of course, like all forms of automation, people tend to sensationalize progress as a crisis because it makes it too easy to shoot yourself in the foot.
But doing things "the old fashioned way" isn't any better. Early particle physics experiments would get armies of undergrads classify photographs of collisions in bubble chambers. These results took thousands of researcher-hours to compile, which might seem all fine and dandy, until you realize that there may have been a systematic bias in your classification. Now what do you do?
Thanks to machine learning, there are a lot of things we can do: we can try to remove the bias and retrain the algorithm, or we can train with extreme examples of bias and use that to quote a systematic uncertainty. We can try a multitude of approaches to estimate uncertainties rerun our entire analysis in a few hours. Good luck doing that with an army of undergrads.
Is machine learning really to blame for the reproducibility crisis? I'm not in academia, but it seemed to me that the problem was entirely present without machine learning being involed. For example, Amgen reporting that of landmark cancer papers they reviewed, 47 of the 53 could not be replicated [1]. I would have assumed that most of them didn't involve 'machine learning' [1] https://www.reuters.com/article/us-scie…
Is machine learning really to blame for the reproducibility crisis? I'm not in academia, but it seemed to me that the problem was entirely present without machine learning being involed. For example, Amgen reporting that of landmark cancer papers they reviewed, 47 of the 53 could not be replicated [1]. I would have assumed that most of them didn't involve 'machine learning' [1] https://www.reuters.com/article/us-scie…
The problem was there before, but there are reasons why Machine Learning is amplifying bad practices. In the past people were manually fishing for results in available datasets. Now they have algorithms to do it for them. In medicine a popular way to use ML is to improve diagnosis. Now there's already a problem in medicine that the benefits of early diagnosis are overrated and the downsides (overtreatment etc.) usual…
It's not an interesting question for many scientists, who prefer focusing on technical solutions over political ones.
Good read. It's also refreshing to see a mainstream article that talks about ML without once mentioning 'AI'.
ML is statistics with a different name using a computer, that should always be mentioned in articles for the general public. On the other hand AI is fantasy BS hype boosting off the fact that ML sounds similar to AI to people who aren't aware Machine Learning is stats. Maybe AI one day but today it is utterly ridiculous. No really. Every single article should mention both of those things at least in passing. Downvote…
AI is real and is a legit field of study, of which ML is currently very popular, so some articles conflate these two. But it is far from bullshit. If you want to learn more read Artificial Intelligence: A modern approach.