Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

False claims in a widely-cited paper

statmodeling.stat.columbia.edu

Post reply on HN
Print view View original

61–70 of 181 posts

Re: False claims in a widely-cited paper

#61

Post by tylerhou »

BobbyTables2 wrote: post #27
paulpauper wrote: post #2

Peer review is a joke still and exists now to please deans (for hiring and promotion) and enrich publishers. Bad papers get published if it reaffirms the biases of editors, and actually good and original stuff gets rejected. Rather than facilitating the exchange of knowledge, it acts as a barrier, especially when it cannot even be relied on for quality control.

Even in more respected journals, peer review is often done by beleaguered grad students who could be still relatively new to the field. They lack the experience to look at things with a critical eye.

Graduate students! Hah! ML researchers can only hope their papers at ICLR/ICML/NeurIPs are reviewed by graduate students!

Re: False claims in a widely-cited paper

#62

Post by throwawaypath »

jasonfarnon wrote: post #31
stinkbeetle wrote: post #20

[flagged]

When was this golden age of western civilization again? like 10 years ago, are you suggesting we were in this golden age? I mean, the paper this link is discussing is from 2014, so I guess it was more like 15 years ago that the golden age sunsetted?

[flagged]

Re: False claims in a widely-cited paper

#63

Post by pjdesno »

Are there any factual allegations on that page? All I could find was "the method described in the paper is not the method the authors actually used", without any elaboration.

I'll add that the reaction of most of academia will be "It's in a management journal - of course it's nonsense."

Re: False claims in a widely-cited paper

#65

Post by pjdesno »

chromacity wrote: post #26
lotsofpulp wrote: post #6

Earlier quoted context omitted.

I have always assumed the further away from math and physics a field is, the higher the probability of any given “research” to be false. Even biology, I might give 50% odds at best, but that is due to the difficulty of observing and measuring in that field. Anything past biology might as well be as fiction due to how easy it is to come up with whatever results you want.

I think that theoretical math and physics are special, but probably not in the way you assume. It's just that there isn't a whole lot of grant money, prestige, or influence associated with them (unless you accomplish something truly exceptional). Computer science is very close to math and should be even easier to verify, but there's plenty of dubious results published every year, simply because it's more profitable t…

I would point out that most products are useless, and either fail or replace other products which weren't any worse. None of which prevented me from cashing my paychecks for the first half of my career when I worked in private industry.

Most scientific research represents about the same amount of improvement over the state of the art as the shitty web app or whatever that you're working on right now. It's not zero, but very few are going to be groundbreaking. And since the rules are that we all have to publish papers[*], the scientific literature (at least in my field, CS) looks less like a carefully curated library of works by geniuses, and more like an Amazon or Etsy marketplace of ideas, where most are crappy.

[* just like software engineers have to write code, even if the product ends up being shitty or ultimately gets canceled]

Neither of us are going to be changing how the system works, so my advice is to deal with it.

Re: False claims in a widely-cited paper

#66

Post by ernsheong »

I'm very confused because there are 2 Andrews, the author in the blog post only states "Andrew", and by the list of Authors the author seems to be Andrew Gelman, but the slug in the first link is "aking", and then there is also Andrew King, lol.

Re: False claims in a widely-cited paper

#68

Post by bluefirebrand »

arjie wrote: post #53

There's this 'criterion of embarrassment' / 'cui bono' sort of standard[0] that really helps judge these things. So many people perform science that seems to always confirm the positions they've held. All the "society is terrible today" people like to quote LendingClub's "70% of Americans live paycheck to paycheck" without knowing it's LendingClub content marketing. The snail darter guy happened to find a novel speci…

> I'm Indian so perhaps it is safe for me to use this as a race of choice but what if it were found that Indians actually are less smart than, say, White people

This is an interesting thought experiment.

I'd like to think that if such a thing was discovered, we would investigate why this is the case. I'd like to believe that we wouldn't just accept this as some kind of de-facto truth, and start treating Indians as lesser

I'm an idealist I guess though

Re: False claims in a widely-cited paper

#69

Post by sigbottle »

lotsofpulp wrote: post #6
OhioMan2943 wrote: post #4

So we're firmly in the era of few people caring about few things now aren't we.

I have always assumed the further away from math and physics a field is, the higher the probability of any given “research” to be false. Even biology, I might give 50% odds at best, but that is due to the difficulty of observing and measuring in that field. Anything past biology might as well be as fiction due to how easy it is to come up with whatever results you want.

Observing, measuring, but also repeatability and ground truth.

Math (and theoretical adjacents like TCS) claim not to make any fundamental claims about the actual world (compared to 17th century philosopher-mathematicians like Leibniz), and physics studies the basest of, well, physical phenomenon.

I don't even know how you would begin actually rigorously studying sociology unless you could start simulating real humans in a vat, or you inject everybody with neuralink. (but that already selects for a type of society, and probably not a good one...)

To be clear, I don't think all sociological observations are bad. However, I tend to heavily disregard "mathematical sociological studies" in favor of just... hearing perspectives. New ones and unconventional ones especially, as in a domain where a lot of theories "seem legit", I want to just hear very specific new ways of thinking that I didn't think about before. I find that to be a pretty good heuristic for finding value, if the verification process itself is broken.

Re: False claims in a widely-cited paper

#70

Post by suzzer99 »

lotsofpulp wrote: post #6
OhioMan2943 wrote: post #4

So we're firmly in the era of few people caring about few things now aren't we.

I have always assumed the further away from math and physics a field is, the higher the probability of any given “research” to be false. Even biology, I might give 50% odds at best, but that is due to the difficulty of observing and measuring in that field. Anything past biology might as well be as fiction due to how easy it is to come up with whatever results you want.

> Anything past biology might as well be as fiction due to how easy it is to come up with whatever results you want.

I used to work for the leading statistical expert witness in the country. Whenever I read something like this:

> The empirical strategy in Eccles, Ioannou, and Serafeim (2014) rests on a demanding requirement: the “treated” and “control” firms must be so closely matched that which firm is treated is essentially random. The authors appear to recognize this, reporting that they used very strict matching criteria “to ensure that none of the matched pairs is materially different.”

I just assume they kept trying different "very strict matching criteria" until they got the matches they wanted. Which is basically what we did all day to support our client (usually big auto or big tobacco). We never presented any of the detrimental analyses to our boss, so he couldn't testify about them on the stand even if asked.

Although in this case it sounds like the authors couldn't even do that, and just fudged the data instead.

Post reply on HN