Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rebuilding after the replication crisis

asteriskmag.com

Post reply on HN
Print view View original

51–60 of 76 posts

Re: Rebuilding after the replication crisis

#51

Post by fock »

bonoboTP wrote: post #36
fock wrote: post #28

Earlier quoted context omitted.

not only social sciences. Except for the very visible areas of ML, the same happens in actual science and engineering... Example: the thousands of fraudulent XRD spectra of made-up compounds.

Even in ML, it's common knowledge that the long tail of papers demonstrate brittle effects that don't really replicate/generalize and often do uncomparable evaluations, fiddle with hyperparameters to fit the test data, use various evaluation tricks (Goodhart's Law) to improve the metrics, sometimes don't cite better prior work, etc. etc. Industry people definitely know not to just take a random ML paper and believe t…

those papers are not "very visible" ML (NeurIPS and co.), but domainspecific "applications" and there's tons of it (as I am acutely aware).

Re: Rebuilding after the replication crisis

#52

Post by nradov »

pa7x1 wrote: post #49

We often hear these days from science commentators if Physics should receive or not funding for the next big accelerator, High Energy Particle physics detector or gravitational wave interferometer. Most frequently brought argument is the lack of interesting findings during the last decades, and if that money would be better spent in other fields that have a higher return on investment given they are more prolific in…

That depends on what you mean by replicability. Climatologists can't exactly conduct controlled experiments, but I think it's still worth putting some public funding into that field.

Re: Rebuilding after the replication crisis

#53

Post by fock »

Qem wrote: post #38
fock wrote: post #28

Earlier quoted context omitted.

not only social sciences. Except for the very visible areas of ML, the same happens in actual science and engineering... Example: the thousands of fraudulent XRD spectra of made-up compounds.

> Example: the thousands of fraudulent XRD spectra of made-up compounds. Interesting, didn't hear about this case before. Can you provide a link?

https://news.ycombinator.com/item?id=32280756 as discussed here ;)

Re: Rebuilding after the replication crisis

#54

Post by bonoboTP »

fock wrote: post #51
bonoboTP wrote: post #36

Earlier quoted context omitted.

Even in ML, it's common knowledge that the long tail of papers demonstrate brittle effects that don't really replicate/generalize and often do uncomparable evaluations, fiddle with hyperparameters to fit the test data, use various evaluation tricks (Goodhart's Law) to improve the metrics, sometimes don't cite better prior work, etc. etc. Industry people definitely know not to just take a random ML paper and believe t…

those papers are not "very visible" ML (NeurIPS and co.), but domainspecific "applications" and there's tons of it (as I am acutely aware).

What I wrote also applies to most papers at top tier conferences, like Neurips and CVPR. There are thousands of papers published per year even just in those top conferences. What gets picked up by the media or even just reaches the HN crowd is just a small tip of the iceberg.

Re: Rebuilding after the replication crisis

#55

Post by thelamest »

3pt14159 wrote: post #48
mattkrause wrote: post #45

Earlier quoted context omitted.

That sneering misses the mark, IMO. Other fields definitely have similar problems: Amgen found that they could only replicate about half of the cancer biology papers they tested. The "replication crisis" showed up in psychology first for a few reasons. First, psychology probably runs the most replications of any field. The experiments themselves are easy to reproduce: all you need to redo a survey project is a Qualtr…

Not going to comment on psychology directly, because I don't know enough about it, but there are some fields where there really is no replication crisis. In structural engineering we literally tore metals apart to find their physical properties. Concrete was mixed. Giant crushing machines were used. Etc. Sometimes I think that half the reason why some fields don't have a replication crisis is because literal lives ar…

Yeah, yet this bonduary traces a fuzzy and surprising route. Lives are on the line in microbiology, yet Bik uncovered thousands of suspect image errors there (https://en.wikipedia.org/wiki/Elisabeth_Bik). Semiconductor physics is relatively tangible and close to production, yet Hsu&Loo and others found that Schön faked his award and attention grabbing recipes (https://en.wikipedia.org/wiki/Sch%C3%B6n_scandal).

Re: Rebuilding after the replication crisis

#56

Post by williamcotton »

zemvpferreira wrote: post #44
williamcotton wrote: post #37

Science is an approach to epistemology. My conjecture is that all truths must be experienced. This aligns with the notion of “nullius in verba”, the original motto of the Royal Society, arguably the birthplace of modern science. Science takes place in a laboratory. Whether or not ink on a page is true depends on nothing other than replicating the methods for oneself. That the current environment is for printing ink o…

Surely this is also a consequence of knowledge expansion? Up to a certain year some people could credibly be said to know everything scientific. There just wasn't much of it. I couldn't imagine verifying with my own eyes even a small subset of one field today. I doubt there's one person alive who's seen 5% of, say, all metallurgy with their own eyes. How would we ever progress without a change in method from observat…

My point is that we’re probably no longer in what future historians will consider the scientific era. “Believe in science” would be an incoherent statement to Newton.

Re: Rebuilding after the replication crisis

#57

Post by mattkrause »

MichaelZuo wrote: post #50
3pt14159 wrote: post #48

Earlier quoted context omitted.

Not going to comment on psychology directly, because I don't know enough about it, but there are some fields where there really is no replication crisis. In structural engineering we literally tore metals apart to find their physical properties. Concrete was mixed. Giant crushing machines were used. Etc. Sometimes I think that half the reason why some fields don't have a replication crisis is because literal lives ar…

There's definitely no replication crisis in jet turbine design research also.

ahem https://retractionwatch.com/?s=turbine

A lot of these appear to be cases of flat-out fraud or dubious research practices. These all get muddled together under the heading of "replication crisis" but I do think they're a bit of a different beast.

Re: Rebuilding after the replication crisis

#58

Post by leereeves »

samus wrote: post #16
dantheman wrote: post #5

Earlier quoted context omitted.

If the result isn't stable across samples how valid is the result?

If the study otherwise follows best practices, results about differences between well-defined subpopulations, and the reasons behind them, are still worthwhile to investigate and report.

If the relevant subpopulations are well-defined, then it should be possible to sample from those subpopulations and reproduce the results.

Re: Rebuilding after the replication crisis

#59

Post by mattkrause »

3pt14159 wrote: post #48
mattkrause wrote: post #45

Earlier quoted context omitted.

That sneering misses the mark, IMO. Other fields definitely have similar problems: Amgen found that they could only replicate about half of the cancer biology papers they tested. The "replication crisis" showed up in psychology first for a few reasons. First, psychology probably runs the most replications of any field. The experiments themselves are easy to reproduce: all you need to redo a survey project is a Qualtr…

Not going to comment on psychology directly, because I don't know enough about it, but there are some fields where there really is no replication crisis. In structural engineering we literally tore metals apart to find their physical properties. Concrete was mixed. Giant crushing machines were used. Etc. Sometimes I think that half the reason why some fields don't have a replication crisis is because literal lives ar…

Right--but that's half the problem!

You can't tear people apart to figure out why they're depressed. Not only will no one volunteer to be torn apart, but it won't tell you much anyway: depression (probably) isn' a property of a single protein or cell. In a sense, this is actually my day job: I work with animals where we can dig down to the level of a single neuron or sometimes even a single molecule....but it's still different enough that you still need to go back to intact, living people to see if your proposed treatment works.

Re: Rebuilding after the replication crisis

#60

Post by krapp »

williamcotton wrote: post #56
zemvpferreira wrote: post #44

Earlier quoted context omitted.

Surely this is also a consequence of knowledge expansion? Up to a certain year some people could credibly be said to know everything scientific. There just wasn't much of it. I couldn't imagine verifying with my own eyes even a small subset of one field today. I doubt there's one person alive who's seen 5% of, say, all metallurgy with their own eyes. How would we ever progress without a change in method from observat…

My point is that we’re probably no longer in what future historians will consider the scientific era. “Believe in science” would be an incoherent statement to Newton.

>“Believe in science” would be an incoherent statement to Newton.

It wouldn't, because Newton considered himself primarily an alchemist and philosopher, and viewed all of his "scientific" endeavors as a means to the end of understanding God through understanding Creation.

Post reply on HN