Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Evidence of Fraud in an Influential Study About Procrastination

datacolada.org

Post reply on HN
Print view View original

161–170 of 276 posts

Re: Evidence of Fraud in an Influential Study About Procrastination

#161

Post by teddyh »

kstenerud wrote: post #139
p-e-w wrote: post #15

Earlier quoted context omitted.

> In 2006, when he was a professor at the MIT Media Lab, Ariely conducted experiments including administering electric shocks with a research assistant who had no human–subject training. […] Ariely confirmed this and said that he wasn't aware that the research assistant did not have the needed one-hour online human–subject training. A one-hour online training is all that’s needed to be considered ready to conduct exp…

> Ariely conducted experiments including administering electric shocks Obligatory https://www.youtube.com/watch?v=A6bJHalRnX4

“Dr. Venkman, we believe that the purpose of science is to serve mankind. You, however, seem to regard science as some kind of dodge or hustle. Your theories are the worst kind of tripe, your methods are sloppy, and your conclusions are highly questionable; you are a poor scientist, Dr. Venkman. And you have no place in this department or in this university.”

Re: Evidence of Fraud in an Influential Study About Procrastination

#162

Post by Otterly99 »

jarenmf wrote: post #133

Science is self-correcting, but sometimes, the correction mechanism it is very slow and inconsequential enough that people are incentivized to just publish anything. Peer review is merely a quality check—and not a very good one. Just the other day, I ran into a paper in my field that claims to have solved a central problem that scientists have been working on for 50 years, single-authored, using some fancy mathematic…

I have been having the opposite problem with my papers lately.

Instead of glossing over mistakes, the reviewer report missing information which is clearly stated in the text.

This is quite frustrating and hard to stay polite after the fifth "please just read the text."

I really wish there was some serious rework of the publishing system. Nobody is happy about it but nothing seems to be moving.

Re: Evidence of Fraud in an Influential Study About Procrastination

#163

Post by dramm »

Dan Ariely is well past the point that he needs to not be a faculty member at Duke University. Data Coda do a fantastic job ferreting out academic fraud. With these fraudulent behavioral science experiments like this relying on printed papers/markups, and coauthors not present and who did not even see the raw data, I wonder if the experiment was ever actually conducted. I would love to see that question answered where fraud is suspected, something I'd hope IRBs would step up to do.

Re: Evidence of Fraud in an Influential Study About Procrastination

#164

Post by kodoman »

trompetenaccoun wrote: post #58
Aurornis wrote: post #6

Dan Ariely has a long history of controversies and studies with falsified data. He wrote 3 popular books and gained a lot of fame before others started uncovering the problems in his studies https://en.wikipedia.org/wiki/Dan_Ariely It’s amazing that problems with his studies are still being uncovered. Equally amazing that Duke still maintains a relationship with him after so many problems have been identified with hi…

We'd find problems with many if not most published studies if we scutinized them the same way. People like him get discovered because they step out of their academic bubble into the light of the world stage. And mind you this is psychology, the entire discipline is only borderline scientific anyway. But even studies in the hard sciences are very often flawed if not outright fabricated. Adjusting data sets to fit the…

The problem in my view is that everything can be measured like hard science in the first place (not to mention the issue with fraud you point out in hard science).

I am think the idea that a thing is only a science if and only if it's empirical is actually harmful, science used to refer to a more generalized idea of study 'scientia' this could also be mathematics or theology. Then natural science begins to eat the term and becomes more limited. I think that the current place we are is one that is actually has a very dim view of anyone thinking about a topic and is naively optimistic about experimentation in general.

For if one looks at Aristotle theories ideas one may laugh given what we know now, despite using empirical reasoning for lots of his observations, Never the less I think where useful at the time as it's great for a theory to be good but failing that if a theory can usefully explain phenomena and provide one with mastery of phenomena in some limited way it's still useful.

Occam's often abused razor is a way of building theories like this and this whole process is similar to compression, given observable phenomena how do we explain phenomena in the most simple way, this process uses experimental and empirical data but requires thinking about the results of all experiments deeply.

From an outsiders view Natural science's largely seem to have pretty good major theories that lots of results of experiments can be used to correct constants of some kind in the theory and a well established field of study of theoretical physics that largely seems to be operating in the way I outlined.

But the social sciences are rather stupidly obsessed by these little experiment and no one has the guts to build bigger theories these days, for any theory of the human mind has to deal with many variables and if one wants to have descriptive theories of human mind then one needs to do a lot of thinking and theorizing and sometimes separate form empirical data and then arguing the theories using the data and reject the theories using the data, but also other kinds of analysis.

I have the most dim view of modern political science and sociology even more so, they tend to adopt this way of operating the most and to actually operate in that realm they need to theorize the most and experimental data is hard to do properly (and do not neglect history one bit, this is are experimental data and is as good as we can get).

Re: Evidence of Fraud in an Influential Study About Procrastination

#165

Post by mike_hearn »

aktenlage wrote: post #159
mike_hearn wrote: post #147

Earlier quoted context omitted.

Science isn't self correcting. This is a harmful meme that we need to collectively put in the trash can, as it's constantly used by academics to evade accountability. The core problem is people conflating science with Science™. The scientific method is a set of techniques that help people discover and report true things. But there are two huge problems with it: 1. What exactly is the scientific method? It's not like…

My take on the self correction is, that later research usually cannot be build on wrong results. So over the decades or centuries the correct results prevail.

You'd think so. I used to believe this too. Surely you try and build on a result which creates an implicit replication study, then your experiments don't work and the problem is discovered pretty quick. Hence, self correction.

It often doesn't happen. The reasons seem to vary by field. We do see it in a few very hard fields like materials science.

In the softer end of social sciences like anthropology, education or the humanities it's obvious why not: they don't make empirically testable claims about reality anyway so there's nothing to replicate. This is the Arday problem, where he was publishing what they call auto-ethnography: effectively just blog posts about the author's own feelings. He even interviewed himself in the third person.

In the harder social sciences it's because nothing actually builds on anything else (see "A problem in theory" [1]). Psychology and related fields like sociology, criminology, etc are basically just a pile of random ideas pooped out in brainstorming sessions. There are hardly any overarching theories generating testable predictions, so studies test whatever random claims the researcher thinks is interesting enough to get in the news. Everything is just rocks lying on the ground rather than towers of understanding, so when one paper is overturned it has no impact on anything else. Citation counts obscure this fact, but do some reverse-citation checks and you'll find a lot of citations are worthless or outright invalid.

In modeling based fields like economics, epidemiology or climatology, it's because they can't put the thing they study in a lab and do high speed experimentation. "Replication" comes to mean repeating the same analysis on the same data set. Often they literally just run the same program on the same input file, get the same result and call it replicated - or sometimes they call it replicated even if the outputs don't match! See [2]. But the program is just a set of assumptions, not real discoveries, so building more papers on top of that program's output doesn't detect invalidity. It will all replicate in some very narrow technical sense without that proving anything useful.

Sometimes outsiders notice what's happening but over time academia has built up a complex web of lore that protects them psychologically. For example, they routinely reject or ignore feedback from outside their field on the basis that it doesn't come from "experts", meaning themselves, or it comes from "right wing" people, meaning anyone who isn't an academic or close political ally.

[1] https://www.nature.com/articles/s41562-018-0522-1

[2] https://dailysceptic.org/2020/05/06/code-review-of-fergusons...

Re: Evidence of Fraud in an Influential Study About Procrastination

#166

Post by raverbashing »

mike_hearn wrote: post #147
jarenmf wrote: post #133

Science is self-correcting, but sometimes, the correction mechanism it is very slow and inconsequential enough that people are incentivized to just publish anything. Peer review is merely a quality check—and not a very good one. Just the other day, I ran into a paper in my field that claims to have solved a central problem that scientists have been working on for 50 years, single-authored, using some fancy mathematic…

Science isn't self correcting. This is a harmful meme that we need to collectively put in the trash can, as it's constantly used by academics to evade accountability. The core problem is people conflating science with Science™. The scientific method is a set of techniques that help people discover and report true things. But there are two huge problems with it: 1. What exactly is the scientific method? It's not like…

Yup

Motte-and-bailey fallacy abound

Nature (not the magazine) doesn't care if you published a peer-reviewed paper in a high-impact journal.

It doesn't care if reviewer 2 thinks you need to quote his friend

It also doesn't care that you think a dewormer works against a virus because it doesn't

> What exactly is the scientific method?

Trying and seeing what sticks. That's literally it.

Of course publishing (and publicity) forces like to jiggle that one way or another

Re: Evidence of Fraud in an Influential Study About Procrastination

#167

Post by mike_hearn »

stackghost wrote: post #152
mike_hearn wrote: post #150

Earlier quoted context omitted.

They wouldn't dread being included there, not at all. The institutions don't care about fraud and reflexively defend it, ignore it or shoot the messenger. Look at the Arday nonsense. People are still discovering new lies he was telling but Cambridge went to the mat for him. Dan Ariely in particular has faced convincing fraud accusations for years: https://en.wikipedia.org/wiki/Dan_Ariely#Controversies_and_c... Nothin…

The real power move would be to start targeting prospective students. Universities will start caring of enrolment goes down.

They might not care even then.

Journalism is an interesting case study. Their revenues went into sharp decline around the start of the century, dropping by more than 50% in some cases. Concurrently polls showed trust in journalism was plummeting, that people were tuning out because they felt that what they read wasn't reliable.

Journalism didn't solve this problem by rebuilding trust and their audience. Instead they started reforming as non-profit institutions and taking huge payments from philanthropists to push specific agendas, without alerting readers to that fact.

https://apnews.com/article/science-business-arts-and-enterta...

These grants aren't tied to readership but rather to the content of what's published, so they can produce lots of articles even if nobody reads them. Those articles are then used as "reliable sources" to control Wikipedia, and from there what search engines and LLMs say.

If students start checking out at scale (already happening), or if governments cut funding (already happening in the USA), their most likely move is to just replace the revenue with money from the Gates Foundation et al, and keep going just as they are. Their primary role will switch from education or research to the propagation of approved narratives.

Re: Evidence of Fraud in an Influential Study About Procrastination

#168

Post by HPsquared »

jarenmf wrote: post #133

Science is self-correcting, but sometimes, the correction mechanism it is very slow and inconsequential enough that people are incentivized to just publish anything. Peer review is merely a quality check—and not a very good one. Just the other day, I ran into a paper in my field that claims to have solved a central problem that scientists have been working on for 50 years, single-authored, using some fancy mathematic…

It's a bit like how people believe markets are self-correcting. Yes, only over an infinite time horizon, in theory. Edit: and like chemical equilibrium, you could say it depends on the "rate of wrongness production" compared to the "rate of correction". Edit edit: self-correction of course must fight against the higher virality of work selected for its impact rather than correctness, and incorrect work can then spawn further incorrect work, like an autocatalytic process or epidemic.

Re: Evidence of Fraud in an Influential Study About Procrastination

#169

Post by mike_hearn »

Quekid5 wrote: post #136
hirvi74 wrote: post #56

Earlier quoted context omitted.

Does some sort of batched style of research already exist? By batched, I mean a study where n number of teams all agree on a single methodology, and then conduct the same experiment simultaneously. Each team could then publish their results independently and the teams could publish one aggregate. I know meta studies attempt to do this in a way, but they can sometimes have their own problems too. I cannot recall comin…

Not quite what you're asking, but pre-registration does alleviate e.g. p-hacking and such without the need for multiple teams. It doesn't do much for outright fraud, but still...

Only if the study matches its pre-registration and someone is checking that. Which it turns out, they often don't.

https://www.compare-trials.org/

Re: Evidence of Fraud in an Influential Study About Procrastination

#170

Post by suddenlybananas »

tyrabound wrote: post #66
trompetenaccoun wrote: post #58

Earlier quoted context omitted.

We'd find problems with many if not most published studies if we scutinized them the same way. People like him get discovered because they step out of their academic bubble into the light of the world stage. And mind you this is psychology, the entire discipline is only borderline scientific anyway. But even studies in the hard sciences are very often flawed if not outright fabricated. Adjusting data sets to fit the…

It is still seared into my mind when a PhD openly stated in a meeting once that she likes to draw the graphs she needs for her papers and then goes looking for the data she needs to support it. I’m sure I can imaging what kind of looking she did based on the fact that she was little more than a slick, fast talking, skin revealing, fast moving, usual suspect con man.

That's just coming up with a hypothesis.
Post reply on HN