Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Evidence of Fraud in an Influential Study About Procrastination

datacolada.org

Post reply on HN
Print view View original

151–160 of 273 posts

Re: Evidence of Fraud in an Influential Study About Procrastination

#151

Post by mike_hearn »

MarceliusK wrote: post #115
hn_throwaway_99 wrote: post #92

Earlier quoted context omitted.

The fraud that Data Colada presents here is unambiguous. It was also no accident in my opinion that they are specifically mentioning that the 3 Excel files were “Last saved by” “Dan Ariely” I'm sure the statute of limitations has expired for this, but I really wish the Feds would prosecute cases like this as wire fraud instead of leaving insular academic institutions to self-police. This type of fraud is done for per…

I'd be pretty wary of stretching wire fraud law to cover this. Prosecutors deciding whether scientific results are fraudulent sounds like it could create its own set of ugly incentives

Fraud law doesn't need to be stretched, academic fraud is covered and criminally prosecutable in the USA for decades. It has to be because they're using these results to apply for grants paid from tax monies.

Occasionally someone is even prosecuted! Unfortunately, even when the government wins, the institutions just treat the resulting fines as a cost of business. See the Dana Farber Cancer Center: prosecuted by a third party on behalf of the government, found guilty of faking their science for a decade+, paid the fine - and nothing anywhere on the DFCC website mentions anything about it.

Re: Evidence of Fraud in an Influential Study About Procrastination

#152

Post by stackghost »

mike_hearn wrote: post #150
rvba wrote: post #145

Earlier quoted context omitted.

Would be nice if you kepy ypur career and created some magazine / journal that focuses on checks and pointing out flaws. That cheaters would dread being included there (would probably be a thick journal too)

They wouldn't dread being included there, not at all. The institutions don't care about fraud and reflexively defend it, ignore it or shoot the messenger. Look at the Arday nonsense. People are still discovering new lies he was telling but Cambridge went to the mat for him. Dan Ariely in particular has faced convincing fraud accusations for years: https://en.wikipedia.org/wiki/Dan_Ariely#Controversies_and_c... Nothin…

The real power move would be to start targeting prospective students.

Universities will start caring of enrolment goes down.

Re: Evidence of Fraud in an Influential Study About Procrastination

#153

Post by kodoman »

People for a long time have had far too much faith in peer review as and had a dim view of anything other then experimental results and work, despite how much fraud and just bad experiments their are. Not to mention how many variables their might actually be to account for in something like psychology or any of the social science's.

Deriving things from axioms and postulates alone is problematic but some one who can do both begins to actually do useful work in my opinion and some one who can accept the difficult their actually is in knowing anything about human psychology simply from studying it and underneath all experiments is assumptions and the people in the experiment who have so many variables. This is not like medicine human psychology is very flexible and for me though not a social scientist I feel of all theories of human psychology they are only applicable to people now and not even 300 years ago, though of course one must concede that the micro psychological principles might be applicable to humans more generally.

Re: Evidence of Fraud in an Influential Study About Procrastination

#154

Post by mike_hearn »

bawolff wrote: post #9

It does seem like its way too easy to do this kind of fraud. The study seems extremely easy to replicate (no fancy equipment needed, just ask some people to proofread), and yet nobody did until now. Is there some way to change incentives to get people to try and replicate? Assign undergrads to replicate random research papers?

Replication is overrated. There's no point in doing more of it. You'd just replicate a ton of bad science:

https://blog.plan99.net/replication-studies-cant-fix-science...

N.B. nobody needed to replicate this study. It only took basic sanity checks to spot that it was very unlikely to be true (and thus, very unlikely to replicate). And they went much further than needed. They could have stopped at "the claimed effect is much larger than the height difference between men and women". This obviousness is a common pattern:

https://www.fantasticanachronism.com/p/whats-wrong-with-soci...

> Prediction markets work well because predicting replication is easy. There's no need for a deep dive into the statistical methodology or a rigorous examination of the data, no need to scrutinize esoteric theories for subtle errors—these papers have obvious, surface-level problems.

So you could also say, how do we change the incentives to get people to peer review properly? That would be a lot cheaper.

The obvious solution is for people to get paid to audit work professionally, and be fired if they sign off on bad results. But that exists already, it's called a corporate lab. Managers are incentivized by carrots and sticks to properly understand what their researchers are doing and if anyone is doing fraud they will be fired, maybe prosecuted. Universities don't do this because so much of what they do exists purely to spend money, with no higher purpose. There's no specific goal or outcome they're trying to achieve, and so no reason to do unpleasant things like enforcing incentives to get there.

So in the end the solution is just to defund university research. The social structures are fundamentally unfixable. You'll never get people to care if the only purpose of their job's existence is to convince voters governments are funding science.

Re: Evidence of Fraud in an Influential Study About Procrastination

#155

Post by ahartmetz »

mike_hearn wrote: post #147
jarenmf wrote: post #133

Science is self-correcting, but sometimes, the correction mechanism it is very slow and inconsequential enough that people are incentivized to just publish anything. Peer review is merely a quality check—and not a very good one. Just the other day, I ran into a paper in my field that claims to have solved a central problem that scientists have been working on for 50 years, single-authored, using some fancy mathematic…

Science isn't self correcting. This is a harmful meme that we need to collectively put in the trash can, as it's constantly used by academics to evade accountability. The core problem is people conflating science with Science™. The scientific method is a set of techniques that help people discover and report true things. But there are two huge problems with it: 1. What exactly is the scientific method? It's not like…

3. Publish or perish, scientific publications (count x impact factor) as careeer credits. What a terrible yet hard to kill idea.

Re: Evidence of Fraud in an Influential Study About Procrastination

#156

Post by daymanstep »

coldtea wrote: post #129
Calazon wrote: post #125

Earlier quoted context omitted.

Perhaps we need a new/updated law to cover this kind of thing. This sort of academic fraud should absolutely be criminalized, but I'm not sure I'm the biggest fan of the "everything is wire fraud" approach to achieving that.

If it gets so bad that a society/academic system needs to criminalize such fraud, they have already lost that race. The only thing that really works is a culture where this rarely if ever happens, because of the peer respect and self-respect of academics.

Peer reputation and self policing works in a system where no money is at stake.

In the old days scientific research was only done by self-funded rich people with tons of money and free time.

Nowadays science is mostly done by people whose income and food depends on getting grants.

When your career and income are at stake, why would you take the risk of opposing Dan Ariely? Only someone who was already rich could do that.

Re: Evidence of Fraud in an Influential Study About Procrastination

#157

Post by 2b3a51 »

The last point in the OA about rounding the estimate of the time spent on the assigned tasks. Just wondering if the response would depend on the wording of the instructions to participants and the nature of the incentivisation.

Me being a bit of a boy scout / Belbin 'company worker', I would tend to use a timer (I'm a pomodoro method person for things like marking or checking references &c).

The rest, oh yes. I stopped paying much attention to studies based on dozens of undergraduates doing weird concocted tasks years ago.

Re: Evidence of Fraud in an Influential Study About Procrastination

#158

Post by Ekaros »

stackghost wrote: post #143
jarenmf wrote: post #133

Science is self-correcting, but sometimes, the correction mechanism it is very slow and inconsequential enough that people are incentivized to just publish anything. Peer review is merely a quality check—and not a very good one. Just the other day, I ran into a paper in my field that claims to have solved a central problem that scientists have been working on for 50 years, single-authored, using some fancy mathematic…

>I made a promise to myself that if no real action is taken, I will seriously consider changing careers. Personally I think you should name and shame the author as well as the publications and the names of the peer "reviewers" who clearly did sweet-fuck-all.

Not to forget the editors responsible at time. Why not leadership and board members if they can be found.

Making responsible people responsible publicly is most effective way. Fully deserved shame can do wonders to world.

Re: Evidence of Fraud in an Influential Study About Procrastination

#159

Post by aktenlage »

mike_hearn wrote: post #147
jarenmf wrote: post #133

Science is self-correcting, but sometimes, the correction mechanism it is very slow and inconsequential enough that people are incentivized to just publish anything. Peer review is merely a quality check—and not a very good one. Just the other day, I ran into a paper in my field that claims to have solved a central problem that scientists have been working on for 50 years, single-authored, using some fancy mathematic…

Science isn't self correcting. This is a harmful meme that we need to collectively put in the trash can, as it's constantly used by academics to evade accountability. The core problem is people conflating science with Science™. The scientific method is a set of techniques that help people discover and report true things. But there are two huge problems with it: 1. What exactly is the scientific method? It's not like…

My take on the self correction is, that later research usually cannot be build on wrong results. So over the decades or centuries the correct results prevail.

Re: Evidence of Fraud in an Influential Study About Procrastination

#160

Post by mike_hearn »

ghm2199 wrote: post #44

The reason why people(or ai agents) have not been able to systematically destroy bad research in fields where fraud is so rampant is because they data are not published or easily accessible without emailing the author and asking them. I sincerely hope that any self respectful journal that publishes papers should have a minimum bar: submit the data on a public link with no paywall or blockers and feature the link prom…

In this case even emailing the author didn't work. The data was obtained via some roundabout method.

But it's not the real reason IMHO. You could use LLMs to write takedowns like this one. I gave ChatGPT the paper PDF and some neutral prompts. Although it didn't go straight to 'this is clearly fraud' - it's too nice for that - it did compute and point out the absurdly huge effect sizes, and give a relatively low confidence the paper's central claim would replicate.

The real reason there aren't bots systematically destroying papers is because universities defend fraud, they don't punish it. You'd just be burning tokens to reveal lots of fraud with no outcome.

Post reply on HN