A Big Study About Honesty Turns Out to Be Based on Fake Data • Hacker News Board

Re: A Big Study About Honesty Turns Out to Be Based on Fake Data

#91

Post by wjnc » Sun, Aug 22, 2021, 6:20 AM UTC

peterthehacker wrote: post #83

wjnc wrote: post #46

Earlier quoted context omitted.

It’s not damning per se. That depends on the sampling (or otherwise it would be a tiny, tiny insurer). I’ve done studies where I sampled an insurance population to get equal group size on a few key parameters because we then did follow up questionnaires and I needed to account for non response. No point in random sampling then because all I probably would get was data on the largest groups in the population. As it tu…

The sample size is in the article: > Nearly 13,500 drivers were randomly sent one of two policy review forms to sign… The distribution referenced is of mileage, which you’d expect to have some kind of right-skewed, continuous distribution.

Look I’m not defending Ariely, just saying that random sampling can be more complex than each record exactly the same sampling chance. And if the population has a few overpopulated groups but you’d like results for all groups, you don’t throw extra samples at it but use smarter sampling.

Re: A Big Study About Honesty Turns Out to Be Based on Fake Data

#92

Post by fastaguy88 » Sun, Aug 22, 2021, 6:24 PM UTC

Sebb767 wrote: post #87

fastaguy88 wrote: post #52

Earlier quoted context omitted.

This seems like a generalization that needs to limited in scope. I think the science is settled about Newton’s laws, Maxwell’s equations, Avogadro’s number, and DNA as the genetic material in bacteria, archaea, and eukaryotes. For pretty much anything taught in a 100 level science course, the science is largely settled. There are controversies in science, and those controversies make for good popular science press, b…

> I think the science is settled about Newton’s laws Except it isn't. These laws have been extended once by general relativity already and there's a non-zero chance that the dark matter problematic will prove them insufficient once again. I don't know enough about the other fields, but it's not unlikely that they have similar problems. The point is, things are really complex and surg edge cases. And if you make final…

Different people may have different ideas about what “the science is settled” means. In many political discussions, the phrase actually is used to argue for or against the quality of the data. Often, when discussing climate change, or perhaps even evolution, the phase is used to say the data is clear, the earth is warming, or all living organisms share a common ancestor. Likewise, I think the science is settled that vaccines save lives. And the science is certainly settled that heavy and light objects fall at the same rate in a vacuum.

Being skeptical that “the science is settled” because of some poorly understood edge case makes it very difficult communicate where there is uncertainty, and where there isn’t (because the data is clear and the science well understood).

Re: A Big Study About Honesty Turns Out to Be Based on Fake Data

#93

Post by 650 » Sun, Aug 22, 2021, 9:15 PM UTC

satisfice wrote: post #84

vegetablepotpie wrote: post #20

Earlier quoted context omitted.

> I’m sick and tired of people saying “the science is settled” or “trust the science”. You’re right, but also, there’s more to it. There is a difference between what people say and what people mean and context is important. When I’ve heard people say “trust the science” it is always directed to people who have demonstrated that they do not understand the scientific method and the meaning of the word “healthy skeptici…

I say “follow the science” about Covid and Global Warming, for instance. Yes, that is a political statement. It’s also rational and reasonable. I am a tester. Science is never complete, just as testing is never complete. But stupid people think they can ignore bad bugs and get away with it; and stupid people spin fantasies that maybe all those experts are completely wrong about Ivermectin or vaccines. Trust science i…

It is difficult to get a man to understand something when his salary depends upon his not understanding it.

Re: A Big Study About Honesty Turns Out to Be Based on Fake Data

#94

Post by whimsicalism » Sun, Aug 22, 2021, 9:15 PM UTC

wjnc wrote: post #89

whimsicalism wrote: post #61

Earlier quoted context omitted.

Sorry, but you seem to be suggesting that non-response bias would be responsible for the uniform distribution seen in the Update mileage digits as well as the uniform miles driven distribution?

Theoretically, yes. I’ve used stratified sampling in this kind of research, since the underlying portfolio is so skewed.

I really don't see how any random sampling mechanism (unless you're literally stratifying based on the last digit of the odometer) would cause these sorts of results, please explain further.

Re: A Big Study About Honesty Turns Out to Be Based on Fake Data

#95

Post by 650 » Sun, Aug 22, 2021, 9:19 PM UTC

BenjiWiebe wrote: post #81

throwawayboise wrote: post #78

Earlier quoted context omitted.

Claiming that the vaccines were necessary to "end the pandemic" without factoring in the possibility of variants for which the vaccines were not as effective was not scientific thinking.

Maybe they expected 99%+ to get vaccinated before a vaccine-avoiding variant got out.

They expected 7 billion+ people to get vaccinated?

Re: A Big Study About Honesty Turns Out to Be Based on Fake Data

#96

Post by jldl805 » Mon, Aug 23, 2021, 5:25 AM UTC

whimsicalism wrote: post #94

wjnc wrote: post #89

Earlier quoted context omitted.

Theoretically, yes. I’ve used stratified sampling in this kind of research, since the underlying portfolio is so skewed.

I really don't see how any random sampling mechanism (unless you're literally stratifying based on the last digit of the odometer) would cause these sorts of results, please explain further.

Or, even better share the data you reference from "this kind of research".

Re: A Big Study About Honesty Turns Out to Be Based on Fake Data

#97

Post by wjnc » Mon, Aug 23, 2021, 10:48 AM UTC

wjnc wrote: post #89

whimsicalism wrote: post #61

Earlier quoted context omitted.

Sorry, but you seem to be suggesting that non-response bias would be responsible for the uniform distribution seen in the Update mileage digits as well as the uniform miles driven distribution?

Theoretically, yes. I’ve used stratified sampling in this kind of research, since the underlying portfolio is so skewed.

These are customers they already know I presume. So they have an inkling of the expected mileage (at least: we price on expected mileage). So you repeatedly sample dropping some samples to get enough filling in all mileage baskets. It’s a stratified sample and random.

Say you have 10k customers with 1k women. You want to take a sample And want enough power to get answers on women. In that case you do stratified sampling.

Re: A Big Study About Honesty Turns Out to Be Based on Fake Data

#98

Post by mzs » Tue, Aug 24, 2021, 5:13 PM UTC

jsnell wrote: post #85

unreal37 wrote: post #24

Earlier quoted context omitted.

I don't think that's "clear" at all. Even the blog link doesn't come to that conclusion.

The trail of the data goes cold at Ariely. He is refusing to tell who supposedly gave him the data, and can't keep his story straight. He is also marked as the creator of the Excel file in the metadata [0]. If he received the data in Excel format, he wouldn't be the creator. If he received it in some other format. It's obvious that Ariely is dirty, and I'd bet we'll hear of other similar cases soon enough from other…

from the submitted article:

>And this is not the first time questions have been raised about Ariely’s research in particular. In a famous 2008 study, he claimed that prompting people to recall the Ten Commandments before a test cuts down on cheating, but an outside team later failed to replicate the effect. An editor’s note was added to a 2004 study of his last month when other researchers raised concerns about statistical discrepancies, and Ariely did not have the original data to cross-check against. And in 2010, Ariely told NPR that dentists often disagree on whether X-rays show a cavity, citing Delta Dental insurance as his source. He later walked back that claim when the company said it could not have shared that information with him because it did not collect it.

from "Dan Ariely was suspended from research at MIT after conducting unauthorized experiment with human subjects" ^

>We talked to sources at MIT, who said that the way in which the placebo experiment was conducted led to dispute and eventually to Ariely's departure. Sources familiar with the study said that Ariely did not request the necessary permissions from the IRB (Institutional Review Board) of MIT that is called CUEHUS, and did not fulfill the required protocols to conduct the experiment, which included administration of electric shocks to 80 participants, and administration of cheaper or more expensive placebo medications.

>When this was discovered after one of the participants complained, the ethics committee reached out to Ariely for an explanation. Emails that have come to our attention reveal that Ariely's reply was amused in tone, indicating he had a general protocol for this type of experiment of this type called "electric shocks", and he followed them.

^ https://www.ha-makom.co.il/post-tomer-dan-ariely-mit-suspent...