Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Wikipedia influences judicial behavior

csail.mit.edu

Post reply on HN
Print view View original

61–70 of 117 posts

Re: How Wikipedia influences judicial behavior

#61

Post by Spivak »

pdonis wrote: post #59

"because randomized experiments are the gold standard for this type of research, we know the effect we are seeing is causation, not just correlation" No, they don't. Correlation is not causation, even if you see it in a randomized experiment. With shoddy reasoning like this, it's no wonder science has a replication crisis.

https://nulib.github.io/moderndive_book/7-causality.html

https://bolt.mph.ufl.edu/6050-6052/unit-2/causation-and-expe...

https://towardsdatascience.com/establishing-causality-part-1...

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6235704/

https://escholarship.org/uc/item/42v4w8k1

http://ippsr.msu.edu/public-policy/michigan-wonk-blog/random...

https://www.cs.cornell.edu/courses/cs1380/2018sp/textbook/ch...

I’ll leave open the possibility that it’s everyone else that’s wrong but RCTs are used to establish causality and are as much “proof” as you’re gonna get in science.

Hell ya know what I’ll just let the actual paper explain it.

> The second, more important advantage of randomized field experiments is that they can distinguish causation from correlation. The ability to prove causal relationships derives from the combination of two characteristics. The first is having a control group, that is, a group unaffected by the intervention (in our case, publication of a Wikipedia article on the topic) that can be used as a counterfactual to estimate the size of causal effects. The second is randomization, that is, random assignment into the control and intervention groups. With sufficient data and a sound experimental design, the experiment can reduce the probability of being misled by correlation or noise to whatever arbitrarily small value is desired.

Re: How Wikipedia influences judicial behavior

#62

Post by asdff »

pdonis wrote: post #59

"because randomized experiments are the gold standard for this type of research, we know the effect we are seeing is causation, not just correlation" No, they don't. Correlation is not causation, even if you see it in a randomized experiment. With shoddy reasoning like this, it's no wonder science has a replication crisis.

What they did was fine with this setup. You might enjoy reading this article that covers the very perceived issue you are having with the research:

https://towardsdatascience.com/establishing-causality-part-1...

Re: How Wikipedia influences judicial behavior

#63

Post by asdff »

mikece wrote: post #3
stevenjgarner wrote: post #2

It seems the greatest value of Wikipedia is consistently as a repository of citations. The reliance of moderation or review of those citations is the question. EDIT: I hear complaints from students all the time that they are not allowed to cite Wikipedia. I tell them no you should instead cite the Wikipedia citations. They invariably tell me how much better they do academically because of that.

What would be the best alternative to Wikipedia at getting a list of citations on a topic quickly? For academic research I consider the content of the page as "Yeah, maybe" but the citations to be more useful in terms of digging to deeper sources faster. The argument could be made that the ability of almost anyone to edit Wikipedia is a form of peer review but for edge topics it's tough to tell who has the chops to b…

It depends on your field, but most fields publish review articles. These are written by a single author or two in the field, and perform no new analysis beyond just listing all the current info about the field. Here is an example for oncology, this one is periodically republished every few years as more info is learned by the field:

https://aacrjournals.org/cancerdiscovery/article/12/1/31/675...

Re: How Wikipedia influences judicial behavior

#64

Post by asdff »

MisterMower wrote: post #54
mountainb wrote: post #47

Earlier quoted context omitted.

The citations on Wikipedia articles are often either broken, in a foreign language, or not up to an academic standard. However, following this practice is what eventually lead me to stop using Wikipedia for anything. When I tried to trace the citations to make my research easier, I found it actually made it a lot slower because I kept running into junk. Before I actually paid attention to the citations on a broad sel…

They have weird requirements for citations. Want to cite a link to a blog where a guy has spent 20,000 hours researching a topic but doesn’t have a degree? Not allowed. Want to post a link to a NYT article written by a journalist with no relevant experience in that same field? Go for it. It’s really strange. It’s like they’re aiming for narrative over fact or something.

To be fair usually when the nyt article is wrong they issue a retraction and when the blogger is wrong they triple down

Re: How Wikipedia influences judicial behavior

#65

Post by the_af »

Spooky23 wrote: post #60
the_af wrote: post #52

Earlier quoted context omitted.

If I understand the article correctly, it's not about incorrect information -- i.e. it's not claiming Wikipedia misleads judges because the articles are wrong -- but about "influence" of Wikipedia on decisions; an influence that could be gamed by an adversarial actor. References themselves are gameable. They argue references mentioned in Wikipedia are more likely to be cited by a judge! This is not about the quality…

Judges would consult encyclopedias and similar resources 40 years ago. The difference is that a crowdsourced resource like Wikipedia is easier to manipulate by people who understand the system. There are plenty of PR specialists who get client articles pushed into Wikipedia or updated to their liking. Wikipedia is a treasure, but it’s also vulnerable to a bunch of different attacks.

Agreed. But that's not about the quality of the articles themselves, and certainly not about their references!

Re: How Wikipedia influences judicial behavior

#66

Post by actually_a_dog »

jasonhansel wrote: post #42
actually_a_dog wrote: post #33

Earlier quoted context omitted.

I disagree. The actual paper shows it was approved by the ethics boards of two universities. Here is a quote from the paper describing the actual methodology for creating the articles: > The experiment featured Wikipedia entries authored by faculty and by law students under faculty supervision, who each had access, through their university library, to all the relevant primary and secondary legal materials available t…

Unfortunately, some IRBs are notoriously bad at preventing unethical research practices, since in practice they are often more focused on ensuring adherence to various bureaucratic requirements. The issue isn't that they were misleading anyone; the issue is that they were, for research purposes, trying (successfully) to influence the outcomes of court cases without the subjects' consent.

No, it absolutely was not an attempt to influence judicial outcomes. It was a test to see whether publishing accurate analysis of precedential cases would cause those cases to be cited more than they otherwise would. Not only is there no harm in doing so (since the information is accurate), neither you nor the researchers can know whether any case outcomes were influenced. And, in fact, the analysis of the paper, which says that the cases were cited more often when the citing judge agreed with the conclusion of the case, which, if anything, suggests that the outcomes of these cases were most likely not affected.

Moreover, if you want to claim this study was unethical, you need to show that there at least could have potentially been harm as a result of it being carried out. Please explain how publishing correct legal analysis of cases anywhere could possibly cause harm.

In short, if you want to make this claim, you need to provide actual evidence, and not just outrage.

Re: How Wikipedia influences judicial behavior

#67

Post by bjornsing »

asdff wrote: post #62
pdonis wrote: post #59

"because randomized experiments are the gold standard for this type of research, we know the effect we are seeing is causation, not just correlation" No, they don't. Correlation is not causation, even if you see it in a randomized experiment. With shoddy reasoning like this, it's no wonder science has a replication crisis.

What they did was fine with this setup. You might enjoy reading this article that covers the very perceived issue you are having with the research: https://towardsdatascience.com/establishing-causality-part-1...

As I understand it a randomized controlled trial (RCT) can determine causation. But I don’t think “what they did with this setup” was an RCT. A real RCT in this context should IMHO assign new cases randomly to a “treatment” or “control” group. In this setup they assign legal precedents randomly, but all new cases are effectively under “treatment”. That’s not the same thing.

Re: How Wikipedia influences judicial behavior

#68

Post by setr »

sien wrote: post #38
jacobolus wrote: post #16

Earlier quoted context omitted.

It is entirely possible to fix those mistakes (one at a time) on Wikipedia.... if you do the research work to find out what happened. But this takes significant effort (like, a half-day of research to sort out one claim), and then sometimes back and forth with other Wikipedians to convince people that you actually chased down the real story. The problem is that for every mistake someone is willing to put effort into…

Not always. Any particular two or more wikipedians who decide they don't want something in can sometimes stop it. For example, in Australia there is a body that does sport participation statistics, Ausplay. They do this every year. It's a great source for sport statistics on Australia. Two wikipedians decided that these statistics were not permissible in the sport in Australia article. They won : https://en.wikipedia…

If I’m reading this right, I’m seeing a vote taking place in the “Survey” section, with 5 votes invoked — 2 no, 3 yes.

And an end result of No Consensus.

Framing this as if 2 Wikipedians exercising outsized power to produce this ruling seems disingenuous at best. And their basic objection (I only bothered skimming) of bias and ambiguity in the source/data/methodology seems fairly reasonable on its face; whether it’s correct I have no idea but it’d be a reasonable objection

As a policy, this whole thing seems like good behavior; the only gap is in the lack of voting participants. I suppose it is a real problem if the vote can’t be recast when more people are willing, but otherwise

Re: How Wikipedia influences judicial behavior

#70

Post by pdonis »

Spivak wrote: post #61
pdonis wrote: post #59

"because randomized experiments are the gold standard for this type of research, we know the effect we are seeing is causation, not just correlation" No, they don't. Correlation is not causation, even if you see it in a randomized experiment. With shoddy reasoning like this, it's no wonder science has a replication crisis.

https://nulib.github.io/moderndive_book/7-causality.html https://bolt.mph.ufl.edu/6050-6052/unit-2/causation-and-expe... https://towardsdatascience.com/establishing-causality-part-1... https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6235704/ https://escholarship.org/uc/item/42v4w8k1 http://ippsr.msu.edu/public-policy/michigan-wonk-blog/random... https://www.cs.cornell.edu/courses/cs1380/2018sp/textbook/ch... I’ll leave…

> RCTs are used to establish causality and are as much “proof” as you’re gonna get in science.

No, they're not. The real "gold standard" in science--the standard that prevails in, for example, physics or chemistry--is a controlled experiment. Not just a "randomized controlled trial", but a controlled experiment, where you can actually dictate exactly what state the things you are going to experiment on start out in. And the eventual output of controlled experiments is a predictive model--a model that can predict, accurately, what will happen if you run further experiments. That is what it takes to truly "establish causality".

But in most other domains, including the one under study here, controlled experiments simply cannot be done and predictive models with any kind of accuracy simply don't exist. The correct response to that unfortunate fact is to realize that we can never achieve the same level of confidence in these other domains as we can in domains like physics or chemistry where we can do controlled experiments. Unfortunately, the response "science" has settled on instead is to pretend that it doesn't matter--that because we can't do controlled experiments in these other domains, the universe will somehow magically lower its standards of what it takes to achieve the level of confidence we want. But the universe doesn't care what we can or can't achieve.

Post reply on HN