Credder Wants to Create an Equivalent to “Rotten Tomatoes” for News

That's a good question. I'm not sure what the other poster had in mind, and I suspect it's not the same, but RT has a bunch of rating problems that would probably become far more egregious for news than they are for movies. (Leaving aside the tricky question of how to translate the formats: is a movie like a news story, or a news source? Does news really have an existing aggregation-worthy source like movie reviews?) RT provides percentage scores for both audience and critic reviews. Those percentages are not averaged scores, but percentage of positive reviews. There's good reason for this in both cases, but it causes a lot of weird side effects.

- For audience reviews, reviewers rate out of 5 stars, and the score is the percentage of reviews at least 3.5 stars.

The goal is to insulate scores from extreme opinions and odd voting patterns. On IMDB, which doesn't do this, a few low scores can destroy a movie's rating, and movies that invite careful analysis are effectively penalized. The Dark Knight ends up rated higher than Schindler's List, but the latter has lots of slightly-imperfect scores with incredibly positive review text.

The basic side effect is the loss of all meaningful score data. That punishes daring movies: 100 4-star reviews equals 100% while 50 3-star and 50 5-star reviews equals 50%, but I would much rather see the second movie. It also hides deep flaws: The Boondock Saints outscores 2001, but a look at the reviews shows that its "rotten" ratings are often 1 and 2-star reviews calling it garbage, while 2001 is full of 3-star reviews like "slow and confusing, but fascinating". Another problem here is that naive binary scores suffer from reviewer skew and vote-bombing campaigns. It's less bad than IMDB because extreme scores are 'softened', but still worse than systems which relate individual scores to the reviewer's other opinions. BeerAdvocate, of all things, takes on this problem with interesting meta-info like "reviewer's average distance from consensus".

- For critical reviews, each review is reduced to a binary opinion and the score is a weighted average of positive review count.

The goal there is to condense varied or absent rating schemes into an aggregation-friendly score and avoid reviewer skew. Metacritic, which doesn't do this, ends up systematically distorted by which outlets review things. Unscored reviews like Ars Technica's (on video games) are skipped, while sites with extremely skewed scores are included by some unspecified weighting system. As a result, Metacritic lists game reviews averaging 75-89 as "generally favorable", but there's a widespread understanding that any major-release product below 85 is terrible. (GameSpot once infamously fired a reviewer for scoring a game 6/10.) It's largely the same problem as Uber and Amazon ratings: does a perfect score mean "exceptional" or "nothing wrong"? RT tries to dodge the problem by saying "good" or "bad", with a weighting to accommodate a reviewer's 'signal strength'.

The side effect here is, again, the total loss of secondary data. RT sometimes gets reviews actively wrong; something like "despite lovely action sequences, this movie is a meandering waste of time" will get parsed as "lovely action sequences". That's probably an automation error, but lots of other reviews can't be clearly declared positive or negative even by humans. And within a categorization, there's a great deal more lost data. Ebert produces 'fresh' for The Avengers because "It provides its fans with exactly what they desire," but he produces 'fresh' for There Will Be Blood by declaring it "A force beyond categories."

- The reason all of this works alright for Rotten Tomatoes is that it's fundamentally answering one question: "will I enjoy watching this?" The Avengers isn't on par with There Will Be Blood, but if you like the description and it's rated fresh, you'll enjoy either one. If you see that critics hate The Boondock Saints and audiences don't, well, you probably know whether you're somebody who likes that kind of cult classic.

But what of news? People already shred Snopes whenever it tries to deal with assessments like "factually correct but misleading" or "incorrect statistic but the real number also supports the same thesis". The entire point of news reviews is to not rely on "will I enjoy this story?", so all the things RT quietly glosses over will become critical problems.

In my opinion, this doesn't really offer any value as the only way to get a good understanding of the news is to read from lots of different places and come to a view. Using this strategy then consuming a truly "random" site is better than consuming pre reviewed site as there is no bias in selection or outcome.

It's not like films, where each one is independent (except marvel and dc comics obviously :)

"yet you state it as a fact and that we should trust mainstream news."

This is the opposite of what I said, I said we should trust the process Mueller used to reach his conclusions, NOT the media, who got a lot wrong.

Please refute the indictments Mueller issued for Russian nationals working to influence the outcome of the election, to back your claim Russia did not make an effort to influence the election.

Unless you are hung up on how massive counts as "massive", which is a pointless and stupid debate.

The media clearly got a lot wrong, but rejecting the Mueller report's conclusions is a much higher bar, and I will want to know how you have more information about the topic or more credibility than Mueller and his team and specifics about which parts of the indictments were incorrect.

Unfortunately what many people consider to be "reputable news" is just whatever source happens to agree with their perspective/opinion at the time

Above all else, RT needs their ranking to be perceived as legitimate. The general public knows there are a handful of old movies that should certainly be in any legitimate Top-100 list. Casablanca, Citizen Kane, The Wizard of Oz, etc. These are movies recognized by the general public as movies that are considered masterpieces. If those movies were not in the list, then the general public would not consider the list legitimate. I'm talking about people who've never seen any of those movies, but are aware of their reputations and will dismiss any Top-100 list that doesn't include those movies.

So the consequence of that is clear. If you are tasked with creating a Top-100 movie list, you better tune whatever algorithm you use to include those movies. You'll butcher your beautiful algorithm to ensure those handful of classic movies always rank high. You might go so far as to put those movies in a special list and inject them arbitrarily into your Top-100 list, circumventing the normal ranking algorithm, because having those particular movies in the Top-100 is a design requirement. It's more important that those movies be in the list than it is for your ranking algorithm be fair and unbiased.