Live data from Hacker News

Credder Wants to Create an Equivalent to “Rotten Tomatoes” for News

mondaynote.com

221–230 of 247 posts

Re: Credder Wants to Create an Equivalent to “Rotten Tomatoes” for News

#221
post #145

Good luck to them if course, but I am not sure this will work. Couple of reasons come to my mind: 1. How will I know that news have been reviewed by qualified people? Technically this should be news agency/source's job. 2. By the time the score will depict the news' quality, the damage may be already done. 3. RT is based on people's opinions, but news should be fact checked. What I think it should do, is to score the…

Yep. Rating news is a terrible idea. The news should ideally be as objective as possible (or otherwise it's called "Fake News"), they don't always come out as a popular / most highly rated one. Movies, on the other hand, are highly subjective. Elon Must has a similar idea where he wanted to rate journalists but later he gave up.[1] https://www.inc.com/brian-hart/elon-musk-wants-to-let-public... Ultimately, you as an…

I'm a life-long (but intermittent) NYT reader. Coverage of SpaceX (in NYT, WSJ, and WaPo) changed how I think of journalism.

Faced with the usual issue of good newspapers getting every topic you know about not-quite right, variously wrong, I used to explain it by an analogy. It was like when newspapers closed their foreign offices. Loss of ground truth, of sanity checking, of insight and nuance. An abundance of misinterpretation and missing the point. My analogy was that for science, and tech policy, and diplomacy, and engineering, and manufacturing, and so much else, the foreign offices were closed. Generalist journalists, not embedded in the various professional communities of expertise, writing from afar, and getting it variously wrong.

What struck me about SpaceX coverage was the process failure. Not the getting things wrong - that was unsurprising. But there was no sign of learning, of process improvement. For instance, there was a successful short-term disinformation PR campaign targeting the press (around Zuma satellite failure attribution). But afterwards, there was little sign of recognition, reflection, remediation. No sign that they wouldn't fail just as hard if the same thing was done again a month later. And journalistic devices to pursue quality, seemed not just poorly executed, but inadequate to need.

So my new analogy is with medicine. Like journalism, medicine has long had folk practices for pursuing quality. But medicine has had it's "oh, shit" moment. The realization that medical error was a leading cause of mortality. That even easy and important standards of care were failing to be consistently executed. That there were decades of experience in other fields, on how to engineer process to systemically achieve quality and avoid failure, to which medicine had been oblivious. Journalism seems still oblivious.

So I suggest that journalism as a profession, and even the best of newspapers as institutions, have not yet realized just how badly they are failing. Or recognized the need for, or even the possibility of, systemic process improvement.

Perhaps the existence of the OP project might help with that.

Re: Credder Wants to Create an Equivalent to “Rotten Tomatoes” for News

#222
post #92

Earlier quoted context omitted.

If you want to go on a rant, is it too much to ask that you list examples and how they apply, rather than assume everyone here is as on top of Internet drama like you might be? Your examples might even make it clear why a RT for news would be even more difficult than the one that exists for movies.

That's a good question. I'm not sure what the other poster had in mind, and I suspect it's not the same, but RT has a bunch of rating problems that would probably become far more egregious for news than they are for movies. (Leaving aside the tricky question of how to translate the formats: is a movie like a news story, or a news source? Does news really have an existing aggregation-worthy source like movie reviews?)…

> BeerAdvocate, of all things, takes on this problem with interesting meta-info like "reviewer's average distance from consensus".

There's a lot to be said for intelligently dealing with ratings and their metadata in ways like this.

For example, the Kappa statistic: https://en.wikipedia.org/wiki/Inter-rater_reliability

Presenting crowdsourced ratings as mere averages -- or really any collapse onto a single scalar -- is a big step up from no info at all, but it's hardly the best one can do even with well-known statistical techniques.

Re: Credder Wants to Create an Equivalent to “Rotten Tomatoes” for News

#223
This is just going to descend into review-bombing chaos (look at what happens to games on Steam, and then think about how much more biased people are about news sources).

A much better idea would be a curated news source registry that does objective fact-checking and scores news sources based on various criteria, with the relevant research/scoring mechanisms shown openly so people can verify the results themselves. Like Snopes meets Charity Navigator (https://www.charitynavigator.org/).

Re: Credder Wants to Create an Equivalent to “Rotten Tomatoes” for News

#225
I think I posted this somewhere else but here it goes.

I want a system that rates journalists not news. I want to see my previous ratings of this journalist and be able to base it on that.

I want the ability to rate a pieces objectivity.

I want to see ratings from users who have been proven to know how to rate objectivity.

I want a system that uses Machine Learning to break me "out of" my information silos by promoting the opposite opinion, and saying as much. "Because you like, Capitalism here's an argument for Socialism."

I want a system that making ratings less of a popularity contest by limiting them in a meaningful way (this is pie in the sky I'll take the first two).

I don't think it's possible to create the things I want, but that's what I want. Not Reddit for news. I've seen how that works.

Re: Credder Wants to Create an Equivalent to “Rotten Tomatoes” for News

#226
People in this thread do not seem to appreciate the real threat to society that censorship and propaganda pose.

A widely used news rating site will be used to suppress political dissent and alternate views.

Maybe take a look at history and the world. Do you think that Chinese authorities would like a tool installed on everyone's computer that could rate the trustworthiness of a news article or site? Of course they would love it because they could just downgrade any reporting that was critical of the party.

Based on the number of young people who seem to be begging for authorities to control their information, it appears that the China model is coming to the United States.

Re: Credder Wants to Create an Equivalent to “Rotten Tomatoes” for News

#227

Earlier quoted context omitted.

While that's true for the possibility of conflict of interests, do you have any proof/data showing they've actually flexed that conflict for their own gain? Undermining the trust in the authority of the aggregate values is a huge issue I'm sure they'd much rather avoid as much as possible in order to extract as much value out of the platform as possible. I know a bunch of folks recently complained that they were tryi…

When avoiding a conflict of interest matters, normally there isn't a standard of requiring evidence of any impropriety for behavior to be deemed inappropriate. Merely not avoiding the conflict of interest is enough. Granted, for something like civil lawsuit or criminal charges this alone isn't enough and evidence is required, but for matters such as violating professional ethics not disclosing a conflict of interest…

Perfectly reasonable. Thanks

Re: Credder Wants to Create an Equivalent to “Rotten Tomatoes” for News

#228
post #40

I wrote something that was trying to do this a back in January that was called desational[1]. What I did was very basic but the process I did went as follows. 1. Break up news articles into sentences 2. Pay Amazon MTurkers to tag sentences as `news` or `not-news` 3. Train a text classifier based on the results for Mturk 4. Process new articles using the trained model to identify and strip out parts pf the article tha…

Very interesting idea. However, having dabbled with Bayesian classifiers myself, I suspect that your decision to break the article into sentences may be a fatal flaw...both the Mturkers and the classifier lose out on valuable context that way. I know because I took a a similar approach in one of my own projects, and the classifier could never gain enough 'confidence' to classify things, even after thousands of traini…

No need to apologize. I'm actually not using a generative classifier like Bayes. I'm using Conditional Random Fields which is a discriminative classifier. I found that based on my training set, CRF produced higher accuracy.

That being said, I've also thought of way more ways to make the processing of the articles a lot better. I had an idea using an LSTM and another idea which would use an RNN to map the sensational article to a non-sensational article. The only problem with some of these approaches is that they would require someone to read an article and write a non-biased form of the article.

I do like your approach, do you have any results from what you've found tinkering around?

Re: Credder Wants to Create an Equivalent to “Rotten Tomatoes” for News

#229

Earlier quoted context omitted.

How would a prediction market help determine whether a news article about current events is accurate?

I believe that the theory is that having "skin in the game" in the form of money would help optimize the news for something other than clickworthiness.

How would a prediction market give you skin in the game on reports about past events? What are you putting your money down on?

Re: Credder Wants to Create an Equivalent to “Rotten Tomatoes” for News

#230

Earlier quoted context omitted.

How would a prediction market help determine whether a news article about current events is accurate?

It would provide a common prior for any empirical claims. Scott Alexander puts it very well here: A democracy provides a Schelling point, … an option which might or might not be the best, but which is not too bad and which everyone agrees on in order to stop fighting. … In the six hundred fifty years between the Norman Conquest and the neutering of the English monarchy, Wikipedia lists about twenty revolts and civil…

Most news isn't about the future, but about things that have already happened. How would you run a retrodiction market?
Post reply on HN