Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Do Social News Sites Deteriorate? (Analysis of 1.8M HN Comments)

blog.effectcheck.com

Post reply on HN
Print view View original

21–30 of 57 posts

Re: Do Social News Sites Deteriorate? (Analysis of 1.8M HN Comments)

#21

Post by rhizome »

hugh3 wrote: post #15
tansey wrote: post #9

Hi everyone, So there seems to be a lot of skepticism about whether our algorithm can actually measure emotional impact accurately. For the long answer, I'll refer you to the about page[1] for EffectCheck. For the short answer: My co-founders are an AI PhD and a Clinical Psychology PhD. They spent three years curating a huge dictionary of words using a methodology similar to the Harvard Psychosocial Dictionary [2], b…

I can easily believe that you can predict quite well ratings of Amazon reviews by a simple keyword-counting algorithm. Just by counting words like "good", "enjoyed", "fantastic" vs "terrible", "boring", "awful" you're gonna get a very strong correlation in that limited domain. But what tests have you done on your broader methodology? What experiments can you really do to figure out the extent to which the use of the…

It seems that the point is to introduce their special-sauce black box, with an argument to authority about its methodology. I think the correlations you ask for are where the problems will lie, in that there is a value judgement that is being hidden. If I can put myself out on a limb here, I'd say that that measurement is going to be fundamentally unscientific.

Re: Do Social News Sites Deteriorate? (Analysis of 1.8M HN Comments)

#22

Post by raganwald »

tansey wrote: post #3
nbashaw wrote: post #2

How does the author measure anxiety, hostility, depression, confidence, compassion, and happiness? That seems to be left out.

Hi! My startup [1] is based on an algorithm that can measure the emotional impact of text. Our blog is where we post interesting results from applying our algorithm. [1] http://effectcheck.com

Speaking of your startup, it appears to have "Oracle Pricing" as its revenue model. You have to talk to a salesperson to get a price, which is often code for "We need to know how much you can afford and how much you are willing to pay before we quote a price."

I like everything else I see.

Re: Do Social News Sites Deteriorate? (Analysis of 1.8M HN Comments)

#23

Post by jlees »

tansey wrote: post #14
jlees wrote: post #13

Earlier quoted context omitted.

You claim that context is not important, but surely in a domain-specific corpus such as HN comments you have to take domain, if not context, into account? For example, words that are generally positive may be only used in a snarky context in hacker circles, whereas words that are generally negative may similarly be used with positive affect (e.g. the word "hacker" itself). Did you customise your lexicon at all for th…

...surely in a domain-specific corpus such as HN comments you have to take domain, if not context, into account? That's a great insight and I believe you are correct. In general, I think the domain does have an impact and I am working on some automatic demographic profiling technology. However, for a first-look, I think our general word dictionary covers an abundant set of terms that do not vary much between domains.…

Is there really much of interest in press releases, or were they just a handy corpus you had around? I started my PhD off looking at affect in scientific papers, and changed tack because there basically is none.

Anyway, I guess I'm not clearly understanding your definition of "anxiety" because few of those words evoke anxiety as I would define it -- not? bag? away? I'll try to dig into whatever you've published to deeper understand your system, but I'm not convinced by the standalone word approach yet, though I see what you're trying to do (I think). I'd really want to check out the bigrams/trigrams, even if you don't do full parsing or any intermediate sophistication. Obviously there's an efficiency aspect, so I guess if your results are good enough already and you have customers, no need to make sacrifices.

On a tangent, how does your system interpret common HN words like "healthcare", "Obama" and "education", I wonder? Could you have an "intellectual" or "hacker-ish" dimension based on manual categorisation of words into "this is stuff HN should be talking about" vs "this is stuff that belongs on reddit"?

Re: Do Social News Sites Deteriorate? (Analysis of 1.8M HN Comments)

#24

Post by tansey »

bhousel wrote: post #18

Any plans to compare against corpora from other social news sites? BTW, I think what you're doing is brilliant, and I can think of about a dozen practical applications for your technology.. Ignore the naysayers.

Thanks! I'd be happy to analyze of corpora like reddit or Digg, but I don't want to scrape the sites as that is a great way to have my IP banned. Are there publicly-available datasets that I can download somewhere?

Re: Do Social News Sites Deteriorate? (Analysis of 1.8M HN Comments)

#25

Post by tansey »

rryan wrote: post #10

Where are the confidence intervals and error bars? In order to be taken seriously when aggregating 1.8MM comments, you need those. The variability in pg's plots makes me think the data for any individual is going to be just as noisy.

We are looking at a time-based analysis of the monthly means of PG and the HN community. The null hypothesis is that the slope of a regression line should be 0. Since we are dealing with a time-based analysis of the mean, the variance of each bucket is irrelevant. I re-ran the analysis to make sure that my results were statistically significant:

Anxiety/Confidence PG: p Hostility/Compassion PG: p Depression/Happiness PG: p Note that for the HN comments, even though I used a 2nd order (parabola) fit for the graphs, the p values above are for a linear regression as that is the more appropriate fit for determining statistical significance here.

Re: Do Social News Sites Deteriorate? (Analysis of 1.8M HN Comments)

#26

Post by mef »

Thanks for this interesting analysis.

I would challenge the assumption that the symptoms of SND include a decrease in the positive emotion measured in the average HN comment.

Is it not possible that there is a growing segment of the HN user base which posts and enjoys the kind of comment that induces negative sentiment in people like pg? Would that not raise the sentiment of the average HN comment, while at the same time lowering the sentiment of the average "good old days" user?

Re: Do Social News Sites Deteriorate? (Analysis of 1.8M HN Comments)

#28

Post by tansey »

hugh3 wrote: post #15
tansey wrote: post #9

Hi everyone, So there seems to be a lot of skepticism about whether our algorithm can actually measure emotional impact accurately. For the long answer, I'll refer you to the about page[1] for EffectCheck. For the short answer: My co-founders are an AI PhD and a Clinical Psychology PhD. They spent three years curating a huge dictionary of words using a methodology similar to the Harvard Psychosocial Dictionary [2], b…

I can easily believe that you can predict quite well ratings of Amazon reviews by a simple keyword-counting algorithm. Just by counting words like "good", "enjoyed", "fantastic" vs "terrible", "boring", "awful" you're gonna get a very strong correlation in that limited domain. But what tests have you done on your broader methodology? What experiments can you really do to figure out the extent to which the use of the…

> Also, as someone else said, where are the error bars? If there really is a correlation between word choice and other metrics, then some simple statistics should give you error bars on your other metrics, right?

Please see my response to the original question about this. [1]

"That joke kills me!" vs. "I'm going to kill you!" (corrected grammar as we do not usually focus on slang)

They do not score exactly the same, though they are both very high on anxiety, hostility, and depression. The former has high marks for happiness and compassion due to the word "joke" being used.

[1] http://news.ycombinator.com/item?id=2603946

Re: Do Social News Sites Deteriorate? (Analysis of 1.8M HN Comments)

#29

Post by pgroves »

What jumps out at me is that the community curves all peak in the months following May 2009, which is when the stock market bottomed. The community was least happy when the economy was at it's "darkest before the dawn" moment.

In other words, I wonder if these graphs are just tracking general sentiment, and would look the same for any site.

Re: Do Social News Sites Deteriorate? (Analysis of 1.8M HN Comments)

#30

Post by tansey »

jlees wrote: post #23
tansey wrote: post #14

Earlier quoted context omitted.

...surely in a domain-specific corpus such as HN comments you have to take domain, if not context, into account? That's a great insight and I believe you are correct. In general, I think the domain does have an impact and I am working on some automatic demographic profiling technology. However, for a first-look, I think our general word dictionary covers an abundant set of terms that do not vary much between domains.…

Is there really much of interest in press releases, or were they just a handy corpus you had around? I started my PhD off looking at affect in scientific papers, and changed tack because there basically is none. Anyway, I guess I'm not clearly understanding your definition of "anxiety" because few of those words evoke anxiety as I would define it -- not? bag? away? I'll try to dig into whatever you've published to de…

>Is there really much of interest in press releases, or were they just a handy corpus you had around?

Actually, both. Corporate marketing departments spend a lot of effort crafting the proper press releases. Entire agencies exist to help companies with press releases. I think we make a great companion tool for professionals in that area. We have an example of how EffectCheck can help with press releases on our blog [1].

>On a tangent, how does your system interpret common HN words like "healthcare", "Obama" and "education", I wonder? Could you have an "intellectual" or "hacker-ish" dimension based on manual categorisation of words into "this is stuff HN should be talking about" vs "this is stuff that belongs on reddit"?

We certainly can do that, and we offer it as a service to clients that want hand-crafted, demographic-specific dictionaries (like political campaigns).

[1] http://blog.effectcheck.com/2011/04/14/delivering-bad-news-a...

Post reply on HN