Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Investigating how the New York Times A/B tests their headlines

blog.tjcx.me

Post reply on HN
Print view View original

71–80 of 131 posts

Re: Investigating how the New York Times A/B tests their headlines

#71

Post by hkai »

CarVac wrote: post #69
mxcrossr wrote: post #52

Earlier quoted context omitted.

If people don’t like click bait I recommend not using news aggregators that only show the headline and are curated based on the votes of users who don’t read the articles!

Sounds like someone needs to make an aggregator that shows only brief auto-generated summaries, no headlines or images.

Even better, these autogenerated summaries should give most weight to paragraphs at the end of the article.

I often find that New York Times and other media put important details that sometimes contradict the headline or challenge the main idea of the article somewhere close to the end of the article.

This lets them still claim objectivity while preserving a highly polarizing message.

Re: Investigating how the New York Times A/B tests their headlines

#72

Post by tqi »

awhitby wrote: post #30

This is interesting but unless I missed it the author doesn't really explain why they believe they observe all A/B tests. They kind of assume that the randomization is over time (so every reader within a window sees the same headline, and then it changes) rather than within cohorts at a fixed time. But the quote included suggests the NYT does do the latter: "Half of readers will see one headline, and the other half w…

Agree, that is absolutely not how AB tests are run. No one runs tests sequentially on 100% of traffic - that would be like a McDonalds offering two versions of the Egg McMuffin, one from 7a-11a and the second from 11a-6p, and declaring the v1 a clear winner because it had more sales.

Assignments are also almost certainly sticky based on a browser cookie.

Re: Investigating how the New York Times A/B tests their headlines

#73

Post by baking »

awhitby wrote: post #30

This is interesting but unless I missed it the author doesn't really explain why they believe they observe all A/B tests. They kind of assume that the randomization is over time (so every reader within a window sees the same headline, and then it changes) rather than within cohorts at a fixed time. But the quote included suggests the NYT does do the latter: "Half of readers will see one headline, and the other half w…

Also, if the content of the story changes with the headline, then it is probably just reflecting a developing story. That's what the SpaceX and Cuomo stories seemed to be doing. Assuming that all headline changes are just because an algorithm thought they were more dramatic seems to be jumping the gun.

Re: Investigating how the New York Times A/B tests their headlines

#75

Post by rkho »

JasonFruit wrote: post #6

Should a news source be optimizing for engagement, or for accuracy in communicating facts? I understand it's a business, but it's analogous to a bakery that labels all its goods as its best-selling items, only to confuse and disappoint the buyer when they open the box and find something different inside. It may sell that product that time, but it goes against the purpose of the organization as a whole, and doesn't se…

It's the goal of Meredith Levien, their current CEO, for the NYT to be considered a "world class tech company"[1]. I believe doing A/B tests is a way to work towards that goal.

[1]: https://www.theinformation.com/articles/meredith-levien-want...

Re: Investigating how the New York Times A/B tests their headlines

#76

Post by bArray »

I would suggest that this can be more 'evil' that just trying to get more clicks.

A number of times I have seen on some place like Facebook where the initial article has some extreme headline, and then hours after when engagement is up, the headline is swapped to one that is less inflammatory. A few more extreme-perspective friends will send me an article saying "see?!?!" - and by the time I see it it's already been rewritten.

A few times I would click on an article going viral and find the headline in the article itself doesn't match the one cached by Facebook. Remember that most people aren't even reading the headlines and just assume that NYT are trustworthy. The first headline is the one they end up internalizing.

Bare in mind, there is zero consequence for doing this either. The newspaper and claim they were "correcting an editorial error", whilst openly spreading misinformation about some hot topic.

I think at the very least it should be mandatory to maintain a list of edits to an article once published - and to indicate to the reader clearly that the article has received a number of edits.

Re: Investigating how the New York Times A/B tests their headlines

#77

Post by [deleted] »

hkai wrote: post #71
CarVac wrote: post #69

Earlier quoted context omitted.

Sounds like someone needs to make an aggregator that shows only brief auto-generated summaries, no headlines or images.

Even better, these autogenerated summaries should give most weight to paragraphs at the end of the article. I often find that New York Times and other media put important details that sometimes contradict the headline or challenge the main idea of the article somewhere close to the end of the article. This lets them still claim objectivity while preserving a highly polarizing message.

[deleted]

Re: Investigating how the New York Times A/B tests their headlines

#78

Post by [deleted] »

motohagiography wrote: post #4

Valuable anecdata. News is defined by conflict, (literally, no conflict = not news) and this shows how headlines that express the most conflict get the most clicks. The integrity issues come up when even though the facts reported are real, the conflict that frames them is manufactured - and this is why people reject news. They don't reject it because of fake facts, they reject it because of fake conflict. When I want…

[deleted]

Re: Investigating how the New York Times A/B tests their headlines

#79

Post by systemvoltage »

bArray wrote: post #76

I would suggest that this can be more 'evil' that just trying to get more clicks. A number of times I have seen on some place like Facebook where the initial article has some extreme headline, and then hours after when engagement is up, the headline is swapped to one that is less inflammatory. A few more extreme-perspective friends will send me an article saying "see?!?!" - and by the time I see it it's already been…

I’d agree with you but it seems like these alterations are pretty benign. They’re still professional and not at all fake/misinformation.

Re: Investigating how the New York Times A/B tests their headlines

#80

Post by xwolfi »

bArray wrote: post #76

I would suggest that this can be more 'evil' that just trying to get more clicks. A number of times I have seen on some place like Facebook where the initial article has some extreme headline, and then hours after when engagement is up, the headline is swapped to one that is less inflammatory. A few more extreme-perspective friends will send me an article saying "see?!?!" - and by the time I see it it's already been…

Having worked in A/B testing in a loan marketing company, where we were pushing 300% payday loans in Mexico (you read that right), I can talk of evil.

I think, when you're in the day to day of a web platform, you really just simplified everything from your hiring to your board meeting to one metric: click.

Everything is number of click, it becomes no more evil than a Lion in the Savanna, you just must increase your number of clicks somehow and you forgot why.

It can lead to interesting "let's remove the legal text away and see what gives", "let's color ever buttons blue vs pink and see who wins", let's reorder the search result independently of what a normal human would expect (as in... cheapest first?)".

I see what you mean, but it's not evil anymore. It's now part of nature, they must hunt your click. They can't stop and they can't explain why. They could put one headline "Myanmar military coup" then replace it with "Myanmar cabinet reshuffle" to see which one you click on with little input from any writer and no philosophical debate whatsoever.

Post reply on HN