Live data from Hacker News

Investigating how the New York Times A/B tests their headlines

blog.tjcx.me

21–30 of 131 posts

Re: Investigating how the New York Times A/B tests their headlines

#21

"And yet it rarely attracts the kind of close scrutiny of, say, a Fox News. And that’s totally reasonable! Fox News is an absurd clown show and deserves every criticism it gets" I find it comical that any write-up critical of a left wing organization needs to include something like this to avoid being flagged, down-voted, whatever, on the platform its being posted on.

If excluded, I doubt anyone would have cared or complained.

Re: Investigating how the New York Times A/B tests their headlines

#22
post #3

If you're interested in observing this in realtime, there's a great Twitter account called @nyt_diff[0] that automatically posts when headlines on the front page change. Really cool to watch articles get updated as stories evolve. It's particularly interesting to witness headline writers grapple with how and how much to editorialize. [0] https://twitter.com/nyt_diff

Cool bot. I'm a keyboard warrior and I've been burned by the NYT and other "mainstream" news site stealth changes multiple times. Now when a big story comes out I save the page using SingleFile to Dropbox and when the edited part comes up in the conversation I compare both versions copied paste with Copy PlainText and Meld which allows me to post nice screenshots of the diff.

Re: Investigating how the New York Times A/B tests their headlines

#23

Earlier quoted context omitted.

Yes, but not at the cost of accuracy.

But multiple headlines can both be accurate with one being more engaging. These aren't mutually exclusive. Article about a bank robbery: "First National Bank Robbed" "Gunmen Rob First National Bank in Daring Robbery"

My favorite this year was from the Times of India on the Jan 6th US capital insurrection.

"Coup Klux Klan: Don triggers mob & rob bid"

Re: Investigating how the New York Times A/B tests their headlines

#24
What we measure, determines what gets maximized (or minimized, if it's something we don't like). The factfulness of an article, or how well informed the reader is after they read it, are not easily measured. The clicks, are now easy to measure. What gets maximized, is the clicks.

It would be nice if we had a news source with a feature (opt in) where the reader got an email quiz the next day, asking just a few questions about the facts in the article. The feedback loop would tell us which articles actually make the reader better informed.

Re: Investigating how the New York Times A/B tests their headlines

#25

Valuable anecdata. News is defined by conflict, (literally, no conflict = not news) and this shows how headlines that express the most conflict get the most clicks. The integrity issues come up when even though the facts reported are real, the conflict that frames them is manufactured - and this is why people reject news. They don't reject it because of fake facts, they reject it because of fake conflict. When I want…

> literally, no conflict = not news

Read local news.

In fact that gives me an idea, a local news aggregator which ignores redundant news and can find the most impactful ones.

Re: Investigating how the New York Times A/B tests their headlines

#26

Should a news source be optimizing for engagement, or for accuracy in communicating facts? I understand it's a business, but it's analogous to a bakery that labels all its goods as its best-selling items, only to confuse and disappoint the buyer when they open the box and find something different inside. It may sell that product that time, but it goes against the purpose of the organization as a whole, and doesn't se…

> Should a news source be optimizing for engagement, or for accuracy in communicating facts?

A journal is a business, with something to sell, the news, and the attention of their readers to advertisers as well.

> The New York Times is a big deal. As they tell their advertisers, the NYT is the #1 news source for young, rich thought leaders:

Obviously ultimately it reflects badly on the profession, since all these news sites are using the same clickbait techniques, from Breibart to the Dailymail to NYT, since they probably hire the same consultancies when it comes to clickbaiting design, or at the very least, people who come from the same marketing circles/education.

Re: Investigating how the New York Times A/B tests their headlines

#27
post #14

Earlier quoted context omitted.

Is there actually an either/or here? Surely it’s possible to write two (or more) headlines for a story that are all equally accurate?

Yes, but did you read the examples in the article? Many of the high-engagement headlines ignored most of the content in favor of a tangential but titillating detail. That's National Enquirer stuff, not what the New York Times has prided itself on being.

I did, but honestly most of the examples confused me. Both the Tesla rocket example and Cuomo story are surely headline changes relating to the story changing? They didn’t add “Cuomo apologizes” to titillate, it’s because… he apologized, and he hadn’t when the story was first published. Which isn’t even A/B testing! The article also more or less states this, hence my confusion.

Re: Investigating how the New York Times A/B tests their headlines

#28
Emotionally charged, slanted or dramatic headlines is one of the reasons I cancelled my NYT subscription a couple of years ago. I'd picked up a student NYT subscription back then and was had FT subscription from school, reading them back to back makes their slant and editorialization pretty obvious even if it isn't the most in your face.

Re: Investigating how the New York Times A/B tests their headlines

#29
The article is trying to paint NYT's A/B testing as nefarious, but unfortunately in the current social-media-oriented landscape, headlines are very, very important for clarity, and A/B testing headlines is something every media publisher with a sufficient tech infra does.

As frequent Hacker News submitters know, headlines alone can determine whether a post gets upvoted.

Re: Investigating how the New York Times A/B tests their headlines

#30
This is interesting but unless I missed it the author doesn't really explain why they believe they observe all A/B tests. They kind of assume that the randomization is over time (so every reader within a window sees the same headline, and then it changes) rather than within cohorts at a fixed time.

But the quote included suggests the NYT does do the latter: "Half of readers will see one headline, and the other half will see an alternative headline, for about half an hour."

So given that the author mostly observes long consistent blocks of time with the same headline, that suggests the NYT is allocating them to a subgroup in a persistent way (by IP or whatever). Then perhaps the cases where they didn't observe A/B testing were just cases where they were randomized into the optimal (hence final) headline subgroup by accident at the beginning, and never saw any different.

Post reply on HN