In several of these the shift is due to a headline written about an upcoming event being changed during and after the event, especially the Trump and Cuomo examples. Interesting tracking but not really A/B, which would be testing two different headlines roughly simultaneously.
Investigating how the New York Times A/B tests their headlines
31–40 of 131 posts
Re: Investigating how the New York Times A/B tests their headlines
#32"And yet it rarely attracts the kind of close scrutiny of, say, a Fox News. And that’s totally reasonable! Fox News is an absurd clown show and deserves every criticism it gets" I find it comical that any write-up critical of a left wing organization needs to include something like this to avoid being flagged, down-voted, whatever, on the platform its being posted on.
[deleted]
Re: Investigating how the New York Times A/B tests their headlines
#33I would say that learning how the NY Post writes their headlines would be a more fun story.
Re: Investigating how the New York Times A/B tests their headlines
#34Integrity and such aside, as someone who runs A/B tests I love seeing this data. Thanks for sharing it!
Re: Investigating how the New York Times A/B tests their headlines
#35Good piece, but damn I was hoping for some details on their implementation. Are the variations coming from the server, CDN/edge, or client? How does this impact SEO? Do they change the article slug if the title is a winner? How much is automated and how much is human curated? When beset with new assignments who has the time to babysit these things?
Re: Investigating how the New York Times A/B tests their headlines
#36The article is trying to paint NYT's A/B testing as nefarious, but unfortunately in the current social-media-oriented landscape, headlines are very, very important for clarity, and A/B testing headlines is something every media publisher with a sufficient tech infra does. As frequent Hacker News submitters know, headlines alone can determine whether a post gets upvoted.
I think the article attempts to demonstrate with its examples that some of the titles don't actually accurately represent what ought to be the core takeaways of a news story.
Selected excerpts:
"I actually watched this interview—all two hours of it—and I can tell you that the first two headlines are a much better summary of what went down. Yes, Meghan does reveal that she contemplated suicide, but it’s a five-minute interlude in an interview that has a lot more going on."
"Trump starts off addressing conservatives and claiming leadership of G.O.P. but in the final headline Trump has a hit list and is firing a warning shot. And sure enough, the bombastic rhetoric propels this article onto the “most viewed” list."
In other words, while you're correct that titles are important for clarity, the claim in the article is that "clarity" can be at odds with what gets clicks (or gets upvoted).
I'm able to viscerally internalize this because, as a tech worker, I've typically used A/B testing as a tool to maximize engagement, but I've never in my career used it as a tool to maximize correctness or "clarity".
Re: Investigating how the New York Times A/B tests their headlines
#37The article is trying to paint NYT's A/B testing as nefarious, but unfortunately in the current social-media-oriented landscape, headlines are very, very important for clarity, and A/B testing headlines is something every media publisher with a sufficient tech infra does. As frequent Hacker News submitters know, headlines alone can determine whether a post gets upvoted.
> headlines are very, very important for clarity
I'm pretty sure engagement is more important than clarity. I don't see how A/B testing helps you optimize for clarity. What are they measuring to determine a clarity score?
Sadly, most of the time engagement is highest when something is more outrage inducing.
Re: Investigating how the New York Times A/B tests their headlines
#38Earlier quoted context omitted.
This is my takeaway. There is a huge incentive for the media to make things more sensational or dramatic because it drives clicks and views. I might also argue that online news drives this more so than print - with print you tend to buy the paper based on the totality of their reporting and reputation versus a zero-cost click of a headline on your screen.
Would you be more trusting of NYT (or other news sites) if they pledged not to look at engagement stats?
It would be a positive signal but it wouldn't be the only signal. You can still do untrustworthy things without engagement stats.
Re: Investigating how the New York Times A/B tests their headlines
#39The value in A/B testing headlines is 30% boosting engagement directly and 70% creating a feedback loop where the newsroom can learn how to write a more engaging headline. Avoiding clickbait headlines requires some restraint and they are well aware of that slippery slope.
Re: Investigating how the New York Times A/B tests their headlines
#40Should a news source be optimizing for engagement, or for accuracy in communicating facts? I understand it's a business, but it's analogous to a bakery that labels all its goods as its best-selling items, only to confuse and disappoint the buyer when they open the box and find something different inside. It may sell that product that time, but it goes against the purpose of the organization as a whole, and doesn't se…
Headlines have become advertisements. They're almost never written by the journalists who wrote the story, they're written for space constraints that change on a whim ... They're very rarely "accurate" the way stories are intended to be for these reasons.
Have journalists ever controlled or written their own headlines? I don't think this is a change from any point in newspaper history.