Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Investigating how the New York Times A/B tests their headlines

blog.tjcx.me

Post reply on HN
Print view View original

121–130 of 131 posts

Re: Investigating how the New York Times A/B tests their headlines

#121

Post by edavison1 »

hkai wrote: post #71
CarVac wrote: post #69

Earlier quoted context omitted.

Sounds like someone needs to make an aggregator that shows only brief auto-generated summaries, no headlines or images.

Even better, these autogenerated summaries should give most weight to paragraphs at the end of the article. I often find that New York Times and other media put important details that sometimes contradict the headline or challenge the main idea of the article somewhere close to the end of the article. This lets them still claim objectivity while preserving a highly polarizing message.

Journalist here. While I love that there's a discussion about burying the lede on HN, the practice you reference is known as a kicker. It's a last thought that is meant to twist, undercut, or add complexity to the narrative thrust of the story. The best ones add a ton of value to stories and reward people who read to the end.

Re: Investigating how the New York Times A/B tests their headlines

#122

Post by [deleted] »

jhugo wrote: post #116
yesenadam wrote: post #113

Earlier quoted context omitted.

Yes and no. The GP probably knows the story that "lede" is a more recent spelling than you might think. e.g. This[0] article from Merriam Webster says "Although evidence dates the spelling to the 1970s, we didn't enter lede in our dictionaries until 2008." It says the different spelling was introduced to differentiate it from lead (the metal) which at that time was talked about often at newspapers re linotype printin…

Sure, lede is very recent, and largely without a solid historical basis, but it's the far more common spelling these days.

[deleted]

Re: Investigating how the New York Times A/B tests their headlines

#123

Post by tqi »

megablast wrote: post #108
tqi wrote: post #72

Earlier quoted context omitted.

Agree, that is absolutely not how AB tests are run. No one runs tests sequentially on 100% of traffic - that would be like a McDonalds offering two versions of the Egg McMuffin, one from 7a-11a and the second from 11a-6p, and declaring the v1 a clear winner because it had more sales. Assignments are also almost certainly sticky based on a browser cookie.

No. It’s like if McDonald’s offered a in the first half hour, and b in the second half hour. I’m sure you would agree that would be a good representation.

Sequential testing will never be truly random, unless you can guarantee that your traffic is uniform across time.

Imagine you're running an AB test on a website, where the number of customers you get increases linearly over time. If you test two features sequentially, there is no duration you could pick that is small enough to where you would not expect the second option to win.

Re: Investigating how the New York Times A/B tests their headlines

#124

Post by [deleted] »

tqi wrote: post #123
megablast wrote: post #108

Earlier quoted context omitted.

No. It’s like if McDonald’s offered a in the first half hour, and b in the second half hour. I’m sure you would agree that would be a good representation.

Sequential testing will never be truly random, unless you can guarantee that your traffic is uniform across time. Imagine you're running an AB test on a website, where the number of customers you get increases linearly over time. If you test two features sequentially, there is no duration you could pick that is small enough to where you would not expect the second option to win.

[deleted]

Re: Investigating how the New York Times A/B tests their headlines

#125

Post by reilly3000 »

novemberyankee wrote: post #42
reilly3000 wrote: post #35

Good piece, but damn I was hoping for some details on their implementation. Are the variations coming from the server, CDN/edge, or client? How does this impact SEO? Do they change the article slug if the title is a winner? How much is automated and how much is human curated? When beset with new assignments who has the time to babysit these things?

cdn; seo impact is also tested for; canonical URL doesn't change; titles are written by humans, testing is entirely automated; small parts of larger UX team just do testing so babysitting these is their job

How do you measure SEO impact in a title split test? Scraping SERPs or monitoring query volume real-time?

Re: Investigating how the New York Times A/B tests their headlines

#126

Post by fwip »

meowface wrote: post #101
fwip wrote: post #96

Earlier quoted context omitted.

Evil done by beauracracy, process, or algorithm, is still evil.

I know you weren't trying to suggest this, but A/B testing doesn't necessarily imply evilness, of course. It's definitely evil in the parent commenter's and the article's cases, but if it's testing color schemes for your blog or a signup flow for your productivity app, there's a pretty good chance it's fine (depending on the underlying ethics of the blog/app). But I overwhelmingly agree that ubiquitous or bureaucrati…

Agreed.

Re: Investigating how the New York Times A/B tests their headlines

#127

Post by bArray »

manicdee wrote: post #102
bArray wrote: post #76

I would suggest that this can be more 'evil' that just trying to get more clicks. A number of times I have seen on some place like Facebook where the initial article has some extreme headline, and then hours after when engagement is up, the headline is swapped to one that is less inflammatory. A few more extreme-perspective friends will send me an article saying "see?!?!" - and by the time I see it it's already been…

The history of edits is of no use to the people whose minds have been poisoned with the original propaganda.

Yeah I somewhat agree, but transparency is surely a good start?

Re: Investigating how the New York Times A/B tests their headlines

#128

Post by bArray »

infogulch wrote: post #99
bArray wrote: post #76

I would suggest that this can be more 'evil' that just trying to get more clicks. A number of times I have seen on some place like Facebook where the initial article has some extreme headline, and then hours after when engagement is up, the headline is swapped to one that is less inflammatory. A few more extreme-perspective friends will send me an article saying "see?!?!" - and by the time I see it it's already been…

Some comments on newly-released YouTube videos will restate the title of the video at the time the comment was made simply to annotate that fact for future viewers, and such comments are often highly upvoted. An interesting reactionary adaptation to this "temporary sensational title" exploit.

Sounds like a plaster on a gaping wound. The comments engagement is quite low, even on very popular videos.

I think more needs to be done for publishing transparency.

Re: Investigating how the New York Times A/B tests their headlines

#129

Post by tomjcleveland »

awhitby wrote: post #30

This is interesting but unless I missed it the author doesn't really explain why they believe they observe all A/B tests. They kind of assume that the randomization is over time (so every reader within a window sees the same headline, and then it changes) rather than within cohorts at a fixed time. But the quote included suggests the NYT does do the latter: "Half of readers will see one headline, and the other half w…

OP here. I don't know if anyone reads old HN threads but thought I'd clarify some things:

1. My scraper runs every 5 minutes, with a randomly-generated user-agent and never sends cookie headers

2. The charts are bucketed by half-hour periods, so even if the headline flips back and forth many times in half an hour, the colors are grouped together

3. Agreed that in the SpaceX situation (and maybe the Cuomo situation) the headlines change because the stories change. But, e.g., in the Meghan Markle situation, the first headline appears _after_ the interview is over. But that's something to watch out for!

And charts like this one[0] look (to me) like a clear example of A/B testing. But would be interested to hear other explanations!

[0] https://nyt.tjcx.me/articles/1163e0c4-e609-5cfa-aff1-b0945f1...

Re: Investigating how the New York Times A/B tests their headlines

#130

Post by rattray »

tomjcleveland wrote: post #129
awhitby wrote: post #30

This is interesting but unless I missed it the author doesn't really explain why they believe they observe all A/B tests. They kind of assume that the randomization is over time (so every reader within a window sees the same headline, and then it changes) rather than within cohorts at a fixed time. But the quote included suggests the NYT does do the latter: "Half of readers will see one headline, and the other half w…

OP here. I don't know if anyone reads old HN threads but thought I'd clarify some things: 1. My scraper runs every 5 minutes, with a randomly-generated user-agent and never sends cookie headers 2. The charts are bucketed by half-hour periods, so even if the headline flips back and forth many times in half an hour, the colors are grouped together 3. Agreed that in the SpaceX situation (and maybe the Cuomo situation) t…

Did you send from the same IP? (Worth including in the post either way).

They may also conduct certain A/B tests only on cookied users, in which case your cookieless bot would not have seen everything.

Post reply on HN