Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Investigating how the New York Times A/B tests their headlines

blog.tjcx.me

Post reply on HN
Print view View original

111–120 of 131 posts

Re: Investigating how the New York Times A/B tests their headlines

#111

Post by jhugo »

godmode2019 wrote: post #94
hkai wrote: post #71

Earlier quoted context omitted.

Even better, these autogenerated summaries should give most weight to paragraphs at the end of the article. I often find that New York Times and other media put important details that sometimes contradict the headline or challenge the main idea of the article somewhere close to the end of the article. This lets them still claim objectivity while preserving a highly polarizing message.

This has a name its called burying the lead, its used because news papers are structured in a way where they know people have limited attention so usually they don't read to the end. All the important bits are typically laied out linearly, often trailing off at the end. Burying the lead takes advance if this and puts the important but inconvenient information at the bottom.

FYI, the name of the introductory section of a journalistic article is "lede", not "lead", and the practice is called "burying the lede".

Re: Investigating how the New York Times A/B tests their headlines

#112

Post by [deleted] »

godmode2019 wrote: post #94
hkai wrote: post #71

Earlier quoted context omitted.

Even better, these autogenerated summaries should give most weight to paragraphs at the end of the article. I often find that New York Times and other media put important details that sometimes contradict the headline or challenge the main idea of the article somewhere close to the end of the article. This lets them still claim objectivity while preserving a highly polarizing message.

This has a name its called burying the lead, its used because news papers are structured in a way where they know people have limited attention so usually they don't read to the end. All the important bits are typically laied out linearly, often trailing off at the end. Burying the lead takes advance if this and puts the important but inconvenient information at the bottom.

[deleted]

Re: Investigating how the New York Times A/B tests their headlines

#113

Post by yesenadam »

jhugo wrote: post #111
godmode2019 wrote: post #94

Earlier quoted context omitted.

This has a name its called burying the lead, its used because news papers are structured in a way where they know people have limited attention so usually they don't read to the end. All the important bits are typically laied out linearly, often trailing off at the end. Burying the lead takes advance if this and puts the important but inconvenient information at the bottom.

FYI, the name of the introductory section of a journalistic article is "lede", not "lead", and the practice is called "burying the lede".

Yes and no. The GP probably knows the story that "lede" is a more recent spelling than you might think. e.g. This[0] article from Merriam Webster says "Although evidence dates the spelling to the 1970s, we didn't enter lede in our dictionaries until 2008." It says the different spelling was introduced to differentiate it from lead (the metal) which at that time was talked about often at newspapers re linotype printing.

That article links to this[1], by Howard Owens:

"It was then I realized, there is no historic basis for the spelling of a lead as “lede.” “Lede” is an invention of linotype romanticists, not something used in newsrooms of the linotype era."

He says he looked at books from the 1940s - 1980s, "The fact is, in none of the dozens of old journalism books that I have examined — none of them — spell it “lede.” I can’t find the definitive first reference to “lede” but it doesn’t start appearing in journalism books until the 1980s. .. It wasn’t until linotype was disappearing from newsrooms across the nation (late 1970s and into the 1980s), that we start seeing the spelling “lede.”

The safest conclusion, then, is that “lede” is a romantic fiction invented by those who were nostalgic for the passing of the linotype era."

[0] https://www.merriam-webster.com/words-at-play/bury-the-lede-...

[1] https://howardowens.com/lede-vs-lead/

Re: Investigating how the New York Times A/B tests their headlines

#114

Post by hntrader »

megablast wrote: post #108
tqi wrote: post #72

Earlier quoted context omitted.

Agree, that is absolutely not how AB tests are run. No one runs tests sequentially on 100% of traffic - that would be like a McDonalds offering two versions of the Egg McMuffin, one from 7a-11a and the second from 11a-6p, and declaring the v1 a clear winner because it had more sales. Assignments are also almost certainly sticky based on a browser cookie.

No. It’s like if McDonald’s offered a in the first half hour, and b in the second half hour. I’m sure you would agree that would be a good representation.

The traffic to a restaurant isn't uniformly distributed over an hour window.

There's a peak 30-min in the morning just before work starts for most people, for example.

Re: Investigating how the New York Times A/B tests their headlines

#115

Post by patleeman »

novemberyankee wrote: post #42
reilly3000 wrote: post #35

Good piece, but damn I was hoping for some details on their implementation. Are the variations coming from the server, CDN/edge, or client? How does this impact SEO? Do they change the article slug if the title is a winner? How much is automated and how much is human curated? When beset with new assignments who has the time to babysit these things?

cdn; seo impact is also tested for; canonical URL doesn't change; titles are written by humans, testing is entirely automated; small parts of larger UX team just do testing so babysitting these is their job

Is the headline testing some sort of multi-arm bandit system or a/b tests that are updated manually?

It seems like MAB would be the ideal approach for something like headlines.

Re: Investigating how the New York Times A/B tests their headlines

#116

Post by jhugo »

yesenadam wrote: post #113
jhugo wrote: post #111

Earlier quoted context omitted.

FYI, the name of the introductory section of a journalistic article is "lede", not "lead", and the practice is called "burying the lede".

Yes and no. The GP probably knows the story that "lede" is a more recent spelling than you might think. e.g. This[0] article from Merriam Webster says "Although evidence dates the spelling to the 1970s, we didn't enter lede in our dictionaries until 2008." It says the different spelling was introduced to differentiate it from lead (the metal) which at that time was talked about often at newspapers re linotype printin…

Sure, lede is very recent, and largely without a solid historical basis, but it's the far more common spelling these days.

Re: Investigating how the New York Times A/B tests their headlines

#117

Post by [deleted] »

jhugo wrote: post #116
yesenadam wrote: post #113

Earlier quoted context omitted.

Yes and no. The GP probably knows the story that "lede" is a more recent spelling than you might think. e.g. This[0] article from Merriam Webster says "Although evidence dates the spelling to the 1970s, we didn't enter lede in our dictionaries until 2008." It says the different spelling was introduced to differentiate it from lead (the metal) which at that time was talked about often at newspapers re linotype printin…

Sure, lede is very recent, and largely without a solid historical basis, but it's the far more common spelling these days.

[deleted]

Re: Investigating how the New York Times A/B tests their headlines

#119

Post by samdfonseca »

JasonFruit wrote: post #6

Should a news source be optimizing for engagement, or for accuracy in communicating facts? I understand it's a business, but it's analogous to a bakery that labels all its goods as its best-selling items, only to confuse and disappoint the buyer when they open the box and find something different inside. It may sell that product that time, but it goes against the purpose of the organization as a whole, and doesn't se…

The article is what communicates facts. The headline is just an advertisement to draw the user in to read it. Optimizing the headline means the article is able to inform more people; in line with the mission of the organization. NYT wants to increase subscribers, not ad revenue, so it's incentivized to not confuse and disappoint readers with clickbait headlines.

Re: Investigating how the New York Times A/B tests their headlines

#120

Post by [deleted] »

samdfonseca wrote: post #119
JasonFruit wrote: post #6

Should a news source be optimizing for engagement, or for accuracy in communicating facts? I understand it's a business, but it's analogous to a bakery that labels all its goods as its best-selling items, only to confuse and disappoint the buyer when they open the box and find something different inside. It may sell that product that time, but it goes against the purpose of the organization as a whole, and doesn't se…

The article is what communicates facts. The headline is just an advertisement to draw the user in to read it. Optimizing the headline means the article is able to inform more people; in line with the mission of the organization. NYT wants to increase subscribers, not ad revenue, so it's incentivized to not confuse and disappoint readers with clickbait headlines.

[deleted]
Post reply on HN