Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Investigating how the New York Times A/B tests their headlines

blog.tjcx.me

Post reply on HN
Print view View original

101–110 of 131 posts

Re: Investigating how the New York Times A/B tests their headlines

#101

Post by meowface »

fwip wrote: post #96
xwolfi wrote: post #80

Earlier quoted context omitted.

Having worked in A/B testing in a loan marketing company, where we were pushing 300% payday loans in Mexico (you read that right), I can talk of evil. I think, when you're in the day to day of a web platform, you really just simplified everything from your hiring to your board meeting to one metric: click. Everything is number of click, it becomes no more evil than a Lion in the Savanna, you just must increase your n…

Evil done by beauracracy, process, or algorithm, is still evil.

I know you weren't trying to suggest this, but A/B testing doesn't necessarily imply evilness, of course.

It's definitely evil in the parent commenter's and the article's cases, but if it's testing color schemes for your blog or a signup flow for your productivity app, there's a pretty good chance it's fine (depending on the underlying ethics of the blog/app).

But I overwhelmingly agree that ubiquitous or bureaucratized doesn't mean "not/no longer evil". If anything, there's possibly a light positive correlation. That comment's espousing one of the worst and most cynical philosophies I've ever encountered.

There might even be some natural/game theoretic phenomena at play here, too. Information wants to be free, and evil wants to be normalized.

Re: Investigating how the New York Times A/B tests their headlines

#102

Post by manicdee »

bArray wrote: post #76

I would suggest that this can be more 'evil' that just trying to get more clicks. A number of times I have seen on some place like Facebook where the initial article has some extreme headline, and then hours after when engagement is up, the headline is swapped to one that is less inflammatory. A few more extreme-perspective friends will send me an article saying "see?!?!" - and by the time I see it it's already been…

The history of edits is of no use to the people whose minds have been poisoned with the original propaganda.

Re: Investigating how the New York Times A/B tests their headlines

#103

Post by dang »

awhitby wrote: post #30

This is interesting but unless I missed it the author doesn't really explain why they believe they observe all A/B tests. They kind of assume that the randomization is over time (so every reader within a window sees the same headline, and then it changes) rather than within cohorts at a fixed time. But the quote included suggests the NYT does do the latter: "Half of readers will see one headline, and the other half w…

Ok, I've consed 'investigating' onto the title above, which hopefully adds an appropriate degree of uncertainty. Thanks!

Re: Investigating how the New York Times A/B tests their headlines

#104

Post by ofrzeta »

throw_m239339 wrote: post #26
JasonFruit wrote: post #6

Should a news source be optimizing for engagement, or for accuracy in communicating facts? I understand it's a business, but it's analogous to a bakery that labels all its goods as its best-selling items, only to confuse and disappoint the buyer when they open the box and find something different inside. It may sell that product that time, but it goes against the purpose of the organization as a whole, and doesn't se…

> Should a news source be optimizing for engagement, or for accuracy in communicating facts? A journal is a business, with something to sell, the news, and the attention of their readers to advertisers as well. > The New York Times is a big deal. As they tell their advertisers, the NYT is the #1 news source for young, rich thought leaders: Obviously ultimately it reflects badly on the profession, since all these news…

> the same marketing circles/education.

Also "young, rich thought leaders", right?

Re: Investigating how the New York Times A/B tests their headlines

#105

Post by penetraitor69 »

solarkraft wrote: post #44
site-packages1 wrote: post #13

Earlier quoted context omitted.

But multiple headlines can both be accurate with one being more engaging. These aren't mutually exclusive. Article about a bank robbery: "First National Bank Robbed" "Gunmen Rob First National Bank in Daring Robbery"

"Engaging" while being highly irritating. The first one is concise.

True, but which is more likely to get your click: a headline you're indifferent to, or a headline that fills you with righteous indignation? I think that if there's one thing that the development of social media has taught us about human nature, it's that any emotion is better than boredom, from an engagement standpoint.

Re: Investigating how the New York Times A/B tests their headlines

#106

Post by 02020202 »

news today is merely a psychology-based domestic terrorism on massive scale without any oversight. hence why people today are constantly stressed out. news became "tabloid" because in the end, it is not about OBJECTIVE reporting of FACTS but rather SUBJECTIVE manipulation and selective truths that will yield the most clicks for the ad-based revenue. it is sickening and one of the most horrid things we have in the 21st century.

Re: Investigating how the New York Times A/B tests their headlines

#107

Post by ls-lah_33 »

refurb wrote: post #88
monch1962 wrote: post #50

Earlier quoted context omitted.

It _usually_ makes sense to optimise for engagement, particularly in the US where there's no state-funded media outlet (cue discussion about socialism that will be ignored). The UK has the BBC, Australia has the ABC, and both are state funded media outlets that aren't (at least overtly) driven by views. I'm sure their funding largely depends on how many people are consuming their product, but it's not as though some…

Based on the amount of click-bait from the BBC that gets posted to HN, I’m not sure your state funded option is much better. I know the CBC in Canada is absolute crap - 80% of the reporting is sensationalistic and for the past 4 years has been focused on the nuances of Trump - not exactly relevant to Canada. At least not deserving of more coverage than domestic issues.

The BBC gets roughly a quarter of its income from commercial sources

https://www.icaew.com/insights/viewpoints-on-the-news/2021/j...

Re: Investigating how the New York Times A/B tests their headlines

#108

Post by megablast »

tqi wrote: post #72
awhitby wrote: post #30

This is interesting but unless I missed it the author doesn't really explain why they believe they observe all A/B tests. They kind of assume that the randomization is over time (so every reader within a window sees the same headline, and then it changes) rather than within cohorts at a fixed time. But the quote included suggests the NYT does do the latter: "Half of readers will see one headline, and the other half w…

Agree, that is absolutely not how AB tests are run. No one runs tests sequentially on 100% of traffic - that would be like a McDonalds offering two versions of the Egg McMuffin, one from 7a-11a and the second from 11a-6p, and declaring the v1 a clear winner because it had more sales. Assignments are also almost certainly sticky based on a browser cookie.

No. It’s like if McDonald’s offered a in the first half hour, and b in the second half hour.

I’m sure you would agree that would be a good representation.

Re: Investigating how the New York Times A/B tests their headlines

#110

Post by lolc »

awhitby wrote: post #30

This is interesting but unless I missed it the author doesn't really explain why they believe they observe all A/B tests. They kind of assume that the randomization is over time (so every reader within a window sees the same headline, and then it changes) rather than within cohorts at a fixed time. But the quote included suggests the NYT does do the latter: "Half of readers will see one headline, and the other half w…

It took me a while to understand this, but I think the long consistent blocks are a sorting artifact. As the observed headlines are bucketed by the hour. You don't see the title flip within the hour, you just see the percentage.

It would have been more interesting to see the title flipping back and forth, because that would reveal how long those tests last. Less neat though.

Post reply on HN