This is interesting but unless I missed it the author doesn't really explain why they believe they observe all A/B tests. They kind of assume that the randomization is over time (so every reader within a window sees the same headline, and then it changes) rather than within cohorts at a fixed time. But the quote included suggests the NYT does do the latter: "Half of readers will see one headline, and the other half w…
OP here. I don't know if anyone reads old HN threads but thought I'd clarify some things: 1. My scraper runs every 5 minutes, with a randomly-generated user-agent and never sends cookie headers 2. The charts are bucketed by half-hour periods, so even if the headline flips back and forth many times in half an hour, the colors are grouped together 3. Agreed that in the SpaceX situation (and maybe the Cuomo situation) t…
Re: Investigating how the New York Times A/B tests their headlines
#131People read week-old posts on Fridays, when the hacker newsletter comes out. Thanks for the clarification!