The math’s not pretty on digital advertising’s future revenues? (2017)

Here is another paper worth reading on it [0] (which cites Lewis & Rao), but provides fairly clear evidence of a distribution of outcomes of digital ad campaigns with a lot of probability mass on the side of positive lift.

The place where issues come in via Lewis & Rao is on the side of poor statistical methodology, particularly in the calculation of necessary statistical power.

But those discussions are all stuck in bad frequentist estimation techniques, and in industry there are practitioners using Bayesian methods, for example such as Gelman’s proposed “type sign” and “type magnitude” errors, with well-calibrated priors that treat the possibility of negative lift seriously.

I strongly disagree with the idea that “ads don’t work” is an “open secret” like you say. Rather, “measuring ads is extremely complicated” is the known conclusion, and marketers are just coming around to the fact that you have to invest in extremely advanced statistical methods to even know if ads are working, let alone to optimize them for an audience, especially inclusive of things like audience privacy, opt-out policies, paychological well being and so on.

[0]: https://marketing.wharton.upenn.edu/wp-content/uploads/2017/...

Maybe part of the reason you're struggling with this concept is because your focus is too narrow. Look beyond political/legal commentary, where small creators are competing directly against international media corporations that provide content in the same category.

Look beyond that, because most of what's out there doesn't fall into that category. There is a LONG tail of independent content being produced that has no competition at all from corporate media. Name any narrow interest or hobby and you can almost certainly find independent creators online that are collectively producing far more content in that topic every week than the corporate media has produced in the past century. Take for instance woodworking. To be sure, there have been TV shows about woodworking, The New Yankee Workshop (Norm Abram) and The Woodwright's Shop (Roy Underhill), both on PBS (which is probably notable, can we expect Disney to produce and commercialize a woodworking TV show?), are very notable excellent examples. But there is so much more content on youtube, the woodworking content on youtube is more diverse, and those independent online creators typically have a closer relationship with their viewers.

Woodworking is a generous example because it's actually a case where traditional media can marginally compete. But change the topic slightly and traditional media leaves you completely out in the cold. What if instead of cutting and joining wood, you're interested in metal? Where is the PBS show for hobby metal working? Would Disney ever give Adam Booth his own machining show? I don't think so.

This is a small list: https://en.wikipedia.org/wiki/Category:Arts_and_crafts_telev...

Nothing per-se, but if you talk to content creators they'll tell you that they prefer Patreon or youtube monetization or similar systems for at least one simple reason: their per-month income is more predictable, which provides them with a sense of financial stability. For many creators this isn't a full-time job, but rather a time intensive hobby that brings in enough money to pay for itself with maybe a little extra. But feeling confident that their expenses will be covered next month provides them with a necessary sense of security to keep doing what they do. If they were to rely on people mailing them cheques every month, many people would simply forget and creators could never be confident of approximately how much money they'll have next month. Patreon and youtube monetization don't provide any guarantees about future income, but in practice creators find that it typically doesn't change that much month from month.

This doesn't mean that youtube monetization or patreon have a permanent monopoly on the payment system for independent creators. Far from it actually, the fact that Patreon broke into the scene previously dominated by youtube monetization demonstrates that if your system provides relative stability to creators, they will be willing to adopt it.

We are still in the infancy of this social transformation, but already we see a clear picture starting to emerge of what creators and their viewers require to make it work.

I’m purposefully discounting TiVo from the future list, as I have not seen them to be a strong OTT platform player in the last few years.

Here is where I think we disagree: I don’t believe that Netflix’s ability (or Google’s, for that matter) to predict new markets is substantively improved by their data set. I understand that that might be a radical perspective among data-driven decision-makers, but I stand by it. Yes, the data set is extremely useful for scaling/marketing an asset (as we have both highlighted), but frankly datasets don’t yet know what the next hot thing will be, and there is a lack of statistical evidence to suggest otherwise (ie Netflix has produced, what, 1-10k shows? That doesn’t necessarily provide much statistical reliability, even if they have a high rate of success, which they don’t).

Furthermore, Netflix picking themes, topics, etc as you suggest to drive successful production runs contrary to their most successful strategy so far, which is to cast a wide net by producing highly diverse content at low costs then scaling what works. While they use data to deduce which shows are successful, the production of those shows occurred with little to no data-driven direction beyond the obvious understanding that a wide net catches more fish.

When a company has a combination of access to scale and quality data, they quickly outpace and overwhelm their competitors. Netflix hasn’t quite accomplished this, suggesting that their dataset isn’t as magical as you imply.

Lastly, their dataset isn’t as private as you might believe, either. To view it as a golden egg that will exclusively produce quality content just for them is naive.