Netflix’s Secret Special Algorithm Is a Human

This is another facet of the "amplified-teams" trend that's been happening over the past few years. This book review of 'Average is over' has some good information about it:

http://ieet.org/index.php/IEET/more/searle20150109

"In his vision intelligent machines will revolutionize everything from medicine to education to business management and negotiation to love. The human beings who will best thrive in this new environment will be those whose work best complements that of intelligent machines, and this will be the case all the way from the factory floor to the classroom."

Very interesting times ahead.

In Health Informatics at least this is taken even further

    Data
      X of Y people have some disease
    Information
      Based on data, I can predict disease likelihood
      given some environmental and personal factors of
      a patient
    Knowledge
      Using informed predictions, I make good inferences
      about how to proceed with diagnosis
    Wisdom
      Using knowledge and experience I choose the right
      approach for treating and diagnosing a patient
      which is efficacious, healthy, and works with the
      patient's actual needs

It's easy to draw these lines in other places or to call the tower a lot of woo woo able to be reduced into inferences atop raw data all combined correctly... but it serves to remind just how difficult it is to combine the right data in the right way to make the right decisions at the right times.

It also serves as a sharp counterpoint to the idea of, say, machine learning patient diagnoses. It turns out that diagnostic accuracy is terrible, but not because people are directly bad at it (even if they are) but instead because knowledge/wisdom dictates that perfect accuracy isn't that valuable---perfect care is and that can involve chasing down treatment and care avenues that would never be predicted or acting on information that is not currently in your model.

What about aggregating movie reviews and classifying movies based on their reviews. Would be an interesting problem regardless of accuracy.

The lesson here isn't "Human Judgement > Algorithmic Judgement". It also isn't "Humans are good at some things and computers are good at others". It's that there isn't a good reason to make the investment in solving this particular problem algorithmically (yet).

Algorithms are great when you need scale, especially in situations where a 10% improvement in prediction accuracy can make a big improvement in the bottom line. Netflix and other studios might greenlight several new shows in a year, out of dozens that receive consideration. And the Pareto Distribution is in full effect. Most of the profits and awards come from one or two big hits. Algorithmic decision making just doesn't make a lot of sense in situations with a small sample size and uneven reward structure.

It doesn't mean that it isn't possible, though. If someone were to make the massive investment necessary to do a more thorough analysis of the content creators, the actors, the scripts and potential audiences and all of the other possible inputs then algorithms could probably do as good a job as humans, if not better. Netflix and others have only taken baby steps in this direction, working with data that is readily available and using predictive techniques that are well tested and understood. Given the nature of the problem, it doesn't make sense for them to approach it any other way at this time. But when it comes to making billions of recommendations to millions of people per day, they still rely heavily on data and algorithmic prediction. There's a time and place for everything. The time and place for algorithms in our daily lives is changing and expanding, but very slowly.