Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Do data-driven companies win?

benn.substack.com

Post reply on HN
Print view View original

41–50 of 63 posts

Re: Do data-driven companies win?

#41

Post by marginalia_nu »

itsoktocry wrote: post #35
Ozzie_osman wrote: post #3

I worked at a self-described "data-driven" company, and the analogy senior leadership liked to make was that the company was like a machine learning algorithm, using data (particularly A/B tests) to do "gradient descent" of the product into its optimal form. My first take-away was that using data to make decisions is tremendously, tremendously powerful. A/B tests, in particular, can help determine causality and drive…

> Being purely data-driven without good intuition and long-term bets (that can't be "proven" with data), and the product loses its soul. This sounds like post-hoc, anti-intellectual rationalization. How do you place long-term bets without a model to measure expectations versus outcomes? How do you know what good intuition is? There is a ton of research on "gut" calls that demonstrates it's random. > But data is not a…

Waiting for data to somehow materialize to support a new action, without actually trying anything new seems like a recipe for just spinning your wheels doing the same things over and over and getting nowhere in a hurry.

Surely the data comes after the action, not prior to it?

Re: Do data-driven companies win?

#42

Post by InefficientRed »

itsoktocry wrote: post #35
Ozzie_osman wrote: post #3

I worked at a self-described "data-driven" company, and the analogy senior leadership liked to make was that the company was like a machine learning algorithm, using data (particularly A/B tests) to do "gradient descent" of the product into its optimal form. My first take-away was that using data to make decisions is tremendously, tremendously powerful. A/B tests, in particular, can help determine causality and drive…

> Being purely data-driven without good intuition and long-term bets (that can't be "proven" with data), and the product loses its soul. This sounds like post-hoc, anti-intellectual rationalization. How do you place long-term bets without a model to measure expectations versus outcomes? How do you know what good intuition is? There is a ton of research on "gut" calls that demonstrates it's random. > But data is not a…

> This sounds like post-hoc, anti-intellectual rationalization.

How is this anti-intellectual?

If you want, I can formalize as a game the problem of choosing business/product strategy in a competitive market with a continuous flow of imperfect information. I can then use ideas from controls to establish some upper bounds on what can be inferred from a continuous flow of information. I can then use that result to prove an impossibility result about the game. I can even tweak assumptions to get bounds on probability distributions which infer we'd be better off flipping a coin or whatever.

I'm not going to do the work, because intuition is almost always enough to identify these situations, but it's absolutely clear to me that results like this obviously exist and correspond to many real-world situations.

> Good judgement requires data.

It used to be that insisting on data-driven decision making was a hard pull. Now it's the opposite. Insisting on data where data cannot possibly provide enough signal to make a decision is the new form of anti-intellectualism. IMO.

Re: Do data-driven companies win?

#43

Post by dasil003 »

itsoktocry wrote: post #35
Ozzie_osman wrote: post #3

I worked at a self-described "data-driven" company, and the analogy senior leadership liked to make was that the company was like a machine learning algorithm, using data (particularly A/B tests) to do "gradient descent" of the product into its optimal form. My first take-away was that using data to make decisions is tremendously, tremendously powerful. A/B tests, in particular, can help determine causality and drive…

> Being purely data-driven without good intuition and long-term bets (that can't be "proven" with data), and the product loses its soul. This sounds like post-hoc, anti-intellectual rationalization. How do you place long-term bets without a model to measure expectations versus outcomes? How do you know what good intuition is? There is a ton of research on "gut" calls that demonstrates it's random. > But data is not a…

> Good judgement requires data.

I would say good judgement requires experience. Data may or may not be available and applicable, but its absence doesn't mean one can't exhibit good judgement.

Re: Do data-driven companies win?

#44

Post by dominotw »

itsoktocry wrote: post #35
Ozzie_osman wrote: post #3

I worked at a self-described "data-driven" company, and the analogy senior leadership liked to make was that the company was like a machine learning algorithm, using data (particularly A/B tests) to do "gradient descent" of the product into its optimal form. My first take-away was that using data to make decisions is tremendously, tremendously powerful. A/B tests, in particular, can help determine causality and drive…

> Being purely data-driven without good intuition and long-term bets (that can't be "proven" with data), and the product loses its soul. This sounds like post-hoc, anti-intellectual rationalization. How do you place long-term bets without a model to measure expectations versus outcomes? How do you know what good intuition is? There is a ton of research on "gut" calls that demonstrates it's random. > But data is not a…

Wouldn't this imply facebook would've built tiktok had they looked at the right data or built right data models.

Re: Do data-driven companies win?

#45

Post by dasil003 »

Like any good idea that sees wide adoption, being "data-driven" has jumped the shark to cargo-cult methodology in most places.

The truth is that data is only powerful if applied judiciously with solid knowledge of statistical fundamentals and careful thinking about causality (which will may not be practically falsifiable!). "Data" can also be misread and applied in big powerpoints with a reckless disregard for reality.

Enlightenment is understanding enough about data to use it correctly, but also acknowledging its limitations and that to create a successful business you also need vision and insight about potentials and trends for which accurate data does not and can not exist (except as a post-hoc trailing indicator).

Re: Do data-driven companies win?

#46

Post by mejutoco »

mrweasel wrote: post #29
mejutoco wrote: post #10

I believe everything else being equal data-driven is better. Of course if data-driven means short-term with bad KPIs this version of data-driven will fail. I think a lot of people picture this A/B short-term testing when they think of it. A data-driven approach is still far from objective and relies on choosing good metrics to optimize for. To this end I would recommend the book "How to measure anything" from Douglas…

> I believe everything else being equal data-driven is better. While I understand your point, it also depends on what you mean by "better". It's certainly safer, less risk, more predictable and makes planning easier. It also limits creativity and high pay offs. Some one else commented that it was like "driving by looking only at the dashboard". You can do that very safely in planes, but you're limited in where you ca…

> It also limits creativity and high pay offs.

I think this fully depends on the metric that you choose. The metrics can be very creative.

> could allow them to move from developing a SaaS product to running a chain of burger joints, because the data indicates that would be a good move.

In this scenario the counterfactual is a company without data is taking the decision based on a hunch.

I think we are saying similar things. For me any decision that is rational is based on data (better or worse data, but _something_ is quantified). What and how to quantify from the real world to incentivise the right behaviour is the creative part, from my point of view.

Re: Do data-driven companies win?

#48

Post by wussboy »

atrettel wrote: post #19
Ozzie_osman wrote: post #3

I worked at a self-described "data-driven" company, and the analogy senior leadership liked to make was that the company was like a machine learning algorithm, using data (particularly A/B tests) to do "gradient descent" of the product into its optimal form. My first take-away was that using data to make decisions is tremendously, tremendously powerful. A/B tests, in particular, can help determine causality and drive…

This idea is sometimes called the McNamara fallacy: https://en.wikipedia.org/wiki/McNamara_fallacy

There are a lot of powerful ideas in that article. Every leader of a "data driven company" should read it and tremble.

Re: Do data-driven companies win?

#49

Post by lumost »

Ozzie_osman wrote: post #3

I worked at a self-described "data-driven" company, and the analogy senior leadership liked to make was that the company was like a machine learning algorithm, using data (particularly A/B tests) to do "gradient descent" of the product into its optimal form. My first take-away was that using data to make decisions is tremendously, tremendously powerful. A/B tests, in particular, can help determine causality and drive…

Something that also tends to be under-appreciated is the constant bias towards successful A/B testing. Successful A/B tests lead to promotions and other positive employee outcomes - negative A/B tests can lead to dismissal in some places. As such people will naturally try to beef up the A/B testing result, or ship a "mixed" result and claim success.

Re: Do data-driven companies win?

#50

Post by buscoquadnary »

Quarrelsome wrote: post #22
KingOfCoders wrote: post #4

I see too many "data driven companies" that are not data driven, but selectivly use data to do what the executive wants to be done. On top of that I see companies with employees who didn't understand college mathematics but now want to be data driven - E.g. basing major decisions on 10 customer feedback data points. Or who don't know the difference between median and average.

> but selectivly use data to do what the executive wants to be done. the thirst of some people when I give them something that sounds like what they want. When they re-parrot last years findings back to me incorrectly because they only listened to the parts that backed up their instinctual beliefs. Makes me angry I waste my time on the data when all they wanted was an excuse. I coulda given them an excuse without hav…

Sounds like you need to embrace the "Zen of Wally" my friend. Just pull up Dilbert.com search for Wally and then read every strip associated with him. At the end you will understand. You will also receive extra points if you do this while you are supposed to be working as that is practicing not just living the Zen of Wally.
Post reply on HN