Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Do data-driven companies win?

benn.substack.com

Post reply on HN
Print view View original

31–40 of 63 posts

Re: Do data-driven companies win?

#31

Post by deltathreetwo »

ggeorgovassilis wrote: post #30

"Data-driven" is a fancier term for "empirical", a method for building knowledge based on observation. The other method is "epistemology", a method for building knowledge by reasoning about things. A successful organisation should not snob the one over the other.

Why not just use the layman terms. Science and logic.

The terms derived from philosophy are the most snobbish in my opinion. Additionally Logic and science formalize these concepts, Epistemology on the other hand has a lot of qualitative mumbo jumbo, with concepts like "belief."

Re: Do data-driven companies win?

#32

Post by deltathreetwo »

everdrive wrote: post #28
Ozzie_osman wrote: post #3

I worked at a self-described "data-driven" company, and the analogy senior leadership liked to make was that the company was like a machine learning algorithm, using data (particularly A/B tests) to do "gradient descent" of the product into its optimal form. My first take-away was that using data to make decisions is tremendously, tremendously powerful. A/B tests, in particular, can help determine causality and drive…

I don't have a way to prove this, but I've long suspected that A/B testing might be one of the major culprits to the modern problem of software churn. ie, constantly-changing GUIs, features, and products. Some percentage of users will always have problems with an application or interface. Constantly chasing every user hurdle might sound like a good idea, but I do wonder if it's similar to listening to too much fan fe…

The churn is the result of the fact that we don't understand what is optimal.

We know how to optimize for things like the shortest distance between two points but we don't understand how to optimize Software Design or even GUI.

This is literally what "Design" means. If something needs to be designed it means there's no theory or notion behind it on what it means to be "optimal." So we "design" a "better" solution but we don't actually know if it's better. So we design another solution and the cycle continues.

A/B testing when done on a population that gives consistent answers should converge on a consistent solution. If the population gives different answers at different times then of course there will be churn.

I would say the methodology of A/B testing is indeed like machine learning, it's quite good and accurate. It's the data source that's the problem. If you have users that don't know what they want or behave differently and inconsistently then your conclusions reflect the data. Perhaps the data is accurate and there is no consistent conclusion, OR the data is inaccurate and you need to get it from a better source other then users telling you what they think is better.

Re: Do data-driven companies win?

#33

Post by Enginerrrd »

davidkuennen wrote: post #8

I connect data-driven to A/B testing. And A/B testing for me is like squeezing the maximum out of an orange, instead of making more oranges. It will give a short term boost, but eventually you will have to look beyond your data and explore new things your customers really want.

I think this is akin to focusing too hard on things that are appear easy to measure over things that are (perhaps incorrectly) perceived as hard to measure.

The parable of the drunk found looking for his keys under the light instead of inside the dark bar where he lost them comes to mind. What's needed is a flashlight, however imperfect.

Re: Do data-driven companies win?

#34

Post by cestith »

jlei523 wrote: post #9
davidkuennen wrote: post #8

I connect data-driven to A/B testing. And A/B testing for me is like squeezing the maximum out of an orange, instead of making more oranges. It will give a short term boost, but eventually you will have to look beyond your data and explore new things your customers really want.

A/B testing can definitely make product teams feel like they're making progress but it should not be a team's only strategy. It's a bit like moving the furniture around the same room.

Nobody has a big enough test set to A/B test every combination of elements for a web page, a product feature set, or even a conference presentation. The combinatorics are just unwieldy.

Someone has to have multiple good ideas and the ability to carry them out properly before an A/B test is even valuable. Otherwise you're measuring one uninformed random change against another. A/B confirms whether you've succeeded in improving something. It can't really suggest what to try next.

Re: Do data-driven companies win?

#35

Post by itsoktocry »

Ozzie_osman wrote: post #3

I worked at a self-described "data-driven" company, and the analogy senior leadership liked to make was that the company was like a machine learning algorithm, using data (particularly A/B tests) to do "gradient descent" of the product into its optimal form. My first take-away was that using data to make decisions is tremendously, tremendously powerful. A/B tests, in particular, can help determine causality and drive…

>Being purely data-driven without good intuition and long-term bets (that can't be "proven" with data), and the product loses its soul.

This sounds like post-hoc, anti-intellectual rationalization.

How do you place long-term bets without a model to measure expectations versus outcomes? How do you know what good intuition is? There is a ton of research on "gut" calls that demonstrates it's random.

>But data is not a substitute for good judgment

Good judgement requires data.

Re: Do data-driven companies win?

#36

Post by itsoktocry »

bratbag wrote: post #20
Ozzie_osman wrote: post #3

I worked at a self-described "data-driven" company, and the analogy senior leadership liked to make was that the company was like a machine learning algorithm, using data (particularly A/B tests) to do "gradient descent" of the product into its optimal form. My first take-away was that using data to make decisions is tremendously, tremendously powerful. A/B tests, in particular, can help determine causality and drive…

Agreed. Data is useful for proving my decisions once I have made them, so what's most useful is having short feedback loops so can I catch bad decisions quickly, or reinforce good decisions.

>Data is useful for proving my decisions once I have made them

I can't figure out what this means, but I have a feeling it's the exact opposite way you should be incorporating data into your decision making.

Re: Do data-driven companies win?

#37

Post by coob »

itsoktocry wrote: post #35
Ozzie_osman wrote: post #3

I worked at a self-described "data-driven" company, and the analogy senior leadership liked to make was that the company was like a machine learning algorithm, using data (particularly A/B tests) to do "gradient descent" of the product into its optimal form. My first take-away was that using data to make decisions is tremendously, tremendously powerful. A/B tests, in particular, can help determine causality and drive…

> Being purely data-driven without good intuition and long-term bets (that can't be "proven" with data), and the product loses its soul. This sounds like post-hoc, anti-intellectual rationalization. How do you place long-term bets without a model to measure expectations versus outcomes? How do you know what good intuition is? There is a ton of research on "gut" calls that demonstrates it's random. > But data is not a…

I think you’re agreeing with OP?

Re: Do data-driven companies win?

#38

Post by itsoktocry »

KingOfCoders wrote: post #4

I see too many "data driven companies" that are not data driven, but selectivly use data to do what the executive wants to be done. On top of that I see companies with employees who didn't understand college mathematics but now want to be data driven - E.g. basing major decisions on 10 customer feedback data points. Or who don't know the difference between median and average.

>I see too many "data driven companies" that are not data driven, but selectivly use data to do what the executive wants to be done.

Bingo. And judging by many of the comments here, it's spoiled people's opinions on what it really means to be "data driven". I mean, imagine advocating for "good judgement" when so much empirical research says it's not repeatable?

Re: Do data-driven companies win?

#39

Post by itsoktocry »

mrweasel wrote: post #29
mejutoco wrote: post #10

I believe everything else being equal data-driven is better. Of course if data-driven means short-term with bad KPIs this version of data-driven will fail. I think a lot of people picture this A/B short-term testing when they think of it. A data-driven approach is still far from objective and relies on choosing good metrics to optimize for. To this end I would recommend the book "How to measure anything" from Douglas…

> I believe everything else being equal data-driven is better. While I understand your point, it also depends on what you mean by "better". It's certainly safer, less risk, more predictable and makes planning easier. It also limits creativity and high pay offs. Some one else commented that it was like "driving by looking only at the dashboard". You can do that very safely in planes, but you're limited in where you ca…

>It also limits creativity and high pay offs.

Why would being data-driven limit either of these things?

Re: Do data-driven companies win?

#40

Post by marginalia_nu »

deltathreetwo wrote: post #32
everdrive wrote: post #28

Earlier quoted context omitted.

I don't have a way to prove this, but I've long suspected that A/B testing might be one of the major culprits to the modern problem of software churn. ie, constantly-changing GUIs, features, and products. Some percentage of users will always have problems with an application or interface. Constantly chasing every user hurdle might sound like a good idea, but I do wonder if it's similar to listening to too much fan fe…

The churn is the result of the fact that we don't understand what is optimal. We know how to optimize for things like the shortest distance between two points but we don't understand how to optimize Software Design or even GUI. This is literally what "Design" means. If something needs to be designed it means there's no theory or notion behind it on what it means to be "optimal." So we "design" a "better" solution but…

Honestly, this comes off more as a model problem than a data problem.

Here are some dubious assumptions:

* There is an optimal answer.

* Human preferences are transitive (if I prefer A over B and B over C, then I prefer A over C).

* Human preferences are internally consistent.

* Human preferences are stable over time.

Apply these assumptions to finding an answer to the question of what you should make for dinner, and you'll quickly see there are problems.

Post reply on HN