Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Do data-driven companies win?

benn.substack.com

Post reply on HN
Print view View original

21–30 of 63 posts

Re: Do data-driven companies win?

#21

Post by tyleo »

Ozzie_osman wrote: post #3

I worked at a self-described "data-driven" company, and the analogy senior leadership liked to make was that the company was like a machine learning algorithm, using data (particularly A/B tests) to do "gradient descent" of the product into its optimal form. My first take-away was that using data to make decisions is tremendously, tremendously powerful. A/B tests, in particular, can help determine causality and drive…

An analogy the company I’m at (Rec Room) has used is “Data can help you climb a mountain, it can’t tell you which mountain to climb.”

Re: Do data-driven companies win?

#22

Post by Quarrelsome »

KingOfCoders wrote: post #4

I see too many "data driven companies" that are not data driven, but selectivly use data to do what the executive wants to be done. On top of that I see companies with employees who didn't understand college mathematics but now want to be data driven - E.g. basing major decisions on 10 customer feedback data points. Or who don't know the difference between median and average.

> but selectivly use data to do what the executive wants to be done.

the thirst of some people when I give them something that sounds like what they want. When they re-parrot last years findings back to me incorrectly because they only listened to the parts that backed up their instinctual beliefs. Makes me angry I waste my time on the data when all they wanted was an excuse. I coulda given them an excuse without having to do all that data digging...

Re: Do data-driven companies win?

#23

Post by puppymaster »

Ozzie_osman wrote: post #3

I worked at a self-described "data-driven" company, and the analogy senior leadership liked to make was that the company was like a machine learning algorithm, using data (particularly A/B tests) to do "gradient descent" of the product into its optimal form. My first take-away was that using data to make decisions is tremendously, tremendously powerful. A/B tests, in particular, can help determine causality and drive…

yep. It is great when you are optimizing a process. But you can't optimize a wrong direction. In some cases it makes it worse.

Re: Do data-driven companies win?

#24

Post by davzie »

Sometimes, but the company will lack soul, purpose and a meaning that it stands for. Sometimes it's better to stand by what you want your company's values to be, even if that means losing some potential customers. As a result, you'll have more passionate and engaged customers who are more likely to stick with you long term.

Re: Do data-driven companies win?

#25

Post by marginalia_nu »

Ozzie_osman wrote: post #3

I worked at a self-described "data-driven" company, and the analogy senior leadership liked to make was that the company was like a machine learning algorithm, using data (particularly A/B tests) to do "gradient descent" of the product into its optimal form. My first take-away was that using data to make decisions is tremendously, tremendously powerful. A/B tests, in particular, can help determine causality and drive…

Another fundamental problem with this type of approach is that science is hard.

If you don't know how to properly construct experiments and interpret data it's honestly not much better than hiring an ornithomancer to spot omens in the skies. To be clear, this is something that even scientists with decades of both education and experience sometimes get wrong. If you're coming at it with an MBA I've got some concerns.

Re: Do data-driven companies win?

#26

Post by szundi »

Ozzie_osman wrote: post #3

I worked at a self-described "data-driven" company, and the analogy senior leadership liked to make was that the company was like a machine learning algorithm, using data (particularly A/B tests) to do "gradient descent" of the product into its optimal form. My first take-away was that using data to make decisions is tremendously, tremendously powerful. A/B tests, in particular, can help determine causality and drive…

Haha. Only thing that gradient hill climbing algos are called “local” as they stuck in local maximums. Probably that’s what happened “on the long term”.

Re: Do data-driven companies win?

#28

Post by everdrive »

Ozzie_osman wrote: post #3

I worked at a self-described "data-driven" company, and the analogy senior leadership liked to make was that the company was like a machine learning algorithm, using data (particularly A/B tests) to do "gradient descent" of the product into its optimal form. My first take-away was that using data to make decisions is tremendously, tremendously powerful. A/B tests, in particular, can help determine causality and drive…

I don't have a way to prove this, but I've long suspected that A/B testing might be one of the major culprits to the modern problem of software churn. ie, constantly-changing GUIs, features, and products. Some percentage of users will always have problems with an application or interface. Constantly chasing every user hurdle might sound like a good idea, but I do wonder if it's similar to listening to too much fan feedback in the case of video games, movies, etc. Implementing every fan suggestion will usually make a terrible game. Instead, the ideal is to figure out which fan suggestions to ignore, and which to listen to. Only some of them will work for a given application.

I wonder if A/B testing is a bit like that. You're suddenly listening to every single issue encountered by users, but lack the wisdom to understand which issues to ignore. As a result, software changes constantly, users cannot really learn a GUI because it changes so frequently, and software teams feel like they're constantly improving things, however the software itself is not actually getting better when a the whole user's experience is taken into consideration.

Re: Do data-driven companies win?

#29

Post by mrweasel »

mejutoco wrote: post #10

I believe everything else being equal data-driven is better. Of course if data-driven means short-term with bad KPIs this version of data-driven will fail. I think a lot of people picture this A/B short-term testing when they think of it. A data-driven approach is still far from objective and relies on choosing good metrics to optimize for. To this end I would recommend the book "How to measure anything" from Douglas…

> I believe everything else being equal data-driven is better.

While I understand your point, it also depends on what you mean by "better". It's certainly safer, less risk, more predictable and makes planning easier. It also limits creativity and high pay offs.

Some one else commented that it was like "driving by looking only at the dashboard". You can do that very safely in planes, but you're limited in where you can go. I feel it's the same in business.

There is the option of you having an absolutely massive dataset available. It just seems a bit far fetch to assume that any company would have data that could allow them to move from developing a SaaS product to running a chain of burger joints, because the data indicates that would be a good move.

Re: Do data-driven companies win?

#30

Post by ggeorgovassilis »

"Data-driven" is a fancier term for "empirical", a method for building knowledge based on observation. The other method is "epistemology", a method for building knowledge by reasoning about things. A successful organisation should not snob the one over the other.
Post reply on HN