The joyless world of data-driven startups
1–10 of 78 posts
Re: The joyless world of data-driven startups
#2The most important skill, in order to be data-driven, is to ask the right questions. If you're looking to get to product market fit, the questions you should be asking are very different from the ones you should be asking if you're looking to grow a "good" product. In both cases, data can help you reach your goal, but only if you ask the right questions.
If an early-stage startup tries growth hacking before it reaches product market fit, it will likely end in disaster.
Re: The joyless world of data-driven startups
#3Oh thank goodness it isn't just me.
Re: The joyless world of data-driven startups
#4I think the article goes a bit too far against data. Hits like Bohemian Rhapsody are by their nature freak events and not easily reproducible. Nobody is suggesting (I hope) that you can achieve that kind of success without a healthy dose of luck.
However I agree that for early stage startups data driven decision making can be difficult. My experience is its expensive and you often have very little data.
The other issue is this kinda uncanny valley of false rigor. On the one extreme you have very informal analysis. For example, we tweaked our blog post template and increased newsletter signup rates but I can't tell you exact %s because at this stage we don't track it. We seem to be getting a lot more signups, but perhaps its illusory. That's ok. At our level of traffic it really isn't important. At the other extreme you try to model non-stationary processes and all that and have rigorous control over sources of error. In the middle is where I see many companies with, say, A/B testing, believing they have a high level of statistical rigor but not actually achieving that rigor in practice due to many uncontrolled sources of error. This middle spot, where you have too much faith in faulty reasoning, is where I believe bad data driven decision making resides.
Oh, and on turd polishing: http://www.dorodango.com/create.html
Re: The joyless world of data-driven startups
#5The so-called data-driven science have not understand the notion of science. In a minimal sense, science is to produce knowledge. There are two things to it: hypothesis generation; testing the hypothesis. As the history and philosophy of sciences have shown, there is no algorithmic way of generating hypotheses. Or if you generate hypotheses algorithmically, you are still left to figure out whether these hypotheses are ad hoc or not. After all, the history of sciences have given powerful heuristics to reduce the solution space to generate hypotheses to solve or explain problems or facts. Here, whether one picks 'solve' or 'explain' depends on which philosophy of science one picks up.
Whenever I see statistics and data-sciences, I see tons of adhoc bullshit masquerading as sciences/knowledge. It is always easy to come up with a hypothesis to explain a set of chosen facts; in order for that hypothesis to be non ad hoc, it has to predict surprising facts.
As the fad continues, we may hear like robots replacing scientists to produce knowledge about various phenomena. For a best critique of AI, check the book by UCBerkeley philosopher Hubert Dreyfus: what computers can't do, a critique of artificial reason.
Re: The joyless world of data-driven startups
#6This article omits one important aspect: the data isn't used only to create. It's also a guide what to delete/abandon. Sure, it might be influenced by chaotic fluctuations but anyway can help make a decision. Sometimes making any decision is better than wandering around with ambivalent gut feelings.
Re: The joyless world of data-driven startups
#7> Sometimes it works. Sometimes it’s critical. But sometimes it fails, or results in unintended consequences that we may not notice for years.
> Data-driven journalism gave us Buzzfeed
> Data-driven music gave us X-Factor and Pop Idol
> Data-driven movies gave us 25 Hollywood sequels planned for this year
> Data-driven education gave us Key Performance Indicators and Teaching to the test
These first three examples are awful examples. A ton of people love all of those things. The only failure of data-drivenness here is the failure to generate content that the author wants. Now, the author can attempt to make some argument about how websites, TV shows, and movies have a moral obligation to strive for whatever objectives the author prefers, but that's a separate issue to settle.
The fourth example is a little different, because we're talking about mandatory education programs for children, rather than products that people choose to pay for or consume. Also, I don't think that data-drivenness itself is a significant contributor to those problems in education.
Re: The joyless world of data-driven startups
#8> Sometimes it works. Sometimes it’s critical. But sometimes it fails, or results in unintended consequences that we may not notice for years. > Data-driven journalism gave us Buzzfeed > Data-driven music gave us X-Factor and Pop Idol > Data-driven movies gave us 25 Hollywood sequels planned for this year > Data-driven education gave us Key Performance Indicators and Teaching to the test These first three examples ar…
The failure is the failure to generate novelty or adequately explore alternatives.
Enough people might reliable go see 25 sequels, but none of those films will be memorable. None will advance the art of film-making. None will change anyone's life or mentality or affect the culture in any meaningful way.
Being data driven means chasing the biggest, loudest signal in your data set. It means pandering to that signal, because it swamps all others. A data-driven approach is not going to lead you anywhere new.
In machine learning / AI we refer to such algorithms as "greedy." A classic example would be simple hill climbing a.k.a. gradient descent. These algorithms are known to be very good at optimizing within the bounds of a simple well-behaved and regular fitness landscape, but they readily become stuck at local maxima when presented with any solution space with any complex structure.
We're living in what I'm tempted to call the dark age of the local maximum, the age of gradient descent.
Re: The joyless world of data-driven startups
#9Premature optimization: still the root of all evil.
One of the better takeaways from the article was the notion that being data-driven means you're aiming for average, and you might not even hit it. Aim for the moon, you might only achieve orbit instead.
I watched a CEO make arbitrary layoff decisions based on what the numbers said should be the size of a development organization and the the ratio of developers to QA. The actual software being built was irrelevant to his figures. He used numbers to justify grinding the dev organization into the ground.
Re: The joyless world of data-driven startups
#10> Sometimes it works. Sometimes it’s critical. But sometimes it fails, or results in unintended consequences that we may not notice for years. > Data-driven journalism gave us Buzzfeed > Data-driven music gave us X-Factor and Pop Idol > Data-driven movies gave us 25 Hollywood sequels planned for this year > Data-driven education gave us Key Performance Indicators and Teaching to the test These first three examples ar…
The failure is the failure to generate novelty or adequately explore alternatives. Enough people might reliable go see 25 sequels, but none of those films will be memorable. None will advance the art of film-making. None will change anyone's life or mentality or affect the culture in any meaningful way. Being data driven means chasing the biggest, loudest signal in your data set. It means pandering to that signal, be…
> The failure is the failure to generate novelty or adequately explore alternatives.
Perhaps you value novelty more than the average person. What might not be "adequate" for you might very well be adequate for a huge portion of people.
Throughout this comment you mention several potential goals of content producers (being memorable, advancing the art, changing someone's life, etc.), but you make no argument for why those goals ought to be prioritized over other goals, other than the implication that you personally prefer those goals.