Both data mining and intuition have their uses. The key is to recognize when you should use each or both.
Data will not tell you what to do
31–40 of 75 posts
Re: Data will not tell you what to do
#32Another point to support the same idea is data can be falacious. For instance, you are making a pretty advanced 3D web app, and notice in your analytics that your userbase is only chrome and safari users. An easy conclusion is to focus your testing on those two platforms, or maybe even drop support entirely for Firefox and Edge by using some webkit specific API. A not so easy conclusion is the experience might be so…
It goes deeper. People learn over time that complex web apps work badly in Firefox, because developers mostly test in Chrome. So they don't even bother trying it in Firefox.
Maybe. Or maybe Firefox users have sophisticated ads & tracker blocker, sometimes even UA spoofing. Then it really depends on your analytics method, if these users show up at all. You might have a lot of FF users, but analytics tells you otherwise.
Re: Data will not tell you what to do
#33Another point to support the same idea is data can be falacious. For instance, you are making a pretty advanced 3D web app, and notice in your analytics that your userbase is only chrome and safari users. An easy conclusion is to focus your testing on those two platforms, or maybe even drop support entirely for Firefox and Edge by using some webkit specific API. A not so easy conclusion is the experience might be so…
And a lot of privacy-friendly browsers don't show up in the stats by design.
Re: Data will not tell you what to do
#34Data is for seeing problems, not for finding solutions.
As by the authors example, you lose 10M in fraud? Good that you monitor that! Otherwise it would be hard to justify spending time on it.
Re: Data will not tell you what to do
#35Another point to support the same idea is data can be falacious. For instance, you are making a pretty advanced 3D web app, and notice in your analytics that your userbase is only chrome and safari users. An easy conclusion is to focus your testing on those two platforms, or maybe even drop support entirely for Firefox and Edge by using some webkit specific API. A not so easy conclusion is the experience might be so…
This is pretty standard survivorship bias - which is to say, a well-studied problem with a famous pithy example (bomber plane armor), that many people and organizations still somehow don't learn.
Re: Data will not tell you what to do
#36I've never worked at a company where we were anywhere close to being able to A/B testing on customers. We just always shot from the hip. I think if you are doing that kind of testing may e you've run out of helpful ideas? Talking to real customers and helping them solve real problems is really potent. And you can get more than just the color of a button. You can get the direction your company needs to go for months.…
When I worked at a company that did a lot of AB testing, we had a lot of people whose main job was to interview users. This was an input in the development process. Then we tested whatever we came up with before releasing it.
I suspect there’s a disconnect here where you are talking about smaller, early stage companies. A lot of the time they don’t have the sample size to do proper AB testing, or the resources to do it properly, and they have less to lose. So shooting from the hip is more likely to be the only reasonable choice.
Re: Data will not tell you what to do
#37Thank you! This guy has obviously read Nassim Taleb. Here you go upvote!
Re: Data will not tell you what to do
#38I think data driven is orhtogonal to opportunity driven and vision driven. With the first, data is used to find opportunities, with the second, it shows you if your strategy to your vision is working or not. Then there is politics driven development. You want to do something and search the trove of data for data that supports what you are doing. Or you look at the data in a strategy meeting, and then ignore it (seen…
I like this explanation. I wonder about this on a philosophical level -- is data analysis just our defense mechanism to gain control over things?
Another thing to say is not all companies should or can be vision driven, some companies are just that copy cat.
Re: Data will not tell you what to do
#39Even if you can tell with absolute knowledge that changing the graphics of that packaging increases conversion (removing the disgusting graphics of cancer patients), perhaps we shouldn't be selling more cigarettes?
Data informs your values, but your values are a choice. Already from the beginning, data will never define your values. Even with perfect knowledge, your decisions are still going to be a choice. Combine that with the fact that our knowledge is imperfect (our data is incomplete, biased, a single and partial perspective)...
"A company should seek to maximize its profits," is a normative statement, not a truth. It is a choice of values.
Re: Data will not tell you what to do
#40A/B testing is valuable to optimise an existing process once the low-hanging fruit has been tackled, but it should also be discontinued when the return on investment becomes negligible. Because of organisational momentum, this rarely happens, as a team would make themselves redundant. This can result in A/B testing noise.
Data also provides valuable insight in a negative manner. If your conversion rate is abysmal, the data tells you to get out of your cubicle and start talking with real customers to find out what the data isn't telling you. It is still a data-driven decision. It is just a negative one.
However, in the end, data isn't going to find your next billionaire dollar opportunity. You need to find a gap in the market that no one has tackled before, and of course there is no data for, otherwise someone else would have jumped on it.