Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data will not tell you what to do

mikkeldengsoe.substack.com

Post reply on HN
Print view View original

41–50 of 75 posts

Re: Data will not tell you what to do

#41

Post by nonrandomstring »

from-nibly wrote: post #7

I've never worked at a company where we were anywhere close to being able to A/B testing on customers. We just always shot from the hip. I think if you are doing that kind of testing may e you've run out of helpful ideas? Talking to real customers and helping them solve real problems is really potent. And you can get more than just the color of a button. You can get the direction your company needs to go for months.…

The direct, old-fashioned approach of;

  - being real

  - talking to customers

  - actually listening to what people say

  - using intuition

  - cutting to the chase (big and meta problems first)

  - doing risky exploration for abductive reasoning
is only as good as the nominal culture we're in. As the author says,

> I’m no longer a believer in decision-by-spreadsheet.

That's nice for you. Me neither. But every day we must interact with dull-headed data crunchers who set the pace and policy.

Re: Data will not tell you what to do

#42

Post by disgruntledphd2 »

fshbbdssbbgdd wrote: post #36
from-nibly wrote: post #7

I've never worked at a company where we were anywhere close to being able to A/B testing on customers. We just always shot from the hip. I think if you are doing that kind of testing may e you've run out of helpful ideas? Talking to real customers and helping them solve real problems is really potent. And you can get more than just the color of a button. You can get the direction your company needs to go for months.…

When I worked at a company that did a lot of AB testing, we had a lot of people whose main job was to interview users. This was an input in the development process. Then we tested whatever we came up with before releasing it. I suspect there’s a disconnect here where you are talking about smaller, early stage companies. A lot of the time they don’t have the sample size to do proper AB testing, or the resources to do…

AB testing works well in B2C, but is much harder to make work in B2B.

You can do it, but you can generally only test really large changes, and often if you have good customer communication you can pick up on what the change means with some interviews and showing the customer(s) what the new thing looks like.

This is generally much faster and cheaper (and I say this as someone who adores designing, running and analysing AB tests).

Re: Data will not tell you what to do

#43

Post by hef19898 »

nonrandomstring wrote: post #41
from-nibly wrote: post #7

I've never worked at a company where we were anywhere close to being able to A/B testing on customers. We just always shot from the hip. I think if you are doing that kind of testing may e you've run out of helpful ideas? Talking to real customers and helping them solve real problems is really potent. And you can get more than just the color of a button. You can get the direction your company needs to go for months.…

The direct, old-fashioned approach of; - being real - talking to customers - actually listening to what people say - using intuition - cutting to the chase (big and meta problems first) - doing risky exploration for abductive reasoning is only as good as the nominal culture we're in. As the author says, > I’m no longer a believer in decision-by-spreadsheet. That's nice for you. Me neither. But every day we must inter…

All of the above needs some basis in reality, aka numbers. Otherwise it is just guess work.

Re: Data will not tell you what to do

#44

Post by carschno »

from-nibly wrote: post #7

I've never worked at a company where we were anywhere close to being able to A/B testing on customers. We just always shot from the hip. I think if you are doing that kind of testing may e you've run out of helpful ideas? Talking to real customers and helping them solve real problems is really potent. And you can get more than just the color of a button. You can get the direction your company needs to go for months.…

And because “science takes too long” (and is expensive, and tedious), people tend to fall back to pseudo-science. There is often no way to derive robust numbers from techniques like A/B testing, for instance because of confounding factors that are not measurable. Given that, I have regularly heard the argument that “these are the only numbers that we have”.

Relying on such numbers, however, is equivalent to falling back on intuition and gut feelings for decision-making (or worse), while believing that the decisions were based on numbers.

Re: Data will not tell you what to do

#45

Post by rightbyte »

makeitdouble wrote: post #21
williamcotton wrote: post #19

Earlier quoted context omitted.

And what does any of that have to do with finding the right services or products for the right customers? Does a customer care if you shave off 2% of the final cost or do they care about having world-class customer support? Does a customer care if a product has a higher conversion rates or if it is the product they were looking for in the first place? Does a customer care more about how long payment processing takes…

> Does a customer care if you shave off 2% of the final cost or do they care about having world-class customer support? Times and times again, they'll tell you the latter and actually choose the former. The same bargain as paying more to have no ads: people vocally push for no ads, some will ponny up the money, and the vast majority will make do with the ad supported model, while ad blocking or giving up on the servi…

> Times and times again, they'll tell you the latter and actually choose the former.

I don't think this is as true anymore.

There was some time between 1990 and 2015(ish?) where physical widget prices was falling faster then the quality decrease, software and computing hardware got better, where the quoted strategy made sense.

Nowadays you will get dropshipped crap (or any service sector equivalent) if you go for the lowest price.

Re: Data will not tell you what to do

#46

Post by nonrandomstring »

hef19898 wrote: post #43
nonrandomstring wrote: post #41

Earlier quoted context omitted.

The direct, old-fashioned approach of; - being real - talking to customers - actually listening to what people say - using intuition - cutting to the chase (big and meta problems first) - doing risky exploration for abductive reasoning is only as good as the nominal culture we're in. As the author says, > I’m no longer a believer in decision-by-spreadsheet. That's nice for you. Me neither. But every day we must inter…

All of the above needs some basis in reality, aka numbers. Otherwise it is just guess work.

What do you think is wrong with guess work?

Re: Data will not tell you what to do

#47

Post by newaccount74 »

I'm a bit torn on this. I think basing product decisions on analytics is a bad idea, because the numbers can only tell you about features your software already has.

But analytics/diagnostics are extremely important to discover bugs, because you can't rely on customers to tell you about them.

Re: Data will not tell you what to do

#48

Post by traceroute66 »

from-nibly wrote: post #7

I've never worked at a company where we were anywhere close to being able to A/B testing on customers. We just always shot from the hip. I think if you are doing that kind of testing may e you've run out of helpful ideas? Talking to real customers and helping them solve real problems is really potent. And you can get more than just the color of a button. You can get the direction your company needs to go for months.…

> Talking to real customers and helping them solve real problems is really potent.

This, this and THIS again !

Example case (of many I could cite) would be Transferwise.

They used to be good, but now they've denigrated into a quagmire. Could they be bothered to talk to their customers, or even just send round some box ticking surveys, they might find that out.

No amount of A/B testing, data lakes or other "data science" buzzwords is going to help them.

But no, instead its the same old story :

Rebranding from Transferwise to Wise because, well, I guess that's the usual shit companies do when they've run out of ideas (Aberdeen rebranding to Abrdn is another fine example from the financial sector).

Doing stuff worse because it benefits the business (read: increase margins) rather than the customer. Transfers take forever. Customer service is non existent.

Funnily enough it all seems to have started going downhill around the same time they floated on the stock market. Funny that !

Whilst I am aware that a company's strict legal definition is to put its shareholders first, it doesn't have to be that way, at least not in a blatant manner. Afterall, disgruntled customers don't do much good for shareholder's pockets.

Re: Data will not tell you what to do

#49

Post by richrichie »

“ Their research shows that a nonlinear approach drawing from anthropology, sociology, philosophy, and psychology, is better at getting to the moment of clarity”

And all of these come from data. There is no non linearity here; just widening of perceptive funnel.

Re: Data will not tell you what to do

#50

Post by ysofunny »

TheAlchemist wrote: post #4

As painful as it is to admit, I've come to the same conclusion, after wasting quite a lot of time and efforts. As Warren Buffett likes to say - "It's better to be approximately right, than to be precisely wrong" This should be a poster in every company doing any kind of data.

I think in some other context it's actually preferred to be precisely wrong than roughly correct

maybe this is the difference between a business (or engineering) mindset of "it must work effectively, how and why it works are secondary"

in contrast with a perhaps more phillosophical (scientific? purely mathematical? reverse engineering?) study goal? in which case whether something works is secondary to having a full theory of what's going on

Post reply on HN