51–60 of 75 posts
Good insights. Before the term was bastardized, then consumed, by Machine Learning, the experimentation component was considered the killer delivery by data scientists (putting the _science_ in the term). Now, most folks assume MLE := DS, rather than MLE ⊂ DS
OT: there's actually a ≔ symbol, so you could write MLE ≔ DS if you wanted to.
I've never worked at a company where we were anywhere close to being able to A/B testing on customers. We just always shot from the hip. I think if you are doing that kind of testing may e you've run out of helpful ideas? Talking to real customers and helping them solve real problems is really potent. And you can get more than just the color of a button. You can get the direction your company needs to go for months.…
> Talking to real customers and helping them solve real problems is really potent. One of the many issues is you only get tot talk to customers willing to honestly talk to you. That means you can't hear from potential new customers you wouldn't know were part of your market. You also don't hear from customers who would want to leave you but just haven't put it into words yet. A/B testing helps get more insight into w…
In the late 2000s I was part of a team that was developing some pretty incredible software to help chip designers manage the added complexity as features got smaller (context: our customers were freaking out about how hard it looked like 45nm was going to be). We did all of these customer satisfaction surveys and shit like that and got... some decent feedback but mostly just all rainbows and unicorns positive reviews.
Chip design software is complex and every customer of ours needed some custom integration, which is where my small group came in: the three of us were dual-degree EE/CS folks. We could sit down with the chip designers and understand their workflow and then go back to our hotel room at night and write the integration code to connect our tool with whatever bespoke workflow they had internally. All of that story leading up to the main point:
The feedback I got talking to random people outside in the smoking area was dramatically more valuable than anything we got from our customer surveys. This wasn't a strategy, I'd just go out for a smoke every hour or two to smoke and there'd usually be a couple of employees out there doing the same. "Hey, I don't recognize you, are you new?" "Oh, no, I'm here helping with the $X integration" "Oh! Hey so maybe you can help me then... in the latest release it looks like feature $X should be able to do $Y but I can't seem to get it to work..."
Pretty much every time I went outside I ended up learning something new, either an interesting way our software was being used or misused, or some other detail about how these guys' day-to-day workflow worked that we hadn't even thought of addressing.
We had some customers in Japan, too, where there's a an interesting social hierarchy when having business meetings. Me and the junior engineer across the table couldn't talk to each other directly in the meetings, all of the questions had to go through my manager, and a translator, and a senior manager on the other side of the table... in a big game of telephone even though we were in the same room. After the meeting I would usually go have a smoke and just happen to find the junior guy from the meeting doing the same. "You know, I do speak English... and have a few questions if you don't mind me asking directly" :D
While I can't recommend picking up a persistent nicotine addiction for doing better user research, I also can't say that I've ever encountered a more organic way to get really good unfiltered user feedback. Surveys, user studies, focus groups, etc... they're all decent tools to varying degrees but don't always get the level of honesty you can get out of someone sharing 5 minutes with you in the smoker's corner.
Maybe I’m missing something. The goal of an A/B test is to test a hypothesis. Where that hypothesis comes from is irrelevant. Sure you can waste your time testing stupid hypotheses that don’t have a lot of business impact, but that’s beside the point.
I watched that company converge on the blandest, clunkiest, least useful features over and over and over again.
Blindly trusting the data without any product vision is just design by committee at scale.
I've never worked at a company where we were anywhere close to being able to A/B testing on customers. We just always shot from the hip. I think if you are doing that kind of testing may e you've run out of helpful ideas? Talking to real customers and helping them solve real problems is really potent. And you can get more than just the color of a button. You can get the direction your company needs to go for months.…
> Talking to real customers and helping them solve real problems is really potent. This, this and THIS again ! Example case (of many I could cite) would be Transferwise. They used to be good, but now they've denigrated into a quagmire. Could they be bothered to talk to their customers, or even just send round some box ticking surveys, they might find that out. No amount of A/B testing, data lakes or other "data scien…
Earlier quoted context omitted.
What do you think is wrong with guess work?
That you are as often wrong as you are right? If you have numbers, good ones, use them during decision finding.
For experts with tens of thousands of hours experience in a specialised field, with 40 years of case studies to extrapolate from?
Let's cast it in more relatable terms:
Such a person is, an enormous collection of data.
The day will come... soon, when people who "believe in technology" (in the very strong sense) will see no problem putting absolute trust in a neural network trained on exactly that same corpus of data.
A neural network is of course, a magnificent black box statistics machine.
And what are statistics machines trained on? Numbers. But they process and relate to them in a fuzzy way.
What is a spreadsheet and data analytics suite? Numbers.
Now your human specialist is going to outperform the numbers machine every time. But the human can often not introspect their ineffable knowledge (most expert knowledge is like that; which is why we developed the entire filed of expert systems to make it legible)
So if we choose to call such knowledge "feelings" of "guesswork" we're making a silly mistake. What does that even mean?
Neither can the neural network introspect. But we choose to label that ineffable knowledge as "calculation".
And so you invoke the magical properties of "NUmbers!" (did you mean real or imaginary ones :)
You see the error we fall into, giving two different labels to the same process only because of what hardware they execute on?
What I'd really like to talk about is the logical process of discovery called "abduction", but I fear I am rambling already :)
I've never worked at a company where we were anywhere close to being able to A/B testing on customers. We just always shot from the hip. I think if you are doing that kind of testing may e you've run out of helpful ideas? Talking to real customers and helping them solve real problems is really potent. And you can get more than just the color of a button. You can get the direction your company needs to go for months.…
You should be using data to invalidate your assumptions, separate the real from the perceived, and to draw those aha moments mentioned in the article. Then use that to prioritize and decide what’s worth iterating on and when its good enough to move on to bigger problems.
As the article says, data won’t tell you everything, which is why your data people need to also be product people, and not just sql monkeys or phds in a backroom doing analyses nobody will understand or read.
Earlier quoted context omitted.
That you are as often wrong as you are right? If you have numbers, good ones, use them during decision finding.
50/50? For terrible guessers maybe. For experts with tens of thousands of hours experience in a specialised field, with 40 years of case studies to extrapolate from? Let's cast it in more relatable terms: Such a person is, an enormous collection of data . The day will come... soon, when people who "believe in technology" (in the very strong sense) will see no problem putting absolute trust in a neural network trained…
Every situation is different, facts change, so I have to evaluate my opinion each and every time (which is hownypu learn and bevome better). And the more data I have, the easier this is.
Earlier quoted context omitted.
50/50? For terrible guessers maybe. For experts with tens of thousands of hours experience in a specialised field, with 40 years of case studies to extrapolate from? Let's cast it in more relatable terms: Such a person is, an enormous collection of data . The day will come... soon, when people who "believe in technology" (in the very strong sense) will see no problem putting absolute trust in a neural network trained…
I consider myself to be rather good in my field. Which is exactly why I take every bit of data I can get before I provide my opinion or decide something. Every situation is different, facts change, so I have to evaluate my opinion each and every time (which is hownypu learn and bevome better). And the more data I have, the easier this is.