11–20 of 121 posts
This is a great point. I've seem teams apply lean startup by testing -> changing something -> testing -> changing something -> testing ...
The problem is that the changes are so small that statistically you end up testing the same thing over and over and expecting different results. You need to make a significant change in your Persona, Problem, Promise, or Product to (hopefully) see promising results.
There are companies out there that probably do none of these things and are x1000 more successful from a revenue or market cap perspective. Seems like the biggest successes are simply being at the right place at the right time and not being a complete idiot. Nobody wants to hear that though.
There are companies out there that probably do none of these things and are x1000 more successful from a revenue or market cap perspective. Seems like the biggest successes are simply being at the right place at the right time and not being a complete idiot. Nobody wants to hear that though.
Sam Altman argued a startup's chance of success is “something like Idea times Product times Execution times Team times Luck, where Luck is a random number between zero and ten thousand”
A lot of the success in startups is not due to getting the initial idea right, but what founders do once they realize that their initial idea is wrong.
This list mentions A/B testing a few times and it's worth noting that A/B testing is great but it's not free. I've seen a nontrivial number of smart engineers get bogged down in wanting to A/B test everything that they spend more time building and maintaining the experiment framework than actually shipping more product and then realizing the A/B testing was useless because they only had a few hundred data points. Dat…
Most orgs should just be shipping features. Before starting an Experiment Program teams should be brainstorming a portfolio of experiments. Just create a spreadsheet where the first column is a one-line hypothesis of the experiment. Eg. "Removing step X from the funnel will increase metric Y while reducing metric Z". And the RICE (Reach-Impact-Confidence-Estimation) score your portfolio.
If the team can't come up with a portfolio of 10s to 100s of experiments then the team should just be shipping stuff.
And then Experiment Buildout should be standardized. Have standardized XRD (Experiment Requirements Doc). Standardize Eligibility and Enrollment criteria. Which population sees this experiment? When do they see it? How do you test that bucketing is happening correctly? What events do analysts need? When do we do readouts?
That's just off the top of my head. Most orgs should just be shipping features.
Sorry, what "successful products"? And I have to say that "Technical Content Marketer " is one of the most dubious job titles I have ever seen.
On their homepage, they have their own logo in the section listing companies that use their products. I mean, if you dogfood it, great, but it doesn't exactly instill confidence if you have to reach for your own company to fill out the list.
Sorry, what "successful products"? And I have to say that "Technical Content Marketer " is one of the most dubious job titles I have ever seen.
Capturing analytics is a no brainer. however, most data in most products at most companies starting out just fundamentally does not matter. It's dangerous to get in the habit of "being data driven" because it becomes a false sense of security and paradoxically data is extremely easy to be biased with. And even with more rigor, you get too easily trapped in local optimums. Lastly, all things decay, but data and experimentation runs as is if the win is forever, until some other test beats it. It becomes exhausting to touch anything and that's seen as a virtue. it's not.
Products need vision and taste.