31–40 of 121 posts
Sorry, what "successful products"? And I have to say that "Technical Content Marketer " is one of the most dubious job titles I have ever seen.
I've used PostHog and it's pretty good. I don't know if I'd classify all of those as different products, you rarely want one of those without the other.
This list mentions A/B testing a few times and it's worth noting that A/B testing is great but it's not free. I've seen a nontrivial number of smart engineers get bogged down in wanting to A/B test everything that they spend more time building and maintaining the experiment framework than actually shipping more product and then realizing the A/B testing was useless because they only had a few hundred data points. Dat…
- You have to make good decisions about what you're going to test
- You have to build the feature twice
- You have to establish a statistically robust tracking mechanism. Using a vendor helps here, but you still need to correctly integrate with them.
- You have to test both versions of the feature AND the tracking and test selection mechanisms really well, because bugs in any of those invalidate the test
- You have to run it in production for several weeks (or you won't get statistically significant results) - and ensure it doesn't overlap with other tests in a way that could bias the results
- You'd better be good at statistics. I've seen plenty of A/B test results presented in ways that did not feel statistically sound to me.
... and after all of that, my experience is that a LOT of the tests you run don't show a statistically significant result one way or the other - so all of that effort really didn't teach you much that was useful.
The problem is that talking people out of running an A/B test is really hard! No-one ever got fired for suggesting an A/B test - it feels like the "safe" option.
Want to do something much cheaper than that which results in a much higher level of information? Run usability tests. Recruit 3-5 testers and watch them use your new feature over screen sharing and talk through what they're doing. This is an order of magnitude cheaper than A/B testing and will probably teach you a whole lot more.
This list mentions A/B testing a few times and it's worth noting that A/B testing is great but it's not free. I've seen a nontrivial number of smart engineers get bogged down in wanting to A/B test everything that they spend more time building and maintaining the experiment framework than actually shipping more product and then realizing the A/B testing was useless because they only had a few hundred data points. Dat…
Most A/B tests should not be done in a production like way. Grab some post-it notes and and sketch out both A and B: then watch users work with it. For a lot of what you want to know the above will give better information than a 100% polished A/B test in production. When people see a polished product they won't give you the same type of feedback as they will for an obvious quick sketch. The UI industry has gone wrong…
Design-time user testing has been a thing for much longer than A/B tests. They are a different thing.
I mean, your point stands. But you can't do A/B tests on anything that is not production, those your are recommending are a different kind of tests.
I know this is not related to the article, which is great, but I am wondering how long "posthog" is going to be the name of this company given what "post hog" means.
This list mentions A/B testing a few times and it's worth noting that A/B testing is great but it's not free. I've seen a nontrivial number of smart engineers get bogged down in wanting to A/B test everything that they spend more time building and maintaining the experiment framework than actually shipping more product and then realizing the A/B testing was useless because they only had a few hundred data points. Dat…
I think A/B testing is one of the most expensive ways of getting feedback on a product feature. - You have to make good decisions about what you're going to test - You have to build the feature twice - You have to establish a statistically robust tracking mechanism. Using a vendor helps here, but you still need to correctly integrate with them. - You have to test both versions of the feature AND the tracking and test…
Steve Blank's quote about validating assumptions: "Lean was designed to inform the founders’ vision while they operated frugally at speed. It was not built as a focus group for consensus for those without deep convictions"
Is the Lean Startup Dead? (2018) https://medium.com/@sgblank/is-the-lean-startup-dead-71e0517...
Discussed on HN at the time: https://news.ycombinator.com/item?id=17917479
Is this like a trial day where you're invited to do a day of work for free?
This list mentions A/B testing a few times and it's worth noting that A/B testing is great but it's not free. I've seen a nontrivial number of smart engineers get bogged down in wanting to A/B test everything that they spend more time building and maintaining the experiment framework than actually shipping more product and then realizing the A/B testing was useless because they only had a few hundred data points. Dat…
I think A/B testing is one of the most expensive ways of getting feedback on a product feature. - You have to make good decisions about what you're going to test - You have to build the feature twice - You have to establish a statistically robust tracking mechanism. Using a vendor helps here, but you still need to correctly integrate with them. - You have to test both versions of the feature AND the tracking and test…
Why though? Can't you have it dynamically look up whether the experiment is active for the current request and if so behave a certain way? And the place it looks up from can be updated however?
I know this is not related to the article, which is great, but I am wondering how long "posthog" is going to be the name of this company given what "post hog" means.
In the first image in the article, what is a "SuperDay"? Is this like a trial day where you're invited to do a day of work for free?
Story time. I interviewed for a job at posthog. I knew that I really loved their communication style. But I hadn't used their product and didn't know a ton about them except that their writing is fantastic.
The 'product for engineers' focus that they is cool but when I had an interview, it was clear that I wasn't a 'product for engineering' person.
When they proposed the Super Day. I was like, I'm not sure because it's awesome to get paid for a day, but it's also not an unstressful event. And I sort of said I had some more questions before we moved on to the Super Day.
And they basically just said: we don't think it's going to work out. It was actually a pretty positive experience. I think that they correctly assessed the situation pretty quickly and my hesitation was a real signal to them.
(This is my recollection - I could have the details wrong.)
But yeah, super day is a day of very structured work in the role that they setup. And its paid.