Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Launch HN: GrowthBook (YC W22) – Open-source feature flagging and A/B testing

growthbook.io

Post reply on HN
Print view View original

41–47 of 47 posts

Re: Launch HN: GrowthBook (YC W22) – Open-source feature flagging and A/B testing

#41

Post by alexdean »

jrdorn wrote: post #23
joecot wrote: post #16

This looks great, but if I can make a suggestion, a guide in your docs on how to use this with Matomo[1] would go far. Growthbook provides Feature Flag and A/B Testing, but you have to bring your own analytics. Matomo provides those analytics and is also open source and self hosted, and would be an excellent match. There's been an issue in about integrating since September[2], and having a guide with SQL examples for…

Thanks for the suggestion. We added MySQL support so we can work with Matomo now, but we're definitely lacking in documentation and tutorials. There's a similar issue with people using GA4, Snowplow, Segment, and anything else with a pre-defined schema. It should be way easier to use these systems with GrowthBook. Like there's no reason to even write SQL most of the time if we know the Matomo table structure in MySQL…

Very cool. Snowplow CEO here - if you need any help on integrating with Snowplow just ask in our forums: https://discourse.snowplowanalytics.com/

Re: Launch HN: GrowthBook (YC W22) – Open-source feature flagging and A/B testing

#42

Post by eduardo_olimpio »

This looks awesome. I have previously struggled on developing an internal tool for A/B testing and that gets complex really fast. At Miggos we have implemented Growth book and we got FF and A/B in our Vue frontend in a few hours. Excited to see the progress of the platform.

Re: Launch HN: GrowthBook (YC W22) – Open-source feature flagging and A/B testing

#44

Post by jrdorn »

wdb wrote: post #43

Looks promising! Personally, I am a happy user of PostHog (gitHub.com/posthog/posthog) which also supports experimentation and feature flags alongside product analytics.

We're big fans of PostHog ourselves. Any tool that encourages people to use data to make product decisions is a huge win in our book.

Re: Launch HN: GrowthBook (YC W22) – Open-source feature flagging and A/B testing

#45

Post by XCSme »

> Can I run multiple A/B tests at a time? Yes! In fact, we recommend running many experiments in parallel in your application

Isn't this a bad practice in general? Not only you split your (potentially limited) sample count, but it is very unlikely that there is ZERO interaction between A/B tests. Even if you test something on one page and another thing on a different page, there might be some interaction between the two changes.

Re: Launch HN: GrowthBook (YC W22) – Open-source feature flagging and A/B testing

#46

Post by jrdorn »

XCSme wrote: post #45

> Can I run multiple A/B tests at a time? Yes! In fact, we recommend running many experiments in parallel in your application Isn't this a bad practice in general? Not only you split your (potentially limited) sample count, but it is very unlikely that there is ZERO interaction between A/B tests. Even if you test something on one page and another thing on a different page, there might be some interaction between the…

Most interactions between tests are really minor and indistinguishable from random noise. Those ones aren't worth worrying about.

There are two ways to deal with the bigger interaction effects. First is to predict which experiments will meaningfully interact ahead of time and split the samples between them (making the tests take longer). Second is to run tests in parallel on all users and look at the data to determine interaction effects after the fact (and potentially need to invalidate some results).

In our experience, a mix of these approaches works best. It's really hard to predict meaningful interaction effects ahead of time, so save that for the really obvious cases (e.g. black text on a black background). For everything else, the benefit of running more experiments usually outweighs the cost of occasionally needing to throw out results because of interaction effects. It's much better to run 10 tests and need to throw out 1 than it is to run 5 tests.

Re: Launch HN: GrowthBook (YC W22) – Open-source feature flagging and A/B testing

#47

Post by vijayer »

chambers wrote: post #40
vijayer wrote: post #39

Congratulations on the launch, guys. Over at Statsig we have tremendous respect for what you have been doing.

Hey Vijaye, will Statsig support diff-in-diff / SEO Experiments in the future? I saw that you got Multi-armed bandit out the door and I'm hoping to see more kinds of experiment pop up on your platform.

No post body was provided.
Post reply on HN