Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Birdy – Twitter Profile A/B Testing

birdy.so

Post reply on HN
Print view View original

51–59 of 59 posts

Re: Show HN: Birdy – Twitter Profile A/B Testing

#51

Post by maximedupre »

laweijfmvo wrote: post #48
shay_ker wrote: post #20

If I’m understanding correctly, this switches your profile on some interval between A and B, whereas a proper A/B test will randomly bucket a user to experiment A or B. Not that it matters - this solution is probably the right way to go without building something into Twitter itself - but the more data/stats oriented folks may be confused or irked by calling this “A/B testing”

Why is bucketing users the better approach? To me it seems like bucketing would ignore "all other factors" that also changed, whereas dynamic (or periodic) switching seems like it would normalize those "other factors" across both A/B (ideally).

I suppose that the issue is that a single user could see both versions, which could skew the data.

I'm not sure I totally understand why, because if a user follows you after seeing the other profile version, it might be because he preferred this other version.

But conceptually it makes sense to eliminate as many variables as you can to isolate the components of the test.

Re: Show HN: Birdy – Twitter Profile A/B Testing

#52

Post by __mharrison__ »

WA wrote: post #24
maximedupre wrote: post #2

I built Birdy to help you optimize your Twitter profile with continual automated A/B testing! Why? More Twitter followers. More website clicks. I'm a huge fan of Twitter since I discovered the indie makers community. I wanted to create a product around this passion of mine, so I started experimenting with the Twitter API and I eventually landed on profile A/B testing as a cool problem to solve :).

What's a good entry to the indie maker community on Twitter?

I have almost 90K followers on Twitter and have a few ideas about how to grow your following.

- Interact with others (w/ high followers) in your niche

- Produce interesting content in your niche

- Consistency

I've created a 30 day course to walk you through these steps:

https://store.metasnake.com/twitter-for-developers

(It is currently part of a Python Humble Bundle where you can get it a little cheaper.)

Re: Show HN: Birdy – Twitter Profile A/B Testing

#53

Post by maximedupre »

sreejithr wrote: post #49

Don't want to rain on your parade but this is in no way a scientific A/B test. If I understand this correctly, Birdy periodically switches profile information b/n A and B. * How do you make sure the same user who was on A variant yesterday doesn't get the B variant today? If the same user gets both variants over different periods of time, how can you say treatment group engaged more with your content. People shuffle…

You understand Birdy's mechanism correctly :D

> How do you make sure the same user who was on A variant yesterday doesn't get the B variant today?

You can't. You make the point that the stats generated by Birdy are not 100% accurate, but they're the best we've got given the limitations of the Twitter API :D

I make the point that the data is good enough, and statistically significant enough (especially if you let the test run enough) to help you find the better profile version.

I agree that this subset of visitors that may come twice or more can skew the data.

Birdy's stats are not to be taken as the absolute truth, but as solid clues as to which profile version it is alternating between is providing the best results.

I'm looking forward to improving Birdy as Twitter adds more capabilities to their API!

Re: Show HN: Birdy – Twitter Profile A/B Testing

#54

Post by daniel-cussen »

maximedupre wrote: post #22
shay_ker wrote: post #20

If I’m understanding correctly, this switches your profile on some interval between A and B, whereas a proper A/B test will randomly bucket a user to experiment A or B. Not that it matters - this solution is probably the right way to go without building something into Twitter itself - but the more data/stats oriented folks may be confused or irked by calling this “A/B testing”

That is correct - it switched your profile version at a regular interval. Indeed, that's the only way I could do it. The Twitter API has its limitations :D I didn't know there was a specific definition of A/B testing. I'll see if I get more complaints about the terms I use ^^. To me, that's still A/B testing - that is I'm testing a version A and a version B and then report on which one does better. I guess the way I'…

That's cool man apologize for what is for you to apologize, and nothing else. Nothing else. And do your thing in the semblance of your vision of your ideas, all the way and show the world. And the world won't like it.

You gotta be sure of what you know. Be sure and be wrong, better than always being unsure.

Re: Show HN: Birdy – Twitter Profile A/B Testing

#55

Post by maximedupre »

daniel-cussen wrote: post #54
maximedupre wrote: post #22

Earlier quoted context omitted.

That is correct - it switched your profile version at a regular interval. Indeed, that's the only way I could do it. The Twitter API has its limitations :D I didn't know there was a specific definition of A/B testing. I'll see if I get more complaints about the terms I use ^^. To me, that's still A/B testing - that is I'm testing a version A and a version B and then report on which one does better. I guess the way I'…

That's cool man apologize for what is for you to apologize, and nothing else. Nothing else. And do your thing in the semblance of your vision of your ideas, all the way and show the world. And the world won't like it. You gotta be sure of what you know. Be sure and be wrong, better than always being unsure.

Thanks Daniel, I guess xD That was quite cryptic, but I think I get it haha. Cheers!

Re: Show HN: Birdy – Twitter Profile A/B Testing

#58

Post by shay_ker »

yeahboats wrote: post #32
maximedupre wrote: post #30

Earlier quoted context omitted.

Thank you! > I think the only downside to this approach is that there might be some effect based on time of day or the interval affecting your results. That is definitely true. I'm about to start alternating the versions every 5m to mitigate this. The closer I can get to 0m, the more accurate the results are. This way even if you get a followers spike (let's say you get a viral tweet), the followers will be properly…

I think what you really want to do is randomize A or B within each 30m (or 5m) interval. This is basically switchback testing. Door dash has a nice write-up here: https://doordash.engineering/2018/02/13/switchback-tests-and...

Yep - we've run many switchback tests so I'm happy to chat more about it. It's a lot more akin to what you're building here, from a stats point of view.

Re: Show HN: Birdy – Twitter Profile A/B Testing

#59

Post by shay_ker »

maximedupre wrote: post #22
shay_ker wrote: post #20

If I’m understanding correctly, this switches your profile on some interval between A and B, whereas a proper A/B test will randomly bucket a user to experiment A or B. Not that it matters - this solution is probably the right way to go without building something into Twitter itself - but the more data/stats oriented folks may be confused or irked by calling this “A/B testing”

That is correct - it switched your profile version at a regular interval. Indeed, that's the only way I could do it. The Twitter API has its limitations :D I didn't know there was a specific definition of A/B testing. I'll see if I get more complaints about the terms I use ^^. To me, that's still A/B testing - that is I'm testing a version A and a version B and then report on which one does better. I guess the way I'…

I hear ya - just giving the data/stats perspective. It's an audience that cares a lot about these details.
Post reply on HN