A/B Test results: Trying different homepage heroes got us 2.6x sign-ups • Hacker News Board

Post by Salim99 » Mon, Jul 13, 2026, 3:43 PM UTC

Last year, we rebranded forms.app, a form builder app. With the rebranding, we had an experimental hero section that we quite liked. It was an AI generator; people write what type of form they like, and even before they sign in, they get it.

I'd have liked to include all images here, but you can check them out here: https://salimdin.substack.com/p/how-an-ab-test-more-than-doubled-our-signups-on-formsapp

After a few months, we realized something was off. We thought it was about rankings, but after a few discussions, we decided to test our homepage hero section against two alternatives.

- Variant A: The AI generator version

- Variant B: Only sign-up buttons (nothing else)

- Variant C: A single button for the app itself (playground without sign-up)

We ran the experiment for 2 weeks and only for new visitors. The results were interesting, to say the least:

- Variant A came last on every single metric

- B got 2.6x the sign-ups of A

- C got fewer sign-ups than B, but those who did sign up activated more; so it's a more distilled result.

Seeing these results, we decided to move forward with Version B, as it has more promise, and agreed that B and C deserve their own testing.

Recently, we changed our pricing, and with it, Version B got an update. Soon, the C version will be updated too, and we'll pit them against each other. I'll be sharing the results when we run that A/B test.

What do you think? Are the results what you expected, or did they surprise you too?

It's not a promotion, just an article where I show the version and the results, but I couldn't submit with the URL (probably due to the Substack URL). So here it is: https://salimdin.substack.com/p/how-an-ab-test-more-than-doubled-our-signups-on-formsapp

Post by ticulatedspline » Mon, Jul 13, 2026, 5:10 PM UTC

Results don't seem that surprising.

What were your engagement numbers between A and C? I would expect similar numbers between A and C maybe more with C since AI stuff can be off-putting to some users.

sounds like A and C are entirely different UX so maybe people just didn't like the results of A compared to C.

Unsurprising that later engagement with C was higher since signup would be a deliberate response to seeing the product instead of "because I had to".

Also unsurprising that B has the most raw signups because looks like with that there's no meaningful way to engage with the product without doing so, given the low impedance of leveraging Oauth (lots of people are signed in all the time anyway) for some users signups are basically zero-care, would be interesting if the ratio of email signups to Oauth on option C were different than B since C is a deliberate choice.