After running an A/B split test, when the results are in, you always have to make the decision:
Do I switch to the new version or not?
If the difference in conversion is large, the decision is easy: just switch to the best version.
If the difference is small, the decision may be harder, but then again, which way you go doesn’t matter much anyway.
However, I have just done an A/B test where the difference in results is large, but still it’s not clear whether I should stick with version A (my control), or switch to version B (my challenger). Here’s my test:
Sign-up for the trial boxes
Our product pages have a “sign-up for the trial” box in the side-panel on the right and an additional big sign-up box in the content area, a little way down the page. The big extra one doesn’t look too great and it kind of disturbs the “flow” of the content.
So I wondered whether removing that extra box would make a difference. The test:
- Version A: “Sign-up for the trial” boxes in both the side panel on the right and in the wide content panel.
- Version B: Removed the big sign-up box, so just a “sign-up for the trial” box in the side panel remains.
These are the numbers for version B over version A, after 14 days.
In other words, removing the big additional sign-up box caused:
- Sign Ups: 15.6% less
- Sales: 1.5% more
- Average Purchase: 5.9% higher
- Total Profits: 8.0% higher
As expected, removing the extra sign-up box caused a decrease in the number of sign-ups. (Though I did not expect the numbers to be this dramatic.)
However, profits are up and considerably so. Not sure why. Maybe it’s the better looking and better flowing content story? Maybe making the trial less prominent causes an increase in the number of immediate purchases?
So now what?
Do I opt for version B and thus remove the extra sign-up box? More profits is good right?
Or should I go for more trial users and I leave my site as is (A), with 2 sign-up boxes?
What do you think? What would you do?