You are running an A/B test (or multivariate test) but are in a hurry to make a decision to pick the winning page. Should you pick a winner based on initial few days of the data before your tool has actually declared a clear winner? This question comes up quite often during the conversations with the clients.
I always warn against such an approach and advice to be patient and get statistically significant results before pulling a plug on an underperforming variation o declaring a winner. I wanted to share few graphs with you to illustrate my point.
The following example is from a test that I am currently running.
First Two Weeks: If you make a decision without completely getting through the test, you will pick “Yellow” as the winner while declaring “Blue” a loser.
Month Later: There does not seem to be a clear winner.
Few More Days Later: Seems like “Blue” is trending higher. Should you pick “Blue” now?
Finally: We don’t have a clear winner yet.
As you can see, picking a winner or dropping a loser in the early stages of test, without having a statistical significant result, would have been a wrong decision.
Looking to fill your Web Analytics or Online Marketing position?
Post your open jobs on Web Analytics Job Board