How do we prove it’s effective?
I believe the ultimate testament to the quality of your product is results. At 1Sale, we saw results when we put the idea behind our App Download plugin to work. Literally thousands more downloads. But that was a little bit different than UserPath’s download plugin, so in order to prove UserPath is effective, we ate our own dog food. However, we don’t have any of our own apps. So, we enlisted the help of a friend. This blog post goes over the results of our first A/B test, using Teepy – a Curated T-Shirt Shop for Amazon. It’s a way to collect, buy, and gift tees for any style or occasion.
We’ll be posting the results of more A/B tests for both App Download and Click-to-call as the results reach statistical significance, because what good are the results of an experiment if they are not reproducible?
The question for our experiment is pretty straight forward – Are people who visit a site for a mobile app on the desktop empowered to download the app? After a bit of looking around, it seems that most of the time, the answer is “No”. However, our previous experience and some anecdata lead us to a hypothesis:
A form that allows someone to send themselves a link to an app from the desktop will increase download intent.
So what exactly do we consider “download intent”? For the purposes of our experiment:
- If there is not a way to text a download link to yourself – then the click of an app store button is considered download intent.
- If there is a way to text a download link to yourself – then completion of a form that texts you a download link is considered download intent on desktop.
Now, let’s look at our A/B test. First, let’s examine our control, which we named “No UserPath”. It’s just the Teepy.co website, without the App Download plugin installed.
Next, let’s see what our independent variable looks like, we named it “With UserPath”. It’s a box that slides in from the right on a 1 second timer with a some text that tells the user they can download the Teepy app by filling out the form.
We ended up running the experiment for 2 weeks and 1 day. We waited that long because we wanted statistically significant results as determined by Optimizely, the A/B testing platform we used.
We believe the results speak for themselves. Teepy was able to achieve an 84.4% increase in download intent with UserPath’s App Download plugin. Additionally, engagement increased 28.4% with UserPath installed.
At the bottom of the funnel, they actually increased downloads 33% when compared to the 15 day time period before we installed UserPath. We asked the co-founder of Teepy, Alex Lopez why Teepy downloads increased so dramatically and this is what he told us: “It was definitely UserPath because I did not do anything else.”
I will be running more experiments in the upcoming weeks & publishing the results here, whether they bode well for UserPath or not. I believe that being honest will help us improve our product & give customers a clear look as to whether or not UserPath would be useful given their scenario.