Achieve High Player Retention And Conversion Rates With A/B Testing
A/B testing is ranked as one of the most effective techniques for conversion rate optimization (Invesp). In addition to increasing conversions, it is commonly used for improving player retention and avoiding bad game feature releases. However, this technique is only effective if you know how to plan and execute your a/b test suite to achieve your set goals. In this article, I will explain what A/B testing is and layout a general template that you can use and tweak, to effectively apply this technique. If you would like to learn more about the benefits of A/B Testing, you can have a look at How To Increase In-App Purchases.
1. Set Your Goal
First, you must set a measurable goal that you want to achieve. For example, “We want to improve conversion rates for our coin-packs from 1% to 1.5%” or “we want to improve our Day 1 retention rate from 30% to 40%”. Having a goal will help you plan and focus your test suite to effectively optimize one metric. It is also important to make sure that the goal is measurable. This makes it significantly easier to perform A/B tests and make the right decisions.
2. Formulate Hypothesis
Once you have your goal, analyze your own game and come up with hypotheses that might improve your metric and bring you closer to your goal. For example, “Improving the UI/UX of our store will improve conversion rates“, or “Adding more ways to spend coins in-game will improve conversion rates for our coin-packs“, or “Decreasing difficulty level will improve player retention“.
When analyzing your game, you might collect data related to how many users are playing your game, which scene/part of the game drives the most traffic, what game mechanics incentivizes in-app purchases, what is the bounce rate of your store UI etc. When formulating your hypotheses, make sure they are supported by both qualitative and quantitative research.
Once you have your hypotheses, you can rank them based on how easy they are to implement and how much impact you think they will have. Hypotheses that are easier to implement and bring more value should be ranked higher.
3. Create Variations
Now you can start by verifying your highest-ranked hypotheses. You must first create a variation based on your hypothesis, and then A/B test it against your current version (also known as a control group). For example, if your hypothesis says “Decreasing difficulty level will improve player retention”, you must implement an easier version of your game and track how it performs against your current version. Tools such as Elecular make it very easy to implement and track variations for your game.
4. Analyze Performance
After releasing the variations to your players, you will start collecting data such as player retention, number of ad clicks, number of in-app purchases, etc for each variation. With this data, you will be able to determine which variation is performing the best. While it is important to pick the variation that is giving you the best results to achieve your goal, you must make sure not to harm other metrics significantly. For example, if your goal is to increase conversion rates, you must still make sure that the variations you pick are not significantly decreasing player retention.
In some cases, your new variation is going to perform worse than the current version. In these cases, you must learn and understand why your experiment failed. Maybe your hypothesis was incorrect, or your implementation was faulty, or your players behaved in an unexpected way. Either way, it is important to learn from these opportunities, document them, and consider them in your next A/B test runs.
5. Rinse And Repeat
After you finish running your A/B test, you will have some extra knowledge that you can use to reprioritize your list of hypotheses. Maybe, your A/B test was very successful and you would like to work on similar hypotheses immediately! Or you were not able to verify your hypothesis, so you would like to deprioritize similar ones. Either way, it is a good idea to take some time to revisit and update your plan based on what you learned.
Ideas For A/B Testing
Here are some ideas (as inspiration) you can consider while formulating your hypotheses.
Call To Action - If you have in-app purchases in your game, it might be a good idea to experiment with your call to action buttons. You can try out different button placement, colors, sizes, text, etc. and see what drives more conversions. You will be surprised how little changes like these can make a huge difference to your conversion rates!
Game Mechanics - You can consider introducing different mechanics that incentivizes in-app purchases. For example, you can implement a VIP system where the game rewards a player with VIP points every time he/she makes an IAP. The game will then offer various perks and benefits based on how many VIP points the player collects. These kinds of game mechanics incentivizes players to make more IAPs, but more importantly, you must keep an eye out for player retention and make sure it does not drop significantly.
Push Notifications - You can send notifications to inactive players and let them know that their beloved game is waiting for them. You can A/B test different messages and timings for your notifications to yield maximum player retention.
A/B Testing is a process of analyzing your game, identifying problems that are potentially leading to lower conversion rates, formulating a data-backed hypothesis to fix the problem, and verifying your hypothesis by creating and tracking variations. If you are using Unity to build your game, you can use Elecular to effortlessly run A/B tests to increase your retention and conversion rates.