Preliminary Booster Use Induction Experiment
Background of the experiment
Introduce the experiments that were conducted with the aim of enabling the purchase-use of the Ready Booster. Like the in-game booster experiment, it is an experiment to induce users to consume as much goods as possible through various events or mission rewards in the game. The key is to encourage use by marking the recommended badge on a particular staging booster to aid in level clear.
The UI of Candy Crush Friends Sagawa Cookie Jam was referenced in the experimental design. However, Candy Crush Friends does not recommend a preparation booster. To be exact, the game does not have a ready booster concept, but instead you can select characters with unique abilities. The same goes for cookie jam. Both games mark recommended badges on the character selection UI to encourage the use of characters to help you clear certain levels or missions. The cookie jam also provides a ready booster. Depending on the level design and mission characteristics, the scale-up and rocking animation is shown when a useful preparation booster is present. Unlike the recommended badge, it is clear that the purpose is to induce the use of a particular warm-up booster. However, there is a prerequisite that the user must have a recommended character or booster to see the character recommendation badge or use-inducing performance in both games.
Candy Crush Friends on the left and Cookie Jam on the right
Using these two reference methods to mark the recommended badge in the Ready Booster area rather than the character selection, we expected to increase the purchase and use of the Ready Booster.
An experimental hypothesis
The recommended badge on the ready booster to assist in clearing a particular level will increase the ready booster consumption per user at that level.
An experimental method
In the Match 3 genre live project, we experimented in two sets (Master/A). The level design and the supply of goods (coin and booster) were applied equally between each set as control variables. We tracked data for a total of 1,800 people for a week with about 900 DAUs per set without a separate marketing campaign. The independent variable was viewed as the amount of ready booster usage per user at the level marked with the ready booster recommendation badge and the dependent variable marked with the recommended badge.
1. Do not mark the ready booster recommended badge for the master set.
2. Set A will mark the recommended badge on the ready booster for each level.
3. Measure the number of unique users and the consumption of the ready booster at the level marked with the ready booster recommendation badge.
The experimental two-type game is a game that has been live for a long time, so the total number of levels is over 1,500. Therefore, it was difficult to identify the design characteristics and intentions of each level and to qualitatively classify boosters to help clear. As an alternative, live game data was extracted to analyze the usage of the prep booster for each level, and based on the analyzed data, the recommended badge was marked only for that level by classifying the level of use of the prep booster. As a result, the level marked with the warm-up booster recommendation badge was less than the overall level, and the level marked with the recommended badge and the booster usage were measured for virus control in the experiment.
Sample design for recommended badges used in the experiment
The results of an experiment
* For security reasons, we do not disclose detailed experimental designs and data.
The ready booster consumption per user in Set A marked with recommended badge on the ready booster was about 6% higher than the Master set marked with nothing. In order to increase the reliability of the results, the re-experimental results from other Match 3 live projects were equally high in Set A. The re-experiment showed that consumption was about twice as high, but the user parameters were somewhat insufficient compared to the first experiment). So the hypothesis was adopted. And what’s noteworthy about this result is that despite controlling other factors, the average D1 retention in Set A for a week was 5-10% higher in both the first and second trials. Ready boosters can be purchased in the Ready pop-up, but they can usually be created in combination with blocks during in-game play. Therefore, the recommended badge may have served as a kind of clear help for users who have difficulty clearing levels, and may have provided a positive experience for users who actually cleared levels using recommended boosters [1]. The results of this experiment may be helpful in most mobile puzzle games that offer a warm-up booster. Or it can be helpful for other genres that need to be encouraged to recommend and use items in addition to puzzle games. The most important thing is to select and recommend a well-prepared booster that can help you clear the actual level.
Suggestion
The most regrettable part is that the number of levels marked with the recommended badge was a bit small (although it was selected as a really helpful booster instead). This is why we didn’t scale to the full level and track the data. It’s also a limitation that I’ve experimented with in a live game, so I’d like to verify it again in a new project if I have a chance. Oh, And if you have time someday, it would be good to use the recommended booster to separate only the user cohort that you cleared the actual level and look at the difference in retention with the control group again.
[1] In fact, there’s a reason to infer this possibility. Prior to the 1st and 2nd trials, pre-experiments were conducted for pilot purposes, where the recommended badges were not limited to specific booster-intensive levels, but rather to the recommended badges if any booster was slightly higher at most levels. As a result, the amount of ready booster usage per user increased, but the D1 retention dropped by more than 2-3%.
Source: https://brunch.co.kr/@jaehyunkim