Aug 18, 2022

Meta Quest Store A/B Testing: Improve Your Conversions and Generate More Revenue

Through A/B testing, the developers of FitXR⁠, Synth Riders⁠, Vacation Simulator⁠, and Spacefolk City⁠ demonstrated that conversion rates—and app revenue—can be increased or validated through changes to product detail page (PDP) assets.

A/B Testing Soft-Launch Cohort Highlights:

FitXR (Free w/ Subscription)

+11.8% conversion rate

Cover art test

Synth Riders ($24.99 USD)

+26.5% conversion rate

Video trailer test

Vacation Simulator ($29.99 USD)

-5.5% conversion rate

Cover art test

Spacefolk City ($24.99 USD)

-29.7% conversion rate

Video trailer test

At Reality Labs at Meta, we’re continuing to work on tools and features that help developers optimize their presence on the Meta Quest Platform. We want to provide as much insight as possible into how and why people choose to purchase or download your app. That’s why we created the A/B Testing Tool in the Developer Dashboard⁠. A/B testing can be an effective method for increasing the total conversions of your app. It works by publishing two versions of an app’s product detail page (PDP) and comparing the conversions between them. Whether your app is new to the Meta Quest Store or a longtime fan favorite, A/B testing can help you understand the impact of your PDP assets and improve conversion—while driving significant increases in revenue.

The A/B Testing Tool currently lets you test across three PDP variables that are seen by audiences in the Meta Quest Store:

Cover Art

Square: 1:1 static image that’s displayed across platforms to market your app on the Store.

Landscape: 16:9 static image that’s displayed across platforms to market your app on the Store.

Descriptions

Short Description: 500-character limit description that’s shown throughout the Platform, primarily in VR.

Long Description: 1,000-character limit description that’s shown throughout the Platform, including on the Meta Quest Store

Video Trailer

The video that displays at the top of your app’s PDP.

The thumbnail image that previews your video trailer.

A/B Tests: A Win-Win Scenario

Our research suggests that people lack familiarity with many of the apps and games available on the Meta Quest Store. This makes your Store assets all the more important in order to both entice audiences and quickly build familiarity. Choosing the “perfect” asset can feel like a high-stakes process. Through A/B testing, you can gain clarity about how your assets affect PDP traffic and conversions—no matter the outcome.

Kluge Interactive⁠’s video trailer test for Synth Riders⁠ resulted in a 26.5% increase in conversion rate (and thus game revenue) and featured noticeable changes. The test trailer includes a new, jazzy song choice and highlights gameplay from a third-person perspective of someone immersed within the app environment, whereas the original trailer featured music with classical instrumentation and heavily emphasized first-person POV gameplay.

Original video trailer for Synth Riders.

Test video trailer for Synth Riders.

The value of A/B testing extends beyond recently-launched apps. As shown by Kluge Interactive’s A/B test for Synth Riders, which was released in 2019, fresh assets can give older apps new life by capturing audiences’ attention and subsequently spurring engagement. This improvement in engagement may boost conversion and can lead to significant revenue increases, even for apps that have been out on the Meta Quest Store for years. A/B testing is a relatively quick and easy way to grow your app’s audience while focusing on your next project. As the VR ecosystem expands, A/B tests also help improve the visibility of your app.

A/B testing also provides value regardless of the outcome. For Beyond Frames Entertainment⁠, an A/B test for Spacefolk City⁠ indicated that people didn’t respond as well to the new trailer they were testing. In its test, Beyond Frames Entertainment shortened the Spacefolk City trailer by roughly 30 seconds. It also featured new and more frequent messaging, choosing to dive deeper into the expected player journey, from building cities and keeping “folk” happy to challenging 3D thinking and, finally, saving “folk.”

Original video trailer for Spacefolk City.

Test video trailer for Spacefolk City.

The test resulted in a 29.7% decrease in conversion rate when compared to the original asset. This result helped prevent potential losses in revenue that would occur if the test trailer had been published without prior A/B testing. It also suggests that gameplay is preferred by audiences over messaging in trailers.

There’s a lot that can be inferred from a false hypothesis, but more importantly, a less effective test asset validates your existing strategy and opens the door for new hypotheses and iteration. While a video trailer test that results in a decrease in conversion rate could indicate that your current trailer is effective, trying a new approach for a trailer or shifting gears to test descriptions or cover art could help determine what specific factors are driving people towards purchasing and downloading your app.

We recommend diving deeper into positive test results to help inform your next hypothesis, but in the meantime, the A/B Testing Tool can help you immediately capitalize on the results from your own tests by enabling you to publish your test asset to the Store—directly from the Developer Dashboard⁠.

Trustworthy Results to Inform Your Strategy

Results are only valuable if they’re accurate. We’ve built the A/B Testing Tool to validate statistically significant results from your test so that you can feel confident about your results. Statistical significance is achieved only when your assets accumulate enough traffic, and the conversion rate between your two assets maintains an error rate of less than 5%, or a 95% confidence interval. To help ensure that your results can be attributed to your assets and not external factors, currently only one type of PDP variable can be tested at any given time.

A/B tests run for a maximum of 30 days, but they generate statistically significant conversion rate results for entitlements in as little as seven days. The Results tab of your test also shows metrics that include reach, unique clicks, conversion rate, and unique conversions. Reviewing these metrics can help indicate what assets may be the most valuable to test.

For instance, your cover art is typically the first asset people see in the Meta Quest Store. If your reach is low, it may be valuable to test new cover art and see if it leads to more visibility and visits to your PDP. If your reach is high but your conversion is low, running A/B tests on either your video trailer or descriptions can help determine which assets are deterring people from purchasing and/or downloading your app.

Hypothesize, Test, Reflect: Rinse & Repeat

To reduce the time it takes for your test to achieve statistical significance, we recommend creating test assets that are noticeably different from your original. With this being said, you don’t need to abandon your creative direction—even subtle differences can lead to big changes in conversion.

Case-in-point is FitXR⁠, which achieved an 11.9% increase in conversion rate for its self-titled app⁠ by removing “TRY FREE FOR 7 DAYS” from its Cover Art. Understanding your assets’ impact on people’s behavior is a never-ending process, and sometimes choosing to start off small can be equally as effective as overhauling your entire asset. Adjusting just one or two variables within your assets can help clarify exactly which factors are responsible for variations in conversion rate.

On the other hand, one of the advantages of A/B testing is that it lets you experiment with bolder creative adjustments while minimizing risk. In its cover art test asset for Vacation Simulator⁠, Owlchemy Labs⁠ maintained the app’s signature logo but drastically changed the overall color scheme, background environment, and objects within the image. Owlchemy Labs also experimented with its messaging, choosing to highlight a DLC in the test asset versus a product update in the original.

These changes led to a 5.5% decrease in conversion rate, but A/B testing allowed Owlchemy Labs to try something new and learn from it.

Depending on your goals, you can choose to show your test asset to 10%, 25%, or 50% of audiences in the Meta Quest Store, enabling you to gather crucial insights while mitigating negative outcomes.

To gain the most value from your test, use your results to update your hypotheses and be as specific as possible. A lot could be inferred from Owlchemy Labs’ results—they could suggest that audiences prefer the aesthetic of the original, or maybe audiences feel confident in purchasing the app thanks to the highlighted hand-tracking update, or maybe the word “job” in the test asset deterred people from purchasing the app. These questions could be investigated with further testing.

As Cubicle Ninjas⁠, the creator of Guided Meditation VR⁠ put it: “The tool is designed to be simple and easy to use, while delivering powerful results. It will help a great deal in maximizing the appeal of our apps.”

Start A/B Testing

Ready to create an A/B test and start boosting your asset effectiveness on the Meta Quest Store? Check out our A/B Testing documentation to get started, and read through our best practices to form a strategy that encourages significant results. To help ensure your test assets are approved and appeal to your target audience, we recommend reviewing our Meta Quest Store Asset Design Guidelines before creating your A/B test.

We’ll continue to enhance the A/B Testing Tool to include more features and metrics—but in the meantime, it’s time to form your hypothesis, gather your test assets, and begin gaining deeper insights.

*This data is based off of A/B tests run independently for each title:

FitXR from 2022-05-12 to 2022-05-27

Synth Riders from 2022-05-19 to 2022-06-03

Vacation Simulator from 2022-05-17 to 2022-06-16

Spacefolk City from 2022-05-17 to 2022-06-07

The tests were run using the A/B Testing Tool available in the Developer Dashboard. The findings reported reflect A/B test results where 50% of Meta Quest Store visitors saw the original asset and the other 50% saw the test asset when browsing the Store or viewing the product detail page (PDP) for the app. All data referenced is statistically significant.

Design

Entertainment

Marketing

Quest

Did you find this page helpful?

Explore more