Twenty Milliseconds Logo

This is Twenty Milliseconds, a site documenting what works and what doesn't in virtual reality design.

What Determines a Good Oculus Share Rating?

Summary: High ratings, high comfort levels, and high download numbers go hand in hand. At the high end of PC performance, frame rates don't affect application rating.

I was curious (and still am) about the determinants of a good rating in the Oculus Share app store. In some cases I have read people claim that “this game is good, people just need to ‘get their VR legs’” (eg, stop being sick) and I want to see if there’s any truth to that.

Let’s see how ratings, download rates, and comfort levels are correlated. Are higher rated games downloaded more often? I ignored apps with less than five ratings for this comparison.

Note: For both ratings and comfort level, Oculus provides the total score (say, 43) and the total number of ratings (say, 13) and you can divide these to get the rating presented on the site (in this example, 3.3). This is not the best way to get ratings and they should really be using a Bernoulli confidence interval instead.

Note 2: I have tried to be careful not to confuse a correlation (two variables move together) with one variable causing another one. There are some implicit assumptions, though, in the positioning of the X and Y variables. Be cautious.

Rating vs. Total Number of Downloads

It looks like games with higher ratings have more downloads on average. It’s important to note there are some complexities at play - just like all Yelp ratings converge to 4 eventually, having more ratings might hurt your score.

Let’s look now at the comfort level. I would predict that applications with a higher comfort level would have both higher ratings and more downloads.

Comfort Level vs. Rating

Yes! There is a pretty clear trend here; applications with higher comfort levels have higher ratings.

Do applications with higher comfort ratings get more downloads?

Comfort Level vs. Number of Downloads

It looks like they do, though this may be driven by a few high download games having high ratings. The absence of any applications with a low comfort level and lots of downloads is noticeable.

How does an application’s frame rate affect its rating?

I spoke with an individual recently who said ~80% of his users were feeling sick within two minutes, and his application had a framerate of 24 frames/second. So I wanted to test the theory that low frame rate would hurt an application’s rating. I used the frame rate data from eVRydayVR’s spreadsheet, and plotted it against the rating each application received in the store. I used the following scripts to download and aggregate everything.

I set the maximum frame rate to 200 for this comparison, because the highest recorded frame rate detectable by the human eye is 220 frames/s, and I would not expect a frame rate higher than that to affect the rating in a noticeable way.

Here are the results; there’s no real correlation. Hover over a game to see a datapoint. There are a few caveats here.

Correlation between frame rate and Oculus Share Rating

Okay, bummer. Let’s look at some other data. Is framerate associated with comfort level? I “normalized” the framerates higher than 200 to just equal 200.

Correlation between frame rate and Comfort Level

A quick least squares analysis shows there is no correlation between DK1 framerate and reported comfort level.

Conclusion

What have we learned?

The code I used to generate this is available here, and in the source code of this page. Send any feedback you have to kevin@twentymilliseconds.com.