What Determines a Good Oculus Share Rating?
August 11, 2014I was curious (and still am) about the determinants of a good rating in the Oculus Share app store. In some cases I have read people claim that “this game is good, people just need to ‘get their VR legs’” (eg, stop being sick) and I want to see if there’s any truth to that.
Let’s see how ratings, download rates, and comfort levels are correlated. Are higher rated games downloaded more often? I ignored apps with less than five ratings for this comparison.
Note: For both ratings and comfort level, Oculus provides the total score (say, 43) and the total number of ratings (say, 13) and you can divide these to get the rating presented on the site (in this example, 3.3). This is not the best way to get ratings and they should really be using a Bernoulli confidence interval instead.
Note 2: I have tried to be careful not to confuse a correlation (two variables move together) with one variable causing another one. There are some implicit assumptions, though, in the positioning of the X and Y variables. Be cautious.
Rating vs. Total Number of Downloads
It looks like games with higher ratings have more downloads on average. It’s important to note there are some complexities at play - just like all Yelp ratings converge to 4 eventually, having more ratings might hurt your score.
Let’s look now at the comfort level. I would predict that applications with a higher comfort level would have both higher ratings and more downloads.
Comfort Level vs. Rating
Yes! There is a pretty clear trend here; applications with higher comfort levels have higher ratings.
Do applications with higher comfort ratings get more downloads?
Comfort Level vs. Number of Downloads
It looks like they do, though this may be driven by a few high download games having high ratings. The absence of any applications with a low comfort level and lots of downloads is noticeable.
How does an application’s frame rate affect its rating?
I spoke with an individual recently who said ~80% of his users were feeling sick within two minutes, and his application had a framerate of 24 frames/second. So I wanted to test the theory that low frame rate would hurt an application’s rating. I used the frame rate data from eVRydayVR’s spreadsheet, and plotted it against the rating each application received in the store. I used the following scripts to download and aggregate everything.
I set the maximum frame rate to 200 for this comparison, because the highest recorded frame rate detectable by the human eye is 220 frames/s, and I would not expect a frame rate higher than that to affect the rating in a noticeable way.
Here are the results; there’s no real correlation. Hover over a game to see a datapoint. There are a few caveats here.
eVRydayVR tested with the Development Kit 1. It would be informative to re-run the tests with a Development Kit 2.
eVRydayVR tested a subset of games, and it’s possible the selection criteria was biased in some way that kills the correlation.
eVRydayVR has pretty good hardware and gets pretty good framerates on the games he tests, even maxing out the polygons etc. Maybe the effect I am looking for is only visible at framerates below 50 frames/s or so, so we’re splitting hairs on the high end of the software scene.The decrease in framerate with worse hardware may not be linear, and so multiplying every one of the numbers here by, say, 0.6, wouldn’t predict game performance on worse hardware.
Correlation between frame rate and Oculus Share Rating
Okay, bummer. Let’s look at some other data. Is framerate associated with comfort level? I “normalized” the framerates higher than 200 to just equal 200.
Correlation between frame rate and Comfort Level
A quick least squares analysis shows there is no correlation between DK1 framerate and reported comfort level.
Conclusion
What have we learned?
High ratings, high comfort levels, and high downloads go together. It’s not clear from this analysis which one of these causes the others. Though the absence of any applications with low comfort levels and high numbers of downloads is suspicious.
At least at the high end, frame rates aren’t correlated with application rating. Though this needs more investigation, see the caveats above.
The code I used to generate this is available here, and in the source code of this page. Send any feedback you have to kevin@twentymilliseconds.com.
