Saturday, March 13, 2010

Does a game's platform influence its EDGE score?

Every time EDGE magazine reviews a Playstation 3 exclusive, they are accused of bias by PS3 supporters, who are then mocked as fanboys who aren't satisfied with anything other than perfect scores for their beloved exclusives. EDGE are known to be harsh graders, and some respect them for using the full 1-10 scale instead of using 7-10 scale like the mainstream gaming press. The counter argument is that EDGE reviews every game harshly and it's pointed out that for every PS3 exclusive that gets low scores like Heavy Rain, there is an equivalent 360 exclusive.

To investigate whether EDGE is just harsh overall, or if it truly judges and scores games based on their platform, I made a comparison chart of the top 30 games from each platform, as well as the top 30 multiplatform games.

To make such comparisons, we need an unbiased standard, so we can measure EDGE scores against it. While we don't live in a perfect world and there is no universal metric that tells how good a game is, the closest to such a standard is Metacritic. It is a weighted and normalized average of all sites that review a video game. Metacritic says that it assigns different weights to different critic sites as well as making sure that the outliers don't affect the average "metascore" too much.

For this analysis, I found the top 30 games for each platform and compared their metascore to the EDGE review score to find the difference. I've limited my analysis to the top 30 due to time constraints and because the top 30 gives a very broad analysis of exclusives available for each system while filtering out the real duds. The #30 exclusive games for both platforms have metascores in the low 70s.

The ground rules used for counting and determining exclusives were as follows:
  1. Only disc based games were counted, downloadable games, expansion packs, and DLC were not included. The most important reviews are for new release disc based games, since they influence the buying decisions greatly.
  2. Games that EDGE did not review were not counted, such as God of War collection on the PS3, MLB The Show series, other sports games, etc.
  3. A game that's on the PC and Xbox 360 is considered an exclusive.
  4. Games that came out on 2005 and early 2006, when there was no PS3, were counted multiplatform if they were also released on PS2, since it was the 360 competition for a year until the PS3 release. A such example is Call of Duty 2, which came out on 360 and PS2. Since Call of Duty has never been a 360 franchise, I felt this was the right thing to do. On the other hand, a game such as Dead Or Alive 4, was not on any Playstation platform, so it was counted as a 360 exclusive.
  5. When comparing the score of a multiplatform game, the version that EDGE reviewed was taken as the baseline. For example Bayonetta has a metascore of 90 for the 360 version and 86 for the PS3 version. Since EDGE reviewed the 360 version, the metascore was taken as 90.
  6. Metascores can change, albeit slightly, over time, so this list of the top 30 games is only accurate as of March 13, 2010.
So without further ado, let's start with the the EDGE vs. Metacritic comparison of the top 30 multiplatform games:

(Click on chart for bigger version)

As we can see from the chart, EDGE's reputation as a harsh critic is much deserved, with the EDGE scores (red) being lower than the metascores (blue) most of the time, although some exceptions like GTA4, The Orange Box and Bayonetta, where the EDGE score was a perfect 10.

Next, we compare the top 30 Playstation 3 exclusives' metascores to their EDGE scores:

(Click on chart for bigger version)

We can see that only LittleBigPlanet, Demons Souls and Disgaea 3 have exceeded their metascore on Edge, while Buzz Quiz TV matched it. Everything else scored significantly lower. EDGE is harsh indeed.

Finally, we compare the top 30 Xbox 360 exclusives' metascores to their EDGE scores:

(Click on chart for bigger version)

It's immediately apparent that many Xbox 360 exclusives received much higher scores from EDGE compared to multiplatform games or especially Playstation 3 exclusives. The blue area in the Xbox 360 graph is much less than the other two graphs.

For the final and the most crucial graph, I calculated the delta's (score differences) between EDGE and Metacritic for each game, and plotted a running average chart:


(Click on chart for bigger version)

We can see from the above chart that EDGE, scores 360 exclusives consistently about 4 points below their Metacritic average, scores multiplatform titles consistently about 8 points below their Metacritic average, and scores PS3 exclusives about 12 points below their Metacritic average. The top 10, 20 and 30 game score difference averages are plotted in the chart.
  1. Top 10 games: These are the cream of the crop for each platform. These are the Halos, Uncharteds, COD4's, GTA4's that sell millions of copies and systems. When comparing the top 10 scores, EDGE gives 360 exclusive games 3.7 points less than their Metacritic average, multiplatform games 6.9 points less than their Metacritic average, and PS3 exclusives a whopping 11 points below their Metacritic average.
  2. Top 20 games: When comparing the top 20 scores, EDGE gives 360 exclusive games 4.6 points less than their Metacritic average, multiplatform games 7.6 points less than their Metacritic average, and PS3 exclusives 11.2 points less than their Metacritic average.
  3. Top 30 games: This list includes games ranging from Mass Effect 2 to Folklore, and a broad range of scores. However, the overall picture remains the same as EDGE gets a little bit harsher to the "above average" games. When comparing the top 30 scores, EDGE gives 360 exclusive games 4.8 points less than their Metacritic average, multiplatform games 8.5 points less than their Metacritic average, and PS3 exclusives a whopping 12,5 points below their Metacritic average.
The consistency of the score differences is remarkable. Whether comparing the top 10,20 or 30 games, the 360 is always scored higher by EDGE, followed by multiplatforms, and followed by PS3 exclusives in the last place. The lines are relatively straight and flat, with no lines crossing each other or switching places after the top 10. If EDGE really scored games irrespective of their platforms, we'd have expected the three lines to cross each other and be all over the place, but this is not the case. There is a strong pattern of discrimination for 360 exclusives and against PS3 exclusives.

While people may have different opinions on a few games, and could indeed review them differently and get different scores, 30 games consistently scoring lower on the PS3 and higher on the 360 means only one thing:

EDGE is biased to give 360 exclusives higher scores and PS3 exclusives lower scores. Their scores should not be taken at face value, as the platform unfortunately influences a game's score on EDGE. EDGE magazine has well written reviews, but their journalistic integrity is questionable.

22 comments:

  1. Wow this is soo tru I always was like WTF when see what the give ps3games on metacritic when I saw them grade great exclusives...:(

    ReplyDelete
  2. now there is some science behind the truth :)

    ReplyDelete
  3. Thankyou for this most insightful article. Loved the analysis.

    ReplyDelete
  4. doesn't explain why they gave ff13 5/10 :)

    ReplyDelete
  5. This comment has been removed by a blog administrator.

    ReplyDelete
  6. havnt dun it in a while....but ..., can you say mmmm. yeeeh

    Motorstorm 3
    Twisted Metal 4
    Ace combat 7
    The mark of kri
    Starhawk
    SSX
    Uncharted 3
    Resistance 3
    Killzone 3
    God of war 3
    Gran turismo 5
    Grand theft auto 5
    syphon filter 5
    The last guardian
    Agent
    infamous 2

    mmmmmmm yeeeh

    and i dont care what games scores games get. i brought infamous, and most places gave that an 8,...would you by an 8 game....YES , of course you would, only gettng 9 and 10 games, is what edge want you to do, so they basically controll what the 'mases' are buying.

    ReplyDelete
  7. but people like me, with a particular taste in games, and wont follow masses..im getting all the games on that list, ALL sony exclusive (notice ssx is on there) ..

    then im getting, blazblue, ive got tekken 6.

    ive got motorstorm, i play god of war colection all last week. im looking forward to last guardian and an ico/collossus double pack . etc etc

    i dont come home, shoot and swear for 2 hours straight, down a 360 headset. if i want to kill people...lemme point you in the direction of quakelive.com

    where it al started, and what every shooter should be marked against.....that halo 3 dont look to quik anymore eh>? twitch gameplay original. suk it 360, its over for you, your just prolonging your death making it slow and painfull for every one.

    hurry up and introduce your 720, new level toaster, and let the battle commense. you have nothing on these games ive listed.


    NOTHING

    ReplyDelete
  8. This is so true, very nice work! By chance I noticed the basis for this earlier today when checking out who owns EDGE these days. I remembered how 'cool' they were back in the days and thought what magazine is the equivalent today.

    I found that Future US INC owns EDGE today and if you check their management team, you'll instantly see where this echoes from:
    http://www.futureus.com/about/management/

    ReplyDelete
  9. This comment has been removed by the author.

    ReplyDelete
  10. Let me just make this clear. Your statement that "EDGE is biased to give 360 exclusives higher scores and PS3 exclusives lower scores" is flat out false. All you proved is a correlation NOT a causation. Any number of other factors could be influencing the scoring data, causing it to skew one way or another. Hell, given the scoring criteria in EDGE, Sony tending to send in review copies early in the month(towards the end of their magazine cycle) would automatically result in a negative data skew because of the increased deadline stress and other variables related to publishing a magazine.

    Come back when you can prove causation.

    ReplyDelete
  11. i love it when academics start fighting. make it interesting for me, and state your titles.I am looking at you Alan.

    ReplyDelete
  12. You don't need all of these graphs to tell that EDGE are biased. If you've ever looked at an actual full issue of their magazines 90% of the ads in them are for 360 games.

    ReplyDelete
  13. @Reece

    Student at Carnegie Mellon University, majoring in Computer Science.

    Have taken statistics courses though...

    And everything I said about correlation not proving causation is entirely accurate. So there really isn't much substance to any sort of "fight" against me. I was just giving an example for one possible distracting variable that could skew the data and cause the data to show what it does. Honestly, I don't know why the scores tend to be lower on the PS3. But I would bet my money on it being something besides some sort of prejudice against the PS3 on the part of the EDGE editors.

    ReplyDelete
  14. I have a feeling Meta is also bias against the PS3.

    Look at their review listing for the Xbox 360 FFXIII -- 39 critic reviews, and they even excluded Edge's 5/10 rating. http://www.metacritic.com/games/platforms/xbox360/finalfantasy13

    Look at the PS3 side and you'll see many many more ratings. Someone must be handing out cash to the meta guys to limit the critic reviews so the Xbox FFXIII score doesn't fall. We all know the Xbox version is greatly inferior to the PS3 version, yet it even has a HIGHER meta score (84 vs 83)! Something is seriously wrong.

    ReplyDelete
  15. @angus

    They gave ff13 low score was because on XBOX 360, FF13 was 576p on graphics and on PS3 it was 1080p. So, because of the hate, they gave it a low score.

    Problem is, it's not Square Enix's fault, blame 360 for being a less-powerful hardware. But they just can't seem to accept that fact.

    ReplyDelete
  16. We know where the industry is heading because you can buy a score and influence the press to get a better review because they have to get a early review copies of games. It would be nice that they send me a review copy of Viagra Online

    ReplyDelete
  17. @AE twisted metal 4 came out on the original playstation the new twisted metal if your counting marks the 6th twisted metal in the series.Most video game sites base video game reviews by their own personal opinions from people that are not very video game design or technology savvy.

    ReplyDelete
  18. Edge are just whores that suck the highest payer (in this case Microsoft).

    Microsoft have got where they are this generation by paying the media.

    Think for a moment if it had been Sony that had 60% RROD failure rate, the press would be crucifying them, but when it's Microsoft it gets mostly swept under the carpet and downplayed.

    PS3 exclusives that get 7/10 would be 10/10 had they been Xbox exclusives.

    Games on Xbox have imperfections ignored, yet games like GT5 are put under a microscope and down pointed over crazy things like how long it takes to initially install.

    American Console, American press. That is the theme of this generation, the few exceptions of blogs that aren't American just get nice big fat advertising budgets (read bribes) from Microsoft to keep them sweet.

    It's all about buying your way forward with Microsoft. Take a look at the 150,000 Xbox Slims that were given away to journalists at E3 as sweeteners.

    http://www.usanewsmonitor.com/news/Microsoft-Gifts-Free-Xbox-360-During-E3-Press-Meet---Bribe--1276630399/

    ReplyDelete
  19. We've already known for a long time that EDGE magazine has a deep-seeded hatred towards the PS3 and Sony platforms. Now we have some statistical evidence to back those thoughts up.

    ReplyDelete
  20. Please can you refresh the graphs on the hosting site or host them somewhere else =it looks like the hosting site deleted them now - all I see are exclamation marks. This is a great article.

    ReplyDelete