How Well Do Goalies Rank Goalies?

In the most recent edition of The Hockey News, a panel of former goaltenders was asked to rank the current crop in terms of their performance this season.

Here' s a link to the list.

Suffice it to say I disagree with several of the selections, since as per usual the guys with the better teams in front of them (Miller, Emery, Toskala, Fleury) were ranked ahead of other more deserving players (DiPietro, Lundqvist, Legace, Huet).

Based on anecdotal viewing of former goalie-turned-broadcasters like Glenn Healy and Greg Millen, it is likely that these rankings depended heavily on subjective viewing, as opposed to statistics. However, I attempted to find out what was impacting the former goalies most in their analysis. I ranked each of the goalies in terms of GAA, save percentage, wins, shutouts, games started, and shot-quality neutral save percentage, to see which variable seemed to be weighted the most heavily.

The completely unsurprising answer? Wins.

The coefficient between goalie rank and wins rank was 0.81. The two lists follow each other closely, as the top 6 ranked goalies are also the top 6 goalies in terms of wins, and 12 of the top 15 goalies are also in the top 15 in the league in wins.

They made one particularly egregious decision that supports the hypothesis:

Goalie A: 38 GS, 2.28 GAA, .912 Save %, 7 SO
Goalie B: 31 GS, 2.45 GAA, .905 Save %, 3 SO

It would appear that Goalie A is having a decisively better year. The panel sided with with Goalie B, putting him in 9th spot. Goalie A was ranked 27th. The reason is that Goalie B, Toskala, has a 24-9-1 record, while Nabokov (A) has gone 17-16-2. This is entirely because of a huge discrepancy in goal support, which a goalie cannot impact in any way.

The correlation with save percentage was 0.5, and the correlation with shot-quality neutral save percentage was 0.16. I guess the goalies probably printed off a list of the league leaders in wins, and made a few subjective adjustments from there.

In summary, it seems that even former goalies are blinded by team factors. Players and former players are often good at identifying the most purely talented individuals, but talent does not equal performance. Marc-Andre Fleury ranks very highly in terms of raw talent, perhaps top 10 or even top 5, but his performance this year suggests that he deserves to be ranked around 25th in terms of actual play. The panel decided to rank Fleury as 10th best in the league.

Former goalies and players are often no better than average in terms of valuing overall performance and contribution to the team. Their viewpoints are coloured by media hype, team success, and the highlight reels, just like everyone else's. Therefore, just because former goalie so-and-so said something, doesn't make it true. A thorough statistical study should be taken to ensure that the subjective viewpoint matches objective reality.