Testing the Teammate Theory

To test how well we can evaluate goaltenders by comparing them to their teammates, I decided to look in some more detail at the last two seasons. This allows for a direct comparison with shot quality data to see if there are parallels, and to see whether the two methods deliver similar results.

I calculated GAA compared to backups and save percentage compared to backups (with adjustments) for all the goalies who had been starters over the last two seasons. I also used shot quality information available from Alan Ryder and Ken Kryzwicki at Hockey Analytics (taking care to attempt to correct for the variance in shot distance reporting around the league recently discovered by Ryder), and used that to find out whether each goaltender's save percentage was above or below predicted levels, and by how much.

It turns out that the methods have quite similar results. The correlation coefficient between save percentage vs. predicted save percentage from shot quality and save percentage vs. teammates' save percentage was 0.73. Six of the top 10 goalies in performance vs. backups were also in the top 10 in performance against the predictions of the shot quality model (Huet, Hasek, Lundqvist, Lehtonen, Kiprusoff, and Legace). In addition, both methods agreed on the top man: Montreal's Cristobal Huet. Here are both lists:

Performance vs. Predicted (Shot Quality), 2005-07:
1. Cristobal Huet
2. Dominik Hasek
3. Tomas Vokoun
4. Henrik Lundqvist
5. Kari Lehtonen
6. Miikka Kiprusoff
7. Roberto Luongo
8. J.S. Giguere
9. Manny Legace
10. Ray Emery

Performance vs. Backups, 2005-07:
1. Cristobal Huet
2. Miikka Kiprusoff
3. Henrik Lundqvist
4. Martin Brodeur
5. Tim Thomas
6. Kari Lehtonen
7. Dominik Hasek
8. Ryan Miller
9. Manny Legace
10. Marc-Andre Fleury

Vokoun illustrates one of the problems with using backup stats: His backup is Chris Mason, a good goaltender, which makes it difficult for Vokoun to rate highly against his own teammates. Giguere is in a similar situation. Emery is knocked down by having backed up Dominik Hasek in 2005-06, while it is likely that Luongo's backups simply overachieved in the few games Roberto was given off, probably also against soft opposition.

On the other list, Brodeur and Miller were just outside the top 10 in shot-quality adjusted save percentage, so they aren't way out of place, although they both made it on because their backups did not do very well compared to their predicted save percentages. Thomas and Fleury benefitted from some very poor play by their backups (mostly Toivonen and Thibault, respectively).

Therefore, this seems to show that while using teammate data cannot replicate shot quality data, it can in many cases do a pretty good job of estimating it, as the results are correlated and the rankings similar. There will always be a few goaltenders rated too high or too low because of the strength or weakness of their teammates, even after adjustments, so this method requires some care in interpreting the results. Nevertheless, a discerning application of the method to the goalies of seasons past should yield some useful and interesting results.