Looking for Outliers

I've been working lately on the issue of shot quality at even strength. Studies of the entire goalie population done by myself and others seem to suggest that there is very little shot quality effect at even strength, with an estimated variance of something like .002-.003.

As someone who has spent a lot of time looking at goaltending team effects, however, these results still don't really make sense to me. Goalie results are usually quite similar by team, and only the top goalies are able to maintain a clear separation in performance compared to their backups year after year. Part of that is because of special teams effects, no doubt, and part of that is because backups often play weaker competition and are subject to more variable results because of fewer games played. However, my intuitive sense is that these results might have a lot to do with the parity and depth of goaltending in the league today, rather than just team effects on shot quality.

I think there are some teams with extreme results that are getting hidden in the population. There are a couple of issues with the type of study I did (which was based on looking at how goalies did from year-to-year when they were on the same team compared to how they did when they changed teams). Some teams used a lot more goalies than others, so it would certainly be possible to have some of the extreme teams not represented or underrepresented by chance. Bad teams also tend to cycle through more goalies, so goalies bouncing around from bottom feeder to bottom feeder could have made the results look like there is less of a team effect.

I decided to look on a team-by-team basis to look for any unusual team results, using the NHL's even-strength save percentage data since 1998-99. We can't simply analyze each team's total even-strength save percentage, since many teams have one or two guys who made up the majority of the minutes. I decided to look only at goalie seasons where a goalie played less than 30 games. This should remove the starters and focus on the backups. Backups are replaced a lot more frequently than starters, which gives us a more varied sample that should be less subject to individual goaltender performance.

The average of the total sample was .908. Seventeen teams were +/- .004 from this result. The typical sample size was around 3,000 shots, so that kind of variance is not statistically significant and would be expected. There were a few teams on either end that either had much higher than average or much lower than average save percentages. I calculated the binomial probabilities that we would see those results if the teams were using average backup goalies, and ended up with 4 teams on the high end and 4 teams on the low end that had probabilities of less than .05, meaning we can be over 95% sure that either they weren't average goalies, or they weren't facing average shot quality against, or some combination of those factors.

This result on its own is pretty good evidence that there are differences between teams, because we certainly wouldn't expect 8 teams out of 30 to significantly deviate from the average if all teams had exactly equal goalie talent on an exactly even playing field. The question is whether it is because of goalie skill, easier than average shot quality, or some other variable like scorer bias.

The 4 teams who had higher than expected save percentages were:

Minnesota: .924, 1343 shots, 8 seasons by 4 goalies
Colorado: .920, 2759 shots, 10 seasons by 7 goalies
San Jose: .918, 2806 shots, 14 seasons by 9 goalies
Florida: .917, 3920 shots, 11 seasons by 10 goalies

The 4 teams who had lower than expected save percentages were:

Atlanta: .900, 5586 shots, 21 seasons by 15 goalies
Toronto: .900, 4898 shots, 18 seasons by 12 goalies
St. Louis: .899, 3837 shots, 23 seasons by 16 goalies
Tampa Bay: .899, 4475 shots, 25 seasons by 17 goalies

That pretty much passes the common sense test for me, especially the bottom 4. On the other hand, goalie quality likely explains a few of these results (about 60% of San Jose's sample was either Kiprusoff or Toskala, for example). It was not unexpected to see Minnesota here, but this unfortunately isn't very good evidence for a "Lemaire effect". Their platoon system meant that Minnesota's goalies almost never met the cutoff, and 90% of the qualifying minutes were played by one guy (Josh Harding) who is likely to be a decent NHL goalie.

If shot quality is primarily dependent on skill rather than style, and I think the evidence supports that position, then it makes sense to me that we would see more deviation at the bottom end of the table where you have the teams who are badly managed or are filled with younger, developing talent or aren't spending up to the cap. Because mediocre talent is a lot more readily available than elite talent, it is much easier to put together a team that is unusually bad than one that is unusually good.

It could be that we are overrating the relative advantage of playing in New Jersey or Minnesota, but I think there does appear to be a disadvantage to play in a place like Tampa or Atlanta, and that should be taken into account when evaluating those goalies.