Shot Quality Refined

I have to throw up a link to the latest work by Ken Krzywicki on shot quality. He tackles the issue of arena reporting bias, and has updated his model to take that into account. I think there are some very good reasons to question the importance and/or the accuracy of shot quality measures, but I still think it is possible that some of the problems are a result of the data or the models rather than the theory that different teams allow different types of scoring chances. That shot quality differences are observed in the game of hockey is beyond doubt, the debate is about whether those differences are significant at the NHL level.

To me, the revised model makes more sense, because we see a smaller spread of values at the team level. The best team has an expected save percentage of .915, while the worst is at .904. Compare that to the .900 - .918 which would be predicted based on the raw numbers. Half of the teams are within .002 of league average, which matches our expectation that most teams do not have significant shot quality effects.

I would like to see the shots separated out by situation into EV and PK. I think that would allow us to better be able to test the results. However, the model must be projecting some even strength variations, since the correlation between expected save percentage and power plays against in 2008-09 was -0.03.

Some of the team results are a bit surprising. In particular, Boston is rated as having allowing the 4th most difficult shots against, which seems to strongly contradict the numbers of Thomas and Fernandez, the Bruins' low number of penalties taken, and the coach and team style of play.

Alan Ryder also has a method for evaluating shot quality in his annual NHL review. He says he has tried to make some adjustments for rink bias as well, yet some his numbers differ from Krzywicki's. One of them is in his measurement for Boston whom Ryder grades as about average, which I suspect might be a more accurate assessment.

By comparing Krzywicki's numbers with Ryder's, and also keeping in mind each team's actual save percentage as an additional check, I think there are about ten teams that we can target as possible shot quality outliers in 2008-09, with New Jersey, Phoenix, Buffalo, Minnesota and Columbus the candidates for easier than average shot quality and Toronto, Dallas, Carolina, Montreal and Calgary possibly allowing harder than average shots against.

Of those, a few could be the rest of measurement error, score effects, randomness, etc., but I think is likely that there are a few teams that must be ahead or behind the rest of the league. To me, Minnesota, New Jersey, Toronto and Dallas are the most likely candidates for 2008-09. For those four teams, the shot quality results match both the subjective perception and the actual results in the crease.

If this is correct, that would have implications for our evaluation of some goalies. That doesn't mean that goalies like Turco or Toskala can be let off the hook, however. The expected save percentage of an outlier team is still likely somewhere in the range of .003-.005 above or below average, which means that outstanding or terrible save percentages cannot be explained by the rest of the team and must fall mostly on the shoulders of the goalie that posted them.