Vic Ferrari put up an interesting post challenging the notion that Minnesota makes their goalies look good. Some of the evidence from the scoring chance charting done by an Edmonton Oilers blogger does seem to suggest that there is little team-to-team variance in shot quality against at even strength. Can that be correct?
I have even-strength save percentages for every goalie in the league since 1998-99. If teams didn't matter at all, then we should see no difference between the year-to-year variance of goalies who played on the same team and goalies who changed teams. I don't think that expectation is realistic even if there really is no team effect, as we would expect a variance in save percentage to be one of the reasons that could cause goalies to change teams. Keeping that in mind, here are the results for a few minimum games played cutoffs (e.g. 20 GP means that a goalie must have played at least 20 games in both seasons being compared):
0 GP: .021 same team, .026 different team (N1=443, N2=204)
5 GP: .013 same team, .017 different team (N1=374, N2=159)
10 GP: .012 same team, .016 different team (N1=330, N2=157)
20 GP: .012 same team, .016 different team (N1=267, N2=95)
30 GP: .010 same team, .012 different team (N1=208, N2=54)
40 GP: .009 same team, .012 different team (N1=158, N2=35)
50 GP: .008 same team, .011 different team (N1=110, N2=15)
Not a huge difference, but I wasn't expecting one as most teams in the league face similar shot quality against. There is a persistently higher variance for goalies who move to different teams than those who play on the same team, suggesting that team situations are not equal. On the other hand, though, it does suggest that most teams are pretty close in terms of difficulty of shots allowed. I don't think this rules out the possibility of a couple of teams being large outliers (a team like Minnesota, for example), but it does look fair to say that most teams face a similar range of shot quality at even-strength.
Another thing that we would expect, if shot quality is significant, is for shot quality measures to correlate well with actual save percentages. Behind the Net has recently posted two years' worth of shot quality data at 5 on 5. I decided to look for goalies who had significant playing time in both of the last two seasons, as well as a prior sample from 2005-06 and 2006-07 that we could use to compare. I ended up setting the cutoffs at a minimum of 30 games played in each of the last 2 seasons, and at least 700 shots against, which gives us 26 goalies.
Correlation between expected and actual save %: 0.34
Correlation between actual save % and career save %: 0.23
Correlation between actual save % and post-lockout save %: 0.38
These numbers bounce around a bit depending on the cutoffs. When I raised the cutoffs for the career numbers, the relationship between career save % and actual save % got a little stronger, which we would expect, and the relationship between expected and actual in 2008-09 got a little weaker. I think that prior save percentage results are likely better correlated with actuals than shot quality predictions are at evens, and that our current shot quality models are still impacted by things like biases and score effects and likely could be further refined, but that there does appear to be some significance to shot quality for 5 on 5 play. That's just a quick check, though, I'll leave it to somebody who has all the shot quality data and can look at a bigger sample size to better test the relationship.