Monday, July 27, 2009

The Demise of Even Strength Shot Quality

Anybody who has read my recent stuff has surely noticed the increasing usage of even strength statistics. That's because there has been a lot of good evidence presented lately that team effects at even strength are pretty minimal. I think Vic Ferrari has dealt the final blow to even strength shot quality, by demonstrating that the observed EV save percentage differences between goalies switching teams is essentially identical to what would be predicted by pure randomness.

This doesn't necessarily mean that there aren't a few teams that are outliers, but if they do exist there doesn't seem to be much room for very many of them.

I tracked some shot quality data during the playoffs using my own quick method, and have to say that I was souring a bit on shot quality already just from going through that exercise. There seemed to me to be a few systemic flaws in the metric, situations where the actual scoring percentages were consistently quite different than expected, although some of it could likely be reduced by using a superior method.

For example, as I calculated it the expected shot quality was roughly equal in all three periods (.913, .917, and .916 respectively). However, the overall save percentage was lower in the second period (.903), which is what we would expect since the teams have the long change. It was also higher in the third period (.928), which is when the playing-to-the-score effects usually become more pronounced. It seems that a goalie might face the same 20 foot wrist shot, but the second period one is more likely to be on a 2 on 1 rush after his team got caught on a bad change, while the third period one might often be taken by an opposing forward focused on defending a lead and primarily looking for a faceoff to keep the puck in the offensive zone. I've argued before that those things would even out over a large enough sample, but it seems likely that some of the situational factors are persistent enough to have more of an impact on the percentages than expected.

The now-familar scoreboard effects were once again observed in the data, but they weren't accurately predicted by my shot quality model. When goalies were on teams that were leading by 1 or 2 goals after two periods of play, I had them at an expected save percentage of .918 in the third, and they did even better that that at .920. When trailing by 1 or 2, the expected was .912 and the actual was just .905. When either team was up by 3 goals or more, the average save percentage jumped to .934 as the leading team usually just ran out the clock, even though the expected save percentage was just .908. Again, these numbers could just be from one unusual postseaon. It's also possible that a more refined model does a better job of predicting these changes. However, it does seem that scoring probabilities consistently differ from overall averages in certain strategic situations.

One thing I wonder about is that if we throw out EV shot quality as useless for evaluating today's goalies, what should we do about goalies from past eras? Can we assume that Grant Fuhr's even strength shot quality against in Edmonton in 1987 was similar to Patrick Roy's in Montreal? Or that Ken Dryden would have faced the same shot difficulty at evens playing for the California Golden Seals as he did behind Montreal's "Big Three"?

I did a similar goalies changing teams analysis for the Original Six era, and it certainly looks to me that there was a pretty major difference between playing in Montreal or Toronto and playing in Boston or New York. I expect that the 1970s had such a lack of parity that there was a similar uneven playing field. I'd guess that it is only in the last 20 years or so that even strength save percentage could be considered roughly team independent, although who knows. After all, not too long ago I thought shot quality was still an important factor at even strength in today's NHL.

We don't have official even strength save percentage numbers from before that so it's difficult to test this out, although I've looked at a couple of ways to try to estimate them. That will have to be a study for another day.

5 comments:

Matt said...

Vic's work is tremendous as usual, but I don't think it's cause to toss out your intuition entirely (even if your intuition is partly an attempt to be painfully fair to your anonymous commenters :) ).

What I get from Vic's work, to the degree that I would have no problem defending it to a general hockey fan who never read a blog (or a good one, anyway), is that most of good defense is preventing shots/chances, not preventing *high quality* shots/chances.

I think the idea that an individual player can improve his own goalie's SV% by 20 points has been blown to smithereens. To reduce your GA, you have to keep it out of your own end, and/or to the outside, where shots never get attempted or never get through.

But it would be too big a stretch to say that there's no such thing as shot quality that can be affected by the defenders (clearly, a very bad Dman will miss the back door pass too often).

I like your guess about the 'olden days'. Most of the stuff about the 'big save' and such was, and remains, horseshit, but I think your suspicion that the gap between average D and awful/awesome D back then was larger than it is today is probably true.

Vic Ferrari said...

I agree with your post and Matt's comment entirely.

I also think that your methodology for estimating EVsave% for days of yore is entirely sensible and fair. More than you suspect, I think. At some point I'll get back to illustrating why, hopefully in a fair way. In short, for all the philosopical banter in an IOF thread from earlier this year, much of it from me ... Sunny was right, shooters decide when to shoot. Whether it's the powerplay or even strength, it's just the way it works.

And you could have accurately said that "Vic Ferrari has shown that it is extremely likely that teams and players DO impact EVsave%". That would have been accurate, but not honest, at least not in the way that I think of the word.

The question was never about "if", it was about "how much", at least to my mind.

I also agree with your thoughts on playing to the score, and you have this year's playoff data to back it up. JLikens work on the subject using scoring chances was terrific except for the fact that using goals ... too volatile over so few games with just one team (the Oilers). Still, a terrific way of looking at it, I think, I would never have guessed that the effect was so profound. At some point I'll write a script to generate shots data by-the-score for Jlikens. He'd need that, I think.

On the other "old NHL" stuff, I think we'll all be surprised how little the game has actually changed. The data is sparse, though we'll get a better look at the Leafs at least (and all the teams they played against) when video archivist Paul Patskou eventually digitizes all Maple Leaf game sheets.

Contrary to popular opinion around here, I think we'll see that Gretzky was a terrific territorial player (granted in an unconventional way) as measured by Corsi. And that Messier floated for the first four months of the season, and that the Oilers' bottom six regularly got bootstomped, territorially, by deep teams like the Flames and Flyers.

We'll see.

The Contrarian Goaltender said...

"But it would be too big a stretch to say that there's no such thing as shot quality that can be affected by the defenders (clearly, a very bad Dman will miss the back door pass too often)."

I'm not saying that shot quality doesn't exist, merely that it does not appear to have big picture importance at the NHL level. Anyone who has ever played hockey on a very good team or a very bad team knows there is a difference in shot quality, especially if they played goalie.

What this evidence suggests is that shot quality is determined by skill rather than by systems or a team's style of play. In competitions with wide variances in skill (e.g. international hockey) we see wide ranges in save percentages, which likely reflect large differences in shot quality. In leagues with more parity, like today's NHL, there is less of a shot quality gap.

That's also why I don't think the assumption of EV numbers being team independent would hold to the same degree if we looked at a playoff sample. Teams play entire series against one opponent, and lower-seeded teams play their entire playoff schedules against the top teams in the league, so I'd assume that differences in skill would lead to a stronger effect.

"Sunny was right, shooters decide when to shoot. Whether it's the powerplay or even strength, it's just the way it works."

That's the way I've always thought of it as well, even as a goalie. I never thought I had an awful lot of control over what the shooter was doing, and that's why my starting position was that goalies had zero effect on shots against or shots that missed the net or anything like that. I've had to revise that slightly when the evidence showed that there was a small effect (although I'm still not really sure what is driving it). But that shooters decide when to shoot is pretty clear. If not the playing to the score stuff wouldn't be as drastic.

"On the other "old NHL" stuff, I think we'll all be surprised how little the game has actually changed."

I agree with that, I don't think the game has changed all that much. Hockey's not that complicated, it's always more or less come down to getting the puck into the other team's end, creating scoring chances, and hoping you can finish more of them than the other guys do. What has changed is the distribution of talent around the league, and that's what leads me to believe that the team dependence of goaltending stats was greater in prior eras than it is today.

Bruce said...

Interesting stuff and conclusions. If shot quality is indeed relatively inconsequential, then let's hearken back to your post Brodeur, Roy and Hasek: The Debate Rages On, in which you posted the following:

Patrick Roy: 2.5% easier shot quality, 1 shot created/gm
Martin Brodeur: 5% easier shot quality, 1 shot prevented/gm
Dominik Hasek: 2.5% harder shot quality, 0.5 shots created/gm
Roberto Luongo: 0% harder shot quality, 1 shot created/gm
Ed Belfour: 0% harder shot quality, 1 shot prevented/gm
Curtis Joseph: 2.5% harder shot quality, 0.75 shots created/gm


At the time I made a comment agreeing with the direction of the arrows in the second column (shot prevention) and my opinion hasn't changed. Your estimates were if anything conservative, but let's continue to use them here.

That leads us to the following results:

The current evidence suggests we entirely throw out the adjustments from the first column (shot quality), so let's restate your results without that adjustment (last column ameliorated).

Goalie Save % SA/60 GAA SA Prev Adj SA SA AdjS%
Hasek .922 28.32 2.20 0.5 27.82 .921
Brodeur .914 25.46 2.20 -1.0 26.46 .917
Luongo .919 31.82 2.58 1.0 30.82 .916
Belfour .906 26.66 2.50 -1.0 27.66 .910
Roy .910 28.24 2.54 1.0 27.24 .907
Joseph .907 29.75 2.79 0.75 29.00 .904


Hmmm, Brodeur with a better adjusted Sv% than Luongo, who knew?

These numbers allow us to compare the adjusted save pecentages to league average and calculate a number of goals above average for each goalie that combines quantity and quality:

Again, updating with shot quality thrown out (note, I have Sv% to just three digits, so the GOA is rounded for all but Belkfour and Luongo whose estimated shot quality was flat.

Rank Goalie AdjSV% LgAvg % Diff SA GOA
1. Hasek .921 .903 +19% 20,220 ~364
2. Roy .907 .895 +11% 28,353 ~343
3. Brodeur .917 .905 +13% 24,720 ~297
4. Belfour .910 .900 +10% 24,751 238.0
5. Luongo .916 .907 +10% 16,051 149.1
6. Joseph .904 .900 +4% 26,676 ~107


So now we have Roy just behind Hasek, and Brodeur within range to catch up with both of them before he's done, although it will likely be close. Belfour is clearly #4, while the other two are well back.

And you know what? That works for me.

The Contrarian Goaltender said...

Bruce: I will certainly have to revisit those numbers, but your conclusions aren't quite correct. The shot quality we are discussing here is even strength shot quality. Brodeur and Roy both faced fewer shots on the penalty kill than the other goalies, which was the primary reason why they had lower shot quality estimates in the first place. That part of the equation doesn't change at all.

To redo everything properly I would have to replace the save percentages with even-strength save percentages, and then include the shot prevention adjustment. I should probably also take penalty kill play into account, but there appears to be more of a team element associated with that.

You are correct that this would benefit Brodeur with respect to Luongo, and would probably put the two of them very close together relative to average, but Brodeur still isn't going to get near either Hasek or Roy. See this post where I estimated even strength save percentages for all of these goalies, and Roy and Hasek were far ahead of the rest of the group.

Assuming no EV shot quality effects and using the same shots against per game figures and based on those even strength estimates I quickly reran the numbers and got the following:

1. Hasek 422
2. Roy 375
3. Brodeur 247
4. Belfour 226
5. Luongo 156
6. Joseph 132

Finally, I'm not convinced yet that New Jersey isn't an outlier. The attempts at measuring shot quality always showed the pre-lockout Devils to be the best in the league. Again, the conclusion about shot quality is that it is generally insignificant, not that it doesn't exist. It might be significant for some teams at both the top and the bottom, and if you were to go looking for teams with some kind of significant effect you would definitely start looking with the team that led every year. Even if New Jersey had a small EV shot quality benefit, that would drop Brodeur relative to the other guys.

We also don't know how far back we can apply this conclusion, I doubt it holds as far back as 1986 when there was less parity in the league, which means that the assumption probably doesn't hold for Roy's early career. Pretty much all of Roy's goalie teammates in Montreal did far better in Montreal than everywhere else, which suggests to me that he probably had an easier time of it, which if true is something that should also be taken into account.