Monday, May 25, 2009

Why Pittsburgh is Better This Time

I've been quite impressed by the Pittsburgh Penguins this playoff season. Their goalscoring expoits are attracting all the headlines, but they are also playing a strong all-around game that is completely overwhelming Carolina at the moment. I think the Penguins will put up a much better fight in the Stanley Cup Final this year, assuming they finish off the Eastern Conference Final series that they currently lead 3-0.

Even though the Pens made it to the Finals last year with many of the same players, their underlying numbers are quite different. Last year's version was not as strong defensively, and depended on goaltending to keep the puck out of their net. The '08 squad was outshot 610-594 in the playoffs, and Marc-Andre Fleury's .933 save percentage was critical for their goal prevention.

This year Fleury has just a .902 save percentage in the postseason, despite facing easier than average shots against and playing on a disciplined team that has not taken many penalties. By my shot quality estimates, Fleury has cost his team about 9 goals compared to average, which is more than any other goalie in the playoffs (although that is partly because most of the other goalies who played poorly went out in the first round). This matches my subjective view that he hasn't been particularly good.

Despite this the Penguins are winning, often in dominating fashion. They have become an outshooting team with a significant 569-460 edge in shots, which has allowed them to average an outstanding 3.7 goals per game despite posting what is for them a typical shooting percentage. Through 2+ rounds this year they have already scored as many goals as they did through 4 rounds one year ago. The Penguins are not relying on percentages but are simply outplaying their opponents, which is the mark of a strong team.

Fleury's play hasn't been a concern as of yet, although it might be soon in what is expected to be a closer, more tight-checking Final. Pittsburgh likely won't be able to count on an outshooting advantage against their Western Conference opponent, which means that the play of their goaltender will become more significant. Fleury is certainly capable of doing better, however, and if he can pick up his game and Pittsburgh is able to add solid goaltending to the offensive fireworks of Crosby and Malkin then they are a strong contender to win it all, even if they have to once again go through Detroit.

Tuesday, May 12, 2009

Looking for Outliers

I've been working lately on the issue of shot quality at even strength. Studies of the entire goalie population done by myself and others seem to suggest that there is very little shot quality effect at even strength, with an estimated variance of something like .002-.003.

As someone who has spent a lot of time looking at goaltending team effects, however, these results still don't really make sense to me. Goalie results are usually quite similar by team, and only the top goalies are able to maintain a clear separation in performance compared to their backups year after year. Part of that is because of special teams effects, no doubt, and part of that is because backups often play weaker competition and are subject to more variable results because of fewer games played. However, my intuitive sense is that these results might have a lot to do with the parity and depth of goaltending in the league today, rather than just team effects on shot quality.

I think there are some teams with extreme results that are getting hidden in the population. There are a couple of issues with the type of study I did (which was based on looking at how goalies did from year-to-year when they were on the same team compared to how they did when they changed teams). Some teams used a lot more goalies than others, so it would certainly be possible to have some of the extreme teams not represented or underrepresented by chance. Bad teams also tend to cycle through more goalies, so goalies bouncing around from bottom feeder to bottom feeder could have made the results look like there is less of a team effect.

I decided to look on a team-by-team basis to look for any unusual team results, using the NHL's even-strength save percentage data since 1998-99. We can't simply analyze each team's total even-strength save percentage, since many teams have one or two guys who made up the majority of the minutes. I decided to look only at goalie seasons where a goalie played less than 30 games. This should remove the starters and focus on the backups. Backups are replaced a lot more frequently than starters, which gives us a more varied sample that should be less subject to individual goaltender performance.

The average of the total sample was .908. Seventeen teams were +/- .004 from this result. The typical sample size was around 3,000 shots, so that kind of variance is not statistically significant and would be expected. There were a few teams on either end that either had much higher than average or much lower than average save percentages. I calculated the binomial probabilities that we would see those results if the teams were using average backup goalies, and ended up with 4 teams on the high end and 4 teams on the low end that had probabilities of less than .05, meaning we can be over 95% sure that either they weren't average goalies, or they weren't facing average shot quality against, or some combination of those factors.

This result on its own is pretty good evidence that there are differences between teams, because we certainly wouldn't expect 8 teams out of 30 to significantly deviate from the average if all teams had exactly equal goalie talent on an exactly even playing field. The question is whether it is because of goalie skill, easier than average shot quality, or some other variable like scorer bias.

The 4 teams who had higher than expected save percentages were:

Minnesota: .924, 1343 shots, 8 seasons by 4 goalies
Colorado: .920, 2759 shots, 10 seasons by 7 goalies
San Jose: .918, 2806 shots, 14 seasons by 9 goalies
Florida: .917, 3920 shots, 11 seasons by 10 goalies

The 4 teams who had lower than expected save percentages were:

Atlanta: .900, 5586 shots, 21 seasons by 15 goalies
Toronto: .900, 4898 shots, 18 seasons by 12 goalies
St. Louis: .899, 3837 shots, 23 seasons by 16 goalies
Tampa Bay: .899, 4475 shots, 25 seasons by 17 goalies

That pretty much passes the common sense test for me, especially the bottom 4. On the other hand, goalie quality likely explains a few of these results (about 60% of San Jose's sample was either Kiprusoff or Toskala, for example). It was not unexpected to see Minnesota here, but this unfortunately isn't very good evidence for a "Lemaire effect". Their platoon system meant that Minnesota's goalies almost never met the cutoff, and 90% of the qualifying minutes were played by one guy (Josh Harding) who is likely to be a decent NHL goalie.

If shot quality is primarily dependent on skill rather than style, and I think the evidence supports that position, then it makes sense to me that we would see more deviation at the bottom end of the table where you have the teams who are badly managed or are filled with younger, developing talent or aren't spending up to the cap. Because mediocre talent is a lot more readily available than elite talent, it is much easier to put together a team that is unusually bad than one that is unusually good.

It could be that we are overrating the relative advantage of playing in New Jersey or Minnesota, but I think there does appear to be a disadvantage to play in a place like Tampa or Atlanta, and that should be taken into account when evaluating those goalies.

Monday, May 11, 2009

The Shot Quality Effect, Part 2

I did some further calculations based on the responses to my last post, so I thought I'd just throw them in here for the record and in case somebody is interested that didn't read the comments to the last one.

It was pointed out that goalies who face fewer shots have results that are more subject to variability, so it would be more correct to adjust for sample size. Vic Ferrari suggested using an adjustment factor of sqrt(1000/shots faced) to normalize all seasons to a 1,000 shot level. I included that adjustment, and I also normalized all years based on league average to remove any seasonal effects.

As an example, if the goalie had a .925 ES SV% last year, and a .930 ES SV% this year, and he faced 500 shots, then he is 2.5 goals better this year than last year. Multiply that by sqrt(1000/500), and we get 3.5 goals, which expresses the result on the scale of a 1,000 shot season, taking into account the significance of the information provided by that sample size.

All that was left to do was sort based on whether the goalie was playing for the same team or a different team. Here are the results (I included the same minimum games played cutoffs as before, 10 GP means that the goalie must have played at least 10 games in both seasons for them to count):

0 GP: 12.0 same team, 12.6 different team
5 GP: 10.4 same team, 11.5 different team
10 GP: 10.3 same team, 12.1 different team
20 GP: 10.0 same team, 11.9 different team
30 GP: 10.0 same team, 10.9 different team
40 GP: 9.8 same team, 11.6 different team

These results suggest a team effect of about 1-2 goals per 1,000 shots, or .001-.002 in save percentage. My look at the average save percentage differences had similar results with a gap of about .002-.003. These results suggest that, for the most part, even strength shot quality is pretty consistent in the NHL. I still expect there to be a few teams who fall a bit outside of the regular curve, but it appears to be pretty clear that even-strength save percentage is a very important statistic for goalie evaluation.

Note, too, that this suggests that the typical variation in goalie performance is 10 goals per 1,000 shots, or .010 in save percentage, even for goalies playing on the same team. Quite often "bad years" or "good years" are simply because of chance. That is something that we need to keep in mind as fans before we make the mistake of calling somebody good or bad after watching them a few times or, as tends to happen at this time of year, based on results from a single seven game series.

Wednesday, May 6, 2009

Shot Quality at Evens

Vic Ferrari put up an interesting post challenging the notion that Minnesota makes their goalies look good. Some of the evidence from the scoring chance charting done by an Edmonton Oilers blogger does seem to suggest that there is little team-to-team variance in shot quality against at even strength. Can that be correct?

I have even-strength save percentages for every goalie in the league since 1998-99. If teams didn't matter at all, then we should see no difference between the year-to-year variance of goalies who played on the same team and goalies who changed teams. I don't think that expectation is realistic even if there really is no team effect, as we would expect a variance in save percentage to be one of the reasons that could cause goalies to change teams. Keeping that in mind, here are the results for a few minimum games played cutoffs (e.g. 20 GP means that a goalie must have played at least 20 games in both seasons being compared):

0 GP: .021 same team, .026 different team (N1=443, N2=204)
5 GP: .013 same team, .017 different team (N1=374, N2=159)
10 GP: .012 same team, .016 different team (N1=330, N2=157)
20 GP: .012 same team, .016 different team (N1=267, N2=95)
30 GP: .010 same team, .012 different team (N1=208, N2=54)
40 GP: .009 same team, .012 different team (N1=158, N2=35)
50 GP: .008 same team, .011 different team (N1=110, N2=15)

Not a huge difference, but I wasn't expecting one as most teams in the league face similar shot quality against. There is a persistently higher variance for goalies who move to different teams than those who play on the same team, suggesting that team situations are not equal. On the other hand, though, it does suggest that most teams are pretty close in terms of difficulty of shots allowed. I don't think this rules out the possibility of a couple of teams being large outliers (a team like Minnesota, for example), but it does look fair to say that most teams face a similar range of shot quality at even-strength.

Another thing that we would expect, if shot quality is significant, is for shot quality measures to correlate well with actual save percentages. Behind the Net has recently posted two years' worth of shot quality data at 5 on 5. I decided to look for goalies who had significant playing time in both of the last two seasons, as well as a prior sample from 2005-06 and 2006-07 that we could use to compare. I ended up setting the cutoffs at a minimum of 30 games played in each of the last 2 seasons, and at least 700 shots against, which gives us 26 goalies.

Correlation between expected and actual save %: 0.34
Correlation between actual save % and career save %: 0.23
Correlation between actual save % and post-lockout save %: 0.38

These numbers bounce around a bit depending on the cutoffs. When I raised the cutoffs for the career numbers, the relationship between career save % and actual save % got a little stronger, which we would expect, and the relationship between expected and actual in 2008-09 got a little weaker. I think that prior save percentage results are likely better correlated with actuals than shot quality predictions are at evens, and that our current shot quality models are still impacted by things like biases and score effects and likely could be further refined, but that there does appear to be some significance to shot quality for 5 on 5 play. That's just a quick check, though, I'll leave it to somebody who has all the shot quality data and can look at a bigger sample size to better test the relationship.

Tuesday, May 5, 2009

Was Kiprusoff Overworked?

I got linked recently from the CalgaryPuck forum, where they are debating whether Kiprusoff's poor numbers this season were because of him being overworked. One of the posters did a nice little study that suggests games played has little impact on the numbers, by comparing save percentages before, during, and after either a 2-season or a 3-season stretch of playing 68 or more games. There was very little difference in the results (save percentages before and after were all +/- .004 from how the goalies did in their high-workload seasons).

One criticism of this technique, however, is that even if goalie fatigue is generally overrated, Miikka Kiprusoff might still be a unique case with a specific weakness in that area. I decided to look at Kiprusoff's results to see if they can tell us anything about his workload:

I pulled the game logs for each of Kipper's last 5 seasons, and looked at the number of days between starts. Here are the overall results:

1: 2.98 GAA, .897 sv%, .565 win %
2: 2.43 GAA, .914 sv%, .626 win %
3: 2.33 GAA, .919 sv%, .636 win %
4: 2.05 GAA, .930 sv%, .635 win %
5+: 2.36 GAA, .915 sv%, .769 win %

Kiprusoff did not do very well in back-to-back starts, but performing poorly in back-to-backs is not unique to him. Teams generally do worse, so part of that is likely to be the rest of the team. If Kiprusoff got at least one day of rest between games, then he played pretty well, and he (and his teammates) did slightly better with additional rest.

Looking at the 2008-09 results, Kiprusoff played 9 back-to-back games, which is the most he played in any single season. His results: 2-7-0, 3.23, .899, suggesting that he was overworked by his coach.

I disagree, however. I think there are two reasons why this is not valid as an excuse for Kipper's overall poor play:

1. Kiprusoff did not do substantially better with 1, 2, or 3 days rest in 2008-09.

1 day rest: 24-11-2, 2.83, .904
2 days rest: 11-6-0, 2.80, .907
3 days rest: 4-2-0, 3.37, .899

2. Calgary as a team obviously played poorly in back-to-back games.

Kiprusoff faced 2.5 extra shots against in his back-to-back starts. That explains why he allowed 0.4 more goals per game with a .005 drop in save percentage. He also had a much worse record, indicating that the rest of the team didn't score much in front of him.

It would have helped Kiprusoff's stats if he didn't play in those back-to-back games. It probably wouldn't have helped the team much, though. The only difference would have been that it would have been Curtis McElhinney getting shelled instead of Kipper. Kiprusoff doesn't have a pattern of strong performance in back-to-back games, but a lot of this appears to be the effect of travel for a West Coast team like Calgary.

If we take out the back-to-backs, it is still apparent that Kipper's best days are behind him. I looked at his numbers for games with between 1 and 3 days rest over the last several seasons, to get a better comparison:

2008-09: 39-19-2, 2.88, .904
2007-08: 33-23-10, 2.61, .908
2006-07: 35-22-7, 2.45, .918
2005-06: 35-16-8, 1.90, .929
2003-04: 20-9-3, 1.61, .938

Unless there is some cumulative wear effect, it's not his seasonal workload that is the cause for Kiprusoff's poor play. He is simply getting worse.

We see the decline even when we break it down further by the number of days rest between games. Here are the seasonal save percentages from 2003-04 to 2008-09:

0 days rest: .929, .889, .901, .893, .899
1 day rest: .940, .928, .906, .907, .904
2 days rest: .927, .928, .931, .903, .907
3+ days rest: .917, .938, .946, .922, .905

To me, the numbers in the last two columns don't suggest an overworked goalie, just a mediocre one. The only year where it really looks like rest was a big help for Kiprusoff was 2006-07. I'd still bet that a lot of that was randomness, however, since the seasonal sample sizes here are pretty small.

An interesting study would be to look at shot quality results for goalies playing in back-to-back games, to see if we can better break down the responsibility of the dropoff between the goalie and the defence. Until then, I think it is likely that the blame is split between both parties, but we can't go much further than that.

Kiprusoff's numbers might have benefitted if Keenan spelled him a few times when the Flames were playing on consecutive nights, as Calgary seems to have played poorly as a team in back-to-backs this year. That does not come close to explaining his regression this year or last year. Kiprusoff hasn't been an elite goalie since 2007, and right now he's probably not even an average goalie, gaudy win totals notwithstanding. If workload has anything at all do with his results, it is far more likely to be the cumulative effects of playing a 70+ game workload for 4 years in a row rather than anything specifically related to 2008-09.

Sunday, May 3, 2009

Point Shots and Possession

One thing I wanted to look at in my playoff shot chart data was which teams were allowing point shots through and which teams were restricting them. I figured this was mostly related to defensive coverage. After having looked at the numbers and thinking it through a little more, however, it seems that the percentage of shots that originate from the point might be more of an indicator of possession than of defensive coverage.

Shots from some locations on the ice are very discretionary, such as for example shots from outside the zone or from sharp angles. The defence is usually relatively indifferent to these shots, because they are very unlikely to go in, which is why you don't see guys going down to block slapshots from the red line.

It is common, however, for teams to pressure the points and have multiple players attempting to block shots. Point shots can go in because of screens or deflections, or lead to other good results like rebounds. Therefore, point shots tend to be less discretionary - the defencemen will take the shot if they can, hoping for a deflection or a rebound. Because the front of the net is usually crowded with players, some puck movement is usually required to set up a point shot that ends up on net, and as a result point shots often occur after some sustained play in the zone. This means that teams that struggle to establish offensive zone possession are likely to have fewer point shot attempts and therefore fewer point shots getting through to the net.

My definition of a point shot is a shot recorded in the ESPN Gamecast as originating from between the top of the circles and the blue line, in between the faceoff dots. Here are the percentage of point shots out of total shots faced by each starting goalie in the playoffs:

Thomas 9%, Price 24%
Varlamov 16%, Lundqvist 20%
Ward 23%, Brodeur 20%
Fleury 15%, Biron 20%

Hiller 18%, Nabokov 17%
Osgood 12%, S. Mason 18%
Luongo 19%, C. Mason 15%
Khabibulin 13%, Kiprusoff 17%

The starting goalie that faced a lower ratio of point shots won 5 out of the 8 series. In all of those series, the winning goalie had a percentage at least 4% lower than the losing goalie. The 3 goalies that won despite facing a higher point shot percentage were Ward, Hiller, and Luongo, who all were excellent in round 1.

Point shot percentage did not have a very high correlation with either expected goals (0.08) or total shots (0.17), however, which suggests that it is not a perfect indicator. It is likely that some teams block more point shots than others, and that some teams attempt more point shots than others, depending on their team strengths and style of play, which of course would affect the numbers. There are also playing to the score effects. At least so far, however, the numbers have been intriguing.

Saturday, May 2, 2009

How The Red Wings Play to Win

NHL teams play to the score, especially in the playoffs. Late in the game, the leading team will take fewer chances and try to choke off the game, while the losing team will press for an equalizer and usually put as many pucks on net as they can.

The Detroit-Columbus series was an interesting showcase for how teams play based on the score, as Columbus didn't spend a single second of the series in the lead. That makes it easy to analyze the numbers, since we don't have to go through and remove any time that the Blue Jackets spent in the lead.

One thing that was striking in both my numbers and those at Hockey Numbers was that the expected goals totals for the Blue Jackets were very close to that of the Red Wings. This seems to indicate that the play was much closer than what observers were reporting. Often I tend to side with numerical evidence over eyewitness accounts, but on this one I think it is pretty clear that the eyeballs were seeing something that the math wasn't taking into account, namely that the Detroit Red Wings playing to win.

It has been well demonstrated that trailing teams tend to shoot more, and that teams with a large lead tend to shut it down somewhat to help preserve their lead late in games. Therefore, we would expect that Columbus' shot totals and expected goal totals to be overstated, and for Detroit's numbers to be understated. This turns out to be the case.

Let's break it down further. Detroit had a large outshooting edge in the first 2 periods (104-73), and shots were essentially even in the third (37-36). That is the kind of result we would expect to see for a talented team like the Red Wings against an overmatched opponent - take control early, score a few goals, and then kill the clock.

As I observed in my last post, point shots may be somewhat of an indicator of possession. Over the first two periods, Detroit had a 19-8 edge in point shots. In the third period, they took just 7 to Columbus' 5. That likely indicates that Detroit spent much of the first two periods in Columbus' zone, allowing their defencemen to get set up and fire at the net, but gave up that possession advantage in the final period.

Perimeter shots are similar in that they indicate possession, although they are more up to the shooter's discretion since they come from less well-defended areas and are therefore probably more likely to end up on net. In first periods throughout the series Detroit threw everything at the net, with 25 perimeter shots on Mason. In the second that dropped to 11, and in the third just 14. Osgood faced few perimeter shots the whole way, just 7 in the first and 10 in each of the second and third periods.

I would have expected Columbus to take more long shots when trailing late. I'm pretty sure that the Red Wings would show an increased perimeter shot rate when trailing late in games. Maybe the Red Wings' forwards were doing a great job of preventing or blocking shots. Or perhaps the Blue Jackets have different offensive tactics, and prefer to try to generate more high-quality scoring chances. They were actually fairly successful at getting high-quality scoring chances against the Red Wings, so if that's the case it might not have been a bad strategy, Osgood just managed to hold up better than expected in net.

In the third period Detroit seemed to be trying extra hard to limit those high-quality chances, and they were pretty successful in doing so. In the first two periods combined, Osgood faced 20 shots from the crease area while Mason faced 13. In the third periods, Osgood faced just 3 while Mason only had to deal with 2. This shows the tradeoff for the leading team: They reduce their chances to score as well as the opposition's. When up a goal or two in the third, however, this appears to be a favourable strategy.

Perhaps the biggest sign that Detroit wasn't trying as hard to score was that in the third period they very clearly stopped trying to shoot high. In the first two periods, 30% of the shots on Mason were high. In the third, just 17% were high. Osgood's numbers went from 14% to 11%.

These numbers show that both goalies were facing much easier shots in the third period. Taking into account both shot location and shot height, my estimate is that over the first two periods the expected save percentages for the two goalies were .915 for Osgood and .929 for Mason. In the third, those numbers rose to .926 and .945 respectively (.921 was the average).

Four games is a small sample size, so some of these results may have been random chance. However, they all seem to support playing to the score effects that have been observed in larger samples, so there is likely something significant there.

In summary, the shot charts make it look like Detroit and Columbus played a fairly close series, and that Mason played poorly while Osgood played well. The underlying numbers, however, suggest that the Red Wings were clearly the better team, and shut down their offence when they had the game in hand. If Columbus was able to keep the games closer, or if the Red Wings had decided to keep trying to score all the way up to the final whistle, the expected goals would have been much more tilted in favour of the defending champions.