Special Teams Play: Luck or Skill?

There is a Vezina Trophy winner hidden in the following table of even strength save percentage leaders from one of the seasons in the last decade. See if you can spot him:

1. Goalie A, .934
2. Goalie B, .931
3. Goalie C, .930
4. Goalie D, .929
5. Goalie E, .929
6. Goalie F, .929
7. Goalie G, .925

Obviously it's not the guy at the top of the list, because that would be too easy and I wouldn't be asking the question in the first place. It's not the guy at the bottom of the list either, I only included him because he is a future Hall of Famer. That leaves the goalies in slots 2 through 6, who have nearly identical save percentages. None of these goalies are backups who got lucky for a few months, they are all their team's clear #1 starter. Goalie E played in 53 games while the rest played between 58 and 67.

Let's add in their save percentages on special teams, and the % of the shots faced that came on special teams:

1. Goalie A: .934 EV, .897 ST, 22.6% on ST
2. Goalie B: .931 EV, .931 ST, 18.5% on ST
3. Goalie C: .930 EV, .889 ST, 23.6% on ST
4. Goalie D: .929 EV, .882 ST, 23.8% on ST
5. Goalie E: .929 EV, .888 ST, 20.6% on ST
6. Goalie F: .929 EV, .877 ST, 25.0% on ST
7. Goalie G: .925 EV, .884 ST, 23.5% on ST

Now I think it's pretty obvious who won the Vezina. Not only that, but he also won the Hart Trophy. Goalie B is Jose Theodore, and the year is 2001-02. The other goalies, in order, are Patrick Roy, Sean Burke, Evgeni Nabokov, J.S. Giguere, Roberto Luongo and Dominik Hasek.

Did Theodore deserve his awards that year? That all depends on how you assess his special teams performance, whether you consider it to be a result of his skill or whether it was mostly luck or random chance or the play of his team's PK unit. You can make the argument that Theodore was the most valuable goalie that year because he faced many more shots against than any of the other goalies behind Montreal's porous defence, but there's not too much evidence to suggest that he was better at even strength than the other goalies who had almost identical rates.

I think Theodore had a career year in 2001-02, but that he still was pretty fortunate on special teams. It's possible that Theodore played better in certain game situations than other goalies and therefore was actually the best that year, but I'm not entirely convinced. The other goalies listed above almost all have much better track records than Theodore. I wouldn't say that Theodore's season was one of the weaker Vezina-winning seasons, as finishing second in EV SV% is still an impressive result, but I would say it is the most "smoke and mirrors" .930+ save percentage season that I am aware of.

In fact, I'm pretty sure that early-career Jose Theodore must have been among the luckiest special teams goalies in league history. In Theodore's first four seasons he stopped 89.3% of the shots he faced on the penalty kill. Since then, his PK SV% has dipped to a mere .850.

That might be attributable at least in part to a decline in Theodore's performance or the play of the team in front of him, but his performance on the power play was luckier still. In his first five seasons in the NHL, Jose Theodore stopped 321 out of 333 shots against while his team had the man advantage. That's a .964 save percentage in a situation where the league average was .917. In the year of his Vezina glory in 2001-02, Theodore peaked with a perfect 69-for-69 on power plays.

This year's Vezina is also likely going to be won by a special teams overachiever. Here are the numbers for this year's Vezina nominees:

Ryan Miller: .928 EV, .929 ST, 19.4% ST
Ilya Bryzgalov: .927 EV, .896 ST, 22.0% ST
Martin Brodeur: .924 EV, .873 ST, 15.7% ST

Here are the same statistics for several other goalies, all of whom had EV SV% equal to or better than Miller in 2009-10:

Tomas Vokoun: .937 EV, .873 ST, 19.2% ST
Tuukka Rask: .937 EV, .906 ST, 19.1% ST
Jaroslav Halak: .933 EV, .885 ST, 18.8% ST
Jonas Hiller: .930 EV, .874 ST, 20.5% ST
Henrik Lundqvist: .929 EV, .886 ST, 18.7% ST
Evgeni Nabokov: .928 EV, .900 ST, 22.1% ST
Miikka Kiprusoff: .928 EV, .888 ST, 21.0% ST

I think it was actually a pretty tight race for the top goalie this season, which is interesting since Miller has probably had the Vezina wrapped up since Christmas. I'm very interested to see how Miller does next season. I think it's a pretty safe bet to expect a great deal of regression to the mean in his special teams totals. I'm not saying he's a Jose Theodore, but I'm also not anointing him the best goalie in the world yet either.

This is not to say that special teams play is pure randomness. It can sometimes seem that way over a short sample, for example a playoff series or two, but PK save percentages generally correlate with 5 on 5 save percentages over multi-year samples. The problem is that typically only 1 in 5 shots come on the penalty kill, which means that evaluating a goalie based on single-season penalty kill performance is roughly the equivalent of evaluating a goalie's overall performance based on 12-15 games. As these playoffs have shown, anybody can run hot or cold over a short stretch like that.

In the long run goalies should be rewarded for persistently strong performance on the penalty kill, but results should be viewed skeptically over a single season. We don't want to be awarding the Vezina every year to the goalie who was the luckiest on the penalty kill, but we also should give some credit to a goalie who was excellent while his team was shorthanded. That makes it difficult for analysis.

Streaks

Here are the best and worst 14-game save percentage streaks during the 2009-10 regular season for each of remaining final four teams:

Philadelphia: .936 and .870
Montreal: .943 and .882
Chicago: .937 and .874
San Jose: .944 and .887

And the same thing for seven game series:

Philadelphia: .946 and .857
Montreal: .952 and .871
Chicago: .955 and .844
San Jose: .967 and .853

The similarity in range illustrates again how the margins are very small in goaltending. Over a short sample the skill element of goaltending can be completely lost in the noise of whether the opposing shooters are missing, the puck is hitting him through screens and traffic, or whether he happens to be in peak form or not. Most teams will have at least one streak of seven games or more where their team save percentage is .940 or better, yet analysts are repeatedly stunned when some lesser goalie hits that mark or a star goalie sees his numbers dive to sub-.900 levels over the course of a playoff series.

With the Canadiens' shooting and save percentages taking a clobbering in back-to-back shutout losses to the Flyers, I think we have a new leader in the clubhouse for the luckiest team in the playoffs. With an 11.8% shooting percentage and a .933 save percentage Philadelphia is absolutely rocking the percentages. Take a look at the PDO numbers (shooting percentage plus save percentage) for the teams that are left:

Philadelphia: 105.1
Chicago: 102.2
Montreal: 101.2
San Jose: 98.5

The only way the Flyers can likely compete with the Western champ is if the pucks keep going in and staying out at ridiculous rates. If Michael Leighton can keep his even-strength save percentage in the .980 range then that should probably do it, but unfortunately the winds of chance tend to be fickle, as the Canadiens are finding out. We don't know how the bounces are going to shake out for the rest of the playoffs, but it's probably fair to say that barring some ridiculous streak occurring everything looks lined up for the end of a long Cup drought in Chicago.

Shot Quality and the Montreal Canadiens

I fooled around with tracking shot quality in last year's playoffs, using shot charts to try to get a sense of where teams were shooting from. One of the most interesting sets of numbers that I happened across was at CBS Sportsline, where they track where a shot was targeted on goal (e.g. low glove, low blocker, five-hole, high glove or high blocker).

This data showed that high shots were much more likely to go in than low shots. The problem was that it was very unevenly done. Some games had no high shots recorded at all, others had just a few, while others seemed like they had a more reasonable number. I ended up just filing it away as something to potentially check on if I was trying to assess the shot quality of a particular team or series of games.

A lot of people are trying to explain why the Montreal Canadiens are having so much success this postseason, and one of the factors they typically point to is that the team is forcing their opponents to shoot from the outside. Often people talk about shot quality when they want to try to justify why one team is running hot with the percentages. In the long run teams tend to converge to the average. However, I think it is entirely possible that shot quality could be a factor when one team is playing against a single opponent over the course of a playoff series.

I thought to look at the high/low CBS numbers for Montreal's series against both Pittsburgh and Washington to see what they tell us. If Montreal's skaters are legitimately doing something to impact the other team's shots, then we would expect to see their opponents have a higher percentage of low shots than the Habs. If this is a bunch of talking head garbage, then the percentages are going to be similar for both teams and Montreal was just lucking out.

What do the data tell us?

Montreal vs. Pittsburgh:

High shots: Pittsburgh 45, Montreal 45
Low shots: Pittsburgh 180, Montreal 128

Montreal vs. Washington:

High shots: Washington 73, Montreal 62
Low shots: Washington 216, Montreal 131

Wow. I'd say that at least warrants a closer look.

It's important to note that we have to be careful with this data. It still seems like whoever tracks this stuff sometimes just falls asleep for a period here and an entire game there. For example, in game one of the Pittsburgh series just 2 out of 55 shots were recorded as being high, and in game three against Washington just 3 of 77 shots were marked down as high shots. Several other games were suspect (games 6 and 7 against Washington, games 3 and 5 against Pittsburgh), all of whom had a total of 8 high shots or fewer for both teams.

I'm inclined to throw those games out, and rerun the results.

Montreal vs. Pittsburgh (G2, G4, G6, G7 only):

High shots: Pittsburgh 37, Montreal 39
Low shots: Pittsburgh 116, Montreal 53

Montreal vs. Washington (G1, G2, G3, G5 only):

High shots: Washington 60, Montreal 56
Low shots: Washington 98, Montreal 61

Montreal won 6 out of these 8 games, so these games may not be exactly representative of the two series. However, it is a good sample to try to figure out whether the Habs' defensive tactics were effective. This evidence certainly suggests that they might have been, assuming that this data is good. Whether it was because they had more pressure on the puck, or because they were clogging the shooting lanes, or because they had more counterattack opportunities or odd-man rushes, these numbers are evidence that Montreal's shots may have been of higher quality than those of their opposition, particularly against the Penguins.

Even if that was true, that of course does not mean there was no luck involved. The Habs have benefitted from a healthy dose of good fortune, especially against the Caps. Today's goalies are very good at taking away the bottom of the net, but low shots aren't by any means harmless, especially ones taken close to the net. Think of Halak's pad save on Evgeni Malkin in the third period of game 7, for example. The Habs have survived a bunch of similar chances in the playoffs.

Montreal is not likely to sustain either their shooting percentage or save percentage numbers so far (13.7% on high shots and 10.5% on low shots in the second sample, compared to 9.3% and 6.1% respectively for their opponents). I'd say they should still be the underdog in the Eastern Conference Finals against whoever wins tonight. Yet maybe there is at least something to Jacques Martin's madness after all.

The Age of Goalie Parity

Here are the post-lockout playoff win/loss records of the top 10 goalies in post-lockout regular season save percentage (minimum 200 regular season games played):

Tomas Vokoun: 1-4
Niklas Backstrom: 3-8
Henrik Lundqvist: 14-16
Tim Thomas: 10-8
Martin Brodeur: 15-22
Roberto Luongo: 17-17
Ryan Miller: 22-18
Ilya Bryzgalov: 12-9
Cristobal Huet: 6-11
Miikka Kiprusoff: 9-16

Combined record: 109-129

That's pretty remarkable, the teams with the top-performing goalies on them have been more likely to lose than to win in the playoffs. And that list certainly makes one wonder why Roberto Luongo seems to take more heat for his playoff team success than the rest of the guys combined, even though only three of them have better win/loss records than he does.

As we all know, factors other than goalie skill contribute heavily to determining the winner, but it's also true that the top performers in a population with a small standard deviation in skill (like NHL goalies) will not have much of an advantage over a relatively small sample size of games (like the NHL playoffs). It's quite possible that the worse goalie outperforms the better goalie in a playoff series, and indeed we've seen it happen several times these playoffs. After all, a difference of .005 in save percentage is just 1 goal every 200 shots, which is a typical number of shots a goalie might face in a seven game series. Niemi outplaying Luongo or Boucher outplaying Brodeur is going to happen a lot more often than most people think.

In today's NHL offence wins championships and pretty much all the goalies in the playoffs are decent. The best goaltenders have not been winning championships and it's not going to happen this year either. Sometimes it really is better to be lucky than to be good.

Why Underdogs Teams Are So Gritty

I've thought for a long time that many hockey observers are far too quick to invoke effort as a major reason for the results of a particular hockey game. A lot of the time I don't think there is too much behind it at all, it just becomes a post hoc explanation for why one team's shots went in and the other team's shots did not. It turns out there could be an additional factor influencing this as well, a bias against the favoured team. Here's a quote from an interesting Slate article about why we love underdogs:

"Why is an underdog so attractive? It may have something do with how hard he tries. Vandello showed subjects a video clip of a basketball game between two international teams said to be playing for a championship. One side was described as the 9-to-1 favorite, having won each of 15 previous playoff matches. After viewing the footage, which showed a close game, students were asked to rate the players according to their ability and effort.

As a rule, the underdogs were characterized as having less "talent" and "intelligence" than the favorites but more "hustle" and "heart." That was true even when subjects viewed the same video clip with the labels reversed. It didn't matter what was actually on the screen—which players jumped higher or who dived for the loose balls. The test subjects attributed more effort to whichever team had the underdog label."

I'm sure these results would be no different in hockey, and often explains why teams like the Colorado Avalanche are considered a plucky, gritty, group of warriors while teams like the San Jose Sharks are often described as lazy bunch of wimps. The problem is that the evidence from the games they played against each other showed San Jose heavily outshoting Colorado. When the underdog that is supposedly trying to so hard and giving it their all can't even get the puck away from a team that is allegedly made up entirely of soft players that don't even care, it either indicates the first team is really, really terrible or that the subjective observation is wrong.

It's possible that an underdog team actually does outwork a favourite, but we need to be careful to ensure that it was not just a convenient narrative but something that actually happened. If you are going to insult the professionalism of a group of players, coaches and managers by implying they don't care about winning or that they weren't prepared to play, you need a lot more evidence to support your point than the fact that the goalie on the other team made a lot of saves.

(P.S. What do you think the results would be if a similar video study was done on a group of Canadian hockey fans, where the viewers were told that one team was Russian and the other team was Canadian? Something tells me that would have a pretty significant impact on which team was showing more "hustle".)

Can You Learn How to Be a Winner?

Once upon a time, there was a goalie in the Quebec Major Junior Hockey League who was a highly-touted prospect. He played 35 games as a 16 year old and then attracted a lot of attention with a great year as a 17 year old. He was named the best defensive player in the Q and his team won two playoff rounds before losing in the semifinals. At the end of the season, the goalie's name was the first one called out in the NHL entry draft. Everything looked very promising indeed.

Then this is what happened to him in big games (playoffs and international hockey) over the next 5 seasons:

18 years old: 0-4, 4.47, .904 in a first round loss, plays fairly well at the world juniors but loses in the gold medal game

19 years old: 1-3, 3.31, .886 in a first round loss to a #7 seeded team, let in 6 goals on 30 shots in brief taste of AHL playoffs, scores the game winning goal on himself while stopping 24 of 28 shots in a loss in the gold medal final at the world juniors

20 years old: 0-2, 4.37, .843 in the AHL playoffs, loses starting job to Andy Chiodo

21 years old: 2-3, 3.47, .883 in the AHL playoffs, loses starting job to Dany Sabourin

22 years old: 1-4, 3.76, .880 in first taste of NHL playoff action

That's nothing short of awful, right there. Surely this guy would never be able to accomplish anything in big games, right? Let's continue:

23 years old: .933 save percentage as his team goes to the Stanley Cup Final

24 years old: Stanley Cup champion

I'm sure you have already recognized the goalie as Marc-Andre Fleury. I don't know of any other elite goalie prospect with a worse pre-NHL playoff career than Fleury. People often like to criticize goalies by saying that they have "never won anything"; Fleury not only didn't win anything, but he pretty much actively helped lose everything that he was involved in. Yet today he is considered by many to be one of the game's best pressure performers. I can recall in particular a few exceedingly dumb columns written by sportswriters arguing that Fleury's amazing clutch abilities should have made him the starter for Team Canada over Roberto Luongo in the medal round after the U.S.A. loss, even though their arguments boiled down to little more than "1 Cup > 0 Cups".

If one was to argue that Marc-Andre Fleury is clutch, then it would seem to be difficult to also claim that past clutch performance is a great predictor of future results. This is a sample of one, I recognize that, but the extreme nature of the results at both ends would still be pretty unlikely if goalies are truly consistent in pressure situations. The only way around this would seem to be to claim that everything that takes place before the NHL doesn't count at all, or to claim that something changed with Fleury as he got older, he "learned how to win" or something like that. The natural follow-up question to that would be that if Fleury was able to so easily escape his past failures, what's to stop any other goalie from deciding to do the same? Again, that would not reflect particularly well on the value of using past playoff successes to predict future ones.

No doubt some clever sportswriter could craft some eloquent narrative about how Fleury learned to "play within himself" and realized that "he didn't have to make all the saves, just the important saves", or possibly that he was "mentored by a veteran backup/goalie coach/teammate who taught him the mindset of a champion." I think he just had Crosby and Malkin on his team. We'll see if the Pens can make a third straight run to the Finals, the door appears to be wide open for them and so far they are 5-2 despite Fleury's .889. No doubt that would lead to a lot more veneration of this "modern-day Grant Fuhr".

Special Teams Variance

In the playoffs you never know how a team or a goalie is going to perform on special teams. Call it a streak, a slump, luck, randomness, variance, or whatever you want, guys run hot and cold and just because you're going one way in one series or even one game doesn't mean it will carry into the next.

Ask Jaroslav Halak. After stopping 50 out of 50 shots taken by the Washington Capitals' power play, the league's #1 ranked unit during the regular season, Halak made just 1 save on 5 shots while shorthanded against the Pittsburgh Penguins in game one of their Eastern Conference semifinal.

The range of even strength save percentages among goalies who have played in at least 4 games these playoffs is .878 to .967. If we drop off the outlier on the top end (Brian Boucher's .967) and the bottom end (Brian Elliott's .878), the range is just .894 to .944, which is not very far at all off of the range of results during the regular season (.896 to .937, counting only goalies who played in at least half of their team's games).

Contrast that with the results on the penalty kill so far. They range from Elliott's awful .667 to Halak's aforementioned 1.000. Halak wasn't even the only guy who has been flawless so far on the PK (Tuukka Rask stopped all 23 shots he faced while his team was down a man), and the next guy up from Elliott was Roberto Luongo at just .730. That means you can drop the top and bottom and still get a range of .270. The regular season range was .853 to .919.

We would expect more variance on the penalty kill just because of the sample size, but we should probably also expect an additional spread in results because of the team factors. Since all a team's games come against the same opponent in each round, a good PK unit up against a weak opposing power play should make life much easier on its goalie, and vice versa. Strength of opposition should therefore be more variable than it is during the regular season where the games are spread around against all levels of opposition, albeit with some divisional effects in play.

At even strength goalies face more shots, and they are likely to face shots from a larger group of players, given that teams will run 3 or 4 lines consistently at even strength and then shorten the bench on special teams. This probably helps reduce the randomness.

It's hard to rate goalies based on special teams play. Was Halak unbeatable in round one because he was terrific, or because the Habs power play was doing such a great job of coverage, or because the Washington shooters were just a bit off? Probably elements of all three. Similarly, did Roberto Luongo play poorly on the PK in round one, or was it more a case of his teammates being mediocre and the Kings playing great? Or is it all just one big roulette wheel of puck luck?

The point is that nobody knows what we will see in the next round. Don't bet on the hot hand, because he might already have gone cold, and if you have a skill guy who is shooting blanks (Alex Semin comes immediately to mind) then keep running him out there because pretty soon the tide is going to turn. The trick is for the team to stay alive in the interim, something that Semin's teammates weren't able to do. Only good teams win the Stanley Cup, but they still need to get some bounces and breaks along the way.