The One Stat Argument

From time to time I get feedback that I focus too much on save percentage. The main criticism is usually that it is folly to rely on only one stat, and that a well-rounded analysis should take all the available numbers into account (e.g. GAA, wins, shutouts, games played, etc.).

The one stat criticism sounds pretty reasonable. It seems intuitive that adding more information is a good thing. It is also undeniably true that save percentage does not tell us everything. It is important to know how many shots the goalie faced to be able to assess the level of randomness in his results. We should also take into account how many of those shots came on the penalty kill, and whether there is evidence of any other team factors like shot quality effects. It also seems clear that goalies have some small impact on shots against, and that should be taken into account.

What is usually suggested in place of a save percentage analysis is to take all the different stats, apply a weighting to each of them, and then calculate the final rankings based on all the different inputs. The problem with this approach is that it simply doesn't deliver on its promise. It does not actually measure a lot of different things that a goalie does. What it does is count save percentage 4 or 5 times, and then add in a bunch of irrelevant team factors. This is because every commonly available goalie stat is essentially a different way of restating save percentage and shots against.

Let's go down the list. GAA, as I've pointed out many times, is equal to (1 - save percentage) x shots against per game. It's handy as a shortcut to compare goalies on the same team, but if you're looking at rivals it's much better to try to make sense of the two underlying components than to try to assess their combination.

Shutouts are very obviously driven by a combination of save percentage and shots against, only we are adding in an arbitrary cutoff (why is 0 goals against worth so much more than 1 goal against when teams win 90% of the time when they allow a single goal?) and a healthy dose of small number randomness (shutouts are infrequent, which means luck is often the difference between something like an ordinary 5 shutout season and a great 8 shutout season).

Wins are determined by goals for and goals against. The goalie has almost nothing at all to do with goals for, and goals against are determined by the same two familiar factors of save percentage and shots against.

Goalie's World magazine has a fairly typical "computer ranking" that puts a weighting on each of the traditional goalie stats. On the surface, it looks like it is measuring a lot of different things. However, break down all the stats into their components and it is essentially only tracking three things: Save percentage, shots against per game, and how good the rest of the team is at scoring goals.

Given that only one of those three variables is mainly determined by the goalie, it makes a lot of sense to me to focus on save percentage. That doesn't mean raw save percentage is a perfect stat, because it is not. Step #2 is to make any necessary corrections (e.g. shot quality, special teams adjustments, shot bias) or try to incorporate any other results (e.g. shot prevention). What does the goalie do to win games other than stop pucks? If we know what those things are, then we should focus on them directly.

Maybe a goalie helps his team with his puckhandling and prevents a shot against per game. Preventing one shot is the same as preventing about 0.1 goals, which based on a typical game with around 6 total goals is about 1.7% of the total team contribution. How are we going to see that small effect show up in the overall win total? We might not see it all through the noise of how good the team is at territorial play, how good they are at shooting, how disciplined they are at taking penalties, how good they are at faceoffs, how good their coach is at matching lines, injury luck, shootout performance, etc. However, if I increase the goalie's shots against totals by the estimated amount of shots he prevented, he will get credit for them and that will boost his save percentage.

One other commonly cited justification for relying on wins is that certain goalies help their teams win by performing better in important situations. The problem is that most of the time we can predict quite accurately what the standings will be based on goal differential. According to Alan Ryder, goal differential explains 94% of wins. This suggests that any "clutch win" effect is likely to be either small or influenced by luck.

Maybe over the course of a season a team wins 2 more games than expected based on their goals scored and allowed. Even in the unlikely event that this is entirely because of the clutch skill of the starting goaltender, that's going to be a net result in a similar range as shot prevention (about 0.1 goals per game, based on the standard goal differential that is usually required to produce 2 extra wins). This again would make up a percentage point or two of the total team effort, and just like the shot prevention effect is likely to be hidden from sight among the hundreds of player interactions, coaching decisions and random bounces that determine the outcome of a hockey game.

For goalies that we suspect have an additional effect on team play, I think we are better off determining the effect and using that to adjust the save percentage numbers. It is of course tricky to estimate things like shot prevention, but it is much more tricky and uncertain to try to somehow estimate a goalie's shot prevention effect directly from his GAA. Or to assess the contribution of a goalie based on his career wins number. Both of those are complete guesswork, and far worse alternatives than some of the methods already proposed for dealing with some of the unknowns (e.g. comparing to backup goalies, or estimating the effect by combining our subjective evaluation with our knowledge of general performance ranges for a particular skill).

I focus heavily on save percentage not because save percentage is perfect and tells us absolutely everything, but because I think the other stats just don't bring an awful lot to the table. They are merely different ways of telling you how good a goalie was at stopping the puck and how good his team was at preventing shots.

Shaking off the Rust?

Heading into the Olympics, the league average save percentage in the NHL this season was .912. That put the league on pace for the highest average save percentage since 1970, higher even than the average during the pre-lockout period with its clutching and grabbing and huge goalie equipment. I'd say that's more evidence that the goalies today are better than ever, and that goalie equipment is not as big of a factor as many make it out to be.

Yet even though it has been 11 days since the closing ceremonies, I think some of the goalies in the league must still think they are on vacation. Goalscoring has been up significantly since the NHL resumed, with an average of about 6 goals per game in the 63 games since Crosby's OT winner. The netminders have not been making the stops of late, combining for an average save percentage of a mere .899.

It's possible that the goalies are rusty from having not played for two weeks. It could also theoretically be possible that the jam-packed Olympic year schedule may be having some effect, but as mentioned they were doing pretty well in December, January and February and most of the league's starters should have been able to rest up while they watched the Olympics on TV.

Much gets made about goalie fatigue and whether certain teams are playing some goalies too much. I'm still not exactly sure of the size of the real fatigue effect, but I suspect it is likely not as significant as generally assumed. To me, it seems like going a long time without playing might very well be more likely to result in poor performance than regularly playing every second or third day. I'm reminded of this post from a while back that showed that October was on average the worst month for goalies. It could be that we tend to show too much concern for starting goalies and their perceived heavy workload, and don't properly appreciate the backups who have to stay sharp despite sometimes going weeks without facing shots in game situations.

On the other hand, even some of the goalies who did play in Vancouver are doing poorly, headlined perhaps by gold medalists Roberto Luongo and Martin Brodeur and their matching .869 save percentages in March. Perhaps there is no underlying cause, and this is just a brief, random streak in the long NHL season where offence comes to the fore at the expense of goaltending.

For All the Wrong Reasons

My brother said to me yesterday, "If Canada wins, everyone will think Luongo is a great goalie...for all the wrong reasons."

Sure enough, there are lots of articles out there today about how Roberto Luongo has overcome his demons and gone from choker to clutch in the last six days. Take this one from NHL.com, for example, that is titled "Luongo now has proof that he's among the game's elite."

What was this proof? Was it the fact that Luongo has the highest save percentage of any goalie in the league with 250+ games played since he broke in as a rookie in 1999-00? Is it that he appears headed towards his ninth consecutive year with an even strength save percentage of .925 or better? Perhaps it was 51 career shutouts, despite playing most of his career in the high shots against environment of New York and Florida? Or being named as one of the top 2 goalies for the best hockey-playing country in the world in the last four best-on-best international tournaments? Maybe the journalist was convinced by Mike Babcock, who pointed out last week that Luongo's bank account shows he is one of the highest-paid goalies in the league?

Nope. It was that the highest-scoring team in the NHL era of the Olympics won the gold medal with Luongo in net. It's amazing how one game can have such a big effect on opinion, either for the good (gold medal win) or the bad (game 6 vs. Chicago).

I did think that Luongo coped very well with the intense pressure of a country that expected and demanded the gold medal. He kept saying how much fun he was having, and clearly relished the moment. He also managed to steer clear of the pitfall that caught other top goalies like Martin Brodeur, Miikka Kiprusoff, Evgeni Nabokov and Henrik Lundqvist, that one poor game that helped sink their team's chances of victory. But I don't think Luongo was really a difference-maker during the Olympics, at least not in the same way Ryan Miller and at times Jonas Hiller were.

Luongo's performance in Vancouver was actually pretty reminiscent of Martin Brodeur's play in 2002. Sorry for being the hundredth person to bring you that completely unoriginal thought, but that doesn't make it any less true. They both began as the team's #2 option, they both never really stood on their heads for any particular game, they both received attention for a big save in a 3-2 game (Luongo on Demitra, Brodeur on Brett Hull), and they both might have saved their best game of the tournament until last, where they gave up two goals in the gold medal game against the United States.

The tournament stats definitely share a resemblance:

Brodeur, 2002: 4-0-1, 1.80, .917
Luongo, 2010: 5-0-0, 1.76, .927

For my money, Patrick Roy still owns the best Olympic performance by a Canadian NHL goalie. Despite that, St. Patrick came home without any medal at all. Goaltending can have a big impact in a short tournament, but there is a very good reason why 20 other players also get to bring home the gold.

I'm interested to see if Luongo picks up the same post-Olympic gold medal "bounce" in public perception and award voting that Martin Brodeur did in 2002. No doubt there will still be a small element out there that will keep doubting the Vancouver goalie until his team achieves some more NHL postseason success, but this year's Canucks squad is probably the best team he has ever played on. Luongo has a special opportunity this season to completely change the way the average hockey fan thinks about him. Even if it is for all the wrong reasons.