Why You Shouldn't Bet on Mathieu Garon

Luck is part of goaltending, and every season there are a few goalies that catch lightning in a bottle and end up having career years that are far out of line of their previous results. This presents the analytical problem of determining whether it was a result of random chance or evidence of a change in ability - e.g. is it a breakout performance or a small sample size fluke?

One goalie who had an interesting season last year was Mathieu Garon. Garon wasn't one of the very top goalies in the league, but his numbers were pretty solid on a non-playoff team, he outplayed veteran Dwayne Roloson, and he grabbed attention by dominating in the shootout. Can Edmonton expect more of the same in 2008-09?

Hockey Numbers posts a save percentage breakdown by difficulty of scoring chances that gives some insight into whether a goalie is lucky or not. Garon ranked 9th in save percentage on easy chances by stopping 98% of them (97% is average) and 6th in save percentage against difficult chances with 72% (average is about 68%). However, he ranked just 30th in save percentage against medium chances (.889, below the league average of .897), the chances that make up the majority of a goalie's workload and are the most representative of his skill level.

So that tends to suggest that Garon got lucky by avoiding bad goals and made possibly more than his share of big saves. Some of the most difficult shots a goalie faces is on the power play, so the next place we need to look at is his situational numbers:

2007-08: .922 at ES, .919 on the PK, .913 on the PP

That .919 leaps off the page as unsustainable. Garon's career averages before last year were .920 at even-strength and .840 on the PK, which makes it look even more out of whack. Garon was actually below average in even-strength save percentage in 2007-08. However, his penalty kill contribution was worth about 10 extra goals compared to league average. If Garon had stopped PK shots at his previous career rate his save percentage would have been at just .897 overall last season.

One place Garon may continue to provide high value is in shootouts. He was a ridiculous 30 out of 32 last season, which is unlikely to be repeated, but he did well in the previous two seasons, stopping a combined 20 out of 26 shots in 2005-06 and 2006-07. Garon will probably come back to earth somewhat, but he appears to be a strong shootout goalie and gives Edmonton a good chance of taking home some extra bonus points this season.

Overall, though, unless we see a Huet-type career trajectory of becoming a very good goalie late in his career, I think Garon was just playing over his head last season. Since Garon was traded for Huet, we can compare them in a couple of different team situations, and the stats suggest that Garon is no Huet.

Montreal: Garon, 2003-04: 2.27, .921; Huet, 2005-06: 2.20, .929
Los Angeles: Huet, 2003-04: 2.43, .907; Garon, 2005-06: 3.22, .894

An improved Edmonton team could mitigate some of the expected fall in Garon's numbers, but I'd expect Garon to drop to about the .905 range in 2008-09.

Greatest of All-Time?

The "Brodeur is the Greatest" argument pops up again in a top 30 goalies list on NHL Home Ice. I expect this to be a taste of things to come in a season where Brodeur will likely break the records for most career regular season wins and career regular season shutouts, two records that are traditionally weighted heavily in the greatest goalie debate. Far be it from me to criticize a list that rates Tom Barrasso ahead of Frank Brimsek, Mike Vernon ahead of Turk Broda, and Chris Osgood ahead of Roberto Luongo, but I obviously disagree with the #1 ranking.

Actually, I've seen worse lists. The top half is not too bad, other than the guy at #1: the goalies he has from 2-5 are the same guys in my top 4, just in a different order. I think the wheels come off from #16-#30, but at that point the margins really do start to narrow.

Wins And Plus/Minus

Plus/minus is a bit of a controversial stat. Certainly it needs to be treated with care, because of issues such as quality of opposition, team strength, empty net goals, etc. However, there are many hockey fans who would prefer to throw it out entirely for these reasons.

What is interesting to me is that many of these same people will rely heavily on goaltender wins to evaluate a goalie. Wins are determined by goals for and goals against, so goalie wins measures whether there are more goals for or goals against when a goalie is in the game. That is conceptually similar to plus/minus, the only difference being that a goalie really only has an impact on results at one end of the ice, while a player can go both ways.

So why do many hockey fans rely heavily on goaltender wins to evaluate a goalie, which is basically a measure of the goals for and against while a goalie is in the game, and yet completely reject plus/minus, a measure (however imperfect) of the goals for and against while a player is in the game? Doesn't make sense to me.

The Continuing Shots Against Investigation

I compared shots per game averages for starters to backup goalies during the 2007-08 season, and was a bit surprised to find that over half the league had a substantial difference in shots faced averages between the #1 and #2 goalies. Sixteen out of the 30 teams had a difference of 1.5 shots per game or better between the starter and the backup, with a few more teams just missing the cutoff at 1.3 or 1.4. So having a sizeable shot differential was not rare but commonplace around the league last season.

I decided to look at just the teams with a substantial shot differential, using 1.5 as a cutoff, trying to isolate any variables that could explain the differences. I broke it down first by starter vs. backup (again using the goalie with most games played as the starter):

Starter: 29.3 SA/60, 26.8 SA/60 5 on 5, 43.2 SA/60 4 on 5
Backup: 29.3 SA/60, 26.1 SA/60 5 on 5, 41.1 SA/60 4 on 5

I also looked at two different measures of shot quality, both from Behind the Net and Hockey Numbers. Behind the Net gives the expected save percentage for each goalie based on the shots they faced. Hockey Numbers gives only the shot-quality neutral save percentage, but we can calculate the expected save percentage from that number. Behind the Net had starters and backups almost equal at .908 and .907 respectively. Hockey Numbers had a bit more of a gap, with starters facing slightly easier shots (.907 expected save percentage compared to .903 for the backups).

Other than the penalty kill discrepancy noted earlier, there isn't much difference there.
I then broke it down into the goalies with high shots against averages compared to those with lower shots against averages.

High shots: 30.7 SA/60, 27.9 SA/60 5 on 5, 42.6 SA/60 4 on 5
Low shots: 28.0 SA/60, 25.1 SA/60 5 on 5, 41.1 SA/60 4 on 5

There is a significant even-strength gap that is driving the results. Let's look at the shot-quality numbers for this group:

High shots: .908 (BtN), .908 (HN)
Low shots: .908 (BtN), .904 (HN)

The goalies who were facing more shots per game were not facing more difficult shots. Again, this does not imply that the extra shots they were facing were rebound shots or other prime scoring chances.

However, those extra chances did have an effect on goals against. The GAAs for the two groups were almost identical, although it was actually the goalies facing more shots that had a slightly lower GAA (2.78 to 2.80). They did much better at stopping the puck by both shot-quality measurements (.30 better in Behind the Net's GAA above average, and .006 better in SQNSV%), but the extra shots they faced resulted in more goals against.

There were two other things I looked at: shots for, and missed shots. Surprisingly to me, the backups had more shots for than the starters, and the goalies facing fewer shots against had more shots for than the goalies facing more shots. Starters had just 25.9 shots for/60 at even-strength compared to 26.7 for their backups, and the teammates of the high shot goalies took 26.1 shots/60 compared to 26.4 for the low shot goalies.

The starters and the backups were very close in missed shots, both at even-strength (10.9 and 11.1) and at 4 on 5 (18.8 and 18.7 respectively). In the high vs. low shot comparison, missed shots were very close at even-strength (10.8 to 10.9), but there was a bit of a gap at 4 on 5 (18.2 to 18.8).

I looked at the numbers for 2006-07 for high shot vs. low shot goalies, and they confirmed the basic principles: the goalies facing more shots tended to have better save percentages (.918 to .914 at even-strength, unadjusted), but the GAAs were almost identical on average (2.89 and 2.88).

So nothing really jumps out as a cause for the shot discrepancies. That these differences were found league-wide makes me think that a lot of it might be random noise, and it is unclear how much of the gap is repeatable from season to season. That is the key question: did the guy who is outperforming his teammates in terms of SA/60 also do it last year, and is he likely to do it next year? I think it is still very difficult to tell which goalie is allowing fewer shots than the other. Try this little quiz: I put together a list of 10 goalie tandems who had a difference of 1.5 or greater in their shots against averages in 2007-08. Guess which of the two goalies allowed fewer shots per game. If you score better than 50% you are outperforming me (I'll post the answers in the comments):

Cam Ward or John Grahame?
Ryan Miller or Jocelyn Thibault?
Tim Thomas or Alex Auld?
Chris Osgood or Dominik Hasek?
Niklas Backstrom or Josh Harding?
Carey Price or Cristobal Huet?
Chris Mason or Dan Ellis?
Martin Gerber or Ray Emery?
Marc-Andre Fleury or Ty Conklin?
Olaf Kolzig or Brent Johnson?

Vanbiesbrouck vs. Vernon

John Vanbiesbrouck and Mike Vernon were two of the more successful goalies of the "pre-Hasek" period (late 1980s/early 1990s). Vernon is probably better known and considered a better goalie by most, mainly because of his 2 Stanley Cups. He even gets a lot of Hall of Fame talk because he is remembered for those championship runs. But this comparison is actually a completely mismatch in favour of John Vanbiesbrouck.

Between the ages of 24 and 34, John Vanbiesbrouck had a better save percentage than Mike Vernon in 10 out of 11 seasons. This was despite playing on a worse team probably every single year in the span.

Vanbiesbrouck won the 1985-86 Vezina, and he had a dominating season in 1993-94 (.924 save percentage compared to a league average of .895) that was Vezina-worthy but unfortunately coincided with the emergence of Dominik Hasek. In his career, Vanbiesbrouck collected a total of 61 Vezina votes and finished in the top 6 seven times. Mike Vernon only finished in the top 6 in Vezina voting three times in his entire career, peaking at 2nd in 1988-89, and was voted for just 32 times.

Because of the strength of his teams, Mike Vernon played in nearly twice as many playoff games over his career, 138 to Beezer's 71. He won a lot more of them as well. However, the two goalies had identical playoff GAAs (2.68), and Vanbiesbrouck had a much better playoff save percentage (.915 to Vernon's .896). Vernon faced an average of just 25.8 shots per game in his playoff career, while Vanbiesbrouck faced 31.4. Mike Vernon won a Conn Smythe Trophy, but Vanbiesbrouck's goaltending in the Florida Panthers' 1996 Cup Final run surpasses anything Vernon did in his playoff career.

Mike Vernon's biggest advantage throughout his career was that he faced very few shots per game (26.9 regular season, 25.8 playoffs). Considering how much of that came in the 1980s, he really did benefit from exceptionally strong team defence in front of him. I don't recall Vernon as being an accomplished puckhandler or having exceptional rebound control or any other quality that could have had a major impact on those totals. It seems to be the case that he just played on some great defensive teams, which meant there were a lot of winning efforts despite Vernon's shortcomings in goal.

Mike Vernon played most of his career on great teams and had a lot of team success, but he really was a very average goalie. John Vanbiesbrouck was a very good goalie who played on average teams.

Shots Against on the Penalty Kill

My pro-Kari Lehtonen post resulted in a bit of a discussion about shot prevention. One interesting fact that came to light was that Lehtonen faced far more shots on the penalty kill than his backups did. I decided to look around the league to see if this result was consistent.

It turns out that that did appear to be the case for the 2007-08 season. Behind the Net gives breakdowns of shots against per 60 minutes at 5 on 5, 4 on 5, and 5 on 4 situations. At 5 on 5, there was little difference between starters and backups (starters faced 26.1 shots per 60, backups 25.9). While at 4 on 5, however, the starters faced 41.6 shots per 60 compared to 40.4 for the backups.

Another interesting result was that the backups' teams actually averaged more shots for per 60 minutes (26.0 compared to 25.6 for the starting goalies).

Just to be clear, there was a wide degree of variability across the teams, and the numbers above are averages. It certainly wasn't the case that every starter faced more shots on the PK than their backups did. Also, my definition of the starter was simply the guy with the most minutes, so there were some teams in a platoon situation that had a "backup" who was either as good as or better than the starting goalie.

I ran the numbers again for just teams that had what I determined to be a clear #1 starter who played the majority of the games at a high level, and compared their numbers again to the backups. The penalty kill gap remained (41.8 to 40.6), the good goalies again faced slightly more shots against at 5 on 5 (26.5 to 26.2), and the shots for totals were very close (25.5 to 25.6).

I did a different filter and looked at just goalies who played at least 65 games. Strength of schedule would be expected to come into play here, since some of the backups played very few games which were likely against weaker than average teams. However, the heavy workload goalies had both better "shot support" 26.2 to 25.7 and faced fewer shots 5 on 5 (25.3 to 25.7). The PK results were again similar (40.9 for starters, 38.6 for backups).

I checked the 2006-07 data, and the starters did again face more shots while down a man, although it was much closer (46.9 to 46.7). At 5 on 5, the shots were virtually identical (29.1 for starters, 29.2 for backups).

We need to look at a few more seasons to be sure, but it looks possible that better goalies face more shots when on the penalty kill. If so, I am not sure if this would be because of a difference in penalty killing tactics or simply that the other team figures they need to shoot more often against a better goalie. Also, with respect to the starters vs. backups shot totals debate, I think the evidence tends to suggest that in general there is no significant difference between #1 and #2 goalies in terms of shots against per game.

The Most Underrated Goalie in the League

For my money, the most underrated goalie in the NHL is Kari Lehtonen. He is yet another example of a very good goalie getting overlooked because of the poor team around him.

Lehtonen has gone 75-61-14 on Atlanta over the past 4 seasons, with an impressive .913 save percentage and a 2.82 GAA. That is much better than his backups have done (.890, 3.31). His save percentage is trending upwards - .906 in 2005-06, .912 in 2006-07, and .916 in 2007-08 even though the team took a substantial step backwards (although their defensive play probably improved). Hockey Numbers had him a solid 9th in the league this past season shot-quality neutral save percentage, and in 2006-07 Alan Ryder ranked Lehtonen as the 3rd most valuable goalie in the league behind only Luongo and Brodeur.

Lehtonen has posted very consistent even-strength save percentages - .930, .926 and .927 over the past three seasons, according to NHL.com. That level of even-strength performance puts him in a class with Kiprusoff, Giguere, Thomas, Luongo, Lundqvist, Vokoun and Brodeur, i.e. the league's elite goalies.

Given those impressive numbers, it is curious how poorly he is rated around the league. The Hockey News did a ranking of the top 30 goalies last spring, and gave Lehtonen the final spot in their ranking, #30, behind every other starting goalie in the league. There was also a recent series of polls on HFBoards asking fans to rank the starting goalies, and Lehtonen came out in just 20th place. I think Lehtonen is probably a top-10 goalie in the league today.

Atlanta will probably be a weak team again this season, but given his recent history and young age (24), expect Lehtonen to continue his strong play and gradually gain more recognition as one of the top young goalies in the league.