Why You Shouldn't Bet on Mathieu Garon

Luck is part of goaltending, and every season there are a few goalies that catch lightning in a bottle and end up having career years that are far out of line of their previous results. This presents the analytical problem of determining whether it was a result of random chance or evidence of a change in ability - e.g. is it a breakout performance or a small sample size fluke?

One goalie who had an interesting season last year was Mathieu Garon. Garon wasn't one of the very top goalies in the league, but his numbers were pretty solid on a non-playoff team, he outplayed veteran Dwayne Roloson, and he grabbed attention by dominating in the shootout. Can Edmonton expect more of the same in 2008-09?

Hockey Numbers posts a save percentage breakdown by difficulty of scoring chances that gives some insight into whether a goalie is lucky or not. Garon ranked 9th in save percentage on easy chances by stopping 98% of them (97% is average) and 6th in save percentage against difficult chances with 72% (average is about 68%). However, he ranked just 30th in save percentage against medium chances (.889, below the league average of .897), the chances that make up the majority of a goalie's workload and are the most representative of his skill level.

So that tends to suggest that Garon got lucky by avoiding bad goals and made possibly more than his share of big saves. Some of the most difficult shots a goalie faces is on the power play, so the next place we need to look at is his situational numbers:

2007-08: .922 at ES, .919 on the PK, .913 on the PP

That .919 leaps off the page as unsustainable. Garon's career averages before last year were .920 at even-strength and .840 on the PK, which makes it look even more out of whack. Garon was actually below average in even-strength save percentage in 2007-08. However, his penalty kill contribution was worth about 10 extra goals compared to league average. If Garon had stopped PK shots at his previous career rate his save percentage would have been at just .897 overall last season.

One place Garon may continue to provide high value is in shootouts. He was a ridiculous 30 out of 32 last season, which is unlikely to be repeated, but he did well in the previous two seasons, stopping a combined 20 out of 26 shots in 2005-06 and 2006-07. Garon will probably come back to earth somewhat, but he appears to be a strong shootout goalie and gives Edmonton a good chance of taking home some extra bonus points this season.

Overall, though, unless we see a Huet-type career trajectory of becoming a very good goalie late in his career, I think Garon was just playing over his head last season. Since Garon was traded for Huet, we can compare them in a couple of different team situations, and the stats suggest that Garon is no Huet.

Montreal: Garon, 2003-04: 2.27, .921; Huet, 2005-06: 2.20, .929
Los Angeles: Huet, 2003-04: 2.43, .907; Garon, 2005-06: 3.22, .894

An improved Edmonton team could mitigate some of the expected fall in Garon's numbers, but I'd expect Garon to drop to about the .905 range in 2008-09.

Greatest of All-Time?

The "Brodeur is the Greatest" argument pops up again in a top 30 goalies list on NHL Home Ice. I expect this to be a taste of things to come in a season where Brodeur will likely break the records for most career regular season wins and career regular season shutouts, two records that are traditionally weighted heavily in the greatest goalie debate. Far be it from me to criticize a list that rates Tom Barrasso ahead of Frank Brimsek, Mike Vernon ahead of Turk Broda, and Chris Osgood ahead of Roberto Luongo, but I obviously disagree with the #1 ranking.

Actually, I've seen worse lists. The top half is not too bad, other than the guy at #1: the goalies he has from 2-5 are the same guys in my top 4, just in a different order. I think the wheels come off from #16-#30, but at that point the margins really do start to narrow.

Wins And Plus/Minus

Plus/minus is a bit of a controversial stat. Certainly it needs to be treated with care, because of issues such as quality of opposition, team strength, empty net goals, etc. However, there are many hockey fans who would prefer to throw it out entirely for these reasons.

What is interesting to me is that many of these same people will rely heavily on goaltender wins to evaluate a goalie. Wins are determined by goals for and goals against, so goalie wins measures whether there are more goals for or goals against when a goalie is in the game. That is conceptually similar to plus/minus, the only difference being that a goalie really only has an impact on results at one end of the ice, while a player can go both ways.

So why do many hockey fans rely heavily on goaltender wins to evaluate a goalie, which is basically a measure of the goals for and against while a goalie is in the game, and yet completely reject plus/minus, a measure (however imperfect) of the goals for and against while a player is in the game? Doesn't make sense to me.

The Continuing Shots Against Investigation

I compared shots per game averages for starters to backup goalies during the 2007-08 season, and was a bit surprised to find that over half the league had a substantial difference in shots faced averages between the #1 and #2 goalies. Sixteen out of the 30 teams had a difference of 1.5 shots per game or better between the starter and the backup, with a few more teams just missing the cutoff at 1.3 or 1.4. So having a sizeable shot differential was not rare but commonplace around the league last season.

I decided to look at just the teams with a substantial shot differential, using 1.5 as a cutoff, trying to isolate any variables that could explain the differences. I broke it down first by starter vs. backup (again using the goalie with most games played as the starter):

Starter: 29.3 SA/60, 26.8 SA/60 5 on 5, 43.2 SA/60 4 on 5
Backup: 29.3 SA/60, 26.1 SA/60 5 on 5, 41.1 SA/60 4 on 5

I also looked at two different measures of shot quality, both from Behind the Net and Hockey Numbers. Behind the Net gives the expected save percentage for each goalie based on the shots they faced. Hockey Numbers gives only the shot-quality neutral save percentage, but we can calculate the expected save percentage from that number. Behind the Net had starters and backups almost equal at .908 and .907 respectively. Hockey Numbers had a bit more of a gap, with starters facing slightly easier shots (.907 expected save percentage compared to .903 for the backups).

Other than the penalty kill discrepancy noted earlier, there isn't much difference there.
I then broke it down into the goalies with high shots against averages compared to those with lower shots against averages.

High shots: 30.7 SA/60, 27.9 SA/60 5 on 5, 42.6 SA/60 4 on 5
Low shots: 28.0 SA/60, 25.1 SA/60 5 on 5, 41.1 SA/60 4 on 5

There is a significant even-strength gap that is driving the results. Let's look at the shot-quality numbers for this group:

High shots: .908 (BtN), .908 (HN)
Low shots: .908 (BtN), .904 (HN)

The goalies who were facing more shots per game were not facing more difficult shots. Again, this does not imply that the extra shots they were facing were rebound shots or other prime scoring chances.

However, those extra chances did have an effect on goals against. The GAAs for the two groups were almost identical, although it was actually the goalies facing more shots that had a slightly lower GAA (2.78 to 2.80). They did much better at stopping the puck by both shot-quality measurements (.30 better in Behind the Net's GAA above average, and .006 better in SQNSV%), but the extra shots they faced resulted in more goals against.

There were two other things I looked at: shots for, and missed shots. Surprisingly to me, the backups had more shots for than the starters, and the goalies facing fewer shots against had more shots for than the goalies facing more shots. Starters had just 25.9 shots for/60 at even-strength compared to 26.7 for their backups, and the teammates of the high shot goalies took 26.1 shots/60 compared to 26.4 for the low shot goalies.

The starters and the backups were very close in missed shots, both at even-strength (10.9 and 11.1) and at 4 on 5 (18.8 and 18.7 respectively). In the high vs. low shot comparison, missed shots were very close at even-strength (10.8 to 10.9), but there was a bit of a gap at 4 on 5 (18.2 to 18.8).

I looked at the numbers for 2006-07 for high shot vs. low shot goalies, and they confirmed the basic principles: the goalies facing more shots tended to have better save percentages (.918 to .914 at even-strength, unadjusted), but the GAAs were almost identical on average (2.89 and 2.88).

So nothing really jumps out as a cause for the shot discrepancies. That these differences were found league-wide makes me think that a lot of it might be random noise, and it is unclear how much of the gap is repeatable from season to season. That is the key question: did the guy who is outperforming his teammates in terms of SA/60 also do it last year, and is he likely to do it next year? I think it is still very difficult to tell which goalie is allowing fewer shots than the other. Try this little quiz: I put together a list of 10 goalie tandems who had a difference of 1.5 or greater in their shots against averages in 2007-08. Guess which of the two goalies allowed fewer shots per game. If you score better than 50% you are outperforming me (I'll post the answers in the comments):

Cam Ward or John Grahame?
Ryan Miller or Jocelyn Thibault?
Tim Thomas or Alex Auld?
Chris Osgood or Dominik Hasek?
Niklas Backstrom or Josh Harding?
Carey Price or Cristobal Huet?
Chris Mason or Dan Ellis?
Martin Gerber or Ray Emery?
Marc-Andre Fleury or Ty Conklin?
Olaf Kolzig or Brent Johnson?

Vanbiesbrouck vs. Vernon

John Vanbiesbrouck and Mike Vernon were two of the more successful goalies of the "pre-Hasek" period (late 1980s/early 1990s). Vernon is probably better known and considered a better goalie by most, mainly because of his 2 Stanley Cups. He even gets a lot of Hall of Fame talk because he is remembered for those championship runs. But this comparison is actually a completely mismatch in favour of John Vanbiesbrouck.

Between the ages of 24 and 34, John Vanbiesbrouck had a better save percentage than Mike Vernon in 10 out of 11 seasons. This was despite playing on a worse team probably every single year in the span.

Vanbiesbrouck won the 1985-86 Vezina, and he had a dominating season in 1993-94 (.924 save percentage compared to a league average of .895) that was Vezina-worthy but unfortunately coincided with the emergence of Dominik Hasek. In his career, Vanbiesbrouck collected a total of 61 Vezina votes and finished in the top 6 seven times. Mike Vernon only finished in the top 6 in Vezina voting three times in his entire career, peaking at 2nd in 1988-89, and was voted for just 32 times.

Because of the strength of his teams, Mike Vernon played in nearly twice as many playoff games over his career, 138 to Beezer's 71. He won a lot more of them as well. However, the two goalies had identical playoff GAAs (2.68), and Vanbiesbrouck had a much better playoff save percentage (.915 to Vernon's .896). Vernon faced an average of just 25.8 shots per game in his playoff career, while Vanbiesbrouck faced 31.4. Mike Vernon won a Conn Smythe Trophy, but Vanbiesbrouck's goaltending in the Florida Panthers' 1996 Cup Final run surpasses anything Vernon did in his playoff career.

Mike Vernon's biggest advantage throughout his career was that he faced very few shots per game (26.9 regular season, 25.8 playoffs). Considering how much of that came in the 1980s, he really did benefit from exceptionally strong team defence in front of him. I don't recall Vernon as being an accomplished puckhandler or having exceptional rebound control or any other quality that could have had a major impact on those totals. It seems to be the case that he just played on some great defensive teams, which meant there were a lot of winning efforts despite Vernon's shortcomings in goal.

Mike Vernon played most of his career on great teams and had a lot of team success, but he really was a very average goalie. John Vanbiesbrouck was a very good goalie who played on average teams.

Shots Against on the Penalty Kill

My pro-Kari Lehtonen post resulted in a bit of a discussion about shot prevention. One interesting fact that came to light was that Lehtonen faced far more shots on the penalty kill than his backups did. I decided to look around the league to see if this result was consistent.

It turns out that that did appear to be the case for the 2007-08 season. Behind the Net gives breakdowns of shots against per 60 minutes at 5 on 5, 4 on 5, and 5 on 4 situations. At 5 on 5, there was little difference between starters and backups (starters faced 26.1 shots per 60, backups 25.9). While at 4 on 5, however, the starters faced 41.6 shots per 60 compared to 40.4 for the backups.

Another interesting result was that the backups' teams actually averaged more shots for per 60 minutes (26.0 compared to 25.6 for the starting goalies).

Just to be clear, there was a wide degree of variability across the teams, and the numbers above are averages. It certainly wasn't the case that every starter faced more shots on the PK than their backups did. Also, my definition of the starter was simply the guy with the most minutes, so there were some teams in a platoon situation that had a "backup" who was either as good as or better than the starting goalie.

I ran the numbers again for just teams that had what I determined to be a clear #1 starter who played the majority of the games at a high level, and compared their numbers again to the backups. The penalty kill gap remained (41.8 to 40.6), the good goalies again faced slightly more shots against at 5 on 5 (26.5 to 26.2), and the shots for totals were very close (25.5 to 25.6).

I did a different filter and looked at just goalies who played at least 65 games. Strength of schedule would be expected to come into play here, since some of the backups played very few games which were likely against weaker than average teams. However, the heavy workload goalies had both better "shot support" 26.2 to 25.7 and faced fewer shots 5 on 5 (25.3 to 25.7). The PK results were again similar (40.9 for starters, 38.6 for backups).

I checked the 2006-07 data, and the starters did again face more shots while down a man, although it was much closer (46.9 to 46.7). At 5 on 5, the shots were virtually identical (29.1 for starters, 29.2 for backups).

We need to look at a few more seasons to be sure, but it looks possible that better goalies face more shots when on the penalty kill. If so, I am not sure if this would be because of a difference in penalty killing tactics or simply that the other team figures they need to shoot more often against a better goalie. Also, with respect to the starters vs. backups shot totals debate, I think the evidence tends to suggest that in general there is no significant difference between #1 and #2 goalies in terms of shots against per game.

The Most Underrated Goalie in the League

For my money, the most underrated goalie in the NHL is Kari Lehtonen. He is yet another example of a very good goalie getting overlooked because of the poor team around him.

Lehtonen has gone 75-61-14 on Atlanta over the past 4 seasons, with an impressive .913 save percentage and a 2.82 GAA. That is much better than his backups have done (.890, 3.31). His save percentage is trending upwards - .906 in 2005-06, .912 in 2006-07, and .916 in 2007-08 even though the team took a substantial step backwards (although their defensive play probably improved). Hockey Numbers had him a solid 9th in the league this past season shot-quality neutral save percentage, and in 2006-07 Alan Ryder ranked Lehtonen as the 3rd most valuable goalie in the league behind only Luongo and Brodeur.

Lehtonen has posted very consistent even-strength save percentages - .930, .926 and .927 over the past three seasons, according to NHL.com. That level of even-strength performance puts him in a class with Kiprusoff, Giguere, Thomas, Luongo, Lundqvist, Vokoun and Brodeur, i.e. the league's elite goalies.

Given those impressive numbers, it is curious how poorly he is rated around the league. The Hockey News did a ranking of the top 30 goalies last spring, and gave Lehtonen the final spot in their ranking, #30, behind every other starting goalie in the league. There was also a recent series of polls on HFBoards asking fans to rank the starting goalies, and Lehtonen came out in just 20th place. I think Lehtonen is probably a top-10 goalie in the league today.

Atlanta will probably be a weak team again this season, but given his recent history and young age (24), expect Lehtonen to continue his strong play and gradually gain more recognition as one of the top young goalies in the league.

Why Aren't There More Brodeurs?

One of the pro-Brodeur arguments is that if it is so easy to rack up huge numbers on a great defensive team, why don't lots of goalies do it? Brodeur's career wins and shutouts will soon surpass every other goalie who has ever played in the NHL. So in a sense Brodeur is a rare breed. Having said that, however, what exactly is rare about him? Does he possess rare goaltending skill? Has he played for a uniquely dominant team? Does he possess any unique characteristics, such as perhaps longevity, loyalty, etc.?

Certainly a goalie's situation is a major factor. Wins and losses are determined by two components - goals for and goals against. Goalies have little impact on goals for, and goals against are largely determined by the number and quality of scoring chances the goalie has to face. Even the best goalies can't singlehandedly save a bad team - Dominik Hasek, Patrick Roy, Jacques Plante, and Glenn Hall have all missed the playoffs in their careers. Without playing on a good team, a goalie will have a much more difficult time making it among the leaders in GAA or piling up a lot of wins and shutouts.

With only a handful of great defensive teams in the league in any given season, the odds are long that any given goalie will end up drawing one of these choice assignments. And even if they do get there, they have to be good enough and durable enough to play lots of games to be able to rack up the huge counting numbers. They also have to stay with that team, which is both a combination of the player being loyal to the franchise as well as the franchise remaining loyal to the player.

So which teams had this type of situation? The two key indicators of strong defensive teams are low goals against and shots against totals. Here are the top 10 teams in goals against during the Dead Puck Era that Brodeur spent most of his career in (say, 1995-96 to 2003-04 to keep league effects roughly constant), with their average shots per game totals in brackets:

1. New Jersey (24.7)
2. Dallas (25.4)
3. Philadelphia (25.0)
4. Detroit (26.3)
5. Colorado (28.0)
6. Buffalo (29.5)
7. St. Louis (24.7)
8. Ottawa (26.0)
9. Washington (27.8)
10. Montreal (29.7)

The effect of really good goaltending becomes obvious in the cases of Colorado and Buffalo. The most similar teams to New Jersey are Dallas, Philadelphia, Detroit, St. Louis and Ottawa. Detroit had the most wins with 431, while New Jersey had 400, Dallas 388, Philadelphia 384, St. Louis 367 and Ottawa 350. All of those teams had seasons when their goaltending was league average or worse in terms of save percentage, so clearly a decent goaltender who started most of the games for any of those franchises would end up "winning" a lot of games.

Here are the total goalie stats for all the teams:

DAL: .596 win %, 2.27 GAA, .910 save %
PHI: .600 win %, 2.27 GAA, .908 save %
DET: .650 win %, 2.32 GAA, .911 save %
STL: .566 win %, 2.44 GAA, .900 save %
OTT: .576 win %, 2.37 GAA, .906 save %*

(*-I chose not to include 1995-96 numbers for Ottawa. The Sens were still an awful expansion franchise and that one season really skews their numbers for the period. Now it is true that if Brodeur broke in with Ottawa he would have had to suffer through the painful early years, but since we are dealing here with "right time, right place" what-ifs let's just say the Ottawa goalie would start in 1996-97).

What were Brodeur's numbers? .626, 2.13, .913. So, expressed in rate stats, he was just slightly better than the goaltending received by Dallas, Philadelphia and Detroit in those years. However, this is comparing Brodeur to all the goalies on the other teams. If another team was going to copy New Jersey's goalie-handling style, they would rely heavily on their starting goalie. So comparing him to the starters only is a fairer comparison.

If you take all the starting goalies (definition: the goalie that played the most minutes in each season) for each of the strong defensive teams, and pro-rate their numbers to Brodeur's level of minutes, here is what you get:

Dallas: 344 W, 60 SO, 2.17, .912
Detroit: 379 W, 54 SO, 2.33, .911
St. Louis: 324 W, 43 SO, 2.42, .903
Ottawa: 311 W, 62 SO, 2.34, .907
Philadelphia: 355 W, 64 SO, 2.13, .913

Brodeur: 355 W, 69 SO, 2.13, .913

Compare those last two lines: Philadelphia and New Jersey had basically identical starting goaltending, the Devils just got more of it. If Philadelphia had played their starter for 70-75 games per season for that entire period and got a similar level of production, they would have matched what New Jersey got out of Brodeur. Now it is probably unrealistic to expect exactly the same performance level with an additional 15-20 starts per season, but I'm not convinced that goalie performance tends to drop off significantly with extra starts (a previous post dealt with this issue). Furthermore, Philadelphia's backups generally did worse than New Jersey's did, yet only one Flyer goalie cracked the 60 game mark in this span. The breakdown:

New Jersey: 649 games for starters at .913, 129 games with backups at .904
Philadelphia: 469 games for starters at .913, 316 games with backups at .901

It looks like New Jersey and Philadelphia simply had differing philosophies on how to handle goaltenders. I think there is little doubt that New Jersey's method is better for both the team as a whole as well as the starting goalie's counting numbers.

St. Louis (Fuhr/Turek/Osgood) and Ottawa (Rhodes/Tugnutt/Lalime) had bad goaltending more often than not, so a good goalie playing on those teams would have likely have surpassed the above numbers. Roman Turek's 1999-00 season (1.95 GAA, 42 wins, 7 SO) and Patrick Lalime's 2002-03 campaign (2.16 GAA, 39 wins, 8 SO) show the kind of numbers a strong goalie could have been expected to put up every season playing behind Pronger/MacInnis or Redden/Chara.

St. Louis in particular was really a wasted opportunity for a star goalie - the Blues allowed the fewest shots of anybody except for New Jersey during the period, but fouled it up by trotting out a line of subpar netminders. If Mike Keenan didn't run Curtis Joseph out of St. Louis in 1995, Joseph could have stayed and won 350+ games over those 9 seasons for the Blues. And in that scenario, as long as he had at least one or two good playoff runs in there, Cujo might be mentioned today in the same breath as Hasek, Roy, and Brodeur in all the best goalie debates. Of course, Joseph would probably still have bolted for a big payday with his home-town Maple Leafs, which just underscores again the rarity of Brodeur being a one-team guy for his entire career.

So was Brodeur really such a special goaltender? The overall quality of his performance was not particularly unique, as the rate stats of the starters for other good defensive teams were very similar to Brodeur, although on the whole Brodeur was certainly a good goaltender with a net positive effect on his team. However, Brodeur really sets himself apart in terms of the quantity of his performances.

Brodeur's career win totals are largely a function of his high number of games played. If Brodeur played a typical starter's workload of 55-60 games per season, he would have likely ended up with 20-25% fewer career wins. His workload was partly a reward for his play, but Brodeur still far surpassed other goalies around the league, even goalies on similar calibre defensive teams who were putting up very similar numbers. Philadelphia, Dallas, Detroit, St. Louis or Ottawa would have produced a goalie with Brodeur-type numbers if they had acquired a decent goalie and given him a heavy workload for a decade or so.

The results show that a strong defence does not necessarily ensure team success or great goaltending (e.g. Ottawa and St. Louis). That Brodeur provided stable enough goaltending to allow a great team to win over a long period of time is certainly a plus, and provides significant value over replacement, even if Brodeur's performance sometimes was barely even above average. Brodeur has never scuttled his team's season like other goalies have, and his performances in the playoffs have been generally good. There have been a few dismal playoff performances in there (including even arguably costing his team the 2001 Stanley Cup), but that is typical of nearly all goalies with a similar length of tenure. Playoff series are short and a lot of luck and randomness is involved. I have probably underrated Brodeur in the past by failing to fully value this type of contribution. Having said that, Brodeur benefits from playing a lot of games, something I tend to believe is more a measure of opportunity than talent, which allows him extra chances to rack up wins and shutouts.

When you allow less than 25 shots per game, you don't need a great goalie to win. But you can still lose with poor goaltending. Brodeur helped prevent the Devils from suffering the repeated playoff failures of the Blues or Senators (Brodeur vs. Lalime was the #1 reason Ottawa did not win the Cup in 2003). On the other hand, he would have never approached his success if he played in somewhere like Long Island or Florida. It is likely that any goalie would have done well in New Jersey between 1994 and 2008 (just look at the backup goalies), but it is unlikely that most other goalies would have been able to stay there that long without getting traded or replaced. The difference between Martin Brodeur and a guy like Patrick Lalime is that Brodeur is a much better goalie, because Lalime had every opportunity to succeed but didn't. But I think the difference between Brodeur and someone like Curtis Joseph or Ed Belfour still comes down more to team factors than individual ones. If you want to rank Brodeur ahead of those guys because he actually did accomplish all those things, while they only hypothetically could have, I don't have a problem with that. But Brodeur is in many ways more of a Joseph with better teammates or a Belfour with more games played than someone who is up there with Roy and Hasek and the greatest who have ever donned the pads.

So in summary, the main reasons for Brodeur's success can be outlined as follows:

1. He is a good goalie, mostly above-average throughout his career and at times great
2. He was drafted by the premier defensive team in the league
3. He played the most games of any goalie in the league
4. He remained a one-team guy, allowing him to play nearly his entire career behind a great defence
5. He played in a low-scoring era, which makes his numbers look better in an all-time context

The $6.25 Million Man

Evaluating goalies is very important, because you don't want to end up paying superstar dollars to a guy who is an average-level talent. Which is exactly the mistake the Sabres made over the off-season when they resigned Ryan Miller to a 5-year, $31.25 million deal.

Miller has a reputation as one of the league's better goalies, but the perception isn't quite in tune with reality. Since the lockout, Ryan Miller is 5th among all goalies in wins, but tied for 23rd in shutouts, 18th in GAA, and 18th in save percentage.

In the playoffs over the same period, Miller has the most games played and the most wins, but he is just 10th in GAA and 11th in save percentage, as well as tied for 12th in shutouts.

What about his team? The Sabres score a lot of goals, does that mean they often leave their goalie out to dry? Apparently not. Alan Ryder has measured the Sabres to be a better-than-average team in terms of shot quality in each of the last three seasons. According to his marginal goals from goaltending measurement (which includes shot quality), Buffalo's goaltending ranked 17th in 2006, 11th in 2007, and 24th in 2008.

Although they regressed last year, the Sabres have generally been a strong team in front of Miller. With their backup goalies in the net, Buffalo is 38-20-6 over the last three seasons, for a .640 winning percentage. Miller's win percentage? .635.

Ryan Miller has done little in his career to prove that he is anything other than an average starting goalie. The Buffalo Sabres overpaid substantially for his services, probably mostly as a result of his high win totals. Having said that, there is a possibility that the 28-year old Miller could improve over the next 5 years, but he would have to improve quite significantly to earn his hefty paycheque.

The Unknown Who Outplayed Sawchuk in His Prime

We've had a heated debate lately in the comments about wins and a goaltender's contribution to a winning team. I thought it would be interesting to take a look back at one of my more interesting findings since I started doing this blog, because it is another case of matching a goalie with a poor record against a goalie with a great record and trying to identify whether the second guy is actually better or not.

Most hockey fans know who Terry Sawchuk was, but few of them know who Al Rollins was. I have posted about Rollins before because he came out very well in my studies comparing goalies to their backups, and I wanted to revisit his case. I have received a fair amount of feedback about that type of study, and a number of concerns have been raised. However, this is one of those situations where a comparison is very easy to make since we can avoid the major drawbacks of the method. Both Sawchuk and Rollins were preceded by Harry Lumley and succeeded by Glenn Hall, two Hall of Fame goalies, so by restricting the comparison to how just those two other goalies did on both teams we are truly comparing apples to apples. Also I took only full seasons played for all the goalies in the sample to avoid any potential strength of schedule advantages for part-time backup goalies. I isolated a time frame (1947-48 to 1957-58) that was both a period of futility for Chicago and a period of excellence for Detroit, so it makes a good point of comparison with fairly constant team effects. I believe this is therefore a good test of which goalie was a bigger difference maker. I have six full seasons for Sawchuk in Detroit, five full seasons for Rollins in Chicago, five full seasons of Lumley/Hall in Detroit and three full seasons of Lumley/Hall in Chicago, which gives us a good bit of data to work with.

So what do the numbers say? Here are how our two goalies made out:

Al Rollins in Chicago: 81-171-56, 17 SO, 3.03, .354 win %
Terry Sawchuk in Detroit: 224-107-77, 59 SO, 2.11, .643 win %

Looks pretty decisive in favour of Sawchuk, doesn't it? That is, until we bring in a bit of context:

Lumley & Hall in Detroit: 165-97-61, 36 SO, 2.30, .605 win %
Lumley & Hall in Chicago: 53-124-26, 12 SO, 3.39, .325 win %

Now we just have to compare the numbers to see which goalie was more dominant compared to his peers, Rollins or Sawchuk.

Rollins: -0.36 GAA, +.029 win %
Sawchuk: -0.19 GAA, +.038 win %

In that comparison, I'd take Rollins. His GAA outperformance is almost twice as large as Sawchuk's. This was partly because he faced more shots per game, however Rollins allowed 11% fewer goals per game than Lumley/Hall, while Sawchuk allowed 8% fewer. Sawchuk did win a higher percentage of games, but Sawchuk had slightly higher goal support than his peers (2.96 per game for Sawchuk, 2.92 for Lumley/Hall) while Rollins had lesser goal support (2.25 compared to 2.34 for Lumley/Hall). In addition, the farther a team gets away from .500, the more difficult it is to be a difference-maker. For example, on a team that scores 8 goals a game or 0 goals a game, a great goalie will have the same record as a terrible one. But on an average team that plays a lot of one-goal games, a great goalie will have more chances to change the result. The other goalies were farther under .500 in Chicago than they were above .500 in Detroit, so Rollins' opportunity to impact the results was probably not as great as Sawchuk's. Even being conservative, I don't think we can say Sawchuk was any better than Rollins in that period. On the other hand, the Red Wings were certainly better than the Black Hawks.

I think Terry Sawchuk's peak is overrated, and his numbers had a good deal to do with the team he was playing on. Sawchuk was more of a consistently good goalie than somebody who was ever really dominant, except maybe in 1950-51 and 1951-52. Other than those two seasons he was never the best goalie in the league. Rollins, on the other hand, only had six full seasons as a starting goalie, but not much of a playoff career (such was the reality of playing on Chicago, although Rollins was a Stanley Cup winner with Toronto in 1950-51). He had to deal with the difficult goalie competition of the Original Six era, and was likely overlooked in his post-Chicago days because of the lack of team success in his early career. Rollins is famous for winning a Hart Trophy in 1954, an award decision that likely was as much a make-up decision for his 1952-53 season and playoffs as an award that was earned in that season, but otherwise he attracted very little recognition and was never named a season-end All-Star. Rollins isn't an all-time great, but merely a goalie who made the most of a less than ideal team situation and for a brief period of time played as well as one of the most celebrated goalies of all-time in his prime.

What Makes A Great Goalie?

"Terry Sawchuk has been unanimously considered the greatest goaltender ever, but Patrick Roy has surpassed everything that he did. In the four Stanley Cups that Roy won, he was the Conn Smythe Trophy winner three times. When his team needed him, he was a difference-maker. He was a superstar in longevity, wins and championships, and that's why he'll be known as the greatest."
Darren Pang, May 28, 2003

I am continually fascinated that only career numbers and team success seem to matter to a lot of people when ranking goalies, even to former NHL goalies like Darren Pang. Notice that Pang never mentioned MVP voting, Vezina Trophies, First Team All-Stars, save percentage, GAA, or the number of times Roy led the league in various categories. The only things he focused on were career length (longevity) and team success (wins and championships).

If players were ranked according to similar criteria, Mario Lemieux and Bobby Orr would be replaced by Henri Richard and Red Kelly in the all-time rankings lists. All kinds of other weird results would follow, like Scott Stevens (1635 games, 3 Cups, 1 Conn Smythe) being better than Ray Bourque (1612 games, 1 Cup, 0 Conn Smythes), Mark Messier (1756 games, 6 Cups, 1 Conn Smythe) ranking ahead of Wayne Gretzky (1487 games, 4 Cups, 2 Conn Smythes), and Ron Francis (1798 career points, 2 Cups) coming out ahead of Bobby Hull (1170 career points, 1 Cup).

There is simply a different standard for goalies. It becomes even more frustrating when the same person who argues that Brodeur was better then Hasek because he was more durable and more reliable will turn around and take Bobby Orr over Gordie Howe and Eddie Shore over Ray Bourque.

The excess focus on goalie longevity makes career records far more prized than single-season records among goalies. Terry Sawchuk's 103 career shutouts is one of the most famous records in hockey, but I doubt many people could identify George Hainsworth as the single season record-holder, much less how many shutouts he had that one season (22). On the other hand, ask someone to name Gretzky's career goal total and then his single-season goal scoring record, and far more people will get the second one right than the first.

This has led to the perception that Martin Brodeur will be some kind of Gretzky-like record breaker (this view is illustrated by a commentor on NJ.com Sports who claims that "Marty will have shattered every goaltending record known to man"). Actually, Martin Brodeur will almost certainly never set or hold any record for either goals against average or save percentage, including career, seasonal, playoff career, playoff season, or number of times leading the league in either one. Even if you look at just results in the modern era, Brodeur would still be shut out of all of the above.

Out of the dozens of major goalie records out there Brodeur will likely end up holding 7 of them, 6 of which are for either wins or shutouts and the last one for the most regular season games played. That is still impressive, to be sure, but implying that Brodeur is completely wiping the slate clean of everyone else past or present is way overstating it.

It doesn't make any sense to take into account both peak and career when evaluating players, but then rely exclusively on longevity and team success to rank goalies. Both peak performance and longevity should be factored in and team success should be secondary to individual performance to arrive at the best possible ranking for both players and goalies.

Did Ftorek and Robinson Affect Brodeur?

There was an interesting comment posted recently by Bruce in the ongoing goalie fatigue debate that I thought deserved a closer look:

As one who doesn't put all my faith in a single column of goaltending statistics, what it indicates to me is that the Devils of 1998-2002 were a different team than the notorious "trapping" teams of the Lemaire years. The Devils of Robbie Ftorek and Larry Robinson played more offensively, took more chances, gave up more chances, and relied on their world-class goalie to limit the damage....But for those three seasons in they were a puck-possession, outscoring type of team. Such teams may or may not allow bunches of shots, but in my observation they do tend to give up a better quality of scoring opportunity when their players are tending towards the other end of the rink. One predictable result of this altered team philosophy is a lower Sv%, and Brodeur's were indeed merely "average" throughout those years.

Sounds plausible. However, all evidence indicates that it is nevertheless wrong. Brodeur's numbers did not appear to suffer from a team that took more chances. Here are his even-strength save percentage numbers from 1998-99 to 2003-04 (along with the league average):

1998-99: .915 (.916)
1999-00: .912 (.912)
2000-01: .919 (.914)
2001-02: .917 (.916)
2002-03: .921 (.918)
2003-04: .924 (.922)

That looks like the same guy doing exactly the same thing. Brodeur's performance relative to the rest of the league was very similar throughout the entire period. The only thing changing is that the league average is gradually rising. If his team was taking a lot more chances between 1998-99 and 2001-02, you would expect Brodeur to do worse relative to the league in that period, and better in 2002-03 and 2003-04 when the Devils were by all accounts a defensive powerhouse.

The main reason for the drop in Brodeur's save percentage under Ftorek and Robinson appears to be his poor play on the penalty kill, a situation that hardly seems to be affected by a team's offensive philosophy. Here are Brodeur's PK numbers (again compared to league average):

1998-99: .864 (.872)
1999-00: .908 (.866)
2000-01: .839 (.862)
2001-02: .849 (.872)
2002-03: .866 (.869)
2003-04: .878 (.867)

Note the particularly poor results in 2000-01 and 2001-02. It seems pretty unfair to blame Brodeur's team for it either - New Jersey's penalty killers were Stevens, Niedermayer, Madden and Pandolfo. In 2002-03, even with a below-average penalty kill save percentage from Brodeur, New Jersey led the league in penalty kill efficiency.

The Devils scored a lot of goals from 1998-99 to 2000-01 (their offence ranked 2nd, 2nd, and 1st in the league). But they still were excellent at preventing shots (allowing the 4th, 4th, and 2nd fewest shots in the league). It might have been a little bit more challenging to play goal for New Jersey in this period because of an increased offensive focus, but it seems unlikely that there was a large difference. Brodeur's even-strength results are pretty consistent through the period, and his save percentage variances can be explained by his performance on the penalty kill.