Two of the Closest Series Ever

Last night's overtime victory by Washington completed what was likely the closest series in NHL history in terms of score.  Not only was every game decided by a single goal, but 99.4% of the entire series was played with a goal differential of one or less.  The only time either team managed to take a two goal lead over the seven games was when Jay Beagle scored in the second period of game five to put the Caps up 2-0, a margin that lasted less than three minutes before the Bruins cut it to 2-1.

The West had a matchup that was also a strong contender for the title of closest ever right up until the final period of the series.  Phoenix and Chicago went to overtime in each of the first five games of the series, and by the time of the second intermission in game six had spent 99.3% of their series tied or with one of the teams leading by a lone goal.  However, in that final period Phoenix finally broke through and pulled away for a decisive 4-0 win.

I thought it would be interesting to look at numbers broken down by score in these two series to see why they were so tight.

Boston vs. Washington:

Washington Up One:
Shots:  Washington 30, Boston 72
Goals:  Washington 1, Boston 7
Save %:  Holtby .903, Thomas .967

Score Tied:
Shots:  Washington 145, Boston 162
Goals:  Washington 11, Boston 7
Save %:  Holtby .957, Thomas .924

Boston Up One:
Shots:  Washington 32, Boston 11
Goals:  Washington 4, Boston 0
Save %:  Holtby 1.000, Thomas .875

There were major score effects during the series, as would be expected.  Including the brief period when Boston was down by 2, the trailing team outshot the team in the lead by the whopping margin of 107-41 and scored 12 goals in just 141.7 minutes of play for an amazing rate of 5.08 goals per 60 minutes.  The leading team managed just one goal (0.42 per 60), scoring on just 2.4% of their shots taken.

It is typical that the trailing team generates more shot attempts, but usually their percentages drop as they put more pucks on the net and take more risks, leaving themselves open to chances going back the other way.  For whatever reason both the Caps and Bruins were able to get away with almost everything in their own end when pushing for the tie.

The overall shot statistics flatter the Bruins a bit because they spent more time trailing.  Boston outshot Washington by 20% overall but by just 12% with the game tied.  It was still a very close series, and Washington was pushed over the top by an impressive effort from rookie netminder Braden Holtby.

The Chicago-Phoenix series was similarly close in terms of scores, but the underlying numbers suggest that balance of play wasn't nearly as tight.

Phoenix vs. Chicago:

Phoenix Up Two Goals or More:
Shots:  Phoenix 11, Chicago 11
Goals:  Phoenix 2, Chicago 1
Save %:  Smith .909, Crawford .818

Phoenix Up One:
Shots:  Phoenix 40, Chicago 76
Goals:  Phoenix 2, Chicago 5
Save %:  Smith .934, Crawford .950

Game Tied:
Shots:  Phoenix 73, Chicago 124
Goals:  Phoenix 9, Chicago 6
Save %:  Smith .952, Crawford .877

Chicago Up One:
Shots:  Phoenix 35, Chicago 30
Goals:  Phoenix 4, Chicago 0
Save %:  Smith 1.000, Crawford .886

In this series the Blackhawks outshot the Coyotes by 51% overall, and an impressive 70% with the game tied, yet somehow managed to get outscored 17-12 over the course of six games, as well 9-6 in tie-game situations.  The strong goaltending of Mike Smith compared to the inconsistent play of Corey Crawford was the decisive factor in the series.

Smith was particularly strong in game six, a game where everything was massively tilted in favour of Chicago except for the scoreboard.  I also found it interesting that Smith had the highest save percentage of any of the four goaltenders while his team held a one goal lead, which was perhaps unexpected given that the Blackhawks fought back three times to tie the game late in the third period.  Several posters in this HFBoards thread specifically downgrade Smith's first-round performance because of his supposed lack of clutch play in allowing late game-tying goals, but to me that's making the classic mistake of evaluating playoff performance:  Letting a few memorable events have too much influence while failing to properly appreciate the larger picture.

For the sake of comparison, Braden Holtby gave up a lead in five out of seven games, including twice in the third period, yet nobody would call him unclutch.  Allowing goals against in the final 10 seconds obviously has a huge negative impact on a team's win probability, but when 99% of the series is within one goal then every goal against has a major impact on the chance of victory.  Other than one third period in Chicago, pretty much every situation these four goalies have faced so far in the 2012 playoffs was a clutch situation.

Just as with Washington and Boston, the trailing team rode both a large outshooting advantage and insanely high percentages.  Overall, shots were 122-81 in favour of the team playing catch-up, with turned into a 10-4 advantage on the scoreboard.  That works out to a rate of 3.16 goals per minute for the trailing team, compared to just 1.26 for the team holding on to a lead.

In some respects it is amazing that Chicago even came as close to winning the series as they did, given that Phoenix's scoring rate per shot was over twice as high.  The repeated late-game comebacks to force OT kept them in it, but unfortunately the goaltending disparity was perhaps never more apparent than in extra time where Smith managed a .923 save percentage while Crawford put up a mere .813 and let in two very soft goals to help the Coyotes to the franchise's first second round appearance since 1987.

It would be interesting to see detailed scoring chance numbers or a breakdown of odd-man rush chances to see whether the leading teams were generating chances and just missing their shots, or whether their defensive focus meant they were not creating many dangerous opportunities to score.  In the absence of compelling evidence, however, I'd guess it was mainly a tremendous run of hot goaltending that kept the scores close (in both series combined, goalies on teams trailing by one stopped 97.3% of the shots against) and created 13 games of razor-thin margins in these two memorable series.

Sitting on a Lead Isn't Working in 2012

Fifteen games in, the story of the 2012 playoffs so far has been third-period comebacks. Twelve out of 15 games have featured a one goal lead at some point in the third period, and in seven of those twelve games the trailing team fought back to tie. Five times the team that was trailing eventually ended up winning the game, four times in overtime and once in regulation (the Flyers' roller coaster 8-5 win over the Penguins on Friday night).

I would have guessed that the trailing teams were riding very high percentages to be able to put up those kind of results, but that's not actually the case. Both the leading and trailing teams so far have shot 8.0% in the third period with one team up a goal, although the leading teams' numbers are slightly inflated because of empty netters (the shooting percentage with a goalie in the net for teams up by one is 6.1%). The primary reason for all the comebacks is that the trailing teams have been absolutely dominant in terms of possession, outshooting the opposition 88-49. That is a rate of 39-22 per 60 minutes of play.

It is normal for teams to play to the score. According to Behind the Net's stats, the Nashville Predators were the only team in the league this season that did not have over 50% of the Fenwick events while trailing by a goal, and teams down by one had an outshooting advantage of 29.6 to 24.5 per 60 minutes. Those numbers include results from the whole game, and would likely show a greater split for third period results only (some of my past findings indicate that the shot rate skews even more in the third period alone).

Despite generating more shots, however, the trailing teams were actually outscored during the regular season (2.54 to 2.43 per 60 minutes), which indicates that playing to the score was the right move for teams holding a late lead. It probably should be noted though that in a regular season sample team strength might also be a factor, given that the better teams would be more likely to spend time in the lead than the league doormats, whereas the playoffs do not have the same broad range of team quality.

The regular season results give a trailing team shooting percentage of 8.2%, very close to the 2012 playoff numbers. During the regular season the numbers for the leading team went up to 10.4%, which indicates that the comeback teams in this year's playoffs have probably been a bit lucky or had their comebacks enabled by strong goaltending in the third period. That said, even if the leading teams' percentages rise as the playoffs go on, there will still be a lot of comebacks as long as the shot ratio remains in the vicinity of 2-to-1 in favour of the trailing teams.

Utilizing a collapsing defence to preserve a late lead is the best strategy in some cases, but some teams (Pittsburgh and Phoenix, perhaps?) are probably currently reviewing their game tape to determine whether their late-game play is doing more to help them than to hurt them. You can trade quantity for quality in terms of chances against, but if the quantity gets too high than some pucks are going to go in anyway and the strategy is no longer optimal. It will be interesting to see whether teams can improve their ability to hang on to late leads or whether the comebacks will continue in 2012.

One of These Does Not Belong...

Goalie	Sv%	Max	Min	Avg	PO Tms
M. Smith	.930	.974	.851	.926	.936
J. Quick	.929	.963	.881	.927	.930
J. Halak	.926	.963	.856	.931	.937
P. Rinne	.923	.960	.880	.921	.919
J. Howard	.920	.940	.881	.919	.919
R. Luongo	.919	.949	.869	.920	.930
A. Niemi	.915	.950	.863	.914	.915
C. Crawford	.903	.931	.867	.901	.896

Sv% is total save percentage, Max and Min are each goalie's best save percentage marks over seven consecutive games during the 2011-12 season, Avg is the goalie's average save percentage per 7 game stretch (naturally, this metric will track very closely with the overall save rate), and PO Tms is each goalie's combined save percentage against the other 7 Western Conference playoff teams.

The obvious takeaway is that there is some really good puckstoppers out West, with six teams possessing demonstrably above-average goaltending. That leaves just the San Jose Sharks with the very average Antti Niemi, and the Chicago Blackhawks, who look to be the one team in the Western Conference that is in a substantial amount of trouble in goal heading into the playoffs.

In a previous post, I pointed out that a typical Cup winning goaltender needs to put together a streak of approximately .930 over 600 shots. This season Crawford barely even managed to hit that during his best seven game stretch. His highest mark over 20 consecutive games was only .912. If the playoff version of Corey Crawford is the same one that showed up during the regular season, it is very unlikely that Chicago will get the percentages needed to overcome four straight difficult opponents and end up with a Stanley Cup. Chicago did manage to win the 2010 Cup despite Antti Niemi's mediocre .910 save percentage, but most observers would agree that version of the Hawks with multiple star players on ELCs was stacked compared to this year's roster.

That said, all hope is not lost for Blackhawks fans. Despite Crawford's mediocre play, Chicago still managed to go 13-10-3 against the other seven Western Conference playoff teams with him in net. That was still the second-worst record of any of the eight expected playoff starters (ahead of only Niemi), but it was not far behind most of the other teams as only Smith and Luongo managed to post a win percentage of .600 or better against that tough slate of opposition. Chicago has a very good team in front of Crawford, especially if captain Jonathan Toews is able to get back into the lineup and contribute at his usual level. In score-close Fenwick, one of the best measures of overall team strength, Chicago ranked 5th in the league.

The other thing to keep in mind is that Crawford acquitted himself quite well in the first round against Vancouver last year and had a much better overall season in 2010-11. Perhaps Crawford's true ability is closer to the .918 he managed in the 2010-11 regular season and playoffs combined than this year's .903, which would make him far more likely to be able to deliver Cup-calibre goaltending over the next two months.

For what it is worth (probably not much), Crawford finished the season strong with a .921 and an 8-2-1 record over the last two months, including six wins over playoff teams, although Chicago's offensive output of 32 goals in the 11 games and stingy shot prevention of 23.5 SA/60 obviously also played a major role in that recent success.

The Western Conference playoffs are as usual going to be a grind filled with tight games where hot goaltending can make a difference. The Blackhawks have a strong team that could be a contender, except for the question mark related to the fact that their goaltending this season has been well behind the rest of the teams in the mix out West. It seems safe to say that the pressure is on firmly on the shoulders of Corey Crawford because Chicago will likely need much better goaltending than they have been given so far in 2011-12 to be able to put together another Cup run.

Why Lundqvist Should Win the Vezina

Even Strength Save Percentage in 2011-12 by Division:

1. Northwest, .925

2. Pacific, .925

3. Central, .923

4. Northeast, .919

5. Southeast, .917

6. Atlantic, .916

Quick and Lundqvist ended up with almost identical situational save percentage stats (both .933 at even strength, with Quick just slightly ahead on the penalty kill .908 to .905). The Kings faced a higher percentage of shots against on the PK, which caused Lundqvist to slightly edge out Quick in terms of overall save percentage by the narrow margin of .001. The closeness of those results, combined with Quick's extra workload and the Kings' late-season playoff charge, has made Quick a recent trendy Vezina pick, an impressive comeback given that it looked like Lundqvist had the award sewn up by the All-Star break.

Assuming both goalies were competing in the same environment, it would indeed be very difficult to separate them by the numbers. However, if each goalie's numbers are adjusted relative to their individual conferences, then Lundqvist opens up a decisive edge over Quick:

Conference-adjusted situational save percentages:

Quick: .930 EV, .907 PK, .941 PP

Lundqvist: .936 EV, .906 PK, .974 PP

Multiplying those out by the league-wide average frequency of shots against in each situation, Lundqvist ends up with a conference and situationally adjusted save percentage of .932 compared to .927 for Quick.

Another option would be to adjust each goalie's numbers relative to their division, although in that case the much smaller samples means it would be important to remove each goalie's results from the overall numbers (i.e. the Pacific without Quick and the Atlantic minus Lundqvist). That calculation only increases the margin in favour of Lundqvist, given that his .933 looks much more impressive when stacked up against the combined .912 put up by the rest of the Atlantic division, with the other netminders in the Pacific still combining for an above-average .924 mark even without the contributions of that division's Vezina candidate. Using the divisional numbers and the same correction for shots against by situation, Lundqvist's adjusted number moves well ahead of Quick .935 to .927.

To me, anything close to a tie suggests Lundqvist should win because his elite track record means that it is much less likely that his terrific season was based on luck or other secondary factors. I would pick Lundqvist even assuming that both goalies faced identical shots against for this reason alone. Adjusting for the east/west disparity only makes the choice that much more obvious, in my opinion.

Relying on historical records to evaluate a single season of goaltending is somewhat unfair to the goalie who is four years younger, but it is simply the reality of dealing with uncertainty in goalie evaluation (and why I don't think single-season awards are all that significant). This was Lundqvist's third season in a row at .920 or better while Quick's previous career best was .918. Future years may well prove that Quick's true talent is in the mid- to upper-.920s, but as of right now that's probably not the smart way to bet.

Another relevant piece of statistical info is that Quick's numbers were much better at home (.936 at home compared to .922 on the road), whereas Lundqvist's numbers were higher away from home (.934) than at MSG (.926). That doesn't necessarily mean much, variance is naturally going to be higher over 900 shot samples than over full-season results, but since road numbers are counted by a variety of different scorers they are less likely to be subject to bias. It is also worth noting because both goalies have been pretty consistent on the road over the last three seasons, with Lundqvist maintaining a steady .011-.013 gap over Quick:

Quick: .915, .916, .922

Lundqvist: .926, .929, .934

This year Quick's home numbers shot up while Lundqvist's improved slightly. Again that could be reflective of improved play for both goalies, but it could also be partially related to shot counting or team factors. The other interesting thing to note is that this is the third straight year that Lundqvist's save numbers have been better on the road. MSG is known for poor stat recording in general and it is possible that his numbers are being at least partially suppressed at home, especially given that Lundqvist faced an average of 5.6 more shots against per 60 minutes on the road compared to at home in 2011-12, coming on the heels of a 6.0 difference in 2010-11.

On the other hand, Quick's shots against rates were 26.5 at home compared to 28.0 on the road in 2011-12. His shots against rate at home has actually increased in each of the last three seasons, at the same time as his road shots against rate has been continually decreasing. That's not the typical statistical profile of a goalie being disadvantaged by his home scorekeeper. Overall, the home/road numbers are just one more reason to be a bit more confident in the Swede than in the American.

Taking their histories into account along with the conference disparity, I think Henrik Lundqvist deserves to win his first best goalie award. It could be argued that the shot quality allowed by the Rangers was not typical of the rest of the Atlantic Division, or that goaltending in the Western Conference in general or the Pacific Division specifically was simply a whole lot better than it was out East, either of which would mean that the adjustments above are unfair to Quick. I would certainly listen to anybody willing to make those arguments, but right now I don't see enough supporting evidence on the table. Shot quality arguments are always particularly murky because of the lack of good data, and subjective comparisons are very difficult, particularly for two teams in opposing conferences.

As an aside, this season has made me wonder at times whether we can continue to rely on the general assumption that EV shot quality is relatively constant between teams. The two major pieces of evidence in that direction are probably St. Louis' 2011-12 goalie stats and the conference splits displayed above. In the aggregate I think 5 on 5 shot quality is still probably not all that important for most of the league, but if there are some significant effects on the margins that would be important to know for goalie evaluation.

Ilya Bryzgalov and Regression

Ilya Bryzgalov, first 27 games of 2011-12: .890

Ilya Bryzgalov, next 27 games of 2011-12: .929

Ilya Bryzgalov, last 27 games of 2008-09: .898

Ilya Bryzgalov, first 27 games of 2009-10: .925

It may still be too soon to proclaim that Ilya Bryzgalov is back in the kind of form that earned him a $51 million contract in Philadelphia, but the results have certainly turned around as of late for the Flyers' quirky netminder (8-1-1, 1.27, .955 so far in March).

As I pointed out in my post on Jon Quick, things that happen at the start of the season always seem to take on much greater significance. And when you have a slump in your first year in a new city that is notoriously tough on its goaltenders after signing a big-money contract in the off season and then end up increasing your exposure through a starring turn in HBO's 24/7 series, then any kind of early season struggles make it dead easy for sportswriters to start crafting narratives. There are all kinds of places to assign blame, from Bryzgalov's weak playoffs last year to his attitude and work ethic after signing his massive contract to his mental toughness and the pressure of the Philly market. It's possible to come up with all kinds of theories about the situation because while the performances were not up to par nobody knew for sure why exactly that was the case.

Is it possible that one of those things was affecting Bryzgalov? Probably, maybe even several of them at the same time. Insiders like Elliotte Friedman are most likely on the right track with some of the possible explanations for Bryzgalov's early-season play. Bryzgalov did look awful at times in games, worse than he did even when going through tough stretches in Phoenix, and it seems to be fairly unanimous within the team that he was struggling to cope with his greatly increased level of public exposure. At the same time, however, it is important to keep in mind the likely role of chance and variance in the goalie's results. As shown above, he has posted similar splits over a similar number of games before, even without the specific factors affecting his current season in Philly.

When a talented goalie goes into a slump, it can seem like they will never get out again. It's not always entirely bad luck, often they have to work on a few issues with their game or they have some off-ice problems, but with enough training and time those things are usually overcome and the goalie ends up returning to his elite form.

That's why regression to the mean is always the way to bet. If it's simply a matter of luck, then that is going to even out over time as shooters will stop making their shots at an unusual rate. If the goalie has issues to fix, then in most cases he will put in the time and effort to deal with them. At the other end of the scale, if a goalie's style of play is unusually effective then the rest of the league will do their homework on them to try to take advantage of any weaknesses and drag them back towards the norm.

Some have criticized stats guys' seemingly relentless focus on regression as removing the human variable and treating players like robots, but I think it is actually a natural consequence of human behaviour. When competition is involved, everyone copies everyone else and unsustainable advantages get erased fairly quickly. Those who excel may not always keep up the effort needed to separate them from the pack, while those who lag behind will often increase their intensity to try to make up ground. This is probably also one of the reasons why it is difficult to find persistent team advantages in terms of shot quality in today's NHL. A new system might initially work wonders but not once everyone else figures it out. In the long run, that often means there is not a lot of separation between the pack, especially when looking at a job like NHL starting goalie which only employs the top 30 best in the world at their craft.

Extended slumps by top level goaltenders are not a rare phenomenon. Sometimes they appear to have an obvious cause and other times they do not. Pretty much every goalie will struggle through them over the course of their pro careers. A particularly good example is when Henrik Lundqvist, whose consistently elite track record stacks up among the best in the game, had a 33 game stretch at .888 in the middle of the 2007-08 season that ended with him finishing third in Vezina voting. Dominik Hasek also found himself below .900 twenty games into the 1997-98 season that concluded with a second straight Hart Trophy. The Dominator recorded an identical .945 save percentage in the twenty regular season games both immediately prior to and immediately following those brief early-season struggles.

My philosophy is to deal with large sample sizes and bet on track record rather than recent form. Others prefer a "what-have-you-done-for-me-lately" approach that tries to take into account all kinds of other factors such as a goalie's mental state, his recent technical performance or the way the rest of the team has been performing. Both methods will get some right and some wrong, but I think the historical evidence suggests that betting on established talent over recent form is a far more successful method (just read Arctic Ice Hockey, Gabe Desjardins has been beating that drum for years). Ellen Etchingham also recently provided a great read on the subject of variance.

Even if there was someone out there who could identify all of Bryzgalov's early-season technical flaws and could see between his ears well enough to perfectly measure how the pressure of his new environment would impact his play, they still might not do any better at predicting his next 27 games than somebody who simply knows that Bryzgalov is a .915 career goalie on 10,000+ shots. All Bryzgalov may need to do is tweak a few things in his game, change his mental approach, or have his team restructure some aspect of his environment and it might be enough to reset everything and put him back to his usual self, leaving that detailed analysis of his struggles pretty much obsolete.

I don't mean to oversimplify the difficulty of fixing holes that have developed in your game, that is not always an easy process or something that happens overnight. And if you are the Philadelphia goalie coach, then obviously nothing is more important than diagnosing exactly what is going awry. But for proven talent, things usually end up working out in the long run, or at least until they hit the inevitable decline because of age or injuries.

Bryzgalov is unquestionably a proven NHL talent, but while he has quieted the critics for the moment the playoffs are certainly going to be a pressure cooker where every small performance sample will be blown ridiculously out of proportion. Hopefully for him when those games roll around he will be closer to his March form than his level of play in October, or the media will no doubt be dusting off the exact same stories about his inability to handle the pressure of Philadelphia.

The Curse of Obama

In the 20 games since the Boston Bruins' White House visit that Tim Thomas famously did not attend, the defending Vezina Trophy winner has gone 9-9-0 with a very uncharacteristic 3.07 GAA and .890 save percentage.

That .890 would look ugly for pretty much any goalie in today's NHL, but it looks especially glaring for Thomas considering his excellence over the past three seasons. Going back to the start of 2008-09, Thomas' numbers have been consistently elite even when divided up into the relatively small sample size of 20 games (playoffs included):

Nov '11-Jan '12: .935

May '11-Nov '11: .943

Mar '11-May '11: .929

Jan '11-Mar '11: .931

Nov '11-Jan '11: .939

Mar '10-Nov '11: .941

Dec '09-Mar '10: .911

May '09-Dec '09: .921

Feb '09-May '09: .936

Dec '08-Feb '09: .930

Oct '08-Dec '08: .937

The .911 and .921 came in the latter half of 2009-10 when Thomas was limited by a hip injury. If that is enough to excuse those two results, it has been over three calendar years since a healthy Thomas had to endure a 20 game stretch below .929. And then he decided to skip a team event for personal reasons and everything fell apart.

Obviously I don't actually think the flap surrounding Thomas and President Obama made any significant impact on his play, but there is no question that regression has hit the Bruins hard in net in 2012. Not only have Thomas' numbers nosedived, but Tuukka Rask is 0-3-1, 3.07, .886 since the All-Star break. The .940 team save percentage that the team was rocking early in the year was never going to hold up, but I didn't expect the wheels to come off this heavily either.

At 37, Thomas is in the age range where performance levels can change pretty suddenly, and he has had a lot of recent mileage considering his workload this year combined his playoff run last season. Thomas is already in the top-10 all-time this year for games played by a 37-year old, and including playoffs only one goalie has ever played more games in his age 36 season than Thomas' 82 last year. That doesn't necessarily mean that an awful 20 game stretch is a sign of impending doom, but it's certainly something to keep an eye on. Thomas' recent form has to be at least somewhat concerning for Bruins fans with the playoffs just around the corner, given that elite goaltending has been a big part of the team's success over the past two seasons.

Why the Counting Trophies Method of Goalie Evaluation Is Flawed

One of the biggest problems with evaluating goalies by career value is that there aren't any good commonly-used counting statistics. A goalie's career line merely shows games played, wins, and shutouts, plus the rate stats of GAA and save percentage. Wins and shutouts are very heavily influenced by the rest of the team, with shutouts also varying widely depending on the level of league scoring. Games played is important in determining overall value, but does not take into account level of performance at all other than reflecting how long the netminder was able to convince an NHL team to keep giving them starts.

There has been a shift towards a greater focus on save percentage, but as long as save percentage remains a rate stat it is difficult to understand intuitively whether a goalie with a higher save percentage but a lower workload is contributing more value than one of his peers with the opposite. There are a number of good ways to turn save percentage from a rate stat into a cumulative stat (typically by comparing to league average or to replacement level and then multiplying by the shots against). Hockey Prospectus' Goals Versus Threshold, which is slightly more complex but based on a similar foundation, is a number that is recognized at least within the online hockey stats community, but there is not a widely accepted standard.

As a shortcut, therefore, many people look at a goalie's trophy case, and use that to determine which netminder had the better career. Intuitively that makes sense, as when we evaluate athletes we want to know things like how many times they were considered the best in the league. However, I do not believe this is the best approach for evaluating NHL goaltenders.

If somebody just sweeps the awards year after year, like Dominik Hasek did in the 1990s, then that is definitely showing something meaningful. Or when Glenn Hall kept getting voted ahead of other Hall of Famers while repeatedly overcoming the heavy bias towards the GAA leader, that's also probably revealing something important about how his play was viewed by his contemporaries. But goalies with a handful of awards are exceedingly rare. When you are comparing two veteran goalies where one has a Vezina or a Smythe while the other one doesn't, I don't think that trophy really adds much information at all. Some trophy-focused individuals might, for example, try to argue that Miikka Kiprusoff's Vezina Trophy means he has had a better career than Roberto Luongo, even though the performance gap between them is likely at least 100 goals in Luongo's favour (Lou's career GVT is more than double Kipper's number).

The problem with goalies is that one season is not a large enough sample to properly rate anybody, because results are not accurately tied to performance. Given that awards are handed out primarily based on results, this means that luck and team factors play a disproportionate role in winning awards for goalies.

For a .914 goalie who faces 1800 shots in a season, the 95% confidence interval of his save percentage based on binomial probability would put his performance anywhere between .903 and .925. To prove that this is not just a theoretical exercise, we can just look at two goalies with .914 career save percentages: Ilya Bryzgalov and Ryan Miller. Both have a low mark of .906 as a starting goalie, and despite their strong career track records both have struggled for the majority of this season. Bryzgalov looks likely to set a new career low mark this year, although Miller looks to have turned things around as of late.

Their peaks, on the other hand, are much higher, with Bryzgalov hitting .921 last season and Miller topping out at .929 during his '09-10 Vezina season. Those ranges are in fact almost exactly what would be expected if their numbers varied by random chance alone. Between the two of them, their average high is .925 and if Bryzgalov ends up at his current .899 while Miller stays above .906 then their average low would be .903, which again would exactly match the predicted range given above.

In addition to simple performance variation, a goalie's teammates could raise or lower his save percentage by up to about .005 or so depending on how many penalties they take and whether they are good at preventing shots against on the penalty kill. The official scorer in the goalie's home rink could also assist in cutting or boosting shot totals by a shot or so per game, which again could have an impact in the neighborhood of .005 compared to a goalie on the other end of the spectrum. And maybe a goalie plays for Ken Hitchcock or Jacques Lemaire and his team has a good year in front of him defensively in terms of reducing scoring chances, which could easily add another few thousandths to the final save percentage number.

Over a career these effects often mostly wash out, as a goalie will benefit from them in some seasons and suffer because of them in others. But when looking at a single season, as is the case when considering awards nominations, these factors can further accentuate the already heavy effect of random chance.

A goalie with at least a .920 save percentage over 1800 shots has about a 50% chance of being a Vezina nominee, based on seasonal results since the lockout. Bump that save percentage up to .925, and you probably have about a 50% chance of winning the trophy, given that half the goalies who met both cutoffs won the Vezina. Consider that binomial probabilities suggest that a goalie with league average talent will put up a .920 or better over 1800 shots 1 time out of every 5 just by chance, and you can see how it is entirely possible that a goalie who carves out a decent NHL career will probably luck into at least one good season somewhere along the line even before considering the other factors that could help his statistics. Fortunately there are selection effects that limit the number of flukes, since most average goalies won't be given that many starts by their teams, but it does still happen with some regularity.

Whether a goaltender wins the Vezina or not depends not just on his own play, but also what other goalies are doing around the league. To return to the Luongo/Kiprusoff comparison above, it's easy to see the impact of external factors when you consider that if Martin Brodeur had his 2006-07 season in 2005-06 and vice versa, Luongo would be the guy with a trophy while Kiprusoff would have been shut out.

It's not hard to find examples of goalies doing more or less the same thing year after year even as their voting numbers vary widely. Take Patrick Roy over the last five seasons of his career when he was the very picture of elite consistency, rattling off 61-63 starts per season, overall save percentage typically in the .915-.920 and EV SV% in the .925-.930 range and a GAA usually 2.20-2.30. In 2001-02, his numbers all improved a bit, particularly his GAA and shutouts, although his EV SV% was just .005 better than his five-year average. His team's improved shot prevention and penalty discipline helped as well. There's certainly a chance that Roy played better than normal in 2001-02, but there is also a pretty strong possibility that he was more or less the same goalie all the way through and the breaks went his way in '01-02.

Over that period Roy typically got a few Vezina votes per season, until 2001-02 when he almost won the award. That year the high-minute goalies (Brodeur, Kolzig, et al) had down seasons, while Hasek was playing at a lower, post-injury level. It turned out that only one starting goalie other than Roy posted a save percentage above .921.

Unfortunately for Roy's trophy case, that goalie happened to be Jose Theodore, who put up a .931 save percentage on his way to the Vezina/Hart combo. If you look at Theodore's 2001-02 numbers in context, they are major outliers. He also had a bunch of the indicators of one-year flukes, including very high special teams numbers and much better numbers at home than on the road.

Knowing what we know now, it is very likely that Patrick Roy was a better goalie in '01-02 than Jose Theodore. That just wasn't immediately apparent from that 82 game sample, and that's why Theodore won the award. Using awards as the primary evaluation criteria, Theodore's first five seasons as a starter ranks ahead of Roy's last five seasons as a starter (unless you're one of those guys who thinks only the playoffs matter and you really love Roy's 2001 Cup/Smythe combo, and even in that case you'd probably give Roy just a slight edge), even though Roy's .929 EV SV% on 6165 SA is quite a bit better than Theodore's .921 on 6522 (about 50 goals better, or nearly two wins per season).

In contrast, you won't find a skater win a scoring title primarily because of luck. Assume a typical first liner with 15 minutes per game at even strength and 3.5 minutes per game on the power play, who either doesn't play much on the PK or doesn't score any points when does get an occasional shorthanded shift. Let's say his team takes shots at an average rate while he is on the ice (27 shots per 60 at 5 on 5, 45 shots per 60 on the power play). That player and his linemates would need to put up a ridiculous shooting percentage, and he would have to be involved in an unusually high number of plays to get on the scoresheet enough to be in Art Ross contention.

For example, if the player's team shot 13% at evens and 20% on the power play, both numbers above what any regular player managed last year, and got a point on 90% of his team's even strength goals and 75% of his team's power play goals, both extremely high and unusual participation percentages, the player would still end up with 97 points. That's a pretty high total and it would end up being near the very top of the league, especially this year, but it still isn't high enough for a top-3 finish in any of the post-lockout seasons. And again, this hypothetical player would need to get all the breaks just to get that close. If any of the luck factors drop off, then he's not in contention, and there are many ways that could happen (e.g. he misses a few games, his ice time decreases, his teammates don't score at a high rate, he gets unlucky with second assists or the power play runs more often through a teammate which reduces his scoring opportunities).

Jordan Eberle is pretty much that guy this year (at even strength his team is shooting 13% at even strength with him on the ice and he has points on 88% of his team's goals), and he's still tied for 10th in scoring. Joffrey Lupul is another guy that has flirted with the top of the scoring charts this season based on unsustainable percentages, and while he still sits just above Eberle in 9th it was always just a matter of time before he was going to left behind by the likes of Malkin, Stamkos and Giroux.

Very high points finishes are meaningful because they are unlikely to be flukes. Pure trophy counting would still undervalue someone like Mats Sundin who was consistently productive although never near the very top of the league, but in general a player with a couple of high finishes can be considered to have had a better peak than a similar player who never climbed the table to the same degree (although of course context and team factors need to be taken into account).

That doesn't happen to goalies. Flashes in the pan have won the Vezina, and average goalies have found themselves in trophy contention simply because fortune ended up favouring them over a 50-60 game stretch. Of course observers can sometimes identify when other factors or luck are at play; they don't always just follow the numbers to the exclusion of anything else. However, it is particularly difficult when rating a young goalie without much of a track record who has a big season. Is he breaking out, or is he getting lucky? Is he Pekka Rinne or Steve Mason? Only time will tell.

It is my belief that goalies can only be properly evaluated in a multiple season context. With that in mind, single season awards should be considered relatively insignificant in terms of career evaluation.