Don't Always Believe Your Lying Eyes

How do sportswriters vote for the year's best goalie? I bet they'll tell you they evaluate lots of different things, like how good he is at stopping the puck, how good he is at controlling rebounds, his mental toughness, how much he helps his team win and a bunch of other criteria that sound pretty good. They'll watch lots of hockey and check out a lot of different goalies, enough that they should be in a pretty good position to be able to evaluate them. And then who do they often end up picking? The goalie on the best team, or at least the best defensive team. That's what they've been doing for years.

If you asked sportswriters the same question back in the 1960s they'd probably give you a similar rundown, describing in detail some technical aspects of Jacques Plante that made him great, or how competitive Terry Sawchuk was in the clutch, or how much Glenn Hall helped his teams win. Some of them no doubt simply copied down the wins leaders, but I bet that a majority of the journalists sat down every year with their detailed opinions and jot notes and went through their memories of the games they had watched and tried to evaluate a dozen different subjective things. They then mailed off their ballots to league headquarters, certain that they had taken everything into account and identified the league's best goaltender.

Were they sincere? I think they were. Yet give me a random stat sheet from one of the seasons back then, with the names of the goalies blacked out, and from that alone I can predict with over 80% accuracy who the voters thought was the best goalie that season. If it wasn't for one particular outlying goalie it would be almost a sure bet.

From 1935 to 1970, there is a very simple algorithm to determine the First Team All-Star goalie. Look at all the goalies who played in at least 75% of their team's games, rank them by goals against average, and take the guy at the top of the list. That's the First Team All-Star. That solution worked in 30 out of 36 seasons, all of them except for 1957, 1958, 1960, 1964, 1968 and 1969. Here are the goalies who were named First Team All-Star ahead of the high-minutes GAA leader in those seasons:

1957: Glenn Hall
1958: Glenn Hall
1960: Glenn Hall
1964: Glenn Hall
1968: Gump Worsley**
1969: Glenn Hall

**-Led the league in GAA by a wide margin but did not play in 75% of his team's games during a season where all of the top 4 guys in GAA were in platoon situations.

Glenn Hall was quite obviously the exception to the rule. That he was the only guy to get named the league's best goalie without playing on one of the league's top defensive teams reflects well on him, and it's even more impressive that he did it 5 times. That shows that he was considered a difference-maker and is evidence of why he is considered one of the best goalies to ever play the game.

Other than Hall, though, it was pretty much a succession of GAA leaders that were named the season's best. Whether or not the voters made a conscious, stat-based decision to rely on GAA in their rankings or whether their subjective views were impacted by the fact that the goalies they watched were less likely to allow goals than everyone else is unclear, I'd guess it was a combination of the two but in the end the result is the same.

That certainly does not mean that the voters never got it right, of course. There were some seasons, perhaps even quite a few seasons, when the best goalie played on the best team or where strong goaltending made the difference between a good and a great defensive team. It remains quite obvious to anyone with a reasonable view of a goalie's role within a hockey team that the best goalie does not always allow the fewest goals against.

The voters were no less predictable in the 1970s, but what had changed was that it became common to platoon goalies. As a result, very few goalies played in 75% or more of the team's games. The voters still, whether consciously or subconscously, voted in the GAA leaders. In every season from 1971 to 1979 the same goalie won the Vezina (which under the old definition went to the starting goalie on the team that allowed the fewest goals) and was named the First Team All-Star.

I'd say the voting has improved at least somewhat in recent years, perhaps partly due to the introduction of the modern Vezina Trophy that allowed GMs to vote on the year's best goalie as well as the sportswriters. Yet even the game's insiders are not immune, as the starting goalie on the team with the fewest goals against is still overrepresented in both recent All-Star and Vezina voting. If you are the starting goalie on the team with the fewest goals against and you play in at least 50 games, you're still almost guaranteed to be named First or Second All-Star at the end of the season. The only post-expansion exceptions to that rule both came in the last decade (Ed Belfour in '99 and Roman Cechmanek in '03, although Cechmanek shared the Jennings with First Team All-Star Martin Brodeur so it's debatable whether that one should even count as an oversight). These results suggest that the bias towards goalies on strong defensive teams is still alive and well.

Some people will tell you that all goalie statistics are flawed because the rest of the team affects them, which means that the best way to evaluate goalies is to watch them play. It is true that a goalie's statistics are affected by his teammates, but what they are failing to take into account is that the rest of the team also affects how a goalie is viewed. This is especially true in the NHL where the margins between goaltenders are so narrow. I'd bet for many sportswriters and most hockey fans this effect is considerably more significant than any team-to-team differences in shot quality. A goalie on a good team that doesn't let in many goals and makes a lot of saves with his team in the lead usually looks better than a goalie on a bad team who faces more shots, allows more goals, and spends more time with his team trailing.

I think that at the NHL level there is some value to goalie scouting, but that there is likely more value in numerical analysis. That's especially true if fans or teams are not able to watch and grade the goalie in every game, which means they will end up seeing certain guys good and certain guys bad. There are other issues with evaluating goalies by watching them, such as the impact of a goalie's style and whether a viewer can accurately assess the importance of various goalie skills (I'd suggest things like rebound control and puckhandling tend to get overrated since they are more obvious and easier to grade than skills like positioning, post play, a goalie's ability to block screened or close-in shots, etc.).

If I was forced to pick only one of the two methods of evaluation (stats vs. watching), I'd prefer to take the numbers. Of course I'd rather watch a guy as well to see if there are any nuances that can be added to what the statistical record is telling me, but the moral of the story is this: Always be aware that your eyes can lie.