How much weight should we give to subjective factors? Can scouts really evaluate the difference between two goalies? Are public perceptions of goalies often right? Do we properly value non-save skills in goalies?
These are important questions, especially for someone like Martin Brodeur who receives more glowing hagiography about his intangibles and his goaltending artistry than anyone since Roy. A lot of the feedback I get is along the lines of, "Haven't you ever seen him play?" The problem with putting a lot of weight into these reports, however, is that these effects are almost never observed in the data, and in a number of cases seems to suggest the complete opposite.
For example, here are a few common debate points, along with the numerical reality:
Brodeur never gives the puck away (According to NHL RTSS stats, he gives the puck away more than the opposing goalie at the other end of the ice in 2005-06, 2006-07, as well as so far in 2007-08).
Brodeur prevents shots (his backup goalies have faced essentially the same shot levels as he has).
Brodeur makes shooters miss more frequently through good positioning (He has been well below average in terms of making shooters miss this year (60th among goalies) and last (45th), source: Behind the Net)
Brodeur makes the big saves at the key times (Little evidence of that, and the playoff overtime performance sample suggests, if anything, the complete opposite).
Brodeur controls his rebounds very well...this one is probably true, but it is difficult to measure and is impacted by shot quality against and defensive play in front of the goaltender.
So there are several oft-repeated refrains in support of Brodeur. Other goalies have received similar treatment - e.g. Grant Fuhr and the "big save at the key time", or Patrick Roy's ability to "carry a team on his back". Both of these claims are just as doubtful as some of the ones listed above.
I think we have to clearly separate what is aesthetically pleasing, skillful and enjoyable to watch from what actually wins hockey games. There is very little margin in goaltending to begin with. The difference between good goalies is often something like one save in 50, a difference so small it is practically unobservable by the human eye. It can then become tempting to latch onto some other factor, like perhaps puckhandling ability, past playoff performance, or some other factor to break the tie, but this results in overweighting a single skill and ignoring the larger picture since by far the most important part of a goalie's job description is of course stopping the puck.
Two of the worst puckhandling goalies I have ever seen are Patrick Roy and Dominik Hasek. Both of them were responsible for many terrible giveaways, sometimes at the worst possible moments. Yet they both stopped a much greater percentage of shots than Brodeur did, which is why at the end of the day they were both significantly better goalies. Good puckhandling goalies are fun to watch, but the evidence is that their skills don't really make much of a difference in terms of winning or losing the game.
Another issue is clutch play, and more specifically whether it exists. Studies in other sports have found evidence of clutch play to be very slight or non-existent, and that generally players perform up to their level of ability in the clutch. Good players or players that have a lot of chances to succeed with the spotlight shining on them gain reputations as being clutch, when in reality their performance at the most important times is no better than the rest of the time.
So it is difficult to reconcile the subjective viewpoints with the objective record. I probably tend to weight the measurables more highly than some, which creates a difference of opinion. I believe that a skill, to be valuable, has to have an impact on winning hockey games, and if it does have an impact on winning games then that will sooner or later show up in some kind of observable way. But the existence of discrepancies between what is observed and what is measured ensures that debates will continue to rage, especially regarding Mr. Brodeur.