The Importance of Asking "What If?"

Alan Ryder wrote an article for Globe Sports contrasting the Luongo acquisition with the Raycroft acquisition.

He analyzed how both Toronto and Vancouver did this year, and concluded, quite reasonably, that Luongo had a major effect on Vancouver making the playoffs, whereas Raycroft gave the Leafs goaltending that was just as mediocre as what they had in 2005-06. He also threw in a jab at Leaf fans, by pointing out that if the Leafs had acquired Luongo instead of Raycroft, they would have been as much as 20 points better in the standings.

What was the response to this article in the comments? By and large, the commenters bashed the article, mostly because it dealt with "what-if" scenarios (which it did, briefly at the very end in a passing comment). Many Brodeur fans respond in a similar manner when people, such as myself, claim that Brodeur simply would not have close to as many wins or shutouts on a league average team.

That argument boils down to something like this: We don't know for sure what would have happened, so let's not even think about any alternate possibilities. Let's all just measure every goalie by wins and shutouts and blindly agree with everything the mediots say. Let's not use our brains to think about possible scenarios or try to separate out what was a team effect and what was a goalie effect. Cam Ward won a Cup, he's better than Rick DiPietro and Henrik Lundqvist. Roberto Luongo never made the playoffs in Florida, he's not worthy to carry Chris Osgood's jock. And Martin Brodeur is the greatest goalie in the history of the world.

Hypotheticals are critical to sports analysis. In scientific experiments, there are things called control variables, and scientists try to keep them constant to measure the effect of changes in the variables they are investigating. In sports, there are no control variables. You can't tell the teams to switch goalies and play the game over again. If you want to compare a player of today with someone who played 40 years ago, you have to compare different eras, different rules, different opponents, different equipment, different playing styles and techniques, and different numbers of teams and levels of talent dilution, among other things. Even in comparing, say, Luongo vs. Brodeur, they have different goal support, different shots against numbers, different quality of shots against, different teammates, different opponents, different team tactics, different special teams situations, different conferences, different travel schedules. Because of all these adjustments that have to be made, everything is really just a "what-if" consideration, or there cannot be any useful ranking or comparison at all other than just looking at the stat sheet and sorting everyone by points or wins.

You cannot properly evaluate anyone without asking "what if". Baseball stat guys use a stat called WARP, which measures how many extra wins a player contributes to the team, compared to if that player had been replaced by a minor leaguer or "replacement-level" player. That is a "what if" scenario (basically, what is the Yankees record if they replace A-Rod with a Triple-A third baseman?), but clearly based on statistical evaluation.

It is the same with goalies. If a goalie plays on the worst team in the league, his performance must be viewed in the context of how other goalies would have been expected to have performed on that same team. No, we will never know for sure how they would have performed, but we can adjust for their actual performance on their own teams, and then apply some correcting factors to fairly reasonably surmise where their expected performance level would have been. For example, we do not know 100% for sure that Martin Brodeur would never have made the playoffs if he played for Florida between 2001 and 2006, but based on his performance record, Florida's defensive and offensive performance, available shot quality information, and formulas for predicting team records given the number of goals scored and allowed, we can be about 99.99% sure that he would not have.

By not asking "what-if" and taking all goalie stats at face value, fans end up penalizing players for being on the wrong team, or conversely, crediting them for playing on a good team. That is poor analysis, and is why good goalies can be considered legends of the game mostly because of their team situations (Brodeur, Fuhr, Billy Smith, etc.).