Monday, October 31, 2011

It's Not Only About the Stats

One of the goalies off to a red hot start this season is Kari Lehtonen (8-1-0, 1.75, .947). Moving to Dallas has rejuvenated Lehtonen's career, and while he obviously won't maintain those lofty numbers for an entire season, the Finnish goalie had a decent year last year and is still only 27 years old. Few people have questioned his talent, but Lehtonen may finally have figured out how to combine that with the hard work and professionalism needed to perform as one of the league's better netminders.

Lehtonen can be used as an example both of the power of statistics and of how other factors can be important beyond the numbers. I touted him for a while prior to his trade to Dallas as a talented NHL goalie who was stuck in a bad situation in Atlanta, based primarily on the strong even strength save percentage results he put up in the early part of his career. However, at the same time observers were pretty much unanimous that Lehtonen was not properly utilizing his talent through a lack of preparation and repeatedly showing up overweight and out of shape, which led to a lot of his injury problems. This year all reports are that Lehtonen has finally put in the off-ice work needed to get into great shape. There is a noticeable difference between what he looks like now and what he looked like when he was playing in Atlanta.

Off-ice training and mental preparation and hours spent on the practice rink working on technique drills are all very important for a top professional goaltender. However, typically goalies who make it to the top levels of the sport have developed the training and work habits they need, especially since for most of them it is a fairly long road to the get to the NHL and if they weren't putting in that time then they would have washed out well before they made it to the show. If everyone is working hard, then it doesn't become much of an advantage for anyone, and it becomes much less likely that a goalie who has been training hard year-round for a number of years will suddenly make a huge leap forward primarily based on those off-ice factors. On the other hand, someone who is very talented but doesn't take the steps to maximize his talent would be a candidate to see his performance improve if he is able to finally put everything together, which may be the case for someone like former second overall pick Kari Lehtonen.

There's always good reason to be skeptical about claims justifying early season success. This is the time of the year where dozens of articles are written by reporters claiming that a good offseason of training is fully responsible for a player's 30% shooting percentage through 10 games, and is the reason why that player is going to hit the 40 or 50 goal mark for the first time in their careers. Needless to say, those players always regress significantly by season's end, as luck was almost certainly a bigger factor than anything that happened in a weight room or on a practice rink. On the other hand, we shouldn't completely dismiss the human factors either. I'm still far from convinced that Kari Lehtonen will end up in Vezina contention, but it will be interesting to see how long he can sustain his sizzling start. At the very least there appears to be a good chance that he is headed for a career year in 2011-12.

Monday, October 17, 2011

The Ecological Fallacy

I found a scholarly name for the tendency to rate goalies on winning teams as better than goalies on losing teams. It's the ecological inference fallacy. You can read all about it on Wikipedia, but in short it is the assumption that all members of a certain group share the same characteristics of the entire group. To quote Wikipedia:
If a particular sports team is described as performing poorly, it would be fallacious to conclude that each player on that team performs poorly. Because the performance of the team depends on each player, one excellent player and two terrible players may average out to three poor players. This does not diminish the excellence of the one player.
Nor does it boost the performance of an average player who happened to have great teammates. Avoid the ecological inference fallacy and give credit where it is due, based on an individual's contribution to the team effort regardless of the final result.

To see this type of thinking in action, go read one of the post-game report cards that fans put up after their team plays. In many cases, when a team wins there are As and Bs across the board. Yet when the team loses, everyone gets Cs and Ds.

Check out, for example, Vancouver blog Nucks Misconduct's report cards from last year's Cup Finals. In Vancouver wins, the average score for Vancouver fourth-liners was 9.1. When the Canucks lost, the average score for fourth-liners was 7.0. Can the Canucks' losses be blamed on a line that barely played and had little impact overall on the series? Of course not. It just so happens that when Roberto Luongo was making saves and the other forward lines were scoring then Victor Oreskovich, Tanner Glass, Jeff Tambellini et al looked better by association. In contrast when Luongo got shelled and the Sedins were shut down, the same guys playing their usual 6 or 7 crash-and-bang minutes without a goal for or against ended up getting hung with the same mediocre grade as the stars who were actually driving the bus. That's the ecological fallacy in action.

I'd say this logical error explains quite a few of the most common mistakes made in rating goalies. Add in the base rate fallacy that causes people to exaggerate their praise or criticism for a goalie's performance by not properly factoring in the play of a typical replacement ("Without Lundqvist, the Rangers would have lost at least 10-1!"), plus the fundamental attribution error which makes people lean towards personality-based explanations for team successes or failures ("Carey Price's teams will never win in the playoffs because he lacks mental toughness"), and finally availability bias ("I don't remember any of Mike Liut's career except for that Canada Cup Final where he let in 8 goals, but that game proves he was an awful clutch performer"), and you've probably covered 95% of the rest of them as well.

Thursday, October 13, 2011

Desire and Success

(I wrote the rough draft for this short post at the end of last season but never ran it, and was recently reminded of it while watching the Winnipeg Jets get beaten 5-1 in their home opener by the Montreal Canadiens. That was not entirely the same situation as the one described here, given that Montreal was certainly looking to add two points just like every other team this early in the season, but one still would have thought that the Jets players would have that extra motivation to kick off a new era of NHL hockey in Winnipeg with some success. Nevertheless, they still came up four goals short.)

There are abundant cliches in sports that attempt to relate winning to effort level. How many times have you heard an announcer say something like "they just wanted it more" in an attempt to explain why one team emerged victorious while the other team did not?

I've posted before about effort-based explanations being largely ridiculous at the professional level given the stakes involved, but there are some situations where there is in fact a clear imbalance in incentives between two teams, such as late in the season where one team is already out and the other is facing a must-win game. What happens in that case, does the team that wants it more always win?

During the last weekend of the 2010-11 regular season, three teams (Carolina, Chicago and Dallas) all controlled their own destinies and all only needed to win their final game to clinch a playoff berth (the Hawks actually only needed to get to OT). None of their opponents had anything to play for, as all three of them were either eliminated or could not change their playoff seeding. Carolina and Chicago were playing in front of their home fans, while Dallas got a non-playoff opponent in the Minnesota Wild. In addition, Detroit was the only one of the three opponents that went with their starting goalie. In every case, the situation looked very favourable for the team that needed to win to get in, especially if "wanting it more" is a good predictor of success in the NHL.

Those three teams combined to go 0-3. Every playoff home date is worth millions to their franchises and earning the opportunity to compete for a Stanley Cup has huge intangible benefits to NHLers yet all three teams squandered their chance. The Chicago Blackhawks did manage to qualify for the postseason, but only because they got lucky when Dallas also failed to seal the deal.

The statistical case for the heavy role of luck in hockey has been well-made, but there remains a resistance for many traditionally-minded hockey fans to accept numbers-based conclusions. That's why sometimes it is good to use other types of arguments (I particularly like the game-charting ones, like this one, for example, because they can't be simply dismissed out of hand by the people who have an ingrained anti-stats outlook). I'd submit that the fact that a team can have a skill advantage and home ice advantage and a starting goalie facing an opposing backup and a huge advantage in incentives and yet can still lose the game is a simple yet powerful observation that supports the heavy role of luck in the sport of hockey.

Monday, October 10, 2011

Goalie Effects on the Jack Adams Award

Last week Hockey Prospectus asked me to predict the 2011-12 award winners as part of its ongoing season preview. I usually just throw out names for that kind of thing because it's a pure guess anyway, and to be honest I'm not at all sure that I know much more than the next guy about who is going to be this year's best rookie or best defenceman or best coach.

Anyway, for Jack Adams I filled in Guy Boucher's name after not much more than a few seconds of thought about it. I figured he did a pretty good job last year and attracted lots of attention during Tampa's run to the Conference Finals, and if the Lightning finish with another 100+ point season maybe people would write his name down on their ballots.

Not too long afterwards, there was an insightful comment on Coppernblue by dkball7: "DeBoer should be everyone's pre-season pick for Jack Adams. As long as the team's PDO regresses to 100%, he will look like a genius."

In hindsight, it makes sense that I should not have picked a coach on a team that performed well the year before, like Boucher's Lightning, but should instead have picked one on a team that underachieved, like DeBoer's Devils. The Coach of the Year often goes to a candidate on a team that massively improved relative to the year before, which often is caused in large part by a significant swing in percentages from one year to the next.

I looked at the teams for each of the Jack Adams award winners since 1990, and compared each team's performance during the year where their coach picked up his hardware to the team's performance during the year prior and the year following. Here are the percentages, with everything adjusting to an average level of 10.0% shooting/.900 save percentage:

Prior to Jack Adams Year: 10.2% shooting, .897 goaltending, 99.9 PDO
During Jack Adams Year: 10.7% shooting, .909 goaltending, 101.6 PDO
Following Jack Adams Year: 10.4% shooting, .905 goaltending, 100.9 PDO

Only one out of 21 of the teams that produced a Jack Adams award winner posted a below-average save percentage during that season. In contrast, during the prior year, 13 out of those 21 teams had a below-average save rate. During the year after, the goaltending still remained strong for the most part, with just three teams dropping back to below-average save numbers.

The following year numbers imply that either the teams had slightly above average shooting and goaltending talent as a whole, or the award-winning coaches themselves combined for a positive effect on the team's numbers.

I think the goaltending numbers in the year after are more likely to reflect goalie talent than the coach's system of play. There were a number of top goalies represented (e.g. Hasek, Brodeur, Luongo, Thomas). Overall, the goalies had an average career save percentage of .906 over a period where the league average save percentage was .902. Considering that some of their careers stretched back further than 1989-90, when the league average was even lower, it seems reasonable that the combined goalies were about .005 better than league average, although it should be at least noted that some of the coaches in the sample have been known to affect shot quality, particularly guys like Burns and Lemaire, which may account for a small part of the above-average result.

It is interesting that many of the teams had the same goalie during the Jack Adams year as they did the season before. Eleven of 21 teams had the same starting goalie, with all of them playing a relatively similar number of games as well. Here are the save percentage numbers for the season prior, during and after, split out by whether the team had the same netminder as the year before (numbers adjusted again to league average with a baseline of .900):

Same goalie: .901, .909, .907
New goalie: .893, .909, .904

The largest improvements came for teams that brought in new goalies, obviously, but a good goalie coming off of an average season can also have a big impact in improving a team's fortunes and getting his head coach some extra attention.

The numbers do suggest that a lot of things simply went right for coaches during their winning years, but I certainly don't want to imply they had no effect at all. There is, for example, the shots for and against evidence, which shows that the teams also had a substantial improvement in their underlying possession metrics during the Jack Adams winning years. Teams playing for a Jack Adams winning coach were also more disciplined than average, as well as more disciplined compared to the year before.

Prior to Jack Adams Year: 1.005 SF/SA ratio, 362 PPOA
During Jack Adams Year: 1.064 SF/SA ratio, 340 PPOA
Following Jack Adams Year: 1.044 SF/SA ratio, 350 PPOA

Goaltending and shooting luck do not determine everything, but a lot of what the best coaches do is difficult to judge and rate, especially from a distance. For that reason, exernal factors can often come into play. As the old hockey saying goes, "Show me a good coach and I'll show you a great goaltender."

In summary, if you want to maximize your chance of being named the NHL's best coach, you should try to get a job on a team that either had awful goaltending the season before and made a move to address that weakness, or where a good goaltender had a down year. Either one of those scenarios would give a coach the best chance to see his team's percentages swing around in a hurry, leading to a significant improvement in the standings. That will in turn cause many people to think there must have been some coaching magic at work, and if you're lucky the awards recognition will soon follow.

To make a better Jack Adams prediction, we should apply this logic to this year's teams, and find a team with good goaltending that had weak goaltending last year and is likely to improve in the standings. The Flyers and Caps brought in new top-flight goaltenders, but both actually had pretty good save percentages last season. Several other teams have also improved in net but are still expected by most to finish near the bottom of the league and as a result are unlikely to produce a Jack Adams winner (Islanders with Nabokov, Senators with Anderson, Avalanche with Varlamov).

There are five teams that had subpar goaltending last season as well as overall PDOs below 100 that could be primed to do better in 2011-12:

New Jersey: 7.3 SH%, .906 Sv%
Toronto: 9.0 SH%, .907 Sv%
Tampa: 9.3 SH%, .903 Sv%
St. Louis: 9.5%, .902 Sv%
Columbus: 8.4%, .900 Sv%

Guy Boucher's Lightning show up on this list, suggesting that my random intuition may not have made that bad of a pick after all. There has to be some concern, however, for the fact that starter Dwayne Roloson is turning 42 next week. That said, he should still be better than the combo of Mike Smith and Dan Ellis (.894 last season), and Mathieu Garon will also provide improved backup goaltending. The Lightning may see their shooting regress slightly, but if they can duplicate last year's outshooting results and if Roloson can hang together to give them better goaltending then they will be definite challengers in the East. That could put Boucher in the conversation for Coach of the Year, but I'm not sure if it would be enough, especially if there is someone else out there who oversaw a much larger improvement in terms of wins and losses.

Age is also a concern for Martin Brodeur, but nobody is expecting a .903 again, and the New Jersey shooters are virtually guaranteed to improve (7.3% is a major outlier for a team shooting percentage). Jaroslav Halak also would be a good candidate for a bounceback year in St. Louis. If James Reimer is the real deal with .920 talent in the NHL he could certainly win Ron Wilson a trophy this year, but that still doesn't look to me like a good bet. As long as Columbus is going to continue to bet all their chips on Steve Mason I'm not sure I'll be expecting above-average goaltending in Columbus, although there likely could be some improvement there, perhaps even enough to get into the playoffs depending on luck and how well the rest of the team plays.

After considering this evidence, I think Peter DeBoer is the best pick for the 2012 Jack Adams, with all signs pointing to the Devils coming back strongly this season. There is always the chance, of course, that a team loses a star player and keeps on trucking, like Pittsburgh did in Dan Bylsma's award-winning campaign, or that one of the league's best teams has a spectacular year and cleans up at awards time, or that a team with a lot of new additions like Buffalo really comes together and climbs the standings. At the end of the day, the most likely winner is probably the coach who saw the largest improvement from the year before, and for this season the team with the best chance to improve is almost certainly the New Jersey Devils.

Monday, October 3, 2011

Why Doesn't Carolina Get Better Backup Goalies?

As I pointed out in my last post, the depth of talent among the league's goaltenders has improved substantially in the NHL over the last 15 seasons. Quality talent has never been more readily and cheaply available than now. At the same time, the salary cap has increased parity across teams, resulting in close playoff races nearly every season in both conferences. The simple conclusion to make based on this fact is that no team should accept awful backup goaltending. It doesn't cost much more to get average goaltending than it does to get replacement level goaltending, and bubble teams that are content to let a washed-up veteran or an over-his-head youngster play backup minutes are jeopardizing their playoff chances in doing so.

There has been one NHL franchise in particular that has seemed to not understand this principle, having been repeatedly burned by weak backup goaltending. That team is the Carolina Hurricanes.

Last year, Carolina finished two points behind the Rangers for 8th in the East, despite a terrific season by Cam Ward (37-26-10, 2.56, .923). Ward actually had a better win/loss record than Rangers starter Henrik Lundqvist (36-27-5), but the decisive difference that sent the Blueshirts to the playoffs at the expense of the 'Canes was what happened when neither #1 netminder was in the net. Solid veteran Martin Biron had a .923 save percentage and an 8-6-0 record in New York, while youngster Justin Peters was lit up in his infrequent playing time in Carolina (3-5-1, 3.98, .875). While Carolina saved money with Peters' $525K cap hit, it would have only cost them an extra $350,000 to pay a guy like Biron.

According to Capgeek, the Hurricanes had $9.5 million in salary cap room last season. Would the team's ownership have been willing to spend an extra $400K if they knew there was a good chance it would have helped the team earn the extra three standing points needed to earn millions in revenue from at least two extra playoff home dates? They would surely have agreed to that deal in a heartbeat. The Canes' management can't be entirely faulted, as Peters was a four-year minor league pro coming off a pretty good season in the AHL and he was probably at least somewhat unlucky to post numbers that terrible. On the other hand, one of the main reasons to get a good #2 option is to minimize the risk of a relying on a unknown quantity.

It was a similar story in Carolina in 2007-08. Ward wasn't quite as good back then, but much of the roster was just two years removed from winning the Cup. Despite a .904 save percentage, Ward's record was 37-25-5, easily good enough to put the 'Canes in playoff position. The problem was that backups John Grahame and Michael Leighton combined for a brutal 6-8-1, 3.58, .878, and the team was again left one win short of making the playoffs.

Backup goaltending left the 'Canes out of the playoff picture for a third time in '99-00, as the team finished an agonizing one point out after their backup goalies combined to go 3-7-1, 3.22, .883. Apparently the organizational indifference to goaltending depth was carried over from Hartford, as the Whalers had more or less the same thing happen in 1996-97 (two points out of the playoffs despite a great year from Sean Burke because the backups combined for .887 and a 10-17-5 record).

Over the last 17 seasons, the numbers are pretty staggering for the Whalers/Hurricanes franchise:

#1 goalies: .532 win %, 2.60, .911
Backups: .413 win %, 3.15, .890

Those splits aren't entirely fair because there may have been a few times when the preseason #1 goalie was supplanted by a backup (as was the case in 1997-98 with Trevor Kidd outplaying incumbent starter Sean Burke, for example). However, it is still perfectly correct to say that Carolina/Hartford has had mostly awful backup goaltending for the better part of two decades, and that has likely had a significant impact in causing the team to fall short of the playoffs on multiple occasions.

Scouting, evaluating and predicting goaltender performance is always difficult. Not every bet is going to pay off, and many organizations get decisions wrong. Take Buffalo, for example, a team that has developed and employed a number of top-quality netminders in recent years, yet still paid Patrick Lalime $2.65 million for three years of service where the Sabres went 9-26-5 in games where he got the decision. Lalime probably cost his team a playoff spot in '08-09, posting a 5-13-3 record as the Sabres fell just two points short.

There have been a number of other teams that were left outside the playoff pictures because of the performance of their backup goalies. Sometimes teams missed out because a goalie they counted on to be a starter or take on a significant workload in a platoon role simply had an awful season (e.g. '06-07 Avs, '08-09 Predators). Others simply had a few options behind their starting goalie ('09-10 Rangers, '06-07 Maple Leafs).

Most of these examples of weak backup goalies are dealing with small sample sizes, so it may not be entirely fair to blame the goalies. All the standard problems of relying on win/loss records for goalies apply, although in nearly all cases they had awful save stats as well. There may also have been other factors at work. Perhaps they weren't playing a favourable schedule, or maybe some of them just had puck luck go against them for 200-300 shots. There is the very large advantage of hindsight available to us now in pointing out some of these teams' decisions. Yet when there is a long-term trend of undeperformance, as is the case in Carolina, a reasonable criticism can certainly be advanced about the way the team handled their goalie situations.

The overall point is that while it is not smart to pay huge money for goalies, the depth of available goaltending talent means that you should never, ever have to settle for bad goaltending. If you have a hole on the roster, it really doesn't cost much more to pick up a veteran or an up-and-coming talent from Europe than it would to roll the dice on an unproven minor-leaguer in your system. For a penny-pinching playoff bubble team, it's probably well worth it to invest some decent money in goalie scouting and development or free agency to reduce the risk of having a not-ready-for-primetime backup come in and sink the season.

Over the summer Carolina signed Brian Boucher to a two-year deal worth $950,000 per year, which may be a sign that the organization is more willing to loosen up the purse strings for their backup position. Then again, the 'Canes haven't always gone with a backup like Justin Peters; over the last 17 years there have been a number of veterans who were brought in but didn't pan out. I think there is a good chance that pattern may repeat again with Boucher, a guy turning 35 in January with a post-lockout save percentage of .902. Boucher is expected to take more of the load off of Cam Ward this season, but he'll need to deliver good enough results, especially if the Hurricanes again find themselves in a dogfight in the middle of the Eastern Conference. If he does not, then the hopes of Carolina fans may yet again be dashed by their backup goaltender.