Thursday, October 16, 2008

Pat Burns: A Goalie's Best Friend

What is the impact of a coach in hockey? That is one of the most difficult analytical questions in hockey, as indeed it is in all sports. Certainly there are many coaches who have reputations either built or ruined because of the talent of their teams. For the most part, I think most observers would agree that talent is the most important factor in determining which teams are successful. However, when it comes to evaluating goalie performance we are not only concerned with overall results (i.e. wins and losses), but also how well the goalie stops the puck and how many goals he allows. This a coach can certainly impact through establishing a team's style of play.

Jacques Lemaire is well-known as a defensive guru, the statistics of his goalies have been consistently excellent in both New Jersey and Minnesota (see this post for some relevant numbers). However, it is difficult to evaluate Lemaire because except for a short stint in Montreal he has worked for only two teams, New Jersey and Minnesota, and as the only coach the Wild has ever employed it is very difficult to estimate how well anyone else would have done in that position. We can guess based on how goalies did on other teams compared to how they did on a Lemaire-coached team, but we don't really know how good the other teams are and the sample sizes are mostly small. Ideally, we would prefer to look at a coach who coached on multiple teams, and evaluate how the team did before he arrived, during his tenure, and after he left. This would allow us to evaluate the overall shot prevention effect and shot quality/goaltending effect of his defensive system.

Pat Burns fits the criteria of a defensive coach with multiple stops, so I decided to focus on him. My method was to look at the two seasons before he was hired, and the two seasons after he was fired, and compare those results to how he did as coach. Since Burns was fired very early in the 2000-01 season, I decided to make the judgment call of counting that campaign as the first season after Burns, rather than his final season.

I looked at how Burns did in Montreal, Toronto, Boston and New Jersey. In the years prior to Burns showing up, his teams allowed 3.05 goals per game, 27.6 shots per game, and posted an .889 save percentage. In the years after Burns left, his teams allowed 2.78 goals per game, 28.7 shots per game, and his goalies stopped shots at a .903 rate. With Burns as coach, his teams alowed 2.57 goals per game, faced 27.5 shots per game and had their goalies post a .907 save percentage. If we look at shutouts, something that often influences Vezina voting, the goalies had .047 shutouts per 60 minutes before Burns, .074 shutouts per 60 with Burns, and .060 shutouts per 60 after Burns, so they were about 25-30% more likely to record a shutout with Burns behind the bench.

Just from the raw numbers, it looks pretty clear that Pat Burns has a positive effect on team defence. The problem here is obviously that most of the years prior to Burns' arrival came in the late 1980s or early 1990s, when scoring levels were higher, whereas his "after" years came when you would expect reduced scoring and improved goalie performance. We therefore need to normalize the numbers to league average. I took the average goals per game, shots against per game, and save percentages for Burns' career (3.08, 29.0, .897) and used them as adjustment factors to remove league effects for each individual season.

Prior: .517, 2.95, .893, 27.5 SA/60
Burns: .567, 2.57, .907, 27.6 SA/60
After: .542, 2.96, .899, 28.9 SA/60

With adjustments, the numbers are clear: Pat Burns helps his goaltenders out to a substantial degree, something in the neighbourhood of a .010 boost in save percentage and a .40 reduction in GAA.

The argument could be made that Burns had better goalies in net. Felix Potvin and Byron Dafoe both became starters in Burns' first year as coach for their respective teams, and they were better goalies than their predecessors. However, both were still around after Burns had left, and we see how the numbers fell back down to earth. I think the small discrepancy between the before and after results can be partly explained by goalie quality, but for the most part the same goalies were employed both during Burns' tenure and in the seasons after he left, so talent does not account for the performance improvement under Burns.

To try to avoid the goalie talent issue altogether, we can look at just the Montreal and New Jersey results, since in both cases Patrick Roy and Martin Brodeur were in net before, during, and after Pat Burns was coach (again the numbers are adjusted to league averages):

Prior: .608, 2.51, 26.0, .905
Burns: .610, 2.37, 26.2, .911
After: .611, 2.75, 28.2, .906

The results are similar, although less striking (which is partly because Montreal was already a strong team and had less room to improve than some of the other teams Burns worked for). Of course player talent is another consideration - for example, the New Jersey Devils would have been expected to suffer some dropoff in defensive play with the loss of Stevens and Niedermayer. However the persistence of the defensive improvement across four different teams makes it pretty likely that Burns' defensive philosophy was responsible for improved play.

Team defence is made up of shot prevention, shot quality against, and goaltending. Pat Burns' defensive effect seems to be primarily based on reducing opposing shot quality. His teams gave up almost exactly the same relative number of shots per game as they did previously, yet the goalies saw their numbers skyrocket under Burns' watch. Pat Burns is known for wearing out his welcome in a city after a few seasons, but I doubt he ever was that unpopular with the goalies. This is more evidence that goalies can receive a significant advantage by playing under the right coach in the right defensive system.


Scott Reynolds said...

Really great post. I always appreciate your commentary. It would be interesting to see the results for a guy like Joel Quenville in St. Louis and Tony Granato in his first Colorado stint to get an idea of what regression or improvements might be attributed to coaching in the terrible goaltending experiment that is this year's Colorado Avalanche.

What formula do you use in order to normalize the data?

The Contrarian Goaltender said...

For each season, I calculated the expected number of goals, shots against, and saves a typical goalie would have made playing exactly the same minutes (based on league averages). That gives us an equivalent league average GAA, shots/game, and save percentage. I then multiplied the raw before/during/after stats by this league average number, and divided by the normalizing number to put everything on the same scale.

Granato hasn't really coached much, and I don't like to use these kinds of techniques in really small sample sizes because it could be luck rather than skill and I'd rather avoid assigning meaning to pure randomness. Quenneville's record, however, might give us some information. The data certainly suggest his style is to reduce shots against but make it difficult on the goalie with increased shot quality against.

I ran Quenneville's numbers quickly:
Prior: .538, 2.68, 28.8, .907
During: .584, 2.45, 24.8, .900
After: .421, 3.35, 30.8, .891

Remember the standard warning about small sample size, and the fact that Quenneville had a terrific defensive team in St. Louis but not a top goaltender likely had a lot to do with the numbers. Ignore the after results too - they are only from the first two post-lockout years in St. Louis where the team would probably have been awful with anyone coaching. But from the before and during numbers there is a big gap in both shots against and save percentage.

I also looked at Alan Ryder's shot quality numbers, and Colorado was worse than average in each of the last three seasons, even though they were above average in shot prevention in all three years. That supports the theory that Quenneville teams focus on shot prevention at the expense of increased quality of scoring chances against.

So if Granato is continuing what Quenneville started, then it could be an ugly year in net in Colorado. The Avalanche goalies aren't very good to begin with, but it doesn't help that apparently play in a system that makes them look even worse.

Bruce said...

Interesting stuff, CG. I wonder how similar comparisons might look for Robbie Ftorek and Larry Robinson?

overpass said...

Good stuff. Like you said, this can only be applied in certain cases because of sample size issues and changes in team composition, but even a limited application suggests that coaches can have a major impact on defensive results.

I bet Ken Hitchcock could give Burns a run for his money as a goalie's best friend - IIRC Columbus reduced their shots against pretty dramatically after they hired him, and a quick check of the numbers suggests he had a similar effect in Dallas and Philadelphia.

Too bad you can't do a similar comparison with Lemaire. I really wonder how many hockey fans realize the crooked numbers that Minnesota goalies put up every year are a product of Lemaire's defensive philosophy. I think it's pretty widely recognized, but I'd like to see Lemaire give Backstrom 70 starts this year and let him get 40 wins with the usual low GAA and SV% that any Minnesota goalie has. I wouldn't be surprised if a lot of people forgot about Minnesota's great defence and started talking Backstrom up as an elite goalie.

Scott Reynolds said...

Thanks for the results CG. This is really amazing stuff. Seeing Chicago hire Quenville is great news. We can maybe test this out over the year to see how Quenville changes things as compared to Savard. It will probably be a little bit difficult given Huet's superiority to Khabibulin, but certainly the difference in shots and shot quality could still be measured with a good degree of reliability.

Host PPH said...

It is the most important part of the team. It is like a movie director or something even better.