First of all, I prefer not to use league rank, but rather the statistics themselves to generate a ranking since that better recognizes dominance. Otherwise somebody can destroy the field by .015 in save percentage and then finish 4th in wins with a couple less than the league leader, and end up ranked behind a guy who finishes 2nd in both categories. Secondly, I don't like the haphazard use of a bunch of related categories, because it ends up double-counting or triple-counting two major statistics: games played (which more or less determines all the counting stats, since obviously more playing time equals more wins, shutouts, saves, etc.), and save percentage (a goalie's ability to stop the puck impacts goals against, and contributes to shutouts and wins).
I think it is fair to include some sort of measure outside of save percentage, and to look at a combination of quantity and quality. I also would prefer to use team context-adjusted stats, rather than simply raw totals.
As such, I decided to use two different measures, both of which reward quality as well as quantity and adjust for the goalie's team context. The first is wins above teammates, calculated by finding the difference between a goalie's winning percentage and the total winning percentage of his teammates, and then multiplying by the number of decisions and dividing by 2 to get a win total. Since we expect that most of the goalie teammates are weaker backup goalies, I also included a .060 winning percentage adjustment for the backup numbers, based on overall league winning percentages for goalies who play backup minutes.
The second is shot quality neutral wins above average, calculated by taking a goalie's shot quality neutral save percentage, subtracting league average, multiplying the difference by the number of shots faced to get a goals above average figure, and dividing that number by 5 since an increase of 5 in goal differential roughly translates to one extra win in the standings based on the Pythagorean expected points theorem. My shot quality numbers were taken from Alan Ryder's Hockey Analytics site. It has been observed that certain rinks around the league vary in the way they record shot distances and types, so I used Ryder's road shot quality results for 2007 and 2008 to reduce rink bias, and I adjusted the 2006 numbers for a few teams that have unusual reporting tendencies. I didn't have the numbers for every individual goalie for each season so I used overall team numbers.
The result is that for each goalie I have two "wins above average" numbers, using two completely different methods. I think the shot-quality neutral method is a superior method, so I decided to give it 2/3 of the weighting, with 1/3 decided by record against backups. If you disagree with the weights or would look at the raw averages I have included all the relevant numbers, but here are the results based on my chosen scoring system:
| Rank | Goalie | SQNSV% | Win%vsTm | SQN Wins | Wins over Tm | Score |
| 1. | Roberto Luongo | .914 | +.165 | 12.0 | 11.8 | 11.9 |
| 2. | J.S. Giguere | .919 | +.143 | 12.9 | 7.0 | 10.9 |
| 3. | Martin Brodeur | .912 | +.164 | 8.9 | 11.8 | 9.9 |
| 3. | Henrik Lundqvist | .916 | +.131 | 11.5 | 6.7 | 9.9 |
| 5. | Miikka Kiprusoff | .909 | +.196 | 5.0 | 15.0 | 8.3 |
| 6. | Tomas Vokoun | .920 | -.041 | 16.5 | -8.6 | 8.1 |
| 7. | Cristobal Huet | .920 | +.051 | 11.5 | -0.6 | 7.5 |
| 8. | Manny Legace | .915 | +.134 | 8.1 | 5.6 | 7.3 |
| 9. | Niklas Backstrom | .916 | +.171 | 5.8 | 5.1 | 5.6 |
| 10. | Tim Thomas | .912 | +.087 | 6.9 | 2.1 | 5.3 |
| 11. | Rick DiPietro | .908 | +.153 | 3.3 | 8.4 | 5.0 |
| 12. | Kari Lehtonen | .915 | -.001 | 9.5 | -4.4 | 4.9 |
| 13. | Dominik Hasek | .914 | +.075 | 6.1 | 1.0 | 4.9 |
| 14. | Mathieu Garon | .910 | +.136 | 5.1 | 4.5 | 4.3 |
| 15. | Martin Biron | .910 | +.042 | 3.9 | -1.1 | 2.2 |
There are 10 goalies that make both Bruce's list and mine. The top 6 are the same on both lists, just reordered, so I think it is pretty clear that those half-dozen guys were the league's best over the given time frame. I don't think there is a clear #1 goalie these days - I think Luongo is a good contender for the title, especially since he has really only had one strong season by his standards in the last three years yet still ranks at or near the top among goalies over that same period, but one could probably make a case for any of Luongo, Brodeur, Lundqvist or Giguere. My estimate is that the difference between any of them is less than one win per season.
Just as a postscript, Mathieu Garon does pretty well by this measure, so I might have been selling him a bit short in my earlier post on him. However, the main reason for his ranking was his success in 2007-08, which as I pointed out was in large part driven by outstanding results on the penalty kill, something which is probably not likely to be duplicated in 2008-09.
11 comments:
Good work, the method looks sound and the results look reasonable. I think this list is certainly as good a list as I have seen.
I agree that the top end of the list looks surprisingly conventional. Giguere is probably higher than most would have him, and Nabokov and Turco are missing. I'm particularly surprised about Turco not being on here. Any comments on that?
Also, what is up with Tomas Vokoun's numbers? He has the best SV% , but his backups have been ridiculously good over the last 3 years. Do you think it's just random and the SV% best reflects his play, or have his teams supported their goalies in some way?
One last request - I think it might be interesting to see a metric of which goalies you predict to be the best this coming year - maybe using the numbers from the last 3 years but weighted in a 3-2-1 pattern or something. While this list is comparable to Bruce's list, most lists of the best goalies are ranking them right now, and a comparison of your metrics and the conventional wisdom would be interesting.
Methodology notes:
I think you are weighting the "Wins above Team" too heavily. I understand why it's included, as this method of comparing a goalie to his backups is probably the best at incorporating everything that a goalie does to help his team win. However, the sample sizes are simply too small to be meaningful over 3 years.
To take an extreme example, Martin Brodeur's backups in New Jersey only recorded 18 decisions over these 3 years. I ran a couple of statistical calculations, and while you are correct that the average difference in W% is 0.164, the standard error is 0.214. This means that we can't even conclude with any certainty that Brodeur was better than his backups, using this method alone!
I ran the same calculation for Rick DiPietro, whose backups saw more playing time. The standard error of 0.135 was lower than Brodeur's, but still not enough to conclude with a high degree of confidence that he was better than his backups.
I would suggest using the W% method only over careers, where the sample sizes may be large enough to give valid results, and use GAA or SV% comparisons against teammates for smaller samples. A GAA comparison specifically would seem to incorporate all aspects of goalie play short of "clutch" play, and I think it might be better for the purposes of this exercise.
Correction to the methodology part of my last comment. I mixed up W% and Pts/G while calculating standard error for the W% difference between the goalie and his backup. The standard error on Brodeur's W% over backups is 0.107, and the standard error on DiPietro' W% over backups is 0.067.
This means that we can in fact conclude that DiPietro has a better W% ability than his backups at a 95% confidence level. Brodeur's advantage isn't quite at a 95% confidence level, but is very close.
I still think that the W% over backups measure is volatile enough that it shouldn't be weighted too much over a 3-year span, but my earlier comment overstated this.
Thanks CG. Another great post.
You said that you think that the shot quality neutral method is superior, but also that you're confident about the top six guys being listed being the top six guys over the last three years. If I order the list according to your preferred method we get:
Vokoun 16.5
Giguere 12.9
Luongo 12.0
Lundqvist 11.5
Huet 11.5
Lehtonen 9.5
Brodeur 8.9
Legace 8.1
Thomas 6.9
Hasek 6.1
Here, Lehtonen and Huet really move up strongly while Brodeur and especially Kipprusoff (out of the top ten) move well down. Which list do you think is a better reflection of performance over the last three years?
regardless of what methods you use its futile when guys like luongo and giguere keep playing like shit, and brodeur takes a team not picked to make the playoffs to a 4-1 start.
mr best boalie in the league luongo is giving up like 4 goals a game. he has yet to put up best goalie in the league type numbers consistently.
Overpass: Regarding the methodology, I would be fine with putting 100% of the weighting on the save statistics, as I agree small sample size is an issue. I generally like to include more seasons when comparing backup stats. For some of these guys the numbers look to be very strongly influenced by randomness or weak/strong backups (Kiprusoff, Vokoun), but I think for someone like Rick DiPietro or Manny Legace it can help show the positive impact on their teams.
I am not in favour of using a GAA comparison, because I don't think you gain any extra insight from the stat. The two inputs are save percentage and shots against per game, so it only adds extra information beyond save percentage if goalies can affect the number of shots they face per game. If that is the case, why not just isolate shots per game in the analysis (or some variant, e.g. shots per game compared to teammates) and use that number, either on its own or as an adjusting factor in save percentage?
Marty Turco has been an average goalie since the lockout. Pre-lockout: 185 GP, 1.91 GAA, .922 save %. Post-lockout: 202 GP, 2.42 GAA, .903 save %. Turco ranked just 26th on my original list, and 18th in the shot-quality wins above average, although he is trending up slightly. His backups have had almost identical stats for three years in a row. I think Turco is an example of how playoff results shape perception - when he was putting up elite regular season numbers and failing in the playoffs he was underrated, and now he is largely overrated because he had a couple of decent playoff runs even as his overall results have dropped off.
The stats say that Evgeni Nabokov is not even an average goalie. He ranked 39th on my original list, and 32nd just based on SQN wins (.898 adjusted save percentage over the last three seasons). He also had the worst record in the league in terms of record against teammates - I calculated him as being 14.9 wins worse than average, and that is based on 84 teammate decisions. Last year was his best year out of the last three, so a forward-looking ranking would rate him substantially higher, but I still think Nabokov is quite overrated.
I think it might be interesting to see a metric of which goalies you predict to be the best this coming year
OK, I'll run a SQNSV% and raw save percentage prediction for the starting goalies based on the last three seasons in an upcoming post.
Scott, there are a couple of things to be aware of when we look at these numbers. One, these shot quality numbers are based on distance only, not distance and angle. I also used team approximations, and there are rink reporting differences that have been minimized but still impact the data. I would not, for instance, declare Kari Lehtonen to be a certifiably superior goalie to Brodeur just because he outscores him by 0.2 wins per season based on my metric. Essentially with every goalie statistic we are looking for what range a guy falls into and who his closest comparables are, because team effects and randomness make it so that we can't trust anything with 100% certainty.
Vokoun, Giguere, Luongo and Lundqvist all rank in the top 4 in the shot quality neutral stat, and Brodeur ranks 7th which is close enough to put him in the top group with everything else considered. The tricky guy is Kiprusoff, and the reason is that he has one great season, one decent season, and one poor season in the last three. He makes my top 6 above because of his winning percentage, but I think poor quality backups and small sample size probably have a lot to do with that. However, the consideration is whether 2007-08 was a down year or Kipper's new skill level. Over 2005-06 and 2006-07 he was right up with the leaders in goals saved above average, but then he dropped from 4th to 13th over the course of last season. I think we will know a lot better after this season whether he is still an elite goalie or not, but early signs are not very positive. I think you are right, though, I was a bit too hasty in declaring the top 6. The saves list is the better list, and as such Kiprusoff probably doesn't deserve to be in with that top group for the last three seasons based on his save record.
Anonymous:
1. These are backwards-looking metrics, evaluating how well a goalie has played, not how well he will play.
2. Anyone not picking the Devils to make the playoffs is not paying attention.
3. Before you bash Luongo and Giguere, recall that Martin Brodeur started last season 2-4, 3.50, .865.
4. It is still very, very, very early.
Thanks CG. I'm also really looking forward to your forward looking analysis, as I'm sure it will be very interesting.
I think your point about ranges is good, and agree that 0.6 wins over three seasons is not much at all. But what do you think the ranges should be?
My intuitive feel would be:
Vokoun 16.5
Giguere 12.9
Luongo 12.0
Lundqvist 11.5
Huet 11.5
Lehtonen 9.5
Brodeur 8.9
Legace 8.1
These guys would seem to be a pretty clear top 8 over the last few years, and Vokoun really does look way ahead. The breaks seem pretty natural to me. Do you think Vokoun is playing that much better than everybody else?
Do you think Vokoun is playing that much better than everybody else?
Judging from the -8.6 in the adjacent column, he's not even outplaying his own creasemates. :o
In a recent post, knowledgeable goalie observer and frequent commenter Bruce did a ranking of the best goalies since the lockout. I think his list is pretty solid, although I do have a couple of issues with his method, which I am going to briefly point out, and then propose my own criteria and ranking.
Thanks for the acknowledgement, the link, and the response.
First of all, I prefer not to use league rank, but rather the statistics themselves to generate a ranking since that better recognizes dominance. Otherwise somebody can destroy the field by .015 in save percentage and then finish 4th in wins with a couple less than the league leader, and end up ranked behind a guy who finishes 2nd in both categories.
I did look for that, and the only guy who destroyed the field was Brodeur, in Wins (135 to 121, or about 10%). He also had a ~10% edge in shutouts, albeit just 21 to 19. Luongo was about 5% ahead in total saves, while Backstrom's advantage in the two percentage categories was less than that.
Secondly, I don't like the haphazard use of a bunch of related categories, because it ends up double-counting or triple-counting two major statistics: games played (which more or less determines all the counting stats, since obviously more playing time equals more wins, shutouts, saves, etc.), and save percentage (a goalie's ability to stop the puck impacts goals against, and contributes to shutouts and wins).
Again I stress my "haphazard" methodology was to use a set of data made available to me, and that at no time did I make any claims stronger than "that list seems pretty darn good". The effective double- and triple-counting of some of the counting stats was acknowledged from the outset, and an attempt was made to mitigate this effect by weighting. In effect I 10X counted your precious Sv% in my third method, whereas the most I could possibly credit minutes played (as manifest in Wins, Shutouts, and Saves) was 6X.
Since we expect that most of the goalie teammates are weaker backup goalies, I also included a .060 winning percentage adjustment for the backup numbers, based on overall league winning percentages for goalies who play backup minutes.
I don't understand why you feel this is necessary. Why convolute the method with adjustments? I'm not even sure which direction the adjustment goes. How much better was the guy than his backup, full stop?
Interesting that Giguere's SQNSv% is .007 higher than Brodeur's when his actual Sv% was lower. Also his MP, Wins, Shutouts, Saves, and GAA were all behind Brodeur, yet somehow Giguere > Brodeur. I'll have to have another look at Ryder's data before I can comment further, but it seems curious.
Sorry, Bruce, I didn't mean to come off as overly critical, because I know you were just taking a quick look at the numbers you had available. The opening paragraphs were meant to just explain the reasoning behind my choice of method.
The reason I included an adjustment was because I wanted the stat to approximate wins above average and be roughly comparable to the shot-quality neutral one. Backup goalies are, in general, worse than the average NHL goalie, so wins vs. backup would result in substantially higher numbers. I debated the point myself, and in the end decided to add in a linear adjustment so that it wouldn't have much affect on the relative rankings. If you take out the adjustment the list is still almost in exactly the same order, and the ranking of the top 6 is unaffected.
Do you think Vokoun is playing that much better than everybody else?
Not really, no. I think Vokoun has been the most consistently good goalie in the league over the last 3 seasons, and he ranks in first place because he has managed to avoid the down year that most of the other goalies have had. I wouldn't say Vokoun is necessarily better than guys like Luongo or Lundqvist or Brodeur when they are all playing at their best, but Vokoun has probably performed better cumulatively over the last 3 seasons. Another thing I'm not sure about, though, is if the shot quality numbers are truly reflecting the Nashville goaltending situation, since Mason and Ellis have also done pretty well there over the last few years.
Post a Comment