tag:blogger.com,1999:blog-8148461224473220694.post3642915314650702670..comments2024-05-21T06:15:26.295-04:00Comments on Brodeur is a Fraud: Shot Quality at EvensThe Contrarian Goaltenderhttp://www.blogger.com/profile/03433370306939690205noreply@blogger.comBlogger26125tag:blogger.com,1999:blog-8148461224473220694.post-13741348668375887312009-05-08T13:59:00.000-04:002009-05-08T13:59:00.000-04:00I mean CG, I agree on all counts. :DI mean <B>CG</B>, I agree on all counts. :DVic Ferrarihttps://www.blogger.com/profile/16425585921916867277noreply@blogger.comtag:blogger.com,1999:blog-8148461224473220694.post-89989751542864339762009-05-08T13:58:00.000-04:002009-05-08T13:58:00.000-04:00overpass:
I agree on all counts.overpass:<br /><br />I agree on all counts.Vic Ferrarihttps://www.blogger.com/profile/16425585921916867277noreply@blogger.comtag:blogger.com,1999:blog-8148461224473220694.post-23903864722438472052009-05-08T13:27:00.000-04:002009-05-08T13:27:00.000-04:00"overpass: Ability persists, randomness doesn't. I..."overpass: Ability persists, randomness doesn't. If the EV shot quality metrics relate well over short stretches, but "even out over a season" as you say, then that is highly suggestive that they are measuring luck."<br /><br />That may be true, but whether or not you are measuring luck the goalie still has to deal with the resulting scoring chance.<br /><br />If Carey Price faced more difficult shots in round one of the playoffs this year than Tim Thomas, for example, which my tracking suggests that he did, then he had the tougher job. That's true whether that was because of Boston's skill, Montreal's defensive style, or pure dumb luck. It would only be fair to take that into account when evaluating his performance.<br /><br />I therefore see shot quality as useful in evaluation, even if it has little predictive ability.The Contrarian Goaltenderhttps://www.blogger.com/profile/03433370306939690205noreply@blogger.comtag:blogger.com,1999:blog-8148461224473220694.post-76582417992638071272009-05-08T10:58:00.000-04:002009-05-08T10:58:00.000-04:00To add, the opposite holds true for measurements o...To add, the opposite holds true for measurements of ability (such as trivia craps, where the trivia ability of the contestent isn't particularly strong over short stretches, but grows with more games played, is the only predictor of future results (though future dice randomness still has the biggest say in future results), and repeats well through any two sets of random samples of trivia crps games.<br /><br />Makes sense, no?<br /><br />So the relationship of scoring chances to a player's results over 10 games has a correlation of only r=0.4 for the Oilers last season. If we pretended that there were no luck in the game, and that the dsitribution of talent was normal, they we would deduce that r^2 = 16% of a player's outscoring results stem from his outchancing numbers.<br /><br />And that his corsi numbers were even less important, in fact trivial.<br /><br />IMO the correct even strength team model should reflect corsi as ability (until we have a better measure for territorial advantage)<br />to the power of 1.2 to correct for the 'playing to the score' effect. And scoring chances are begat from corsi in random fashion (surprising, but true for the Oilers last season at least), resulting in a 65 scoring chances per 100 shots at net, as averaged in the population. <br /><br />And scoring chances turn into goals in random fashion at a 22 in 100 clip. Corrected up or down a bit depending on the historical ability of the team's players to finish and the team's goalie to stop pucks at even strength.<br /><br />I'm sure that model will end up with a lot of randomness in results. But so does hockey at evens, even over 82 games. And they will be eerily similar.<br />The relaVic Ferrarihttps://www.blogger.com/profile/16425585921916867277noreply@blogger.comtag:blogger.com,1999:blog-8148461224473220694.post-43048699743735015252009-05-08T10:38:00.000-04:002009-05-08T10:38:00.000-04:00Outstanding stuff, CG.
As you seem to have it in ...Outstanding stuff, CG.<br /><br />As you seem to have it in spreadsheet form and to hand, what are the standard deviations of the two samples?<br /><br />I suspect that they are even more similar than the year to year changes. Implying that the difference in mean from the two samples explains much of the .001 to .002 difference now remaining. This being indicative of an imperfect assessment of the goalie market by GMs.<br /><br />overpass: Ability persists, randomness doesn't. If the EV shot quality metrics relate well over short stretches, but "even out over a season" as you say, then that is highly suggestive that they are measuring luck.Vic Ferrarihttps://www.blogger.com/profile/16425585921916867277noreply@blogger.comtag:blogger.com,1999:blog-8148461224473220694.post-5485712990680078722009-05-08T10:21:00.000-04:002009-05-08T10:21:00.000-04:00ILR: You're right about that, Aebischer played mo...ILR: You're right about that, Aebischer played most of the season in Colorado that year. The NHL data that I am using doesn't separate by team for goalies who were traded mid-season. I would expect that could have contributed to slightly reducing the observed spread.<br /><br />It might be worth taking out any who switched teams midway through the year and then re-running the numbers to check if that was a factor.The Contrarian Goaltenderhttps://www.blogger.com/profile/03433370306939690205noreply@blogger.comtag:blogger.com,1999:blog-8148461224473220694.post-32703018194857316942009-05-08T03:22:00.000-04:002009-05-08T03:22:00.000-04:00Umm... wasn't Aebischer-Theodore a deadline deal i...Umm... wasn't Aebischer-Theodore a deadline deal in 2006? If so, how should that be reflected in the same team / different team distinction?<br /><br />I mean, if my memory is correct, he played 2/3ths of the season in Colorado so he's bound to have racked up some of that statistical drop-off from there as well. Although you could make the case that Colorado before and after the lockout were de facto different teams to begin with.ILRnoreply@blogger.comtag:blogger.com,1999:blog-8148461224473220694.post-5915386582352613102009-05-07T23:53:00.000-04:002009-05-07T23:53:00.000-04:00It does appear that the differences between even-s...It does appear that the differences between even-strength save percentage and shot-quality metrics mostly even out over a season. I don't have any insight into the reason or the implications. <br /><br />Despite this result, I don't think there's any reason to throw the shot quality metrics out. It seems to me they probably still add value when looking at individual games or short stretches of play like a playoff series, where we can't necessarily expect things to even out as they do over a full season.overpasshttps://www.blogger.com/profile/05106889069832791092noreply@blogger.comtag:blogger.com,1999:blog-8148461224473220694.post-40401225707628279732009-05-07T21:14:00.000-04:002009-05-07T21:14:00.000-04:00Assuming these results are valid and there isn't a...Assuming these results are valid and there isn't anything we're missing, I have to admit it doesn't look there is a whole lot there in terms of EV team shot quality differential. Certainly less than I was expecting. I don't think it rules out a few outlier teams, but it suggests that if there are some then probably not very many.<br /><br />It still seems odd that average shot distance against, or blocked shot frequency, or defensive zone coverage system might not have much of an effect. What exactly are shot quality metrics measuring then?<br /><br />I also have run across results that suggests certain coaches have higher save percentages when they are in charge (Pat Burns, for example). Is this entirely a special teams effect? Also, in my work on playing to the score effects I looked at third period results in the playoffs, and it seemed like shooting percentages dropped slightly for the trailing team and rose slightly for the leading team. I highly doubt that had anything to do with special teams, most of that sample was from the Dead Puck Era where the refs swallowed their whistles and you could probably get away with killing a guy late in the third of a tie game.<br /><br />If EV SV% is the key stat in today's NHL, how far back can we make that same conclusion? These results make it look like today's NHL has a high degree of parity, but I find it hard to believe that the 1977 Montreal Canadiens had the same shot quality against as the 1977 Washington Capitals. We know from international results, junior results, playing against the top team in the local beer league, etc. that at some point a major skill gap does result in a real difference in shot quality.The Contrarian Goaltenderhttps://www.blogger.com/profile/03433370306939690205noreply@blogger.comtag:blogger.com,1999:blog-8148461224473220694.post-55323786515828581612009-05-07T20:55:00.000-04:002009-05-07T20:55:00.000-04:00OK, I normalized to league average, and used Vic's...OK, I normalized to league average, and used Vic's formula to express everything in terms of goals relative to average over 1,000 shots.<br /><br />Let's use Aebischer as an example again to demonstrate and to make sure I'm doing it right.<br /><br />Step 1: Convert yearly difference to goals<br /><br />2003: (.926-.929) * 447 = -1.2<br />2004: (.932-.926) * 1319 = +7.5<br />2006: (.907-.932) * 1061 = -26.6<br />2007: (.914-.907) * 700 = +5.0<br /><br />Step 2: Adjust for sample size<br />2003: -1.2 * sqrt(1000/447) = -1.7<br />2004: +7.5 * sqrt(1000/1319) = +6.5<br />2006: -26.6 * sqrt(1000/1061) = -25.8<br />2007: +5.0 * sqrt(1000/700) = +5.9<br /><br />Step 3: Take absolute values of differences<br /><br />Step 4: Compare averages for goalies who played on the same team and goalies who played on different teams.<br /><br />I used the same minimum game cutoffs again, just to see what the results would say. I think results are too variable to be worth considering when goalies are playing 1 or 2 games, but with the sample size adjustment even results from 5 games seem to be on a similar scale to full season results.<br /><br />0 GP: 12.0 same, 12.6 different<br />5 GP: 10.4 same, 11.5 different<br />10 GP: 10.3 same, 12.1 different<br />20 GP: 10.0 same, 11.9 different<br />30 GP: 10.0 same, 10.9 different<br />40 GP: 9.8 same, 11.6 different<br /><br />Looks like the adjustment works. The goalies playing the same team as the year before average a difference of 10 goals per 1,000 shots, and the guys who switched teams average a difference of 11-12. This suggests that team differences may account for only 1-2 goals per 1,000 shots at even strength, or about .001-.002 in save percentage.<br /><br />One thing that might still need to be accounted for is if we have reason to expect more regression to the mean for either sample. I looked at averages of the actual goal differentials instead of taking the absolute value, to see if the net effect was positive or negative.<br /><br />For goalies on the same team, the average was about -2, or 2 goals worse than the year before, and that was pretty consistent when I tried different minimums.<br /><br />For goalies who switched teams, the averages fell as the cutoffs rose.<br /><br />5 GP: -0.5<br />10 GP: -0.8<br />20 GP: -1.6<br />30 GP: -2.8<br />40 GP: -3.6<br /><br />What I guess that means is that they were slightly more likely to be coming off of subpar seasons if they hadn't played many games, i.e. their team sold them low, but were also slightly more likely to be coming off good seasons if they had played lots of games, i.e. someone bought them high. Or could there be another reason, backups playing weaker competition or something like that? I'm not really sure, does anyone have any thoughts?The Contrarian Goaltenderhttps://www.blogger.com/profile/03433370306939690205noreply@blogger.comtag:blogger.com,1999:blog-8148461224473220694.post-73880531614087256442009-05-07T19:06:00.000-04:002009-05-07T19:06:00.000-04:00Christopher: Actually, it is assuming that there ...Christopher: Actually, it is assuming that there are no shot quality effects at all at 5 on 5. That is the point of this exercise, to try to provide evidence for the existence of shot quality effects at even-strength.<br /><br />Vic and others have suggested that shot quality has a very minimal effect at even-strength. If that is the case, we would expect that there should be little difference between goalies playing on the same team as the prior season compared to goalies who switch teams, since they all would be effectively in the same boat. On the other hand, if the goalies changing teams see much greater save percentage swings, then we know that something must be impacting the results, most likely shot quality effects.The Contrarian Goaltenderhttps://www.blogger.com/profile/03433370306939690205noreply@blogger.comtag:blogger.com,1999:blog-8148461224473220694.post-40628362459533211862009-05-07T18:08:00.000-04:002009-05-07T18:08:00.000-04:00isnt this also assuming that each team's level of ...isnt this also assuming that each team's level of shot quality is fairly even year to year?Topher0820https://www.blogger.com/profile/04248864107532023162noreply@blogger.comtag:blogger.com,1999:blog-8148461224473220694.post-58861430695134098992009-05-07T13:30:00.000-04:002009-05-07T13:30:00.000-04:00Another thing I thought of that might be influenci...Another thing I thought of that might be influencing the data is the league context. League averages are fairly steady in this period at even strength, but they do move a few thousandths here and there and since that is likely to be the kind of difference we are talking about it I should be adjusting for that to remove that variable.<br /><br />I'm going to add that correction in and then address some of your follow-up questions, Vic.The Contrarian Goaltenderhttps://www.blogger.com/profile/03433370306939690205noreply@blogger.comtag:blogger.com,1999:blog-8148461224473220694.post-82194393460662059062009-05-07T10:49:00.000-04:002009-05-07T10:49:00.000-04:00OK, I see what you are getting at Vic. That makes ...OK, I see what you are getting at Vic. That makes sense to me.<br /><br />I'm not certain it will make a difference, as CG's cutoffs should eliminate most of the sample size issues, but it would be a more correct way of looking at it.overpasshttps://www.blogger.com/profile/05106889069832791092noreply@blogger.comtag:blogger.com,1999:blog-8148461224473220694.post-71428521810660162672009-05-07T10:07:00.000-04:002009-05-07T10:07:00.000-04:00Ah, thanks for the example, CG.
Did the goalies w...Ah, thanks for the example, CG.<br /><br />Did the goalies who changed teams seem be more likely to be coming off of subpar seasons, CG?<br /><br />And do backups move teams more than starters? It seems that way to me, though I may well be wrong. Now seems like the time to ask though, as you have the data to hand.<br /><br />overpass or others will correct me if I'm wrong. But I think you still have to correct for sample size (which is the number of shots each goalie faced, here).<br /><br />So If Aebischer faced 500 shots in a season and saw a swing in EVsave% of -.030, that's a swing of 15 extra goals.<br /><br />That swing of -.030 is, for certain, much more dramatic if it happened in a year that he faced 1500 shots. That would be an extra 45 goals!<br /><br />if you level each season to 1000 games, so if he faced 500 shots, then 15 extra goals * sqrt(1000/500) = 21.2 extra goals, corrected for 100 shots.<br /><br />Put another way, if you take your spreadsheet and sort it by number of games played, then you'll see that the guys who faced more shots see substantially smaller swings in save%.<br /><br />That's not because they're more consistent, it's sample size effect. Use the method I've shown above and the swings in expected goals will show no material difference between guys who played a bunch and guys who played much less. Because we've levelled the field for sample size (in this case sample size would be shots against per goalie).<br /><br />Does that make sense. You could post the raw data in chart format here, or email me the spreadsheet if you like, my explanation may still not be clear (and may be flawed, I'm wide open on alternate <br />suggestions to solve the sample size problem, we still haven't allowed for the fact that the previous season may have had the small sample).<br /><br />still, I suspect that the change from year to year by 'change in goals allowed based on correction to 1000 shots' should remove much of the 'change in team effect'.<br /><br />Really, what we are looking for is a model. <br /><br />So, alternatively, if you weighted a coin to flip a head 92.1% of the time (expected EVsave% of the goalie from three prior years, knowing that NHL.com calc that wrong in their script some years). And flipped it 300 times and counted the tails, then flipped it 500 times again and couted the tails. How much did the 'flipping percentage' change from one sample to the other? Run a thousand trials and what was the average change?<br /><br />This gives us a reasonable expectation, because the guy who flips more coins in a season (say 1300 in one trial and 1100 the next) will see a smaller shift in flipping% change, sample to sample.<br /><br />Does that make sense?Vic Ferrarihttps://www.blogger.com/profile/16425585921916867277noreply@blogger.comtag:blogger.com,1999:blog-8148461224473220694.post-3372978486496305342009-05-07T02:07:00.001-04:002009-05-07T02:07:00.001-04:00Although again if you have any more advanced calcu...Although again if you have any more advanced calculations you would like to test out, I have the data, so let me know.The Contrarian Goaltenderhttps://www.blogger.com/profile/03433370306939690205noreply@blogger.comtag:blogger.com,1999:blog-8148461224473220694.post-32600024905510187152009-05-07T02:07:00.000-04:002009-05-07T02:07:00.000-04:00Confusion because of my careless use of terms, pro...Confusion because of my careless use of terms, probably. I should have written "difference", not "variance". Sorry about that.The Contrarian Goaltenderhttps://www.blogger.com/profile/03433370306939690205noreply@blogger.comtag:blogger.com,1999:blog-8148461224473220694.post-84442740273978189142009-05-07T01:13:00.000-04:002009-05-07T01:13:00.000-04:00Sounds good. My guess is that there was a confusio...Sounds good. My guess is that there was a confusion of terms here, and Vic suggested the sample size correction because he thought you were working with statistical variance, where it's true that a smaller sample size will show more variance. However, if you've just taken a simple average of the y-t-y differences, I don't think that sample size will affect the results.<br /><br />Is that right, Vic? Or am I missing something?overpasshttps://www.blogger.com/profile/05106889069832791092noreply@blogger.comtag:blogger.com,1999:blog-8148461224473220694.post-21448036921692769692009-05-07T01:04:00.000-04:002009-05-07T01:04:00.000-04:00Overpass: That's right, the absolute value of the...Overpass: That's right, the absolute value of the difference in save percentages between year 1 and year 2. Nothing particularly fancy.<br /><br />Take David Aebischer, for example, because he's the first guy alphabetically.<br /><br />2002: COL, .928<br />2003: COL, .928, variance of .000, same team<br />2004: COL, .937, variance of .009, same team<br />2006: MTL, .905, variance of .032, different team<br />2007: MTL, .914, variance of .009, same teamThe Contrarian Goaltenderhttps://www.blogger.com/profile/03433370306939690205noreply@blogger.comtag:blogger.com,1999:blog-8148461224473220694.post-65605993488303967052009-05-07T00:53:00.000-04:002009-05-07T00:53:00.000-04:00What I did was go through every goalie season sinc...<I>What I did was go through every goalie season since 1999-00, and compared the goalie's even-strength save percentage that season with their number from the previous season. I then sorted them by whether they were playing on the same team, or on a different team. This gives us a bunch of save percentage variances, which I averaged to get the typical year-to-year difference between a goalie playing on the same team as the year before, and a goalie playing on a different team the year before.</I>By "save percentage variances" do you mean absolute value of the difference in save percentage between year 1 and year 2? That's how I read it, and if that's the case I don't believe you need to correct for sample size. It's just a simple average.<br /><br />If you mean something different (which I think Vic is assuming) then I'm not sure what you did.overpasshttps://www.blogger.com/profile/05106889069832791092noreply@blogger.comtag:blogger.com,1999:blog-8148461224473220694.post-89156729551245750632009-05-06T22:18:00.000-04:002009-05-06T22:18:00.000-04:00I'm sure that you're right about having to level t...I'm sure that you're right about having to level the playing field. I just subtracted the rates basically with a couple of Excel formulas, kind of as an acid test more than anything, but it does make sense that the smaller sample would have more variance.<br /><br />What do you suggest to correct for the sample size issue?The Contrarian Goaltenderhttps://www.blogger.com/profile/03433370306939690205noreply@blogger.comtag:blogger.com,1999:blog-8148461224473220694.post-71661659030795076362009-05-06T22:01:00.000-04:002009-05-06T22:01:00.000-04:00squared should read 'square root'squared should read 'square root'Vic Ferrarihttps://www.blogger.com/profile/16425585921916867277noreply@blogger.comtag:blogger.com,1999:blog-8148461224473220694.post-61648080624630920532009-05-06T21:59:00.000-04:002009-05-06T21:59:00.000-04:00Okay, still not dead sure that I'm clear, but this...Okay, still not dead sure that I'm clear, but this is terrific stuff.<br /><br />Wouldn't you need to sum up the saves and shots for each goalie? <br /><br />N2 is always going to be smaller than N1 of course, so the group coming from the smaller sample will be expected to show more variance, no? If all goalies faced the same number of shots (clearly not the case), then n2/n1 squared would be the correction, the equalizer. But we have more math to do to level the playing field here I think.<br /><br />Does that make sense?Vic Ferrarihttps://www.blogger.com/profile/16425585921916867277noreply@blogger.comtag:blogger.com,1999:blog-8148461224473220694.post-10301140610089656212009-05-06T21:44:00.000-04:002009-05-06T21:44:00.000-04:00Sorry, I wrote this post pretty quickly and that r...Sorry, I wrote this post pretty quickly and that really wasn't very well explained at all.<br /><br />What I did was go through every goalie season since 1999-00, and compared the goalie's even-strength save percentage that season with their number from the previous season. I then sorted them by whether they were playing on the same team, or on a different team. This gives us a bunch of save percentage variances, which I averaged to get the typical year-to-year difference between a goalie playing on the same team as the year before, and a goalie playing on a different team the year before.<br /><br />I also included the numbers of each just so we know what kind of sample was involved (N1 = same team, N2 = different team).<br /><br />The first one included every goalie season, even the ones where a guy faced like 2 shots in 5 minutes of play, so I re-ran it with a few minimum games played cutoffs as well.<br /><br />The table shows that even strength save percentages vary more when goalies change teams than when they stay on the same team, which is evidence for at least some kind of shot quality effect. Not a large one, granted, at least in the majority of cases, but it looks like there is something there.The Contrarian Goaltenderhttps://www.blogger.com/profile/03433370306939690205noreply@blogger.comtag:blogger.com,1999:blog-8148461224473220694.post-53627700153334514122009-05-06T18:21:00.000-04:002009-05-06T18:21:00.000-04:00I'm talking about the first table, btw.I'm talking about the first table, btw.Vic Ferrarihttps://www.blogger.com/profile/16425585921916867277noreply@blogger.com