Friday, July 6, 2012

The Value of Subjective Evaluation

I've written many times about how I do not put a high degree of emphasis on subjective evaluations of goaltenders because there are extraneous and often subconscious factors that can impact how a performance is judged. Even on the individual game level, where the task should presumably be at its very easiest, luck and the performance of the goalie's teammates still have a big impact on how they are rated.

TSN analyst and former NHL goaltender Jamie McLennan scored every goaltending performance in the 2012 Stanley Cup playoffs based on a subjective scale from 1-5, which gives an interesting point of comparison between what the numbers show in terms of saves and goals against and what an informed expert concludes from their individual judgment. I do not question McLennan's scouting ability or his knowledge of goaltending, but the data set he has provided has some interesting properties that force me to question how much value his analysis actually adds.
  • In 86 playoff games, McLennan never gave the losing goalie a better score than the winning goalie.  He gave the two the same score only 6 times, meaning that 93% of the time the goalie on the winning team was judged to have played a better game.  Eleven times the two goalies had a save percentage within .005, suggesting there was likely very little difference in their play, yet the winning goalie routinely received a higher degree of recognition.  When you factor in score effects that typically end up inflating the winning goalie's numbers slightly, it is likely that the losing goalie often managed to match or outperform the winner, yet they never got a better score and only rarely were even graded on the same level as their counterpart
  • The correlation between a goalie's average game score and his overall save percentage was 0.924, which is an extremely high degree of correlation
  • Goalies received a 5 every single time they had a save percentage of .970 or better, regardless of how many shots they faced
  • If you compare the save percentage rankings with the game score rankings, only 4 out of the 17 goalies with at least 3 games played in the 2012 playoffs have a ranking differential of more than two (Niemi, Howard, Brodeur and Holtby)
  • Using regression, each goalie's average game score can be predicted from their overall save percentage.  Every goalie's predicted average score was within 0.25 of their actual average game score, with the exception of Jon Quick, who was 0.37 higher, and Antti Niemi, who came in a whopping 0.67 lower
In summary, McLennan's rankings did not seem to add much information.  He really liked Jon Quick, but he was hardly unique in that viewpoint; the Conn Smythe Trophy is proof enough that most observers liked what they saw from the Kings' goalie. McLennan really wasn't a fan of Niemi's playoffs, giving him scores of 2 for games of .906 and .923, both very unusual for his ranking system (there were only two other games total where a goalie got a 2 for a .900+ save percentage outing).  Perhaps we can conclude from this that the numbers flattered the Sharks' netminder a bit relative to his actual performance.  However, it is also at least possible that McLennan has some sort of bias against Niemi, who had strong numbers at even strength (.940) but was ventilated on the penalty kill (.806), a unit that has been a point of serious weakness for San Jose over the past two seasons.

Beyond those two, comparing the save numbers to the game scores indicates that McLennan thought that Marc-Andre Fleury, Cory Schneider, Jimmy Howard and Martin Brodeur were all a bit better than their numbers suggest, while Tim Thomas, Jose Theodore and Corey Crawford were all a bit worse.  That makes sense for Fleury, awful performances do tend to disproportionately drag down a goalie's overall averages, as well as Brodeur who is noted for his non-save skills (although McLennan also openly admitted to giving the veteran a few sympathy marks for his play in the Stanley Cup Finals).

I would subjectively agree that Thomas and Crawford may both have been a bit worse than their numbers suggest, particularly if you factor in situational leverage and their impact on win probability.  A previous post of mine looked at save percentage by game score, and Thomas and Crawford were certainly outplayed by their opposite numbers when the game was close.  How much of that is randomness and how much of that was each goalie's fault is an open question, but I'm not surprised that someone subjectively rating goalies would take their situational performance into account to some extent.  Still, if I turn the analysis around to predict save percentage based on game score, McLennan was subjectively downgrading both goalies by only about .006-.008, not a very significant change at all for a 150-200 shot sample.

Imagine two individuals, one who didn't watch a second of playoff hockey but was armed with detailed stat sheets showing the save efficiency of every goalie, and one who also didn't watch a second of playoff hockey but still somehow faithfully tuned in every night to hear Jamie McLennan's Post2Post segment on TSN.  Which one would be in better position to rate the goaltenders in the 2012 playoffs?  I think it probably would be the McLennan fan, but their advantage would be very slight as the two would still agree on almost everything.

Over and over, analysis tends to show that the subjective factors often hyped by broadcasters and hockey insiders (shot quality, clutch saves, etc.) really do not have that much of an impact at the end of the day.  That's not always the most intuitive finding for us narrative-obsessed sports fans, but it's hard to argue with the evidence.  I think we should be careful to completely dismiss potential factors that over time could have a significant impact, particularly at the career level, but given that nearly every observer seems to vastly overrate the value of goaltending contributions that are not encapsulated in save percentage I think it reinforces the idea that a goalie's save rate should remain the primary and most trusted method of evaluation.  Subjective opinions from people who know what they are talking about should not be disregarded, but they should at least be treated with some level of skepticism and compared with the statistical record to test their validity before given much weighting in the final evaluation.

24 comments:

Anonymous said...

you are kinda clueless , aren't ya?

Anonymous said...

Different Anonymous here. I want to know if I'm the only one who thinks Quick's performance these playoffs was a tad overrated. Sure he had great numbers but most nights, his team minimized chances against very effectively. Also, on most nights his team received favorable calls.

I'm not entirely sure Martin Brodeur or Tim Thomas wouldn't have been able to put together a similar run this year with those advantages in front of them.

The minder I am most impressed with this playoffs is Mike Smith, followed by Holtby.

a witness said...

Hey man. I was just now catching up on your posts - love the Sutter analysis. Have you done that same analysis with Ken Hitchcock? And even more, follow the goalies, before KH and after KH. He sure seems to improve a goalie's numbers everywhere he goes...

Thanks for the time.
Bryan

Anonymous said...

Massage in London
Massage London

Anonymous said...

Body to Body Massage London
Erotic Massage London

C said...

Any analysis for the shortened 2012-2013 season? Thanks!

Gerrard said...

COOL

Host PPH said...

that is pretty appealing, and I have not done anything like that before and I think I will use this technique, thanks buddy!

Anonymous said...

While I agree the numbers game works for the most part, the one part you all seem to miss is simply the goalie who can do those numbers year after year and not just once then again in a few years again. Brodeur,Roy,Belfour,Joseph and Fuhr to name a shot list did the numbers year after year. So to judge greatness is simply consistancy year after year.

Anonymous said...

True Hockey fans, I know it blows about the whole Coyote thing, but you must know this...NEVER GIVE UP AT ALL! Where there's a way, there's a will.

This thing is far from over and we can end it long before five years are up by reuniting millions and working on the Coyotes downfall instead of whining and doing nothing.

First, not everything has been approved, and the Coyote owners have until August 5th to get everything approved or else!

Despite what the Goldwater Institute said recently. there is one thing that would get them to go after them anyway...

Money.

Donation money that is. With billions of us true hockey fans out there, we can each spare a ton of money to donate to Goldwater.

Here's a link (to copy and paste) below:

http://goldwaterinstitute.org/

In the upper right corner is a link where you can donate money to Goldwater in order to stop the Coyote thing RIGHT NOW instead of waiting five years. Make sure you tell Goldwater that they did a great job stopping Hulsizer a couple of years ago and that the current deal is no different than the one out there right now. I'm already gathering money up to send there way.

Also, don't forget about:

http://www.moodys.com/research/Moodys-downgrades-Glendale-AZs-GO-rating-to-A2-senior-excise--PR_261198

These guys have downgraded Glendale's bond rate before (as mentioned in the article at the link above) and they can certainly do it again! Please contact Moody's like I plan on doing and show them your support for the continuing of Glendale's bonds being downgraded.

And finally:

https://twitter.com/ReverendMaupin

This man is trying to start a movement in order to stop this horrible Phoenix Coyote deal once and for all. Please show him your support and help him anyway you can.

Copy and paste this message to other websites as often and frequent as possible. Together we can end this boondoggle and get the Coyotes into Canada where they belong!

dody Love said...



شركة تسليك المجارى بالرياض
شركة اتنظيف الخزانات بالرياض
شركة تنظيف الشقق بالرياض
شركة تنظيف الفلل بالرياض
شركة ارش المبيدات بالرياض
شركة كشف تسربات المياه بالرياض
شركة مكافحة النمل الابيض بالرياض
شركة نقل الاثاث بالرياض
شركة عزل الخزانات بالرياض
.

dody Love said...

مكافحة صراصير المطبخ
شركة نقل اثاث بجده
شركة نقل عفش بجده
شركة مكافحةالنمل الابيض بأبها
شركة رش مبيدات بأبها
شركة تنظيف بأبها
شركة تنظيف شقق بأبها
شركة تنظيف فلل بأبها
شركة تنظيف بالمدينه المنوره
شركة تنظيف فلل بالمدينه المنوره
شركة تنظيف شقق بالمدينه المنوره
شركة نقل اثاث بالمدينه المنوره
شركة تنظيف عفش بالمدينه المنوره
شركة رش مبيدات بالمدينه المنوره
شركة تسليك مجارى بالرياض
.

marketing for all said...

الشقق التي تحتاج شركات التنظيف فى مكةإلى تنظيف الأوساخ شركة ابادة الحشرات فى مكةو التفكيرشركة تنظيف بالبخار فى مكة في تنظيف يلزم شركة نقل عفش بمكة حتى استخدام الملح شركة نقل اثاث بمكةفي إزالة البقع

على باب الله said...

السلام عليكم ورحمة الله وبركاته اما بعد ، اهالي المنطقبة الشرقيه بالمملكة العربية السعوديه يسر شركة شام للخدمات المنزليه ان تعلن لجميع اهالي المنطقه عن بدأ تقديم خدماتها الخاصه بالمسابح للموسم الصيفي الجديد وذلك من خلال فروعا في مدينة الدمام ومدينة الخبر ومدينة القطيف ومدينة الجبيل ، واما عن فرعنا في مدينة الدمام فسوف تجدونه علي المتصفح تحت اسم
شركة تنظيف مسابح بالدمام
وايضا تحت اسم
شركة صيانة مسابح بالدمام
مقترنا برقم هاتف مندوبنا بالدمام لذل فللفحصول علي اي خدمه تخص المسابح تنظيفها او تشغيلها او صيانتها فليس عليك الا التصال بنا وسوف نصلك فورا ، واما اذل كنت من اهل مدينة الخبر وتحتاج الي ان تستفيد من خدماتنا في مدينة الخبر فسوف تجدنا علي المتصفح تحت اسم
شركة صيانة وتنظيف مسابح بالخبر
وفي هذا الفرع نسعد بتقديم كل الخدمات التي تخص المسابح لاهالي الخبر من خدمات تنظيف او صيانة او تشغيل برك السباحه ،ولاننا قد عملنا في شركة شام علي تغطية المنطقة الشرقيه كافه فقد حرصنا علي ان يكون لنا فرع في مدينة القطيف يختص بتقديم خدمات المسابح لجميع اهالي مدينة القطيف والذي سوف تجدونه علي المتصفح تحت مسمى
شركة تنظيف وصيانة مسابح بالقطيف
وهنا سوف تجدون كل ما يخص المسابح من خدمات تنظيف المسابح بالقطيف وصيانتها وتشغيلها ‘واما عن مدينة الجبيل فلنا ايضا هناك فرع يختص بتقديم خدمات المسابح لجميع اهالي الجبيل من تنظيف وصيانه وتشغيل وهذا الفرع سوف تجدونه علي المتصفح تحت اسم
شركة صيانة وتنظيف مسابح بالجبيل
بهذا اعزائي نكون في شركة شام قد استطعنا ان نغطي كافة انحاء المنطقة الشرقيه وتقديم كافة خدمات المسابح التي يحتاجها اهالي المنطقه باحدث الاساليب والطرق العلميه وباستخدام افضل مواد التنظيف واحسن اجهزة ومعدات الصيانه والتعقيم ، فلا تقلق عزيزي العميل ولا تحتار فشركة شام هي افضل اختيار .

seocom said...

شركة عزل الاسطح بالمدينة المنورة
تقدم لك شركة عزل اسطح بالمدينة المنورة جميع أنوع العزل و بأرخص الأسعار و في نفس الوقت نقدم أفضل خدمة لكي نكون الأفضل دائماً و لنحصل علي مزيد من التطور و النجاح فدائماً نسعى لتقديم الخدمات بأقصى كفاءة و مهارة و تملك الشركة جميع المعدات اللازمة للقيام بعملية العزل و لدينا العمالة الماهرة التي تدرب دائما علي أن يقوموا بجميع أنواع عوازل الأسطح التي لها دور كبير و فعال في تنفيذ خدمات عزل الأسطح و الأسقف و الخزانات و عزل مائي و عزل حراري و عزل صوت و عزل فوم ويجب الحفاظ على السطح من الحرارة المرتفعة و الأمطار الغزيرة لكي تتجنب ارتفاع درجه الحرارة في المنزل.

على محمد said...



شركة تنظيف بالدمام
شركة تنظيف بالخبر
شركة تنظيف بالجبيل
شركة تنظيف بالاحساء
شركة تنظيف بالقطيف

Anonymous said...

outsourcingall.com is a Porn Training site. outsourcingall.com is the largest Amateur porn training institute with the hottest selection Online and offline real life cam.
free bangla sex video dhaka

amr said...

مركز صيانة توشيبا بالبحيرة

بيت العز said...

تاتش
بلاط متداخل في دبى
تركيب انترلوك متداخل في دبى
افضل مقاول انترلوك في دبى
رقم معلم تركيب انترلوك بدبى
انترلوك رخيص في الامارات
تركيب طابوق دبى
شركة تركيب طابوق دبى
افضل معلم طابوق بدبى
تركيب طابوق زجاجي في دبى
تركيب طوب حراري بدبى
رقم معلم طابوق جيري بدبى
اسعار تركيب طابوق دبى
تركيب طابوق جيري في دبى
تكسية واجهات مباني بدبى
عزل اسطح دبى
وتر بروف بدبى
شركات عزل مائي في دبى
شركات العزل المائي في الامارات
شركات العزل الحراري في الامارات

بيت العز said...

الانامل-البيضاء
تركيب رخام دبي
تركيب سيراميك دبى
فني تركيب سيراميك دبي
تركيب صحى بدبى
تكسير و ترميم فلل دبي
ترميم فلل دبي
تركيب جبس بورد دبي
فني تركيب جبس بودر دبى
تركيب فورسيلنج دبي
تكسير وترميم منازل بدبي
ترميم منازل بدبي
ترميمات المنازل فى دبى
ترميمات الفلل فى دبى
شركات صيانة المنازل فى دبى
صيانة عامة بدبى
شركات تشطيبات في دبى
شركات صيانة المباني فى دبى
ترميم واجهات المبانى بدبى

بيت العز said...

تاتش
مقاول عزل دبى
-تاتش
رقم شركة عزل اسطح دبى
فني عزل اسطح بدبى
أفضل شركات عزل اسطح في دبى
عزل خزانات دبى
عزل مسابح دبى
عزل بالفوم في دبى
سباك دبى
كهربائى منازل دبى
عامل بلاستر بدبى

لمار للتسويق الالكتروني said...

شركة مكافحة حشرات فى عجمان
شركة مكافحة حمام في عجمان
شركة تنظيف خزانات بعجمان
شركة تنظيف بعجمان
شركة مكافحة الفئران فى عجمان
شركة تنظيف سجاد في عجمان
شركة تنظيف كنب عجمان
شركة مكافحة الرمة فى عجمان
اتصل الآن : 0503881920
https://bahr-elehlam4cleaning.com

Monnika Jacob said...

Thank you Philip Myrland! I understood very well value of subjective evaluation by example that you given n your post. It clears the concept more easily. Buy Essay Online

California Air Conditioning Systems said...

Are you looking a trusted agency for Heating Repair Compton? Then California Air Conditioning Systems, Inc. is last search for you where our experts provide best AC repairing, Heater repairing & installation and many more.

Contact Us

California Air Conditioning Systems, Inc.

Address: 25217 Narbonne Ave. Lomita, CA 90717, USA

Phone: (310) 530-0504 (phone lines are 24 hours)

Fax: (310) 530-4004

Email: General Inquiry: caaircondi@aol.com (CC: info@californiaac.com)