New-Wave Analytics Research: DIGR
The following is a brief summary of a research paper that was presented this past weekend by Michael E. Shuckers at the MIT Sloan Sports Analytics Conference in Boston, MA.
DIGR: Defense-Independent Goalie Rating
Goaltenders are generally analyzed and rated using save percentage as a key indicator of performance. One of the common issues that crops up when using save percentage as an analytic metric is comparing one goaltender against another. Varying degrees of the quality of defense from team-to-team, and the distribution of shots faced by each goaltender, create a large margin of error for goaltender-to-goaltender comparisons.
Michael E. Shuckers, of St. Lawrence University and Statistical Sports Consulting, has done some fascinating research into ways in which these issues can be resolved so that save percentage can be a more effective comparative statistic. What he devised, and subsequently presented at the conference, was the concept of DIGR (Defense-Independent Goalie Rating).
For the sake of simplicity (as well as copyright), I will avoid getting into the complex mathematical formulas behind the methodology and instead focus on the theoretical aspects of Mr. Shuckers’ approach.
As we all know, save percentage as a statistic is derived by number of saves made divided by the number of shots faced [ SV% = SV / SH ]. The problem is each goaltender faces a different distribution of shots, or in other words, varying degrees of shot quality.
The example given in this circumstance is Johan Hedberg and Tim Thomas during the 2009-10 season, where shot distribution charts suggested Hedberg faced more difficult shots than Thomas … yet both would up with a .915 save percentage. In other words, the statistic did not reflect the difference in performance between the two, and was in many ways a product of the difference in the quality of the defensive units in front of each goaltender. The question Mr. Shuckers asks is, could one generate a quantifiable measure of exactly how much better Hedberg performed than Thomas?
Those who are familiar with Ken Krzywicki’s work may remember that he and others have in the past proposed models by which one could project how a goaltender would fare when faced with a particular shot distribution, by comparing a given goalie’s performance against certain shot types to how the league average fared against those same shot types.
Inspired by these ideas, Mr. Shuckers used ESPN data to chart the shot type and location of each shot faced by each goaltender who faced over 600 shots during the 2009-10 season, and created a mathematical equation combining the goaltender’s save percentage for each shot type and location (e.g. save percentage when facing a wrister in the high slot) with the percentage of total shots of that type faced by the goaltender (shot distribution). The shot-type percentage is important as it produces a weighted result … after all, simply running the numbers for each shot type and adding them together would result in the same totals as the basic formula that is currently employed. This is known as Shot-Quality Adjusted (SQA) save percentage.
Taking this a step further, Mr. Shuckers noted that substituting the league average Sv% for each particular shot type in place of the goaltender’s individual Sv % in the weighted formula results in an average goaltender’s save percentage when facing a particular shot distribution. In other words, one can look at whether a goaltender under-performed or over-performed the league average for the shot distribution he faced by comparing his SQA Sv% against the league average for that same distribution.
While this model is useful in terms of establishing a more equitable scale by which to rank goaltenders against the league average, it does not allow one to accurately compare one goaltender directly against another as it does not account for the difference made by a defensive unit. For instance, Antero Niitimaki and Mikka Kiprusoff both outperformed the league average SQA Sv% by the same measure (+ 0.007) for the distribution of shots they each faced. But which was the better goaltender? Nittimaki had a 0.902 SQA Sv% (avg 0.895) and Kiprusoff had a 0.913 SQA Sv% (avg 0.906). Yet, Niitimaki faced a more difficult shot distribution than Kiprusoff. And therein lies the chief limitation of SQA Sv%: accounting for the impact of team defensive play on the quality of shots faced by a goaltender.
In order to compare one goaltender accurately against another, Mr. Shuckers concluded that a metric which did not depend upon the shots faced by an individual goalie would have to be created. Long story short, he flipped the SQA Sv% formula: Instead of replacing a goaltender’s Sv% for each shot type with league averages, he replaced the goaltender’s shot distribution (shots faced of each type) with the league-average distribution. As a result the formula now computes a save percentage for each goaltender based upon a standardized distribution of shots, or, compares the performance of each as if they faced the exact same total and quality of shots (thus eliminating the impact of team defense). Hence the term Defense-Independent Goalie Rating (DIGR).
The DIGR results were quite interesting; while one would naturally expect the DIGR % to be higher than the Sv % (most goaltenders tend to outperform their defense — therein lies the impact of defensive gaffes such as screens and giveaways), there were some oddities. Ryan Miller’s DIGR % and Sv % were identical, an indicator that defensive play made no difference to his performance (it’s no coincidence he won the Vezina). Despite his rookie season acclaim, Tuukka Rask’s DIGR % was significantly lower than his Sv %, evidence of the high quality of the Bruins’ defensive play. Similarly, Tomas Vokoun’s lower DIGR % suggests an underrated defensive performance in Florida.
From a Toronto perspective, Jonas Gustavsson, Jean-Sebastien Giguere and even Vesa Toskala all had higher DIGR %’s than Sv %’s, which should surprise absolutely no one.
And to complete our example from earlier, Johan Hedberg’s DIGR % was slightly greater than his Sv % … and vastly superior to Tim Thomas’ DIGR %, suggesting that Hedberg did indeed outperform Thomas last season despite their standard save percentages suggesting they performed on par with one another.
Top Ten DIGR, 2009-10 NHL Season:
| Rank | Goaltender | DIGR | Sv % |
|---|---|---|---|
| 1 | Ryan Miller (Buf) | 0.9285 | 0.9285 |
| 2 | Ty Conklin (StL) | 0.9280 | 0.9215 |
| 3 | Jaroslav Halak (Mtl) | 0.9269 | 0.9242 |
| 4 | Jonas Hiller (Ana) | 0.9243 | 0.9183 |
| 5 | Henrik Lundqvist (NYR) | 0.9237 | 0.9238 |
| 6 | Evgeni Nabokov (SJ) | 0.9227 | 0.9216 |
| 7 | Ilya Bryzgalov (Phx) | 0.9226 | 0.9204 |
| 8 | Tuukka Rask (Bos) | 0.9218 | 0.9312 |
| 9 | Antii Niemi (Chi) | 0.9215 | 0.9124 |
| 10 | Tomas Vokoun (Fla) | 0.9191 | 0.9246 |
| Maple Leafs‘ goaltenders | |||
| 32 | J-S Giguere (Ana, Tor) | 0.9110 | 0.9069 |
| 35 | Jonas Gustavsson (Tor) | 0.9087 | 0.9023 |
| 49 | Vesa Toskala (Tor, Cgy) | 0.8969 | 0.8797 |
| Hedberg vs Thomas | |||
| 11 | Johan Hedberg, Atl | 0.9190 | 0.9151 |
| 40 | Tim Thomas, Bos | 0.9064 | 0.9148 |
_
An interesting extension to the research, for which I won’t go into too much detail, was to use the shot distribution data to create spatial maps to show the probability of a goalie giving up a goal from locations across the playing surface. This involved the gathering of precise x and y shot location coordinates, as well as the use of weighted scatterplot smoothing software (not all goaltenders face shots from the same precise coordinates) to create the maps. For the sake of copyright I have not included the sample maps shown in the presentation … but it is certainly an intriguing approach to providing a visual comparison of goaltenders’ strengths and weaknesses by shot type, location and game situation (e.g. 5-on-5 backhands, PP slap shots, etc).
In terms of practical application, DIGR still has a few flaws which would need to be addressed (as does all research in its primary stages). Notably:
- Screened shots are not accounted for, as that data is often not recorded by the ESPN or NHL shot trackers. The contrasts between DIGR, SQA and standard Sv% would be much more pronounced were a method to be devised of tracking and normalizing screens.
- Rebounds are not accounted for in the analysis, due to a lack of available data (as above). Although one could argue they are technically included with shots taken near the crease, it would theoretically be more effective to separate rebound opportunities into their own category as they are among the highest-probability scoring chances. Similarly, penalty shots are not accounted for either, but arguably should be for the same reason.
- The tracking of shot types and zones can vary depending on which organization is recording the data. It has long been known that the stat trackers at Madison Square Garden can be rather generous in their recordings; for the purposes of this research, Shuckers did come up with a mathematical equation to eliminate the MSG effect.
- There are also inconsistencies between the way the NHL and ESPN record shot data; shot type, distance can vary between the two. There are cases where the NHL will adjust the data on scoring plays after the fact, and a slap shot may change to a tip-in, etc. ESPN does not update in the same manner.
DIGR analysis is useful on two fronts:
(1) evaluating goaltender performance against the rest of the league, and
(2) comparing individual goaltenders against each other in a given season.
In both cases, the analysis occurs after-the-fact; that is, it is not predictive so much as it is evaluative. SQA, on the other hand, remains a useful predictive tool insofar as predicting one goaltender’s performance when facing another’s shot distribution.
Whether teams will adopt and / or implement statistical measures such as these in their own evaluations — or if they already use a similar brand of metrics — remains to be seen (teams are, naturally, notoriously tight-lipped about their evaluation techniques). Nevertheless, a study such as this is indicative of the creativity of thought and research that exists beyond the league offices and within the fanbase itself. And that can only be a good thing for the continued growth and future of the game.
Looking forward to your thoughts as always,
[email protected]
twitter.com/garrettbauman
Note: For anyone wishing to reference or extend upon the concepts discussed above, please be sure to provide credit to the author of the originating research.
