Let’s talk about Randy Carlyle. He is, by most observations, not a stellar NHL coach these days. When he got extended, it drove Leafs fans to drink. Like, for real. So why did he get extended? The simplest answer is that the got lucky. The Leafs have been able to ride a ridiculous sky-high PDO for two seasons in a row, making the playoffs once and then blaming defeat on James Reimer. For two seasons now, the Leafs were supposed to regress. But they didn’t. Were they lucky, or is there a more complicated picture that should be painted?
To start, let’s examine Randy Carlyle a little closer. Most measures for coaching actually seem to paint him as being a pretty good coach. He has a NHL record that is good enough to speak for itself (343-244-77), has been to the playoffs more often than not, and even won a Stanley Cup. If he is such a bad coach, but has such a good record, what can the explanation be? The only answer seems to be that he has been incredibly lucky.
Well, according to many advanced stats people, he has been incredibly lucky. Since 07-08 (the beginning of the advanced stats era), he has never been the coach of an elite puck possession team. In fact, he led a bottom-10 puck possession team more often than not. However, in only 1 full season that he’s coached with the same team has he had a PDO less than 100. His average PDO? A whopping 101.3. Almost all signs have pointed to him coaching hockey on the edge – all he has been is a slight regression to the mean away from merit-based banishment from the NHL.
But here’s the problem. I don’t buy it. You can’t go a season in the NHL without a team having a late-season collapse, and in almost all cases the cause appears to be regression of PDO. But for every instance of collapse due to regression, teams seem to find ways to ride sky-high, regression-bound PDO for way longer than expected. The Boston Bruins have had only one season with a PDO less than 100 (and it was 99.9) since 07-08. The odds of that happening? About 1 in 250, assuming that PDO is a coin flip between above or below 100, and that of course doesn’t even take into account how high their PDO has been – as high as 103.3 in 08-09. Randy Carlyle has made a career based off of very similar odds, as well as Bruce Brodreau. How about the fact that the Florida Panthers and the New Jersey Devils have both had 2 seasons in a row with luck-defyingly low PDO? To me, there are too many instances of extreme good luck and bad luck to cast PDO as a purely luck-based statistic.
To me, these cases represent 2 irreconcilable ideas that pertain to PDO:
- PDO is a statistic that is a proxy for luck, and is therefore subject to heavy regression and is not repeatable based on skill.
- PDO, while certainly able to be a red flag for lucky teams, is in fact partially based on skill and can be repeatable.
In my experience, the advanced stat users around the internet believe in Idea #1 and use PDO as such. I believe, however, that Idea #1 is used and regarded to an extent that is not only misleading (especially to those who don’t understand advanced stats as well), but actually entirely incorrect. While I do agree that PDO is partly luck-based and still subject to regression, I am going to make an argument that there is a specific window inside of which PDO no longer represents luck, and actually represents deviation from the mean based on skill.
Note: If you are unclear as to what I am arguing, skip to the end of this piece and read the conclusion. Then it should be clear. The next section of this post is purely dedicated to how I arrived to my conclusions in a mathematical sense, so if you don’t like math or don’t care, skip it.
My Theory and Application
In order to show that PDO is skill based, there should be a quantifiable statistic that correlates to PDO that represents skill. My theory is that we can analyze how large these fluctuations in shooting and saving percentage based on skill are, and then use that to define regression to the mean in a new way for PDO. Rather than imagining it as a flat regression to the mean, imagining a PDO boundary inside of which teams vary based on skill, and outside of which teams are most likely subject to regression, or at least need closer inspection.
Luckily, finding that boundary will be actually pretty simple. Because PDO is comprised of both shooting and saving percentages, all you have to do is find the reasonable level of talent-based fluctuation for both shooting and saving, then add them up to get your reasonable level of talent-based fluctuation for PDO.
Trouble is, statistics that have a meaningful correlation with PDO haven’t been found yet, at least to my knowledge. However, I think I may have found it. If you haven’t heard about Corey Sznajder’s zone tracking project, check it out. I’ll skip explaining it and instead just offer a link to his description (link). David Johnson from hockeyanalysis.com got some of Corey’s data and created a stat that I call NCI%, which stands for Net Carry-In Differential Percentage. Basically, it is the differential a team has in successful zone carry-ins for and against, expressed as a percentage. You can read his work here, but the premise is that teams that gain the zone well have a higher winning percentage. However, nothing he did dealt with PDO in particular.
The goal is to find talent-based deviations from the mean in both shooting and saving percentage, and I figured that teams that gain the zone well would have a higher shooting percentage, so I decided to test it. What I found was that there is in fact a correlation.
There is a positive linear correlation between NCI% and SH%. In fact, the odds that those two things are not correlated but lead to an r^2 value of about .25 is about 15%, which, in my opinion, indicates that shooting percentage is repeatable and skill based. That means that PDO, which is comprised of shooting and saving percentages, must be as well. Given that PDO does have random fluctuations in it (that make any correlation appear to be weaker than it is), I would argue that this correlation is strong enough to be safely considered causation.
We now know that good zone entries cause increased shooting percentage. So how much fluctuation can we expect? Well, the slope of the best-fit linear line is pretty much 8, so that means that for each variation of 100% (which in terms of raw numbers is equal to 1) in zone entries, we can expect an 8% increase in shot percentage. Well, for the last season, basically every team was within 8% (.08 in terms of raw numbers) from the mean in NCI%, so multiplying that by the slope of the line gives us an expected talent-based deviation in shooting percentage of .64 (because .08*8=.64). We’ll call the expected deviation .65 because rounding is fun, easy, and for lazy people like me. Keep this number in mind.
Now for save percentage. Unfortunately, there isn’t really a good stat to reflect talent-based fluctuation in goaltending other than save percentage, which fluctuates a lot from season to season for almost all goaltenders. What I decided to do was simply look at historical data. Using this chart of 7-year save percentage for all NHL teams, I figure that NHL save percentage is average at about .9215, and that normal fluctuation based on talent (because luck will become a non-factor over a large, 7-year sample), is from .915 to .930. That means that you can expect a deviation from the mean, based on goaltending talent, to be .75 for save percentage.
So now all there is left to do is add them together and get your expected talent-based fluctuation for PDO. The shooting deviation is .65, and the saving deviation is .75, which means that the overall PDO skill-based deviation is 1.4. That means that a team’s talent (or lack thereof) can lead to their expected PDO (not factoring in luck) to be anywhere between 98.6 and 101.4.
So there you have it. That is my theory for PDO fluctuation based on talent. Looking at the chart for 7-year averaged PDO (once again a large enough sample size that I argue luck is a non-factor) (Here it is), my +/- 1.4 actually fits pretty well. In fact, remarkably well.
Note: Just to be clear, I did not intend for the numbers to work out the way that they did. I analyzed shooting and saving independently and didn’t even look at the PDO table until I came up with my number. In fact, it was looking at the table when I was finally convinced that I’m on to something pretty precise here.
However, there are limitations. Just because a team has a PDO in the talent-based fluctuation doesn’t mean that luck plays no role in the number they have. PDO fluctuations will still arise due to luck and skill. So, exactly how should this be interpreted? Surely there are still limitations.
The power of PDO still remains in predicting what will happen, mostly raising red flags for teams that have been absurdly lucky. Keeping with that mentality, I would advise analyzing PDO by grouping teams into 3 categories:
- Teams with PDO within 1 of the mean (99-101): Expect no regression. While they could be lucky and regress, it is just as likely that they have been unlucky and will deviate further based on their actual talent level.
- Teams with PDO from 1 to 1.4 away from the mean (98.6-99, 101-101.4): These teams should be analyzed on a case by case basis. While they could be lucky, they could also have high levels of talent in both (or neither) goaltending and zone entries. A team like Boston with a PDO of 101.2 would not be expected to regress, while a team such as the Islanders with the same PDO would probably regress a little.
- Teams with PDO greater than 1.4 away from the mean: Expect regression. However, the regression will almost certainly be weaker than writers and bloggers will make it out to be. At the halfway point of a season, a team with a PDO of 102, no matter how much deviation is skill-based of luck-based, will almost certainly finish the season still in this category. While it remains more likely they will finish the season with a PDO lower than 102, those forecasting impending doom are as likely to be as wrong as those who cast regression to the mean off as a myth or a non-factor (see why here). It is also worth noting that regression will get stronger as you get farther away from the mean, so a team with a PDO of 101.5 will not regress as strongly as one with a PDO of 102.7.
So could I be wrong? Of course! It is entirely possible that the link between zone entries and PDO is simply correlation and not causation, and that there are, in fact, a lot of absurdly lucky and unlucky teams in hockey. However, given the precise way the numbers worked out, I think to argue coincidence and look at PDO in the same way would be foolish. At the very least, I think it’s time to stop pretending that PDO’s regression to the mean is strong and/or inevitable. That’s just not the way it’s worked.