What Should We Expect from NHL Goaltenders?

Analytic minds of the blogosphere are starting to converge on a new frontier: analyzing goaltending. We at Eyes on the Prize have been blessed to have Chris Boyle give us incredibly in depth analysis on the position that is seen nowhere else on the internet, but something came up recently in his piece on Marc-Andre Fleury that got me thinking… what exactly is replacement level goaltending?

Now, I’m not the only blogger out there that is trying to answer this question. Cam Charron earlier this year used it to try and give comparables to Winnipeg Jets starter Ondrej Pavelec, using other metrics other than league save percentage like measuring quality starts and blow-ups. I didn’t go this route with my own analysis, but it’d be worthwhile to build a database of these over time. Scott Reynolds of The Copper & Blue earlier this month decided to calculate a replacement level save percentage using all NHL goalies that had 15 or fewer games in a season. He split up even strength and shorthanded save percentages and found these goalies had a much lower threshold than the league average. It makes a lot more sense than just using the league average: clearly there are different expectation levels for goalies who are called up from the minors than those that have guaranteed NHL jobs, and largely the NHL does a decent job of picking who their best goalie is.

My methodology started with a similar threshold. Reynolds posted that on May 1, while I had been working on the charts in April, so they were come up with independently. I wanted to separate goalies by their usage: starters, platoons, backups, and replacements. I came up with my own cut-offs to determine this: anyone who played more than 60% of the minutes in an 82 game regulation schedule would be considered a starter, platoon goaltenders played between 40 and 60%, backups played in over 10 GP a year and up to 40% of a team’s minutes, and the rest were replacements. I broke out every year since 2003-04, since I was doing this for a project I’m working on about the 2003 Entry Draft that I’ll be posting on throughout June. I looked at save percentage only, but broke it between regular save percentage (all situations) and even strength only.

If you’re wondering why splitting up the situations is important, team effects can greatly contribute to a goalie’s save percentage on the Penalty Kill. While it is often said that your goaltender has to be your best penalty killer, teams with good penalty kills have to be better at preventing shots and prime scoring chances in order to succeed over the whole season. Therefore, a goalie’s save percentage on the penalty kill is less of a factor of his individual skill level than at even strength, although I do agree with Chris Boyle in that teams can design systems that make it a little easier on a goaltender to stop the puck at a higher rate at even strength. However, those fluctuations won’t be to the same level a good PK unit can effect a goalie’s save percentage. Even strength is a better indicator of a goalie’s skill level over time, but we’ll include both measures here.

After the jump are some of the results.

There are 4920 minutes in an 82 game season, excluding overtime. I can’t possibly be bothered to add all the overtime minutes together and figure out a threshold based on the actual minutes, so I kept it simple: all goalies had the potential to play 4920 minutes for their team, so we’ll base our classifications on them. 60% of 4920 minutes is 2952 minutes, which is just over 49 full games. The least amount of appearances a goalie made in this study that classified as a starter was Nicklas Backstrom in 2010-11, with 51 GP. I think that’s an acceptable figure for what we’d consider a starting goaltender, a good sample size. 40% of minutes played is 1968 minutes, or just under 33 GP, and Michael Leighton in 2003-04 had the fewest appearances of any goalie in this range with 34 GP, although a number of goalies had 35 GP and didn’t reach 1968 minutes. These are definitions I’ve made up, if one doesn’t like them, they’re free to adjust them, they just made sense to me. Generally, platoon goalies played in at least 35 games and up to 50. Backups, then, appeared in 11-35 GP. Here’s the sample size for each year:

My educated guess for 2005-06 having more platoon goalies than starters is this: with the NHL lockout, teams lost a year of up close evaluation of their goaltenders and it took longer for them to figure out who their better goalie exactly was (there were more backups that year than any other as well). Every year since has been more normal, with 2010-11 being a bit of an extreme in favour of teams employing a pure starter/backup system. You’ll also notice a slight CBA effect: 2003-04 had the largest amount of goalies called up, when there were less restrictions on callups than there were post-lockout.

From here, I figured out the average save percentages for each group of goalie, but you’ll notice that it didn’t quite work out as linear as I had hoped:

As you can see, for the most part, the divisions work, but due to the random nature of sample sizes, sometimes teams are forced to play really bad backups due to injuries, or a call-up has a particularly strong run and it muddles up the numbers at the bottom end. It also shows that it is now harder to score a goal on the top goalies in the NHL today than it was the year before the lockout, and goalies are making one more save every 100 shots than the first year after the lockout, depressing goal totals significantly.

Due to this finding, I decided to expand the sample size for defining a replacement level goalie. Fortunately, the NHL has their own standard definition of what an acceptable sample size is for goalies. It’s based on Games Played, not minutes, but who am I to argue against Big Hockey? For determining the awards for the William M. Jennings Trophy (Top Team GAA) and the Roger Crozier Saving Grace Award (Top Save Percentage), the NHL believes a goalie must appear in 25 games to be considered. So, according to the NHL, goalies who play less than that are not really statistically relevant. Perfect.

Therefore, I calculated a Replacement Level SV% based on all goaltenders that appeared in less than 25 GP. Here are the results:

I provide all these figures for future comparisons of goalies by save percentage. From here, I developed a RelSV% metric, basically a +/- figure comparing goalies to others in their peer group. Here are the RelSV% figures for the 2011-12 starters, as well as how they performed compared to a replacement level goalie:

As you can see, Cam Charron was right to put Ondrej Pavelec’s season into perspective: he’s in the replacement level range as a starter, while Martin Brodeur and Corey Crawford were definitely at that level this year. Mike Smith didn’t get the love from the General Managers and missed out on a Vezina nomination, but the other three finalists were right up there as well.

For the most part, teams are employing starters that are above replacement level. Even Marc-Andre Fleury is a step above that level despite Chris’ takedown of him last month. Carey Price had a down year, but 10 points above replacement level overall is a decent showing in those circumstances (and average for a starting goalie).

I came up with another metric for evaluating goalies as well that I’ll introduce in a follow-up piece. I’m sure there is plenty of material on statistically evaluating goalies that will be coming this summer.