Comments / New

Analytics, Goodhart’s Law & why scorers stop scoring

Before I offer a hypothesis on why improved possession stats haven’t led to more wins in Toronto, Edmonton and Carolina in 2014-15, let me share a brief anecdote:

The company my father works for (a tech start-up in Southern California) did something ill-advised recently. Someone in management thought it was a good idea to buy a fitness bracelet for every employee and track their physical activity over time as a friendly competition.

Now, there is nothing wrong with encouraging people to exercise. The intention was good and “gamifying” the whole experience can serve as motivation for people to get out and move around. But having the competition based around a number on your wristband creates a serious problem.

Two days ago, I was doing intervals on a stationary bike at the gym, when my dad sat down on a bench across from the room and started waving his arms around as if he was trying to get a spider out of his armpits. He realized that the fitness bracelet was nothing more than a glorified step counter, and that small, rapid arm movements were enough to keep the numbers rolling upwards.

When I asked him what the heck he was doing, he told me that a coworker had been averaging 10,000 points a day, and that he needed to catch up.

He was padding his stats at the expense of doing actual exercise.

I couldn’t help but wonder whether one of his coworkers had thought of putting that stupid little bracelet on a paint shaker at Home Depot, turning the shaker on and just leaving it there for a few hours.

In an attempt to promote healthy living, has my dad’s company actually incited its employees to cut corners and behave in counterproductive ways? And was the problem caused by the tools and metrics used to approximate “exercise”?

These two questions don’t actually matter all that much in my dad’s case; he does plenty of exercise as it is, and posting a lower number than his colleagues won’t get him fired (I hope…). But these questions matter a lot when talking about advanced stats and how hockey players interact with them.

Proxies can be (and often are) manipulated

When I started compiling advanced stats at McGill, our head coach Peter Smith (former Team Canada Women’s team coach, 2x Olympic Golds) was unusually open to what I was doing. But from the get-go, he insisted that the data I collect remain within the coaching staff. For a long time, I had doubts about his rule – our players are smart people, maybe they can get something out of knowing their Corsi, deployment, time and ice and quality of competition.

But then it made sense. It’s a natural human instinct to manipulate numbers used to measure their work (like my dad at the gym). Remember that Corsi is only a proxy for possession, and if you know how it works, you can hack it to your own benefit while defeating its purpose. Take shots from anywhere in the O-zone and be over-aggressive at your blueline, and your Corsi goes up, your PDO goes down, and your team is likely worst off than it was. It’s like waving your arms around at the gym.

I can’t say for certain how teams like the Maple Leafs, Oilers and Hurricanes have been using possession proxies in coaching, but I get the impression that at least one of those teams was freely sharing Corsi information with its players (any NHLer could go on War on Ice to get his advanced stats, anyway). Over the course of 2014-15, all three teams became noticeably better at puck possession, as measured by shot attempt differentials. But they didn’t score more goals, or allow fewer goals, or win more games. Maybe that’s just luck. But maybe not.

If players changed their on-ice behavior in unproductive ways just to boost their raw shot count, I could see where things can go wrong. But I don’t know for sure.

So, as it stands, my personal feelings about using advanced stats in coaching is this:

It’s very important to track them, and it’s very important to keep them in mind when setting up a roster and teaching the game, but it’s not very important for players to know and care deeply about their stats, because it’ll only distract them from the most important thing – internalizing the coaching staff’s message and doing the right things on the ice.

The other side of the token

All that being said, I think that many NHLers can still get a lot out of understanding the concepts behind advanced statistics. During the off-season, being able to see exactly when you’ve thrived and struggled can be an incredible eye-opener.

For a few players in the league, knowing “who they are,” as encapsulated by stats, can be a life-changer.

I have a person in mind when I am talking about this. Let’s call this person Bobby.

Bobby is currrently in his late twenties. A high draft pick, Bobby broke into the league early with an improving team, and had four 30-goal seasons under his belt when many of his high-school classmates were still working at fast food joints.

At a certain point, Bobby was traded to another team in exchange for a significant package. In the coming years, he will be unfavorably compared to one of the players going the other way in this deal, which will cause him a lot of unnecessary and undeserved grief.

Because of his excellent track record and valuable skill set, Bobby’s new GM gave him a lot of money and a lot of term to play for his hockey team.

Bobby must like where he is now. He is rich, he is successful, and he believes that he will keep scoring goals well into his thirties.

I know for a fact that Bobby does not like the idea of advanced stats, and that’s totally okay, but there are some crucial things that stats tell us about him, that he deserves to know.

What Bobby should know

For instance, Bobby, you deserve to know that players like you – high-scoring forwards with elite finishing skills but only average puck-possession numbers – do not age terribly well.

Players like Dany Heatley, Thomas Vanek, Vincent Lecavalier, Michael Ryder, Rene Bourque and Daniel Briere were all great snipers in the first half of their careers, but saw their on-ice performance decline sharply past the age of 30, never to recover.

The decline starts out innocently enough. Goalscorers are streaky by nature, but at some point, the scoring droughts last a little bit longer. The hot streaks get a little bit cooler. Maybe your coach starts giving more time to younger players on the powerplay; maybe minor, nagging injuries make shooting the puck just a bit more difficult.

When the goals dry up, these players’ role changed. They did not “do change” all that well. They were booed, benched, scratched, traded, bought out and ultimately left jobless. They’re chewed up and spit out by a league which had treated them so well just a few years prior.

Unless you change, that could be you three years from now.

At this point, Bobby, you might be tempted to get a little defensive – and that’s normal.

You think of yourself as a 30-goal scorer (which you are, obviously), and think that you can score 30 goals every single year if you train hard and play hard. Goalscorers are stubborn, and you trust the gifts you were born with and the work that you’ve put in.

Look up at the list of names I’ve mentioned. Former All-Stars, scoring title winners, good guys who have an obvious talent for the game, but guys who weren’t successful at sustaining their early-career from, to the detriment of their financial and personal well-being.

But you don’t have to end up that way. You’re just 28 and you’ve got some time.

There’s nothing more depressing, up-close, than scorers who just can’t score anymore. They think they’ve lost “the magic,” when in reality they’ve just lost half a step. A half-step they could get back by taking in the right information.

Learn from the guys who’ve been able to maintain their level well into their late-thirties. Your former captain is one of them and he played the same position as you. Ask him how he trains, how he eats and how he thinks the game. Ask him to teach you how to play better ping-pong. His advanced stats were great – find out why and it’ll change your life.

Stats aren’t just for geeks. They’ll show you things that no one else can.

It’s your future; take charge of it.

Jack Han is the Video & Analytics Coordinator for the McGill Martlet Hockey team. He also writes occasionally about the NHL for Habs Eyes on the Prize. You can find him on Twitter or on the ice at McConnell Arena.

Support Habs Eyes On The Prize by signing up for Norton 360