How NBA Coaches Are Approaching Their New Challenge Capabilities

Early in the second quarter of a preseason game between the New Orleans Pelicans and Utah Jazz, Jazz power forward Jeff Green was called for a shooting foul on young Pelicans sensation Zion Williamson's layup attempt. Green and several Jazz teammates were immediately dubious of the whistle, but what can you do? Such is life. You have to accept the call and move on.

Except starting this year, maybe you don't. With the advent of the coach's challenge, a one-year NBA experiment that's also been tested in the G-League and Summer League competition, there was another avenue available. Jazz coach Quin Snyder chose to sacrifice a timeout and challenge the play - correctly, as it turned out.

Asked about the call and the new rule later on, Snyder actually second-guessed himself - even despite getting the challenge right.

"Later in the game, I didn't think Royce [O'Neale] fouled Lonzo Ball on a three-point shot - that would have been a better opportunity to use it because it was three free-throws," Snyder said.

But because a coach's challenge evaporates after one use whether or not it was correct, Snyder couldn't do anything.

This is but one example of a fascinating probabilistic exercise facing NBA coaches this year: How to properly utilize their challenge. The preseason is serving as the primary testing ground, with differing approaches found around the league.

Per league sources familiar with the data, coaches have attempted a challenge 33 times so far this preseason (through games completed on Wednesday, Oct. 16). A breakdown of their success rates based on the type of call being challenged:

· Foul calls: 24 challenges, 10 successful (42% overturn rate)

· Out-of-bounds calls: 7 challenges, 4 successful (57% overturn rate)

· Goaltending or basket interference: 2 challenges, 1 successful (50% overturn rate)

· Total: 33 challenges, 15 successful (45% overturn rate)

The returns are encouraging relative to leaguewide expectations. Coaches in the G-League hovered just below the 33% accuracy mark on their challenges for the 2015-16 and 16-17 seasons, and multiple league sources expected roughly similar results at the NBA level. So far, albeit on a tiny sample that could change drastically when it expands, NBA coaches have done better.

At the same time, it's clear everyone is still wrapping their heads around the rule, its practical use and the optimal times to consider it. Some are taking, shall we say, a more relaxed approach than others.

"I've taken two - one was for the hell of it," joked Portland Trail Blazers coach Terry Stotts before a preseason game.

Comedy aside, Stotts and several others are preaching caution here. Both he and Sacramento Kings coach Luke Walton raised concerns about the time remaining in a given game, logically suggesting that they'd prefer to use the challenge at later points when leverage is at its highest.

Some also aren't thrilled with the narrow scope of the rule, which limits the kinds of calls that can be challenged to the three categories listed earlier. Coaches can't challenge non-calls, for instance, often huge turning points in a game.

There are also concerns about timeouts, which many NBA coaches treat as a more valuable commodity than gold. Using the challenge eats up a timeout, again regardless of whether it succeeds.

"Even if you win a challenge, it may end up hurting you," Stotts said. "I took a challenge, [it was] successful, [but I] got charged a timeout... But the timeout was at six [minutes], and I ended up having to call another one because [our opponent] made a run. So I was down a timeout."

This is mostly anecdotal stuff, even if it's all grounded in simple logic. Some are taking a more scientific approach to the question of optimal challenge use.

Snyder and his staff are among them, to the surprise of no one who knows him well. Utah's coach heaped heavy praise on his analytics team for helping him dissect the probabilities involved, trying to get to the heart of when a challenge is a positive expected value event. They looked at everything from the fouled opposing player's free-throw percentage to the time of game, the number of fouls their own player had at the time (i.e. could they prevent a fifth foul on an important player early in the fourth quarter?), and even the player-coach dynamic that will often be at work in these situations.

What happens, for example, if a player is adamant that the call on the floor was wrong - but the coach doesn't deem the situation valuable enough to warrant a challenge? Could there be hurt feelings involved?

"We've tried to talk to our team about it," Snyder said. "There may be a situation, whether it be in the first half or the third quarter or whatever, where the ball gets knocked out of bounds and one of our guys is like, 'No, it wasn't off me, it was off him.' Well, you challenge, and what happens with that? It's a marginal advantage that's gained. Does that happen to Mike [Conley], and I don't have his back? Do [players] perceive that as a lack of support?"

While the precise nature of Utah's analysis obviously remains behind closed doors, others have been done in the public sphere. The most notable of these was just last month at the NBA's yearly Hackathon event, which was won by recently-graduated Syracuse student C.B. Garrett and Notre Dame student Peter Zanca for their research on this precise issue.

Garrett and Zanca utilized publicly available data from two sources: NBA Last Two Minute Reports that have been issued for the last several seasons, plus play-by-play information found in modern NBA box scores.

They used this data to synthesize several connected elements, including call accuracy, call frequency, free-throw percentages and more. They also segmented calls more specifically - would a given out-of-bounds call gain a team a new possession or simply maintain a previous one? Would a particular loose-ball foul that's challenged result in a jump-ball between the two teams or possession awarded to one?

Based on these elements plus related probabilistic calculations, the researchers were able to chart five "tiers" of challenges, ranked in terms of expected points gained from each:

· Tier 1: Rare calls like goaltending, basket interference or a made three that's turned into an offensive foul. Expected points gained: 0.40 to 0.93

· Tier 2: Out-of-bounds calls with a high shot clock and made block or charge calls. Expected points gained: 0.28 to 0.40

· Tier 3: Missed block or charge calls, three-point shooting fouls and low shot-clock situations. Expected points gained: 0.15 to 0.23

· Tier 4: Two-point shooting fouls and and-one fouls. Expected points gained: 0.07 to 0.12

· Tier 5: Non-shooting fouls on the floor. Expected points gained: 0.00 to 0.04

According to this research, the "safest" strategy - that is, the one that yields the highest average expected points gained value - involves mostly challenging high-confidence out-of-bounds or made block/charge calls. Goaltending or basket interference calls could be more valuable, but they're so rare that building a challenge strategy around them likely isn't optimal.

Why not shooting fouls? Two reasons: They aren't overturned very often, and many of those that are overturned will just lead to a jump-ball rather than an actual change in possession. He may not have known it at the time, but this was one of the issues with Snyder's challenge against the Pelicans: Even though he was right, the result was a jump-ball New Orleans won to retain possession anyway.

Now, the researchers noted several additional important crux points. They referenced other value-adds like preventing an important player from foul trouble or saving the challenge for later in the game, plus areas the available data simply couldn't speak to: Evolving coach confidence in their challenges as they move along in the season, which coaches or officials are "good" at certain call types and more.

They also noted the limitations of the data, namely that Last Two Minute Reports leave out 46 minutes or more in every game - the full league-wide data set could look very different.

Teams have access to that full data set, and those are the questions staffs like Snyder's are trying to answer. The challenges facing them (pun somewhat intended) are twofold: Not just to arrive at logical conclusions on how to use challenges, but conclusions a coach can easily reference and recall in the heat of the moment.

This isn't baseball, where umpires patiently wait while coaches radio their video team and challenge a call only once they're sure. NBA coaches have to be faster, and are working with incomplete information. They won't have extra time to calculate expected point values in their head - they have to know the framework ahead of time.

The theoretical parts of this conversation could go for hours. Even researchers Garrett and Zanca noted that, while the numbers paint one picture, the true name of the game is finding the right incorrect call to challenge within a given game - not just an incorrect call to challenge.

We're still in the early stages here for everyone, from coaches to players and even referees.

"There [are] gonna be opportunities in a very visible way to analyze those situations," Snyder said. "We'll analyze the heck out of them, no question. And sometimes we're not going to have a right answer."