Evaluating Play Callers in the NFL

Quantifying a Play Caller's Worth

The 22 players on the field in every NFL game have a direct impact on the outcome. The coaches that design various schemes also play an important role. There are a multitude of metrics that inform us of how well passers, rushers, receivers, linemen, and defenders perform. What about coaches? How do you place a value on someone who isn’t on the field?

There are a couple of different frameworks that try to quantify the worth of a coach. Eric Eager with Sumer Sports has looked at the win probability added from head coaches making the correct in game decision. While this is certainly insightful, it doesn’t account for play calling ability which captures additional value for a coach.

Other attempts try to capture player and team contribution and credit the residual to the head coach or play caller. These structures can penalize coaches who get the most out of their players (Shanahan, Tomlin, etc.), making it hard to distinguish between the system and the personnel. In this article, a slightly different approach is taken.

Offensive Framework

Since 2016, 11 personnel (1 RB, 1 TE, 3 WR) is the most popular package to run an offense out of in the modern NFL. Sean McVay and several of his prodigies operate practically their entire offense out of this personnel package. This has led to defensive coordinators playing smaller personnel groups and lighter boxes to try and suffocate these high-flying offenses. At times it’s almost a dare for the offense to run the ball.

Part of what makes a great offensive play caller is their ability to solve the same problem in different ways. For example, Andy Reid and Kyle Shanahan design efficient offenses in a multitude of personnel packages. To counter lighter boxes and two high shells, Andy Reid put two or more TEs on the field (12 & 13 personnel) for roughly 38.5% of their offensive snaps with a pass rate around 65% last season. As a result, several front offices invested in the TE position this offseason to gain a similar edge.

Kyle Shanahan loves running his outside zone concepts out of 21 personnel (2 RB, 1 TE, 2 WR). The threat of the run in a few cases opens the passing game up. Many analysts would disagree with this sentiment (and for good reason), but this is something Shanahan’s players have even attested to. Mike McDaniel brought those same concepts to Miami. As a result, Tua Tagovailoa dropped back to pass at the highest rate in 21 personnel last season.

Along with personnel decisions, play callers also influence the number of pass attempts called. Using an offense’s pass rate doesn’t account for various in game situations. Passes Above Expected from nflfastR are a better reflection of how aggressive a play caller is.

What about the team around the coach? This analysis considers how good the quarterback is along with the team’s cap allocation and draft capital spent. To account for quarterback value, the average ELO rating for starters is taken for each play caller. Similarly, a team’s yearly cap allocation on the offensive side is factored against the play caller. The quarterback’s cap hit is not included to avoid double counting.

Since the 2011 CBA, rookie contracts are structured over four years. The resources spent on offense over the previous four drafts are a good proxy for the front office’s investment in young talent. In this analysis, the Fitzgerald-Spielberger Draft Trade Value Chart is used to provide an accurate value for each pick. In future iterations, snap counts will be considered as a weight for the total draft capital invested.

The last and probably the most controversial aspect is crediting points over market expectation to the play caller. The expected points for both sides of the ball is calculated with the closing total and spread lines.

Defensive Framework

Overall, there aren’t many differences between the offensive and defensive play calling models. Four personnel packages are analyzed for the offense; 11, 12, 21 and Other. Whereas three personnel groupings are considered on defense; Base, Nickel and Other. Base includes both 3-4 and 4-3 looks.

Instead of looking at Passes Above Expected, Completion Percentage Over Expected (CPOE) from nflfastR is included. Defenses that typically allow a lower CPOE tend to play lighter boxes. This intuitively makes a lot of sense. A defensive play caller that drops more players in coverage will be in a better position to defend against the pass.

On offense, play callers are penalized for having really good quarterbacks. It’s the reverse on the defensive side of the ball. The better the quarterback the defense faces, the more credit the play caller receives.

Model Overview

Principal Component Analysis (PCA) is a multivariate statistical technique that is particularly useful with datasets that have a lot of features. This method reduces the dimensionality within the data and helps to observe coaching trends, clusters and outliers. One of the underlying assumptions with PCA is that the data is normally distributed. In this analysis, all the variables discussed are standardized.

The subset of variables involving efficiency in different personnel packages is measured by the annual sum of Expected Points Added (EPA). Aggressiveness in these packages is measured by the average Passes Above Expected for the offense and Completion Percentage Over Expected for the defense. These Efficiency and Aggressiveness metrics are then weighted by the percentage of snaps by personnel grouping.

Since 2016, there have been 85 offensive play callers who have called at least 500 plays. This analysis omits eight play callers over this duration. The biplot displays the relationship of the variables and coaches over the first two dimensions.

Four dimensions in both models have an eigenvalue greater than one. This rule known as the Kaiser’s Latent Root Criterion identifies which components to keep and helps to filter out noise. Bartlett’s Test of Sphericity is another thing to consider with PCA. It checks for redundancy between the variables. Both the offensive and defensive models meet the significance criteria (p-value < 0.05).

73 coaches have called least 500 plays on defense. Two play callers have failed to meet this criteria since 2016. Both biplots tell a similar story. The Aggressiveness metrics correlate with the lack of draft resources a team spent and the Efficiency variables are in line with how well a unit performed above market.

Now there is one final step before coaching scores can be generated. An orthogonal rotation is applied to the significant principal axes which makes linear combinations easier to interpret. These rotated components no longer correspond to orthogonal axes or capture the maximal amount of variance. The variance is then redistributed to the other components.

The proportion of variance of the newly rotated components are taken and multiplied to the corresponding factor scores for each play caller. These scores are then summed over the four significant dimensions to create the final ranking.

Offensive Play Callers

Andy Reid and Kyle Shanahan are the best in the business at calling offenses. Four first year play callers made the top 12; Mike McDaniel, Ken Dorsey, Ben Johnson and Mike Kafka. We’ll see if they can pick up where they left off in 2023.

The last time Mike McCarthy, Todd Monken and Bill O’Brien called plays for a full season in the NFL was pre-COVID. The front offices for all three teams made it a point of emphasis to add pass catchers this offseason. The Cowboys traded for Brandin Cooks while the Ravens signed Odell Beckham and drafted Zay Flowers. The Patriots even brought in Juju Smith-Schuster and Mike Gesicki.

14 teams have a new offensive play callers, nine of whom have prior play calling experience. Four out of the five first-time play callers; Eric Bieniemy (WAS), Bobby Slowik (HOU), Drew Petzing (ARI) and Dave Canales (TB) were all hired by defensive minded head coaches. The only exception was the Eagles’ quarterbacks coach, Brian Johnson, who was promoted internally after Shane Steichen left for Indianapolis.

Sean Payton and Bill O’Brien are two of three biggest upgrades this year. The general public would consider Kellen Moore to be one of the major upgrades across the league, but that might not be entirely the case. The model sees Moore as a downgrade to Lombardi and McCarthy. The main reason for this is because the Charger’s new play caller hasn’t been creative or aggressive enough in packages outside of 11 personnel.

The response will inevitably be one of two arguments. The first is McCarthy held Moore back. The second will point to Dak Prescott missing five games last year and eleven games in 2020. The last two seasons McCarthy (GB; ‘17 - ‘18) and Moore (DAL; ‘21 - ‘22) called plays, tells a different story.

This idea that McCarthy will run the ball exponentially more I don’t believe is supported by his prior tendencies. Despite saying, “Kellen wants to light the scoreboard up. But I want to run the damn ball so I can rest my defense.” McCarthy actually called more passes over expected in 11 personnel (+3.26 compared to +2.28) and 12 personnel (+9.39 compared to -8.11) his last two years in Green Bay. Moore only passed at a higher rate in 21 personnel (+2.5 compared to +2.13). Now to Moore’s credit, he was more efficient during this two year span in each personnel group aside from 12 personnel.

Perhaps Moore had to dial up more runs because Cooper Rush started five games last year. A backup quarterback can limit an offense and influence conservative play calling. In 2017, Brett Hundley started nine games for the Packers when Aaron Rodgers was out. If McCarthy is truly more conservative than Moore, why were his passing numbers still higher despite having a backup quarterback start in almost twice as many games?

Defensive Play Callers

Bill Belichick is still the best defensive mind in the game and should orchestrate a dominant unit this fall. The Vikings had several salary cap casualties this offseason, but upgraded at an important position; defensive coordinator. If Brian Flores can get around league average production out of the Vikings’ defense, that will be a huge win.

There is a lot of hype around Vic Fangio’s style of defensive play in the modern NFL. It’s a big reason why Brandon Staley landed the head coaching gig with the Chargers. While Raheem Morris and Ejiro Evero have not coached directly with Fangio they do operate similar systems.

Over the past two years, there is a considerable amount of hype around Dan Quinn and Lou Anarumo. When you factor in the playmakers they’ve had at their disposal and their body of work over this duration their impact isn’t as significant.

To kickoff 2023 there will be 11 new defensive play callers across the league. Nine of those coaches have previous experience calling plays. The first time play callers are Ryan Nielsen (ATL) and Nick Rallis (ARI). Nielsen was the Saints’ defensive line coach last year and Rallis was the Eagles’ linebackers coach.

Sean McDermott is an example of two of the larger pitfalls within this framework. McDermott’s last year calling plays for the Panthers wasn’t his best, but it’s the only year recorded in this analysis. The data is limited to seven years and doesn’t accurately represent several play callers with experience prior to 2016.

Another issue is that Leslie Frazier ends up receiving the credit for the Bills’ defense even though Sean McDermott was the head coach. If the play caller isn’t the head coach, but shares the same background (offense/defense) how should the credit be divided? This is something that I’m still wrestling with and trying to appropriately account for.

Wrap Up

Kyle Shanahan rightfully deserves the hype he’s received and has earned his seat at the table with first ballot Hall of Famers; Andy Reid and Bill Belichick. The only question that remains; will Shanahan lead the 49ers to a Super Bowl victory?

Two of the top five play calling duos are going into year two of their current systems. Mike McDaniel is an integral part of the play calling in Miami, whereas Brian Daboll has relinquished those responsibilities to his offensive coordinator in New York.

Since 2011, at least five new head coaches are hired each year. The annual average over the last 13 years is exactly seven head coaches. Almost a quarter of the league hires a new head coach every season. Finding better ways to evaluate play callers on both sides of the ball could improve how well teams identify game changers.

A potential direction to take this analysis on coaches could be to use pre-snap alignments, route combinations, zone coverages, and blocking assignments to distinguish between different styles and in turn how efficient they are. Unfortunately, player tracking data is not easily accessible and could stay that way for the foreseeable future.

Resources

A special thanks to Sebastian Carl and Tan Ho who developed the nflverse data repository; nflreadr. It played a vital role in this analysis.

Also could not have done this without salary cap numbers from Over the Cap and the Draft Trade Value Chart from Jason Fitzgerald and Brad Spielberger.

Last but not least, a special thanks to Sam Hoppen for compiling a list of offensive and defensive play callers for every NFL team dating back to 2013.