Evaluating Play Callers (Part Two)
An attempt to quantify a play caller's worth
Six months ago, I discussed a framework in detail for quantifying the value of a play caller. While the overarching framework has maintained a similar structure, a few modifications have been made, which are discussed in detail here.
Y-Aware Approach
The most significant difference between the old and new approaches lies in how the variables are standardized. In the original framework, all variables were normally standardized.
In the new method, variables are scaled based on their relationship to points scored or allowed. This new method, known as a Y-Aware approach, was implemented around week 4 of last season and increased the accuracy of the results.
The lower the p-value (indicating statistical significance), the better. In this context, Expected Points Added (EPA) and Points Over Market Expectation (POE) tend to be the features most correlated with points scored or allowed in the model. This is intuitive and suggests they should be weighted more heavily than metrics like Passes Over Expectation (PASS) and Completion Percentage Over Expectation (CPOE). While the QB's ELO Rating plays a significant role on the offensive side, it does not have the same impact defensively, which is fascinating.
The three most significant differences on the offensive side involve Andy Reid, Pete Carmichael, and Shane Steichen. In the original framework, Reid is recognized as a top three play caller, while Carmichael is considered average. Shane Steichen is viewed as a top three play caller in the new method, a notable change from being considered average in the old one.
On defense, there was significant discrepancy between the traditional PCA framework and the Y-Aware approach. The three biggest movers were Matt Patricia, Dennis Allen, and Derrick Ansley. In the old framework, Matt Patricia and Dennis Allen were viewed as some of the worst play callers this past season, whereas Ansley was viewed as the best. In the new framework, Dennis Allen is considered the best, Ansley is average, and Patricia is below average.
Changes to the Model Structure
These changes are not the only modifications to the model. Before Week 16 last year, there was public access to Next Gen Stats (NGS) personnel data, which has since been restricted. This led to the use of data charted by FTN Network. FTN documents the five skill position players on the field, allowing for offensive personnel decisions to be analyzed in a manner similar to the original model.
Across the first 15 weeks, 27,514 offensive snaps were charted by both NGS and FTN. There was a discrepancy in only 388 plays (~1.4%). It is important to note that these 27,514 plays do not include instances where a penalty occurred. While NGS accounted for player personnel on these plays, FTN does not. A proxy for penalties is now utilized in the model to account for this; total Expected Points Added (EPA) weighted by the percentage of penalties over the number of plays.
The defensive framework had a complete overhaul since the positions for all 11 defensive players on the field are not available. However, coverage types and box counts are provided by FTN and became the new parameters within the model. EPA and CPOE for man and zone coverages (on pass plays) are incorporated along with EPA for light and heavy boxes (on run plays). Light boxes are defined as 7 or fewer defenders while heavy boxes include 8 or more defenders.
Outlook for Recent Hires
Three of the top ten defensive play-callers landed head coaching positions; Mike Macdonald in Seattle, Dan Quinn in Washington, and Raheem Morris in Atlanta. Despite being excellent play callers last year, Quinn and Morris will not call plays as head coaches.
Dave Canales, who ranked 16th among all play-callers last year (including interims), certainly helped maximize the offense in Tampa Bay. However, he didn't appear to be the best option available when Carolina hired him as their next head coach.
Jim Harbaugh, Antonio Pierce, and Jerod Mayo (potentially) will not be calling plays this fall. Brian Callahan didn’t call plays in Cincinnati, will he do so for Tennessee? The CEO style of head coaching is starting to become a thing of the past. The most notable head coaches who still operate in this manner include John Harbaugh, Mike Tomlin, and Nick Sirianni.
Kellen Moore, along with Vic Fangio, Mike Caldwell, and Clint Hurtt, are heading to Philadelphia. All four have experience calling plays this past year, making their collective presence on one staff quite impressive. Shane Waldron was hired by Chicago and Ken Dorsey heads to Cleveland, both being potentially impactful hires that could offer significant benefits moving forward. It will be interesting to see if Kevin Stefanski decides to relinquish play-calling duties for the Browns.
Across all 32 franchises there are 16 new offensive coordinators and 15 new defensive coordinators entering the 2024 season. Among these coordinators, 6 offensive and 6 defensive play callers have prior experience. It will be particularly interesting to observe how Greg Roman (LAC), Kliff Kingsbury (WAS), and Mike Zimmer (DAL) adjust after a few years away from play-calling duties.
On offense, Zac Robinson (ATL), Alex Van Pelt (NE), Klint Kubiak (NO), Ryan Grubb (SEA), and Liam Coen (TB) are heading into their first season as NFL play callers. Meanwhile, there are some new faces on defense such as Jimmy Lake (ATL), Zach Orr (BAL), Jeff Hafley (GB), Jesse Minter (LAC), Chris Shula (LA), Anthony Weaver (MIA), DeMarcus Covington (NE), Dennard Wilson (TEN), and Joe Witt Jr. (WAS).
With significant turnover among teams across the league, there are a lot of new faces taking on play-calling duties. This top 10 list focuses on teams with play callers who had NFL play-calling responsibilities last season. Notably absent are teams coached by Kyle Shanahan, Sean McVay, Matt LaFleur, and Mike McDaniel, all of whom now have new defensive coordinators (DCs) without prior play-calling experience and are therefore omitted from this list. Despite their omission, it's worth mentioning that they each have a score greater than 0.25.
Addressing Concerns
Some major concerns regarding the outcomes (not the methodology) have arisen in regards to where certain offensive play callers rank. Notably, Press Taylor's placement at 7th has been deemed too high, while Andy Reid's ranking at 17th is considered too low. While I don’t personally concur with where these two play callers rank, I believe there are logical explanations as to why.
Many of the Jacksonville Jaguars' offensive struggles can be attributed to variance and bad luck, as seen in Ben Baldwin's Adjusted Expected Points Added (EPA) per Play. Trevor Lawrence and the Jaguars' passing offense were actually notably effective. While the model doesn’t account for these factors it does favor play callers who prefer a passing-oriented approach, which is an area where Press Taylor excelled.
On the other hand, Andy Reid and the Kansas City Chiefs experienced sporadic struggles and did not perform up to expectations according to the market, despite having a superstar quarterback. The model factors in quarterback value and market performance, which, in this case, is a disadvantage for one of the greatest coaches of all time.
One of the more valid criticisms of the model is its week-to-week and year-to-year stability. The main objective moving forward is to reduce variance through the development of more nuanced metrics and approaches.
Wrapping Up
This offseason, I plan to incorporate several proxies to help refine this analysis. I intend to consolidate draft capital and cap allocation into a singular metric reflective of GM/front office efficiency (that hopefully accounts for injuries) and introduce a unit strength variable, such as the Football Power Index (FPI) from ESPN or the Simple Rating System (SRS) from Pro Football Reference.
Additionally, I may consider removing variables that have shown little to no significance, such as Pass Over Expectation in 21 and other personnel groupings known to be run-heavy. Overall, I’m truly excited about far this project has come and eager to take it to the next level.