Pigskin Pythagoras
A guy from Framingham tries to remake the muddy field of football statistics
By Jascha Hoffman | February 1, 2004
IN FOOTBALL, not all 15-yard passes are created equal.
Say it's the end of the 4th quarter of Super Bowl XXXVIII and the Panthers are trailing by three on their own 35-yard line. It's 4th-and-16 with a minute left. Delhomme passes to wide receiver Steve Smith, who is tackled one yard short of a 1st down. While the crowd roars, NFL statisticians quietly record the pass as a 15-yard gain, although in this situation it is clearly worth the same as an incomplete pass: absolutely nothing.
Aaron Schatz thinks there is a better way. Adapting methods from baseball fanatics and pro football franchises alike, this 29-year-old Framingham resident recently invented his own system of statistics to record this pass -- and all other NFL plays. Forget touchdowns, total yards, and red-zone efficiency. The Sunday afternoon armchair general should worry about DVOA (defense-adjusted value over average) and line-yards. By measuring each of the 40,000-odd plays made in a typical NFL season and evaluating them according to a complex array of situational factors, Schatz aims to do nothing less than revolutionize football writing and analysis.
By day, Schatz compiles the "Lycos 50" daily report on Internet pop culture. By night he tends his 6-month-old daughter and a serious football addiction. (On the Sunday she was born, he says, "I complained to the hospital that it didn't get ESPN and I couldn't watch the Bills-Chiefs game.") A year ago, he joined with several fraternity brothers from Brown University to found the website FootballOutsiders.com, which offers wry commentary, discussion threads, and reams of stats.
Schatz compares his "ridiculously geeky venture" to the mimeographed copies of the "Baseball Abstract" that the legendary Bill James compiled by hand from box scores in 1978, setting off the revolution in baseball statistics known as sabermetrics. (The name reflects the acronym of the Society for American Baseball Research, SABR). Schatz says a colleague calls their system "safermetrics," but he calls it "my giant time-sucking hobby."
Schatz started in late 2002 with a simple question: Were the Patriots really losing games because they couldn't establish the run, as many Boston sportswriters claimed? He started out counting by hand, then made some spreadsheets over Christmas vacation, and before long he had compiled a database of every single play in the 2002 NFL season.
Sure enough, he found no correlation between 1st-quarter rushing attempts and winning football games. "Teams, with rare exceptions, run when they win instead of winning when they run, so the Pats aren't hurting themselves by running less often early in games," he says.
For years, NFL franchises have kept in-house statisticians to help compile a winning playbook. But while amateur baseball analysis floods the Internet, football fans don't have much beyond the stats at NFL.com and ESPN.com and the offerings on gambling and fantasy sports sites. Though a well-respected sabermetrics site has launched its own gridiron analysis at FootballProject.com, Schatz's numbers are unique in that they evaluate each play against the league average for plays of its type, adjust for the strength of the opponents' defense, and even try to divide credit for a given play among teammates.
"We're trying to see through the biases inherent in a game where the basic situation is constantly changing," says Schatz. "We're trying to create an intelligent community for discussing the NFL -- not just point spreads, not just fantasy, but the game on the field."
Then the "three wise men" of football -- Bob Carroll, Pete Palmer, and John Thorn -- published "The Hidden Game of Football" in 1988, they were riding a current of sports analysis initiated by Bill James.
James's approach to baseball, as implemented by Billy Beane's Oakland A's, inspired Michael Lewis's 2003 bestseller "Moneyball" (W.W. Norton), and in November 2002 he was hired by the Red Sox. James's key move was to measure individual performance more accurately, compensating a pitcher's Earned Run Average for hitter-friendly parks and incompetent fielding, for example. Similarly, for Carroll, et al., and for Schatz, the key is to break down each game into individual plays, and to rate these plays based on their point-scoring value, even adjusting for tough opponents.
Gridiron analysts face a particularly daunting task. First, while baseball unfolds at a leisurely cadence of pitch-hit-field in mostly good weather, the football statistician squints out through fierce winds and snowdrifts at "twenty-two players in constantly shifting alignments, all of them racing against the clock into new configurations," as Carroll and his coauthors put it. Play-by-play records kept by the NFL and ESPN sometimes disagree on who rushed in which direction on a given play.
Second, the short 16-game schedule ensures that some teams will get off easy playing weak opponents, and threatens the analyst with a shortage of data. Finally, there's the challenge of seeing the play in context. As the authors of "The Hidden Game" put it, "The situational side of football statistics is like the weather; everybody knows it's there, but nobody does anything about it."
Schatz thinks he has solved the data problem by treating each season's 40,000-odd plays as separate events and adjusting for lucky schedules. And he has developed a statistic that he says cuts through football's situational fog.
Suppose you want to measure a quarterback's season performance. You could start by asking, all other things being equal, how many yards he averaged per pass. Unfortunately, because the offense has two immediate goals -- achieve a 1st down, and get closer to the end zone -- all things are never equal. Would you rather your quarterback complete an eight-yard pass on 3d-and-10, or a six-yard pass on 3d-and-5? The first boosts his total passing yards, but the second gets a new 1st down. "On any given down and distance, it is always better to get more yards," explains Schatz. But with just a few yards to go, less can be more.
To measure both yards and 1st downs in one statistic, while adjusting for the specific game situation, Schatz adapted a measure from "The Hidden Game." DVOA, or defense-adjusted value over average, compares every single play a team makes to the league averages for that particular situation -- down, distance to go, current score gap, location on the field, opponent -- and rates it on how it improved the team's chances of scoring. DVOA measures a team's point-scoring (or point-stopping) ability as a percentage above or below league average; a strong offense will have a positive DVOA, while a strong defense's DVOA will be negative.
For example, this season the Patriots' offense was 12th in the NFL with a -0.2 percent DVOA, but their -20 percent defense, ranking 3d in the league, more than made up for it. (In their four-interception playoff game against the Colts, the Pats' defense had an impenetrable DVOA of -80 percent.) The Panthers, on the other hand, were 14th on defense with a -2.5 percent DVOA, and a pitiful 20th on offense with -9.1 percent -- but have drastically improved in the playoffs.
According to Schatz, the DVOA should mitigate a big problem with the short 16-game season, which is that luck does not even out. "Games turn on one turnover, one big play, one lucky break, and so teams do not necessarily have a record that reflects their true ability," he says. "I think that as we do this for three or four years we will discover that DVOA is more consistent from year to year than actual wins and losses." As an example, he cites the Kansas City offense, which was No. 1 in DVOA last year when they were 8-8, and was No. 1 in DVOA this year when they were 13-3.
There remains the problem of untangling football's unfathomably complex teamwork. DVOA solves some problems easily: If Tom Brady passes for 80 yards and then Antowain Smith runs it in from the one-yard line, Smith gets credit for the touchdown -- but Brady gets the DVOA boost. But how, for example, can you evaluate the contributions of individual offensive lineman?"
We're not sure that we can even imagine a set of statistics that would make it possible to rate these players individually," say Sean Lahman and Todd Greanier, who run FootballProject.com, in their comprehensive yearbook, "Pro Football Prospectus 2003." Schatz is a bit more hopeful, using a quantity called line-yards to separate initial rushing yards earned by the strength of the offensive line from subsequent rushing yards earned by the running back's agility.
So what can Schatz tell us about Super Bowl XXXVIII?
First, according to the Pythagorean theorem of football, the only team that can beat New England is New England itself.The Pythagorean what? Early on, Bill James came up with the following formula: The ratio of wins to losses for a given baseball team is roughly equal to the square of the ratio of runs scored to runs allowed. Because it looked a bit like the familiar geometrical formula for computing the hypotenuse of a right triangle, and expressed a similarly basic relation, he named it after the Greek mathematician.
Later, statistician Daryl Morey of STATS, Inc. found that the formula applied to many sports if the exponent was tweaked: for the NFL, the exponent is 2.37 instead of 2. For example, the Patriots scored 348 points and allowed 238 in the 2003 regular season, for a ratio of 1.46 points scored for every one allowed. Raise this number to the power of 2.37 and you have an expected win-to-loss ratio of 2.46, which comes out to 11.4 wins for the 16-game season. Of course, the Pats won 14.
History shows that teams "lucky" enough to win more games than their combined offensive and defensive abilities should allow tend to find their luck running out in the post-season. For instance, in 1990 Oakland lucked into a stunning 14-2 record, but they lost in the playoffs. The Super Bowl was played between New York and Buffalo, the top two teams in Pythagorean wins that year.
The bad news, Schatz says, is that the Patriots are the "luckiest" team in the NFL this season, their inflated 14-2 record dwarfing their 11.4 Pythagorean wins. The good news, however, is that the Patriots also lead the league in Pythagorean wins, making them the first team since 1982 -- and perhaps ever -- to earn the double distinction of being both the luckiest and the best team in the NFL at the same time. "All year long, the argument has raged over whether the Patriots are lucky or good," says Schatz. "It turns out they are both."
The next question is, which Panthers will the Patriots be playing: the regular-season team, or the playoffs team? According to Schatz, the two are like night and day.
Carolina scored an anemic 8.6 Pythagorean wins in the regular season. But they won the season 11-5, a good indication that, among other things, they got lucky with an easy schedule. But when it comes to DVOA, in the playoffs Carolina had the best three-game stretch of any team in any part of the season.
Such a wild comeback is not apparent using traditional NFL stats. But Schatz's metrics show the offense improving on every down and distance except 2nd-and-short, with Jake Delhomme and company passing an unbelievable 26.4 percent DVOA over their negative 5.7 percent DVOA of just five weeks ago.
"If the last three weeks are just part of the standard ups and downs of the regular-season Carolina Panthers," Schatz opines, "then we are probably headed for a New England rout. But if Carolina's playoff performance truly represents advancement to a higher level of skill, then we have one heck of a good game on our hands."
Jascha Hoffman is a writer living in Brooklyn.