Modeling Score Events from Play by Play Data
I develop and explore an expected points model at the play level for evaluating college football offenses and defenses. The goal of this analysis is to place a value on offensive/defensive plays in terms of their contribution’s to a team’s expected points.
The data comes from collegefootballdata.com, which has play by play data on games from 2000 to present. Each observation represents one play in a game, in which we know the team, the situation (down, time remaining), and the location on the field (yards to go, yards to reach end zone). We have information about the types of plays as well in a text field.
Due to data quality issues, I focus my analysis on the years from 2007 and onwards.
For each play in a game, I model the probability of the next scoring event that will occur within the same half for either team. This means the analysis is not at the drive level, but at what I dub the sequence level. For any given play, the next scoring event can take on one of seven outcomes:
Suppose we have two teams, A and B, playing in a game. Team A receives the opening kickoff, begins drive #1, drives for a few plays, and then punts. Team B takes over, which starts drive #2, and they drive for a few plays before also punting. Team A then manages to finally put points on the board, scoring a TD on drive #3.
tibble(
offense = c("A", "B", "A"),
defense = c("B", "A", "B"),
drive = c(1, 2, 3),
drive_result = c("PUNT", "PUNT", "TD"),
score_result = c(NA, NA, "OFFENSE TD")
) |>
gt_tbl()| offense | defense | drive | drive_result | score_result |
|---|---|---|---|---|
| A | B | 1 | PUNT | — |
| B | A | 2 | PUNT | — |
| A | B | 3 | TD | OFFENSE TD |
All plays on these three drives are one sequence. The outcome of this sequence is the points scored by Team A - if they score a touchdown, their points from this sequence is 7 (assuming for now they make the extra point). Team B’s points from this sequence is -7 points.
This means that each one of these play was leading up to the Next Scoring Event of Team A scoring, which was the outcome we assign to each drive (and play) in that sequence.
tibble(
offense = c("A", "B", "A"),
defense = c("B", "A", "B"),
sequence = c(1, 1, 1),
drive = c(1, 2, 3),
drive_result = c("PUNT", "PUNT", "TD"),
score_result = c(NA, NA, "OFFENSE TD"),
next_score_event = c("team A TD", "team A TD", "team A TD")
) |>
gt_tbl()| offense | defense | sequence | drive | drive_result | score_result | next_score_event |
|---|---|---|---|---|---|---|
| A | B | 1 | 1 | PUNT | — | team A TD |
| B | A | 1 | 2 | PUNT | — | team A TD |
| A | B | 1 | 3 | TD | OFFENSE TD | team A TD |
If the team on offense drives down and scores a TD/FG, this will end the sequence. If the team on offense does not score but punts or turns the ball over, the sequence will continue with the other team now on offense. The sequence will continue until either one team scores, or the half comes to an end. From this, a sequence begins at kickoff and ends at the next kick off. When Team A kicks off to Team B to start drive #4, we start our next sequence, which will end either with one team scoring or at the end of the half.
Why model the outcome of sequences rather than individual drives? Individual plays have the potential to affect both team’s chances of scoring, positively or negatively, and we want our model to directly capture this. If an offense turns the ball over at midfield, they are not only hurting their own chances of scoring, they are increasing the other team’s chance of scoring. The value of a play in terms of expected points is function of how both team’s probabilities are affected by the result of the play.
A team’s expected points is sum of the probability of each possible scoring event multiplied by the points of that event. For this analysis, I assume that touchdowns equate to 7 rather than 6 points, assuming that extra points will be made. I can later bake in the actual probability of making extra points, but this will be a simplification for now.
For a given play \(i\) for Team \(A\) facing Team \(B\), we can compute Team A’s expected points using the following:
\[\begin{align*} {Expected Points}_A = \\ Pr(TD_A)*7 \\ + Pr(FG_A)*3 \\ + Pr(Safety_A)*2 \\ + Pr(No Score)*0 \\ + Pr(Safety_B)*-2 \\ + Pr(FG_B) * -3\\ + Pr(TD_B) * -7 \end{align*}\]
How do we get the probabilities of each scoring event? We learn these from historical data by using a model - I train a multinomial logistic regression model on many seasons worth of college football plays to learn how situations on the field affect the probability of the next scoring event.
The outcome for our analysis is the next scoring event. Each play in a given sequence contributes to the eventual outcome of the sequence. Here we can see an example of one game and its drives:
sample_games = tibble(game_id = 322520245)
cfbd_drives_tbl |>
inner_join(sample_games) |>
select(season, game_id, offense, defense, drive_number, drive_result) |>
gt_tbl()| season | game_id | offense | defense | drive_number | drive_result |
|---|---|---|---|---|---|
| 2012 | 322520245 | Texas A&M | Florida | 1 | FG GOOD |
| 2012 | 322520245 | Florida | Texas A&M | 2 | RUSHING TD |
| 2012 | 322520245 | Texas A&M | Florida | 3 | RUSHING TD |
| 2012 | 322520245 | Florida | Texas A&M | 4 | PUNT |
| 2012 | 322520245 | Texas A&M | Florida | 5 | RUSHING TD |
| 2012 | 322520245 | Florida | Texas A&M | 6 | FG GOOD |
| 2012 | 322520245 | Texas A&M | Florida | 7 | END OF HALF |
| 2012 | 322520245 | Florida | Texas A&M | 8 | FG GOOD |
| 2012 | 322520245 | Texas A&M | Florida | 9 | PUNT |
| 2012 | 322520245 | Florida | Texas A&M | 10 | PUNT |
| 2012 | 322520245 | Texas A&M | Florida | 11 | PUNT |
| 2012 | 322520245 | Florida | Texas A&M | 12 | PUNT |
| 2012 | 322520245 | Texas A&M | Florida | 13 | PUNT |
| 2012 | 322520245 | Florida | Texas A&M | 14 | RUSHING TD |
| 2012 | 322520245 | Texas A&M | Florida | 15 | PUNT |
| 2012 | 322520245 | Florida | Texas A&M | 16 | PUNT |
| 2012 | 322520245 | Texas A&M | Florida | 17 | FUMBLE |
| 2012 | 322520245 | Florida | Texas A&M | 18 | PUNT |
| 2012 | 322520245 | Texas A&M | Florida | 19 | PUNT |
| 2012 | 322520245 | Florida | Texas A&M | 20 | END OF HALF |
For this game, we can filter to the plays that took place in the lead up to first score event. In this case, the first sequence included one drive and ended when Texas A&M kicked a field goal.
# function to display plays in a drive
drive_plays_tbl <- function(data) {
data |>
select(season, game_id, offense, defense, drive_number, down, distance, yards_to_goal, play_text, next_score_event_offense) |>
group_by(season, game_id, drive_number) |>
gt_tbl() |>
gt::cols_label(next_score_event_offense = "next_score_event") |>
gt::cols_align(columns = c("offense", "defense", "down", "distance", "yards_to_goal"), align = "center")
}
# show first drive
prepared_pbp |>
inner_join(sample_games) |>
filter(drive_number == 1) |>
drive_plays_tbl()| offense | defense | down | distance | yards_to_goal | play_text | next_score_event |
|---|---|---|---|---|---|---|
| 2012 - 322520245 - 1 | ||||||
| Texas A&M | Florida | 1 | 10 | 75 | TEXAS A&M penalty 5 yard False Start on Patrick Lewis accepted. | FG |
| Texas A&M | Florida | 1 | 15 | 80 | Christine Michael rush for no gain to the TexAM 20. | FG |
| Texas A&M | Florida | 2 | 15 | 80 | Johnny Manziel pass complete to Ryan Swope for a loss of 2 yards to the TexAM 18. | FG |
| Texas A&M | Florida | 3 | 17 | 82 | Johnny Manziel rush for 16 yards to the 50 yard line, FLORIDA penalty 16 yard Personal Foul accepted for a 1ST down. | FG |
| Texas A&M | Florida | 1 | 10 | 50 | Johnny Manziel pass complete to Kenric McNeal for 8 yards to the Fla 42. | FG |
| Texas A&M | Florida | 2 | 2 | 42 | Christine Michael rush for 2 yards to the Fla 40 for a 1ST down. | FG |
| Texas A&M | Florida | 1 | 10 | 40 | Johnny Manziel pass complete to Kenric McNeal for 3 yards to the Fla 37. | FG |
| Texas A&M | Florida | 2 | 7 | 37 | Christine Michael rush for 6 yards to the Fla 31. | FG |
| Texas A&M | Florida | 3 | 1 | 31 | Christine Michael rush for no gain to the Fla 31. | FG |
| Texas A&M | Florida | 4 | 1 | 31 | Johnny Manziel pass complete to Nehemiah Hicks for 8 yards to the Fla 23 for a 1ST down. | FG |
| Texas A&M | Florida | 1 | 10 | 23 | Johnny Manziel pass complete to Ryan Swope for 2 yards to the Fla 21. | FG |
| Texas A&M | Florida | 2 | 8 | 21 | Johnny Manziel pass complete to Christine Michael for 14 yards to the Fla 7 for a 1ST down. | FG |
| Texas A&M | Florida | 1 | 7 | 7 | Johnny Manziel pass incomplete to Kenric McNeal, broken up by Matt Elam. | FG |
| Texas A&M | Florida | 2 | 7 | 7 | Johnny Manziel pass incomplete to Mike Evans, broken up by Marcus Roberson. | FG |
| Texas A&M | Florida | 3 | 7 | 7 | Johnny Manziel rush for 3 yards to the Fla 9, TEXAS A&M penalty 5 yard Illegal Forward Pass on Johnny Manziel accepted. | FG |
| Texas A&M | Florida | 4 | 9 | 9 | Taylor Bertolet 27 yard field goal GOOD. | FG |
If we look at another sequence in the second half, we can see there were multiple drives before a team was able to score in that sequence.
The next scoring event is always defined from the perspective of the offense.
prepared_pbp |>
inner_join(sample_games) |>
filter(drive_number > 8 & drive_number < 15) |>
drive_plays_tbl()| offense | defense | down | distance | yards_to_goal | play_text | next_score_event |
|---|---|---|---|---|---|---|
| 2012 - 322520245 - 9 | ||||||
| Texas A&M | Florida | 1 | 10 | 73 | Christine Michael rush for 6 yards to the TexAM 33. | Opp_TD |
| Texas A&M | Florida | 2 | 4 | 67 | Christine Michael rush for 1 yard to the TexAM 34. | Opp_TD |
| Texas A&M | Florida | 3 | 3 | 66 | Johnny Manziel pass complete to Mike Evans for 1 yard to the TexAM 35. | Opp_TD |
| Texas A&M | Florida | 4 | 2 | 65 | Ryan Epperson punt for 42 yards, fair catch by Andre Debose at the Fla 23. | Opp_TD |
| 2012 - 322520245 - 10 | ||||||
| Florida | Texas A&M | 1 | 10 | 77 | Jeff Driskel pass complete to Frankie Hammond for 10 yards to the Fla 33 for a 1ST down. | TD |
| Florida | Texas A&M | 1 | 10 | 67 | Mike Gillislee rush for 4 yards to the Fla 37. | TD |
| Florida | Texas A&M | 2 | 6 | 63 | Jeff Driskel pass complete to Omarius Hines for 7 yards to the Fla 44 for a 1ST down. | TD |
| Florida | Texas A&M | 1 | 10 | 56 | Mike Gillislee rush for 3 yards to the Fla 47. | TD |
| Florida | Texas A&M | 2 | 7 | 53 | Jeff Driskel sacked by Spencer Nealy for a loss of 1 yard to the Fla 46. | TD |
| Florida | Texas A&M | 3 | 8 | 54 | Jeff Driskel sacked by Sean Porter and Damontre Moore for a loss of 12 yards to the Fla 34. | TD |
| Florida | Texas A&M | 4 | 20 | 66 | Kyle Christy punt for 48 yards, fair catch by Dustin Harris at the TexAM 18. | TD |
| 2012 - 322520245 - 11 | ||||||
| Texas A&M | Florida | 1 | 10 | 82 | Johnny Manziel pass complete to Mike Evans for a loss of 2 yards to the TexAM 16. | Opp_TD |
| Texas A&M | Florida | 2 | 12 | 84 | Trey Williams rush for a loss of 2 yards to the TexAM 14. | Opp_TD |
| Texas A&M | Florida | 3 | 14 | 86 | Johnny Manziel rush for 3 yards to the TexAM 17. | Opp_TD |
| Texas A&M | Florida | 4 | 11 | 83 | Ryan Epperson punt for 53 yards, downed at the Fla 30. | Opp_TD |
| 2012 - 322520245 - 12 | ||||||
| Florida | Texas A&M | 1 | 10 | 70 | Jeff Driskel pass complete to Jordan Reed for 5 yards to the Fla 35. | TD |
| Florida | Texas A&M | 2 | 5 | 65 | Mack Brown rush for no gain to the Fla 35. | TD |
| Florida | Texas A&M | 3 | 5 | 65 | Jeff Driskel rush for 14 yards to the Fla 49 for a 1ST down. | TD |
| Florida | Texas A&M | 1 | 10 | 51 | Matt Jones rush for 5 yards to the TexAM 46. | TD |
| Florida | Texas A&M | 2 | 5 | 46 | Jeff Driskel pass incomplete. | TD |
| Florida | Texas A&M | 3 | 5 | 46 | Jeff Driskel rush for no gain to the TexAM 46. | TD |
| Florida | Texas A&M | 4 | 5 | 46 | Kyle Christy punt for 37 yards, fair catch by Dustin Harris at the TexAM 9. | TD |
| 2012 - 322520245 - 13 | ||||||
| Texas A&M | Florida | 1 | 10 | 91 | Johnny Manziel pass complete to Mike Evans for 14 yards to the TexAM 23 for a 1ST down. | Opp_TD |
| Texas A&M | Florida | 1 | 10 | 77 | TEXAS A&M penalty 11 yard Personal Foul on Kenric McNeal accepted. | Opp_TD |
| Texas A&M | Florida | 1 | 10 | 88 | Johnny Manziel pass complete to Thomas Johnson for 2 yards to the TexAM 14. | Opp_TD |
| Texas A&M | Florida | 2 | 8 | 86 | Johnny Manziel pass complete to Mike Evans for 5 yards to the TexAM 19. | Opp_TD |
| Texas A&M | Florida | 3 | 3 | 81 | Johnny Manziel sacked by Lerentee McCray for a loss of 4 yards to the TexAM 15. | Opp_TD |
| Texas A&M | Florida | 4 | 7 | 85 | Ryan Epperson punt for 47 yards, fair catch by Andre Debose at the Fla 38. | Opp_TD |
| 2012 - 322520245 - 14 | ||||||
| Florida | Texas A&M | 1 | 10 | 62 | Mike Gillislee rush for 5 yards to the Fla 43. | TD |
| Florida | Texas A&M | 2 | 5 | 57 | Jeff Driskel pass complete to Omarius Hines for 39 yards to the TexAM 18 for a 1ST down. | TD |
| Florida | Texas A&M | 1 | 10 | 18 | Solomon Patton rush for 6 yards to the TexAM 12. | TD |
| Florida | Texas A&M | 2 | 4 | 12 | Mike Gillislee rush for 12 yards for a TOUCHDOWN. | TD |
Our goal is to understand how individual plays contribute to a team’s expected points, or the average points a team should expect to score on average given their situation (down, time, possession).
For instance, in the first drive of the Texas A&M-Florida game in 2012, Texas A&M received the ball at their own 25 yard line to open the game.
The simplest intuition of expected points is to ask, for teams starting at the 25 yard line at the beginning of a game, how many points do they typically go on to score?
One way to answer this is to look at all starting drives with 75 yards to go and see what the eventual next scoring event was for each of these plays - we simply take the average of all of the points that followed from teams in this situation.
prepared_pbp |>
filter(yards_to_goal == 75 & drive_number == 1) |>
group_by(next_score_event_offense, yards_to_goal, drive_number) |>
count(sort = T) |>
ungroup() |>
mutate(prop = n / sum(n)) |>
mutate_if(is.numeric, round, 3) |>
gt_tbl()| next_score_event_offense | yards_to_goal | drive_number | n | prop |
|---|---|---|---|---|
| TD | 75 | 1 | 2807 | 0.385 |
| Opp_TD | 75 | 1 | 2528 | 0.346 |
| FG | 75 | 1 | 982 | 0.135 |
| Opp_FG | 75 | 1 | 914 | 0.125 |
| Safety | 75 | 1 | 25 | 0.003 |
| No_Score | 75 | 1 | 24 | 0.003 |
| Opp_Safety | 75 | 1 | 20 | 0.003 |
prepared_pbp |>
filter(yards_to_goal == 75 & drive_number == 1) |>
summarize(
expected_points = round(mean(next_score_event_offense_diff), 3),
n = n(),
.groups = "drop"
) |>
gt_tbl()| expected_points | n |
|---|---|
| 0.297 | 7300 |
In this case, we find teams with the ball at their own 25 to start the game generally obtained more points on the ensuing sequence than their opponents, so they have a slightly positive expected points.
But, expected points is also a function of the down. If we look at the expected points for a team in this situation in first down vs a team in this situation for fourth down, we should see a drop in their expected points - by the time you hit fourth down, if you haven’t moved from the 25, your expected points drops into the negatives, as you will now be punting the ball back to your opponent and it becomes more probable that they score than you.
prepared_pbp |>
filter(yards_to_goal == 75 & drive_number == 1) |>
group_by(yards_to_goal, down) |>
summarize(
expected_points = mean(next_score_event_offense_diff, na.rm = T),
n = n(),
.groups = "drop"
) |>
mutate_if(is.numeric, round, 3) |>
gt_tbl()| yards_to_goal | down | expected_points | n |
|---|---|---|---|
| 75 | 1 | 0.617 | 5487 |
| 75 | 2 | -0.308 | 1164 |
| 75 | 3 | -0.711 | 429 |
| 75 | 4 | -2.527 | 220 |
The fact that expected points changes based on the situation (down, yard line, time remaining) allows us to calculate the difference in expected points from play to play. That is, before the ball is snapped, we ask, what is the expected points given the current situation?
Then the ball is hiked and the play occurs; what is the expected points now? Did it increase or decrease? This difference in expected points from play to play, positive or negative, is what we refer to as Expected Points Added.
For any given play, we get a sense of the expected points a team can expect from their situation. For instance, if we look at all total plays in a game, how do expected points vary as a function of a team’s distance from their opponent’s goal line?
prepared_pbp |>
filter(yard_line < 100 & yard_line > 0) %>%
group_by(yards_to_goal) |>
summarize(
expected_points = mean(next_score_event_offense_diff, na.rm = T),
n = n()
) %>%
ggplot(., aes(
x = yards_to_goal,
y = expected_points
)) +
geom_line() +
geom_point(aes(size = n)) +
geom_hline(
yintercept = 0,
linetype = "dashed"
) +
scale_x_reverse()
This should make sense - if you’re backed up against your own end zone, your opponent has higher expected points because they are, historically, more likely to have the next scoring event - either by gaining good field advantage after you punt or by getting a safety. We can see this if we just look at the proportion of next scoring events based on the offense’s position on the field.
prepared_pbp |>
filter(yard_line < 100 & yard_line > 0) %>%
group_by(yards_to_goal, next_score_event_offense) |>
count() |>
ggplot(aes(x=yards_to_goal, y=n, fill = next_score_event_offense))+
geom_col(position = 'fill')+
scale_x_reverse()+
scale_fill_viridis_d()+
ylab("proportion")
From this, when we see an offense move the ball up the field on a given play, we will generally see their expected points go up. The difference in expected points before the snap and after the snap is the value added (positively or negatively) by the play.
But, it’s not just position on the field - it’s also about the situation. If we look at how expected points varies by the down, we should see that fourth downs have lower expected points.
prepared_pbp |>
filter(yard_line < 100 & yard_line > 0) |>
group_by(yards_to_goal, down = as.character(down)) |>
summarize(
expected_points = mean(next_score_event_offense_diff, na.rm = T),
n = n(),
.groups = 'drop'
) |>
ggplot(aes(
x = yards_to_goal,
y = expected_points,
color = down
)) +
geom_point(aes(size = n)) +
geom_hline(
yintercept = 0,
linetype = "dashed"
) +
scale_x_reverse()+
scale_color_viridis_d()+
guides(size = 'none')
We also have other features like distance and down.
prepared_pbp |>
filter(yard_line < 100 & yard_line > 0) |>
filter(distance <= 30, distance >=0) |>
group_by(down, distance, next_score_event_offense) |>
count() |>
ggplot(aes(x=distance, y=n, fill = next_score_event_offense))+
geom_col(position = 'fill')+
scale_fill_viridis_d()+
facet_wrap(down ~.)+
ylab("proportion")
And we also have info on time remaining in the half - as we might expect, the proportion of drives leading to no scoring goes up as the amount of time remaining in the half goes down.
prepared_pbp |>
filter(yard_line < 100 & yard_line > 0) |>
group_by(
seconds_in_half = round_any(seconds_in_half, 5),
next_score_event_offense
) |>
count() |>
ggplot(aes(x = seconds_in_half, y = n, fill = next_score_event_offense)) +
geom_col(position = "fill") +
scale_fill_viridis_d() +
ylab("proportion")+
scale_x_reverse()
We use all of this historical data to learn the expected points from a given situation, then look at the difference in expected points from play to play - this is the intuition behind how we will value individual plays, which we can then roll up to the offense/defense/game/season level.
How do these various features like down, distance, yards to goal, and time remaining affect the probability of the next scoring event? I use a model to learn this relationship from historical plays.
First, I set up training, validation, and test sets based around the season. I rely on data from the 2007 season onwards, as the data quality of plays starts to get worse the further back we go, though I can later do some backtesting of the model on older seasons.
# load in pieces from model
tar_load(split_pbp)
tar_load(pbp_last_fit)
# get wflow
pbp_fit = pbp_last_fit |> extract_workflow()
# show plan
split_pbp |>
plot_split_plan()
The outcome of interest is next_score_event, which is always defined from the perspective of the offense.
split_pbp |>
validation_set() |>
pluck("splits", 1) |>
pluck("data") |>
mutate(next_score_event_offense = factor_class(next_score_event_offense)) |>
group_by(season, next_score_event_offense) |>
count() |>
ggplot(aes(x=n,
y=reorder(next_score_event_offense,n),
fill = next_score_event_offense))+
geom_col()+
facet_wrap(season ~.)+
scale_fill_viridis_d()+
guides(fill = 'none')+
scale_x_continuous(breaks = scales::pretty_breaks(n=3))+
ylab("Score Event")+
xlab("Plays")
I train a multinomial logistic regression on the next scoring event for each play (TD, FG, Opp. TD, Opp. FG, etc) as a function of the situation in the game (down, distance, yards to goal). I use the probabilities from this model to compute expected points at the play level, which I then aggregate to the team, game, and season level in order to measure each team’s raw offensive/defensive efficiency.
I examine the model’s performance on the validation set.
pbp_last_fit |>
collect_metrics() |>
mutate_if(is.numeric, round, 3) |>
gt_tbl()| .metric | .estimator | .estimate | .config |
|---|---|---|---|
| roc_auc | hand_till | 0.714 | Preprocessor1_Model1 |
| mn_log_loss | multiclass | 1.221 | Preprocessor1_Model1 |
What I really care about is the calibration of the predictions - does the observed incidence rate of events match the predicted probabilities from the model? That is, when the model predicts that the next scoring event has a probability of 0.5 of being a TD, do we observe TDs occur about half of the time?
pbp_last_fit |>
collect_predictions() |>
plot_pbp_calibration() +
labs(
title = "Model Calibration",
subtitle = stringr::str_wrap("Observed vs predicted incident rate of next scoring event from classification model", 90)
)
Understanding partial effects from a multinomial logit is difficult, and I’ve thrown a bunch of interactions in there to make this even more unwieldy. I’ll extract the coefficients and take a look (excluding the intercept), but really in order to interpret this model I’ll use predicted probabilities.
pbp_fit |>
tidy() |>
mutate(class = factor_class(class)) |>
filter(term != "(Intercept)") |>
ggplot(aes(x=estimate, y = reorder(term, abs(estimate)))) +
#y=tidytext::reorder_within(term, estimate, class)))+
geom_point()+
facet_wrap(class~., ncol = 4) +
ylab("Feature")
I’ll look at predicted probabilities using an observed values approach for particular features (using a sample rather than the full dataset to save time). This amounts taking historical plays and setting features to specific values for every observation, predicting those observations with the model, then finding the average predicted probability as the feature changes.
The following visualization the predicted probability of each scoring event based on field position.
set.seed(1999)
samp <-
split_pbp |>
training() |>
slice_sample(n = 10000)
v <- expand.grid(
yards_to_goal = seq(0, 99, 3),
down = c(1, 2, 3, 4)
)
df <-
map(
seq(0, 99, 2),
~ {
samp |>
mutate(yards_to_goal := .x) |>
nest(data = -yards_to_goal)
}
) |>
list_rbind() |>
unnest(data)
est <-
df |>
estimate_pbp_effect(fit = pbp_fit) |>
summarize_pbp_effect(vars = "yards_to_goal")
est |>
pivot_pbp() |>
mutate(class = factor_class(class)) |>
ggplot(aes(
x = yards_to_goal, color = class,
y = prob
)) +
geom_line() +
scale_color_viridis_d() +
facet_wrap(class ~ ., ncol = 4) +
coord_cartesian(ylim = c(0, 1)) +
ylab("Pr(Outcome)") +
xlab("Yards to Opponent End Zone") +
labs(
title = "Predicted Probability of Scoring Event by Offensive Field Position",
subtitle = stringr::str_wrap("Predicted probabilities from classification model trained on historical play by play data. Displaying probabilities using observed values approach from a random sample of plays", 90)
) +
guides(color = "none")
How is this affected by the down?
df2 <-
map(
c(1, 2, 3, 4),
~ {
df |>
mutate(down := .x) |>
nest(data = -c(yards_to_goal, down))
}
) |>
list_rbind()
est2 <-
df2 |>
unnest(data) |>
estimate_pbp_effect(fit = pbp_fit) |>
summarize_pbp_effect(vars = c("yards_to_goal", "down"))
# plot probabilities
est2 |>
pivot_pbp() |>
mutate(
down = factor(down),
class = factor_class(class)
) |>
ggplot(aes(
x = yards_to_goal,
color = down,
y = prob
)) +
geom_line() +
scale_color_viridis_d() +
facet_wrap(class ~ ., ncol = 4) +
coord_cartesian(ylim = c(0, 1)) +
ylab("Pr(Outcome)") +
xlab("Yards to Goal") +
labs(
title = "Predicted Probability of Scoring Event by Offensive Field Position and Down",
subtitle = stringr::str_wrap("Predicted probabilities from classification model trained on historical play by play data. Displaying probabilities using observed values approach from a random sample of plays", 90)
)
We can then translate these predicted probabilities into expected points, multiplying the predicted probabilities for each scoring event by their point value.
est2 |>
mutate(down = factor(down)) |>
calculate_expected_points() |>
ggplot(aes(x = yards_to_goal, y = expected_points, color = down)) +
geom_line() +
scale_color_viridis_d() +
coord_cartesian(ylim = c(-3, 7)) +
geom_hline(yintercept = 0, linetype = "dashed") +
labs(
title = "Expected Points by Offensive Field Position and Down",
subtitle = stringr::str_wrap("Expected points using probabilities from classification model trained on historical play by play data. Displaying expected points using observed values approach from a random sample of plays", 90)
)
With the model in hand, I can predict the probability of the next scoring event for every play in order to compute the expected points within each game situation. The difference in expected points from play to play is the currency by which we can evaluate players/teams/offenses/defenses.
Going back to the game between Texas A&M and Florida in 2012 as an example (chosen entirely at random and not at all because it was the final home game of my last year at A&M), we can examine the first drive in terms of expected points.
sample_plays <-
pbp_fit |>
augment(
prepared_pbp |>
inner_join(sample_games) |>
filter(drive_number == 1)
) |>
calculate_expected_points() |>
calculate_points_added()
plays_points_tbl <- function(plays) {
plays |>
rename(
ep_pre = expected_points_pre,
ep_post = expected_points_post,
ep_added = predicted_points_added,
) |>
select(season, game_id, offense, defense, period, yards_to_goal, down, distance, play_text, ep_pre, ep_post, ep_added) |>
mutate_if(is.numeric, round, 3) |>
gt_tbl() |>
gt::cols_align(
columns = c("period", "down", "distance", "yards_to_goal", "ep_pre", "ep_post", "ep_added"),
align = "center"
) |>
gt::data_color(
columns = c(
"ep_pre",
"ep_post",
"ep_added"
),
method = "numeric",
domain = c(-10, 10),
palette = c("orange", "white", "dodgerblue"),
na_color = "white"
) |>
gt::tab_options(
data_row.padding = px(20),
table.font.size = 12
) |>
gt::cols_label(
yards_to_goal = "ytg"
)
}
sample_plays |>
group_by(season, game_id, period) |>
plays_points_tbl()| offense | defense | ytg | down | distance | play_text | ep_pre | ep_post | ep_added |
|---|---|---|---|---|---|---|---|---|
| 2012 - 322520245 - 1 | ||||||||
| Texas A&M | Florida | 75 | 1 | 10 | TEXAS A&M penalty 5 yard False Start on Patrick Lewis accepted. | 0.622 | -0.209 | -0.831 |
| Texas A&M | Florida | 80 | 1 | 15 | Christine Michael rush for no gain to the TexAM 20. | -0.209 | -1.144 | -0.935 |
| Texas A&M | Florida | 80 | 2 | 15 | Johnny Manziel pass complete to Ryan Swope for a loss of 2 yards to the TexAM 18. | -1.144 | -2.401 | -1.257 |
| Texas A&M | Florida | 82 | 3 | 17 | Johnny Manziel rush for 16 yards to the 50 yard line, FLORIDA penalty 16 yard Personal Foul accepted for a 1ST down. | -2.401 | 2.615 | 5.016 |
| Texas A&M | Florida | 50 | 1 | 10 | Johnny Manziel pass complete to Kenric McNeal for 8 yards to the Fla 42. | 2.615 | 3.434 | 0.820 |
| Texas A&M | Florida | 42 | 2 | 2 | Christine Michael rush for 2 yards to the Fla 40 for a 1ST down. | 3.434 | 3.265 | -0.169 |
| Texas A&M | Florida | 40 | 1 | 10 | Johnny Manziel pass complete to Kenric McNeal for 3 yards to the Fla 37. | 3.265 | 2.973 | -0.292 |
| Texas A&M | Florida | 37 | 2 | 7 | Christine Michael rush for 6 yards to the Fla 31. | 2.973 | 3.462 | 0.489 |
| Texas A&M | Florida | 31 | 3 | 1 | Christine Michael rush for no gain to the Fla 31. | 3.462 | 1.436 | -2.026 |
| Texas A&M | Florida | 31 | 4 | 1 | Johnny Manziel pass complete to Nehemiah Hicks for 8 yards to the Fla 23 for a 1ST down. | 1.436 | 4.183 | 2.747 |
| Texas A&M | Florida | 23 | 1 | 10 | Johnny Manziel pass complete to Ryan Swope for 2 yards to the Fla 21. | 4.183 | 3.796 | -0.387 |
| Texas A&M | Florida | 21 | 2 | 8 | Johnny Manziel pass complete to Christine Michael for 14 yards to the Fla 7 for a 1ST down. | 3.796 | 4.973 | 1.177 |
| Texas A&M | Florida | 7 | 1 | 7 | Johnny Manziel pass incomplete to Kenric McNeal, broken up by Matt Elam. | 4.973 | 4.382 | -0.591 |
| Texas A&M | Florida | 7 | 2 | 7 | Johnny Manziel pass incomplete to Mike Evans, broken up by Marcus Roberson. | 4.382 | 3.526 | -0.855 |
| Texas A&M | Florida | 7 | 3 | 7 | Johnny Manziel rush for 3 yards to the Fla 9, TEXAS A&M penalty 5 yard Illegal Forward Pass on Johnny Manziel accepted. | 3.526 | 2.283 | -1.244 |
| Texas A&M | Florida | 9 | 4 | 9 | Taylor Bertolet 27 yard field goal GOOD. | 2.283 | — | 0.717 |
In this case, we can see how the results of plays added or lost points in expectation.
The play with the most points added was Manziel’s rush for 16 yards on 3 and 17 combined with a personal foul on Florida for an additional 15 yards. A long third down inside your own territory is a negative points situation for an offense; teams in this situation usually have to punt, which leads to the other teams being more likely to score with favorable field position. Converting on a long third down and moving to midfield results in a big change in the expected points you would get from the drive.
An example of a negative points play is the previous play, with Manziel’s pass to Swope for 2 yards on 2nd and 15. Even though this play gained yardage, the end result left them with a 3rd and long in their own territory.
The result of this drive was a field goal, where the expected points added (~0.75) equals the actual points (3) minus the expected points from the situation (~2.25).
However the metric Expected Points Added typically doesn’t include scoring plays. In analyzing an offense, I’ve seen others working with these types of model make a theoretical distinction between plays that add points vs plays that only result in a shift in the expected points of situation.
Predicted Points Added uses both scoring plays and non scoring plays, taking into account both expected points and actual points added as a result of the play.
Once we’ve scored plays, we can examine individual games to examine which plays had the biggest impact in terms of expected points. Here I’ll examine a few games, chosen completely at random and in no way influenced by my fandom.
Kind of interesting - this game is remembered for a lot of plays by Johnny Manziel, but the most impactful plays in the game in terms of expected points changes were actually turnovers forced by the A&M defense.
pbp_fit |>
augment(
prepared_pbp |>
filter(game_id == 323150333)
) |>
calculate_expected_points() |>
calculate_points_added() |>
top_plays_by_game() |>
plays_points_tbl()| offense | defense | period | ytg | down | distance | play_text | ep_pre | ep_post | ep_added |
|---|---|---|---|---|---|---|---|---|---|
| 2012 - 323150333 | |||||||||
| Alabama | Texas A&M | 4 | 54 | 3 | 6 | AJ McCarron pass complete to Amari Cooper for 54 yards for a TOUCHDOWN. | 0.929 | — | 6.071 |
| Alabama | Texas A&M | 4 | 38 | 1 | 10 | T.J. Yeldon rush for 8 yards to the TexAM 30, fumbled, forced by Steven Terrell, recovered by TexAM Dustin Harris, Dustin Harris for 4 yards to the TexAM 34. | 3.472 | -1.462 | -4.935 |
| Alabama | Texas A&M | 1 | 52 | 3 | 5 | AJ McCarron pass intercepted by Sean Porter at the TexAM 43, returned for 16 yards to the Alab 41. | 1.242 | -3.365 | -4.606 |
| Texas A&M | Alabama | 4 | 68 | 3 | 8 | Johnny Manziel pass complete to Ryan Swope for 28 yards to the Alab 25, ALABAMA penalty 15 yard Personal Foul on Ha'Sean Clinton-Dix accepted for a 1ST down. | -0.157 | 4.393 | 4.550 |
| Alabama | Texas A&M | 4 | 88 | 1 | 18 | AJ McCarron pass complete to Amari Cooper for 50 yards to the TexAM 38 for a 1ST down. | -0.660 | 3.472 | 4.133 |
| Texas A&M | Alabama | 1 | 10 | 3 | 10 | Johnny Manziel pass complete to Ryan Swope for 10 yards for a TOUCHDOWN. | 3.296 | — | 3.704 |
| Texas A&M | Alabama | 1 | 59 | 3 | 6 | Johnny Manziel rush for 32 yards to the Alab 27 for a 1ST down. | 0.718 | 4.288 | 3.570 |
| Alabama | Texas A&M | 2 | 2 | 3 | 2 | Eddie Lacy rush for 2 yards for a TOUCHDOWN. | 3.714 | — | 3.286 |
| Alabama | Texas A&M | 4 | 60 | 1 | 10 | AJ McCarron pass complete to Kenny Bell for 54 yards to the TexAM 6 for a 1ST down. | 1.178 | 4.437 | 3.259 |
| Texas A&M | Alabama | 4 | 24 | 1 | 10 | Johnny Manziel pass complete to Malcome Kennedy for 24 yards for a TOUCHDOWN. | 4.172 | — | 2.828 |
Another game chosen completely at random.
pbp_fit |>
augment(
prepared_pbp |>
filter(game_id == 401282103)
) |>
calculate_expected_points() |>
calculate_points_added() |>
top_plays_by_game(var = predicted_points_added, n =10) |>
plays_points_tbl()| offense | defense | period | ytg | down | distance | play_text | ep_pre | ep_post | ep_added |
|---|---|---|---|---|---|---|---|---|---|
| 2021 - 401282103 | |||||||||
| Alabama | Texas A&M | 1 | 56 | 3 | 1 | Brian Robinson Jr. run for 1 yd to the Alab 45 Brian Robinson Jr. fumbled, recovered by TexAM , return for 4 yds to the Alab 41 | 2.240 | -3.582 | -5.821 |
| Texas A&M | Alabama | 3 | 81 | 4 | 4 | Nik Constantinou punt blocked by Ja'Corey Brooks blocked by Ja'Corey Brooks for a TD, (Will Reichard KICK) | -2.385 | — | -4.615 |
| Texas A&M | Alabama | 4 | 25 | 1 | 10 | Zach Calzada pass complete to Ainias Smith for 25 yds for a TD, (Seth Small KICK) | 2.839 | — | 4.161 |
| Texas A&M | Alabama | 2 | 74 | 1 | 10 | Zach Calzada pass intercepted DeMarcco Hellams return for no gain to the TexAM 40 | 0.764 | -3.396 | -4.160 |
| Alabama | Texas A&M | 2 | 1 | 3 | 1 | Bryce Young pass intercepted Demani Richardson return for 3 yds to the TexAM 3 | 5.502 | 1.451 | -4.051 |
| Alabama | Texas A&M | 3 | 29 | 2 | 7 | Bryce Young pass complete to Jameson Williams for 29 yds for a TD, (Will Reichard KICK) | 3.775 | — | 3.225 |
| Texas A&M | Alabama | 2 | 77 | 2 | 12 | Zach Calzada pass complete to Devon Achane for 33 yds to the Alab 44 for a 1ST down | -0.270 | 2.836 | 3.106 |
| Alabama | Texas A&M | 3 | 75 | 2 | 10 | Brian Robinson Jr. run for 24 yds to the Alab 49 for a 1ST down | -0.152 | 2.844 | 2.996 |
| Texas A&M | Alabama | 2 | 15 | 1 | 10 | Isaiah Spiller run for 15 yds for a TD, (Seth Small KICK) | 4.050 | — | 2.950 |
| Alabama | Texas A&M | 3 | 51 | 3 | 9 | Bryce Young pass complete to Brian Robinson Jr. for 16 yds to the TexAM 35 for a 1ST down | 0.711 | 3.639 | 2.927 |
pbp_fit |>
augment(
prepared_pbp |>
filter(game_id == 333200002)
) |>
calculate_expected_points() |>
calculate_points_added() |>
top_plays_by_game(var = predicted_points_added, n = 10) |>
plays_points_tbl()| offense | defense | period | ytg | down | distance | play_text | ep_pre | ep_post | ep_added |
|---|---|---|---|---|---|---|---|---|---|
| 2013 - 333200002 | |||||||||
| Auburn | Georgia | 4 | 73 | 4 | 18 | Nick Marshall pass complete to Ricardo Louis for 73 yards for a TOUCHDOWN. | -0.301 | — | 7.301 |
| Georgia | Auburn | 4 | 5 | 4 | 5 | Aaron Murray rush for 5 yards for a TOUCHDOWN. | 2.087 | — | 4.913 |
| Auburn | Georgia | 1 | 68 | 4 | 4 | GEORGIA penalty 15 yard Personal Foul on John Taylor accepted. | -1.318 | 2.854 | 4.172 |
| Auburn | Georgia | 2 | 24 | 1 | 10 | Tre Mason rush for 24 yards for a TOUCHDOWN. | 3.357 | — | 3.643 |
| Auburn | Georgia | 3 | 51 | 3 | 4 | Nick Marshall pass complete to Ricardo Louis for 44 yards to the Geo 7 for a 1ST down. | 1.535 | 5.092 | 3.558 |
| Georgia | Auburn | 4 | 24 | 1 | 10 | Aaron Murray pass complete to Arthur Lynch for 24 yards for a TOUCHDOWN. | 3.690 | — | 3.310 |
| Georgia | Auburn | 2 | 75 | 1 | 10 | Aaron Murray pass complete to Michael Bennett for 42 yards to the Aub 33 for a 1ST down. | 0.683 | 3.973 | 3.290 |
| Georgia | Auburn | 3 | 83 | 1 | 10 | Aaron Murray pass complete to Rantavious Wooten for 40 yards to the Aub 43 for a 1ST down. | 0.189 | 3.467 | 3.278 |
| Georgia | Auburn | 3 | 16 | 2 | 10 | Aaron Murray rush for 16 yards for a TOUCHDOWN. | 3.930 | — | 3.070 |
| Auburn | Georgia | 1 | 21 | 2 | 10 | Corey Grant rush for 21 yards for a TOUCHDOWN. | 3.946 | — | 3.054 |
pbp_fit |>
augment(
prepared_pbp |>
filter(game_id == 400763542)
) |>
calculate_expected_points() |>
calculate_points_added() |>
top_plays_by_game(var = predicted_points_added, n = 10) |>
plays_points_tbl()| offense | defense | period | ytg | down | distance | play_text | ep_pre | ep_post | ep_added |
|---|---|---|---|---|---|---|---|---|---|
| 2015 - 400763542 | |||||||||
| Michigan | Michigan State | 4 | 47 | 4 | 2 | Blake O'Neill run for a loss of 15 yards Blake O'Neill fumbled, recovered by MchSt Jalen Watts-Jackson , return for 38 yds | 0.273 | — | -7.273 |
| Michigan State | Michigan | 4 | 75 | 1 | 10 | Connor Cook pass complete to Trevon Pendleton for 74 yds to the Mich 1 for a 1ST down | 0.776 | 6.437 | 5.660 |
| Michigan | Michigan State | 1 | 82 | 4 | 19 | Blake O'Neill punt for 80 yds, downed at the MchSt 2 | -3.311 | 1.266 | 4.577 |
| Michigan State | Michigan | 1 | 77 | 3 | 12 | Connor Cook pass complete to Aaron Burbridge for 27 yds to the 50 yard line for a 1ST down | -1.769 | 2.599 | 4.368 |
| Michigan | Michigan State | 3 | 64 | 3 | 9 | Jake Rudock pass complete to Amara Darboh for 32 yds to the MchSt 32 for a 1ST down | -0.094 | 4.056 | 4.150 |
| Michigan | Michigan State | 4 | 69 | 3 | 11 | Jake Rudock pass complete to Amara Darboh for 24 yds to the MchSt 45 for a 1ST down | -0.611 | 3.379 | 3.989 |
| Michigan State | Michigan | 3 | 30 | 2 | 8 | Connor Cook pass complete to Macgarrett Kings Jr. for 30 yds for a TD, (Michael Geiger KICK) | 3.583 | — | 3.417 |
| Michigan State | Michigan | 1 | 74 | 3 | 10 | Connor Cook pass complete to Aaron Burbridge for 13 yds to the MchSt 39 for a 1ST down | -1.177 | 2.066 | 3.244 |
| Michigan State | Michigan | 1 | 51 | 3 | 18 | MICHIGAN Penalty, Defensive Holding (Jabrill Peppers) to the Mich 41 for a 1ST down | 0.563 | 3.607 | 3.044 |
| Michigan State | Michigan | 1 | 28 | 4 | 8 | Connor Cook pass incomplete to Aaron Burbridge | 1.615 | -1.254 | -2.869 |