British biologist Carl Bergstrom recently gave an interview to the Guardian on the topic of “bullshit”. In it, the interviewer asked Bergstrom about shortcomings of existing epidemiological models as well as their use (and misuse) by political decision makers.
[Guardian] If you had the ability to arm every person with one tool – a statistical tool or scientific concept – to help them understand and contextualize scientific information as we look to the future of this pandemic, what would it be?
[Bergstrom] I would like people to understand that there are interactions between the models we make, the science we do and the way that we behave. The models that we make influence the decisions that we take individually and as a society, which then feed back into the models and the models often don’t treat that part explicitly. Once you put a model out there that then creates changes in behavior that pull you out of the domain that the model was trying to model in the first place. We have to be very attuned to that as we try to use the models for guiding policy.
In the context of the coronavirus, the problem was this: Early models such as the one by the Imperial College in London predicted between 1.1 and 2.2 millions of Americans could die from COVID-19, depending on the severity of mitigation efforts. This eye-popping number jolted the political decision makers (Trump, Congress, the Governors, etc.) into action, locking down schools and businesses and issuing stay-home orders. The media publicity around the study probably scared many people which made them take the social distancing measures much more seriously. All of this probably helped in slowing the spread of the disease such that the same researchers had to revise their predictions downward only weeks later.
That is, the publication of the initial predictions changed the behavior of people which rendered those predictions obsolete.
Bergstrom seems to say that the problem here is with the general public. They don’t understand that the models rely on behavioral assumptions which no longer hold once people learn about the models’ predictions and adjust their actions accordingly.
But, with apologies to Shakespeare: The fault, dear Bergstrom, is not in the general public, but in your models!
The problem with those epidemiological models (at least with the SIR-types of models) is that some of their key parameters (such as the reproduction rate R0, for instance) depend, in various ways, on people’s expectations about the future path of the disease. If you don’t take that into account, your predictions will be way off.
And way off they were! Here’s the summary of a statistical evaluation of a model similar to the one used in the Imperial study:
In excess of 70% of US states had actual death rates falling outside the 95% prediction interval for that state (Figure 1)
The ability of the model to make accurate predictions decreases with increasing amount of data. (figure 2)
You might say that prediction is not the point with those models. Maybe their only purpose is to produce scary headlines to make people listen to the experts. But that is a weird proposition. If experts want the general public to take them more seriously, making wildly erroneous predictions seems like a bad strategy.
So how are we going to take people’s expectations into account in epidemiological models? Let’s see.
March: Imperial predicts 2 million deaths. Government imposes lockdown. People are scared and stay at home.
April: Imperial revises his model, now predicts 50,000 deaths. Government partially re-opens the economy. People cautiously start going out again.
May: Imperial revises his model, now predicts 200,000 deaths. Government re-imposes some lockdown measures. People are scared again.
June: Imperial revises his model, now predicts 75,000 deaths. Government opens up again. People relax again.
And so on until we have converged to a situation in which the number of deaths Imperial predicts is consistent with the government’s (and the people’s) expectations and actions.
Such a situation is what economists call a rational expectations equilibrium. I think that trying to model people’s expectations in a consistent way would improve the usefulness of epidemiological models. This is, of course, a tall order. But perhaps if economists, statisticians, and epidemiologists would put their heads together, we could move in this direction.