When representatives from the drug company Pfizer say that they could know as soon as the end of October if their Covid-19 vaccine works, here’s what they mean: If their trial, involving perhaps as many as 44,000 people, pops just 32 of them with mild Covid-19 symptoms and a positive test—and if 26 of those people got a placebo instead of the vaccine—that, potentially, is it. According to the guidelines laid out by the Food and Drug Administration, that would be an “effective” vaccine: 50 percent efficacy with a statistical “confidence interval” that puts brackets around a range from 30 percent to 70 percent. At that point, per Pfizer’s protocol, the company could stop the trial. Technically, that vaccine would be successful.
Now to be fair, nobody, least of all those selfsame Pfizer representatives, is explicitly claiming that will happen—or that if it does, Pfizer would take those numbers to the FDA and ask to start giving people shots. “The protocol only specifies that the study would stop in the case of futility, and does not outline a binding obligation to stop the study if efficacy is declared,” a Pfizer spokesperson told me by email. Translation: They have wiggle room to keep going. On the other hand, they could ask for an emergency use authorization, which the FDA and President Donald Trump seem to be angling for—and which could, for various ethical and practical reasons, then become a roadblock in front of all the other trials in progress. It’s hard to tell!
Which is a problem. Now that several pharmaceutical companies have released detailed plans for how they’re testing their Covid-19 vaccine candidates, researchers are asking questions about these protocols. Even if anyone can reliably say whether a particular vaccine works—for various definitions of “works”—it’s less clear that the trials will be able to tell which one works better, and for whom. No one is yet testing vaccines head-to-head. The goal here hasn’t changed: To get one or more vaccines that protect lots of different kinds of people against Covid-19. At issue is how the many candidate vaccine trials are designed, what the trials will actually show, and how the vaccines compare to each other.
Big vaccine trials all depend in part on defining “end points,” the signs of infection or illness that the researchers say they’re going to count. Basically, the setup is: You give tens of thousands of people the vaccine and a few thousand other people a placebo, and you see who gets to those predetermined end points. If more people who got the placebo do—by a mathematically predetermined proportion—you got yourself a vaccine.
The tricky bit is, what really constitutes an end point? Obviously a big one is “infection with the virus SARS-CoV-2.” But after that, reasonable minds could disagree. You could also choose “correlates of immunity,” like antibodies found in a blood test. Or you could use symptoms, as these trials do. That’s common practice. But does it matter if someone gets a little sick, with mild illness like a cough or muscle aches, versus a lot sick, with severe illness that requires a ventilator or an intensive care unit? Pfizer and the other companies with trials underway are using mild symptoms and a positive Covid-19 test as their primary end points, and severe illness as a secondary end point, something for later statistical analysis.
But incidence of mild cases might not be the most useful thing to count. If you’re looking for vaccines meant to eventually reach billions of people, maybe you actually want to first ensure they beat back the most severe symptoms, not the mild ones. “What you’d like, in this very small number of events, going to the planetary population, is to have the most confidence you possibly can. That would be suppressing the worst events, sickness that requires a hospitalization and anything worse than that,” says Eric Topol, a professor of molecular medicine at the Scripps Research Institute who has been watchdogging the trial protocols. Mild, coldlike symptoms, he says, “are not very good signals of efficacy. And my understanding is there was tremendous internal debate about that when these protocols were being discussed, but I think they made a bad decision.”
What’s the problem? Nominally, the trials are looking for what scientists call vaccine efficacy, or VE—calculated as “1 minus the relative risk of infection of someone who gets vaccinated versus someone who does not.” The end point is the difference between those groups. But the reason that trials need tens of thousands of people is that they’re trying to pick up subtle differences in the effectiveness of the medicine. Also, different people—kids versus the elderly, say—get sick differently. So the potency of the vaccine might matter too. If a vaccine isn’t a grand-slam success—and few are—tiny variations in how sick people get and how infectious they are to others will make a real difference across hundreds of millions of people. Technically the FDA has requested a vaccine with 50 percent efficacy, with a margin ranging from as low as 30 percent to as high as 70 percent. “Anything 30 to 50 percent is pretty darn low efficacy, but that’s just one measure,” Topol says. “Another level of efficacy is: What type of events is it suppressing? Are you suppressing important infections? That’s a measure of potency. We don’t know that blocking only mild ones is the same here. We just don’t know that.”
Even weirder, the trials all use slightly different definitions for those mild symptoms, which means they’re tracking different end points. The vaccines from Pfizer and Moderna—both based on similar technologies that deliver the genetic material mRNA packaged in a sort of bubble of fat, both requiring two doses—will have subtly different criteria for success. So, for example, Pfizer defines its primary end point as a positive Covid-19 test and any one of these: fever, cough, shortness of breath, chills, muscle aches, loss of sense of taste or smell, sore throat, diarrhea, or vomiting. It also has a second definition it’s working with based on CDC definitions that includes fatigue, headache, stuffy or runny nose, and nausea. Moderna says the primary end point, in addition to a positive test, is cough, shortness of breath, pneumonia, and any two of these—fever, chills, muscle aches, headache, sore throat, and loss of the sense of smell or taste. But the Janssen/Johnson & Johnson protocols invert all that, using severe symptoms like respiratory failure and hospitalization as primary end points, and mild ones as secondary—with lots of follow-up evaluations and frequent testing for the virus itself.
Now, maybe that’s not a huge problem. “They have approached the problem similarly with respect to the primary end point of disease of any severity,” says Natalie Dean, a biostatistician at the University of Florida and vaccine trial design expert. “The FDA would have accepted an infection end point and probably a severe-disease-only end point, but they’ve all landed at disease of any severity. There’s some details that vary on the particular definition of disease, but it’s not like a striking difference.”
More striking, perhaps, is when the companies decide to count up those end points. Vaccine trials define in advance whether and under what conditions they’ll conduct “interim analyses,” when a data monitoring team—independent of the researchers—looks at the numbers. If, after running the stats, the data is either terrible or amazing, the data committee can call a halt to the trial. That’s called “futility”—it means there’s no point in continuing. But so does amazing, or “efficacy.” It works, goes the logic, so we can stop. But stopping early has a cost, both in terms of having rock-solid numbers to prove you’re right, and in an inability to see possible safety implications or side effects that would’ve only shown up later.
Moderna and the other companies have only built in one or two interim analyses; Pfizer has four, and the company’s CEO has said they might have the data from the first one as soon as late October. That would be the one that could show efficacy with just a couple dozen “events,” if they’ve happened: 26 mildly symptomatic infections in the placebo group out of 32 in total. That’s it. “What if those 26 events are headaches and sore throats? Would that give you great confidence we have an effective vaccine? It sure wouldn’t give me confidence,” Topol says.
It’d be statistically significant. But that’s not the only criterion to look at in a trial. “They could declare overwhelming efficacy at that time,” Dean says. “The question then is: If the trial declares efficacy after an early analysis, then do they stop the trial at that point?”
That would have tremendous significance. That Pfizer spokesperson told me they might not. But it seems like they could. It wouldn’t even violate the multi-company pledge Pfizer and others signed to produce a safe and effective vaccine via ethical trials without political influence. (Senior FDA officials pledged much the same in an opinion piece in USA Today in mid September, though President Trump has threatened to ignore tougher standards for approval.) Technically, even those seemingly small event numbers would show overwhelming efficacy. Pfizer could go straight from there to a request for an emergency use authorization. If that goes through, they could start giving people shots. But the trial will have stopped so early that it might miss potential side effects or variability in how different demographic groups respond to the vaccine. “They can technically meet that definition the FDA has set: 50 percent efficacy and a 30 percent lower bound,” Dean says. “But the big risk I see in having relatively few events is that you could have very little information about other secondary end points or other things that are going to be important for decisionmaking.”
And worst of all, nobody knows what effect an EUA will have on the other ongoing trials. In the strictest possible definition of medical ethics, having a vaccine that works means other labs and companies with other formulas might stop testing theirs, because technically those trials would pose unnecessary risks to the subjects. That probably isn’t true here—more than one Covid-19 vaccine will almost certainly be necessary, if for no other reason than the logistics of distributing hundreds of millions of doses. But still: What happens to the other trials? “My guess is that if somebody gets an EUA, nobody is stopping their trials. It doesn’t become unethical to continue,” says Elizabeth Halloran, a vaccine trial design expert at the Fred Hutchinson Cancer Research Center and another vaccine trial design expert. “But if at a point somebody has a licensed vaccine that went through Phase III and gets a licensure, then you do have an ethical issue.”
An FDA spokesperson said that the agency plans to put out more guidance about what pharmaceutical companies will need to get an EUA, and potentially how it would affect other ongoing trials.
Meanwhile, though, all of this could have perhaps been avoided if the federal government hadn’t authorized multiple simultaneous trials of multiple vaccines. The government jump-started these trials with billions of taxpayer dollars. Yet it made few demands on the design of the trials—and manifestly failed to set up a head-to-head, multiarm adaptive trial with a shared placebo group, which could’ve assessed not only safety and efficacy but also “non-inferiority,” which is to say, which one’s better. Since each trial defines slightly different end points, it’s that much more difficult (though, to be sure, not impossible) to compare the trials to each other. “It’s reasonable to argue that you would care about picking a vaccine that’s 2 percent more effective. Two percent is thousands of lives, so all things being equal, you’d rather have that one,” says Peter Bach, director of the Center for Health Policy and Outcomes and the Drug Pricing Lab at Memorial Sloan Kettering Cancer Center. “These things use very different platforms, very different technologies; in some sense the biological hypotheses underlying them are different. There’s a reasonable chance that they won’t all work the same.”
In Europe, the World Health Organization is organizing just that kind of trial—the Solidarity trial, aiming to test multiple vaccines against Covid-19 and against each other. But it’s fair to note that while several companies are in Phase III trials of their vaccines, the Solidarity trial hasn’t even fully gotten off the ground yet. “By sharing a placebo arm, you can have fewer participants overall or fewer people receiving placebo, or you can facilitate a direct comparison,” Dean says. “But the other side of that argument is that this has gone really fast [in the US], and I do think we need to be realistic about the added time it takes to get all these companies on the same page.”
That’s a true thing, but it also sucks. “It’s worth it to do these things. And the companies don’t want us to do it. They’d much prefer being oligopolists than to compete,” Bach says. Head-to-head tests would let the market compare their products, and the companies would have no way of spinning the results. (He pitched the idea in an op-ed in Stat.) “They don’t want binary events that would cause their market to evaporate,” Bach says. “Here we have a situation where we have financed a lot of the development, there’s a lot of government IP, we’ve given advance marketing commitments—which are guarantees of revenue—and we’re paying for the distribution. We’ve run the table on reasons why the government should have an interest in managing and guiding the science.”
Yet that only happened in one case—the government-run trial of the antiviral drug remdesivir. It didn’t happen with any other therapeutics, and hasn’t happened with vaccines. Instead, the regulatory agencies let the pharmaceutical companies define the terms of their own trials. “It makes me bonkers that we think we should let the companies decide on the study designs, because their incentives are off,” Bach says. “When we know definitively that X or Y are not what we want, and we want something slightly different, that’s where the government is supposed to step in and modify the market’s behavior.”
To be clear, these are scientifically rigorous trials taken on their own terms; the people who work for these companies are good at their jobs, and nobody wants to see a world without a vaccine, or with one that doesn’t stop the disease or has untenable side effects. Those are two very real fail modes. But there’s a third one that isn’t getting addressed. It seems like one so-so vaccine could actually preclude a better one. Or maybe worse: Multiple vaccines that are protective against Covid-19 could get approved, but people might get one or the other almost capriciously. It could depend on factors as prosaic as which one their local pharmacy happened to get access to, or whatever was available at the terrifying dystopian distribution center, rather than which one was better, or perhaps better suited to a person like them. And the people who need the shot (all of us) might not even have the right data to know the difference. The vaccines will be a success—and a failure.