Results from ongoing trials are closely guarded so that researchers (and investors) have no opportunity to mess with them. Under strict trial rules that vaccine makers and the FDA set up in advance, there are only a few predetermined times when a data monitoring board is scheduled to look at the data. The purpose of those check-ins (known as “interim analyses”) is to see if there’s already enough evidence to conclude that the vaccine either works or doesn’t.
According to the trial rules that Pfizer released two weeks ago, it has four of these check-ins before the final analysis. The first one occurs when 32 people in the trial get sick with COVID-19.
Yes, that is a tiny number in a study designed to enroll 30,000 participants. But it could be enough to show that the vaccine works if far more of the infections occur in people who took the placebo than in those who got the vaccine, so much so that it’s probably not random.
The vaccine “would have to be way, way better to meet an interim stopping boundary,” said Frank Harrell Jr., professor of biostatistics at Vanderbilt University.
If six or fewer of the first 32 cases are people who got the vaccine, that suggests the vaccine reduced COVID-19 cases by 76%. Under Pfizer’s trial rules, the company can then conclude that its vaccine is effective enough to submit its application to the FDA. If, on the other hand, 15 or more of the first 32 infections are people who got the vaccine, it would mean the vaccine doesn’t work and the study ends.
The better the vaccine works, the longer it would take to reach 32 cases, because fewer vaccinated people would get sick and more of the infections would have to occur in the placebo group.
So when exactly will Pfizer get to 32 cases?
Analysts at JPMorgan estimate that this first readout would occur on Oct. 31 if the vaccine is 70% effective, or on Nov. 2 if the vaccine is 80%. Their model also estimates that Pfizer is more than twice as likely to be able to file for approval at 80% effectiveness vs 70% effectiveness.
The JPMorgan analysts’ prediction is roughly in line with official public statements from Pfizer, though the company has projected even more confidence. “We have a good chance that we will know if the product works by the end of October,” CEO Albert Bourla said Sept. 13 on CBS’ “Face the Nation.”
The key thing we don’t know is how fast people in Pfizer’s trial are getting sick. The trial doesn’t intentionally expose people to the virus; it waits to see who catches it on their own. Even though the pandemic is far from under control, the coronavirus is not spreading as fast as it was a few months ago. The study started with a baseline assumption that just 1.3% of participants would be infected over the course of a year.
It’s possible that the thousands of people who signed up for Pfizer’s study could be getting sick faster or slower than that, depending on how bad the outbreak is where they are. If Pfizer’s trial infection rate is 1.5 to 2 times faster than publicly reported case counts, then Pfizer’s first readout could be Oct. 12-19 with a 70% effective vaccine, or Oct. 14-22 with 80% efficacy, according to JPMorgan’s model.
“When the trials started there were a lot more cases in the U.S. at that time,” said Dr. Vamil Divan, an analyst at the bank Mizuho. “That’s a key unknown. How many of these participants are enrolled in hotter areas versus New York or Boston?”
To protect the integrity of trials like this, the drugmakers running them aren’t supposed to know what the data is showing as it comes in. And yet, company executives have led some observers to believe they know how many people have tested positive for COVID-19.
“They’re making projections based on how rapidly they’re accruing data, and they probably know the total number of events,” said Susan Ellenberg, professor of biostatistics at the University of Pennsylvania.
“When they say things like that, first of all, it does some wonders for their stock prices,” she added.