The conversation around COVID-19 has put vaccine development under the public’s microscope. This discussion has left many people wondering how a vaccine is tested and what is different, if anything, about the way COVID-19 vaccines are being developed.
In this Q&A, a group of biostatisticians from Wake Forest School of Medicine answer some core questions about vaccine trials. There are a lot of steps involved in clinical research, and an important part of the process is how decisions are made about what data is needed and how it’s collected, analyzed, and then translated into medical recommendations.
Whether leading research or validating that of others, biostatisticians play a crucial role in the research process by taking responsibility for the collection, analysis and interpretation of data. Whenever research is pushed into clinical care, it is backed by experts who ensure the research is sound and that the health care recommendations that emerge from it make sense for the intended population of people. This is the role of a biostatistician.
As the world receives the first COVID-19 vaccines, some of our leading faculty from the Department of Biostatistics and Data Science within our Public Health Sciences division, describe how clinical trials, particularly for vaccines, are moved from a research question into data that drives medical practice.
What is the first step in any clinical research study and how does that apply to COVID-19 vaccine trials?
The first step in any research study is the development of a research question. This question narrows the scope of work, drilling down to the who and what, and will inform where and how the research is conducted.
In the case of COVID-19 vaccines, the question is rather simple: Does the vaccine protect people from COVID-19 infections? However, once there is a path determined for testing the research question, the focus is next on safety and efficacy.
What does it mean to test for safety and efficacy?
For a clinical trial testing the effectiveness of a COVID-19 vaccine, the first research question will involve safety.
- What side effects do patients experience after receiving the vaccine and how often do these occur?
- Do the side effects of the vaccine outweigh the positive potential for preventing the spread of COVID-19?
Because vaccines are often given to millions, or hundreds of millions, of healthy people, safety is paramount. For example, temporary side effects like headache, fever or redness at the injection site are not considered serious if they resolve without medical intervention. So even if such side effects are common, which may be true for some of the COVID-19 vaccines, they are not likely to keep a vaccine from being approved. However, any side effects that are serious, even if they are rare, could prevent a vaccine from being deemed safe.
After safety has been established, efficacy is assessed, which is determining if the new treatment, vaccine or intervention being introduced has the expected outcomes. In the case of COVID-19, answering this question means determining if the vaccine consistently works to prevent the spread of COVID-19.
- Does the vaccine lower the number of COVID-19 infections in a specific group of individuals?
The World Health Organization recommends that vaccines should result in at least a 50% reduction of COVID-19 cases in order to be determined efficacious. This means a COVID-19 vaccine is efficacious if it drastically lowers the spread of COVID-19 relative to a similar group without the vaccine. For instance, in a clinical trial with an equal number of participants receiving vaccine and placebo (see description below), we would expect to see half or fewer of the COVID-19 cases in the vaccine group compared to the placebo group.
In addition to safety and efficacy, how the study is designed is important to ensure that the research question can be adequately answered and that the conclusions will accurately reflect the group of people who will receive the treatment.
How are vaccine trials typically designed?
Vaccine trials are typically tested through a study design called a randomized, double-blind and placebo-controlled trial. This means that participants are randomly assigned to receive either the experimental vaccine or a placebo. A placebo is an inactive injection that mimics the real experience to the patient. In the case of the vaccine, the placebo would be an injected salt water solution or something similar. The person receiving the treatment and the care provider giving the treatment do not know which group the participant belongs—whether the placebo or the treatment group.
This type of trial helps minimize bias and makes the results more accurate because no one can alter results, intentionally or unintentionally, based on what they believe will work best. Data need to be collected on people with and without the vaccine so that the groups can be compared to determine if the vaccine is better than receiving nothing (the placebo).
Additionally, details about the specific number of study participants, called a sample size, are very important when later analyzing the results. The sample size needs to be large enough to sufficiently answer the research question, and whether it is or is not will be a factor in the conclusions of the study.
How do we ensure the analysis and conclusions of the study team are accurate?
As data are collected from the participants, a team of scientists/statisticians ensure the appropriate data were collected and then conduct the subsequent analysis of the data. There are various questions this team will ask.
- Did the researchers provide enough evidence to justify the study design and analysis?
- Did researchers provide details on which statistical methods were used to analyze the data and were these methods appropriate?
Results of a study need to be interpreted correctly within the context of the research question.
- Are conclusions reasonable and supported by the data?
- Are the results clinically meaningful? In other words, will these results actually make a difference in the real world or is the effect we are seeing not likely to matter?
Concluding that a COVID-19 vaccine is effective needs to be supported by observed results. The researchers need to look at how many participants who were vaccinated later contracted COVID-19 versus how many participants who received the placebo later contracted COVID-19. If fewer people contracted COVID-19 who were vaccinated compared to those in the placebo group, the researchers must determine if the reduction in cases is enough to warrant widespread recommendations to use the vaccine.
After conclusions are made based on the study data, are the results generalizable to other people outside of the study?
This question circles back to the study design and understanding the group of people who participated in the study. If the participants are representative of a group of people, then the results should apply to that specific group of people generally. Ethical researchers are careful to not use the results to make recommendations for groups that do not reflect the participants included in the study.
For example, if the COVID-19 vaccine trial only involves individuals between the ages of 18-65 years old, then what does this mean for individuals who are 70 years old? In this case, the vaccine recommendations should not include anyone over 65 years. This same line of reasoning applies to other individual characteristics as well if they were not included in the study design, such as gender, medical history (e.g., asthma or diabetes) or whether someone lives in an institution (e.g., people is assisted living).
How are the study results and conclusions influenced by other research on the same topic and societal trends?
The study results and conclusions should be placed within the context of current research and society.
- Where do the results fit in to what is already known on the topic?
- Do the results refute or support findings from previous studies?
- What additional studies need to be done before action can be taken based on the results?
- What is the big picture and where do we go from here?
Information emerging from other research or established knowledge will often play a role in determining whether the medical or therapeutic intervention studied is recommended for the intended population. If, for example, the conclusions contradict current knowledge, it is likely more studies will be called for rather than if the results confirm what we already understand to be true. If the conclusions agree with current knowledge, then further studies may need to evaluate how the new treatment or intervention is implemented into real-world medical practice.
Though our understanding of COVID-19 may be incomplete, we can still ask how a specific vaccine study compares to others, which is one of the benefits of multiple vaccines being developed at one time. We can show similarities and differences.
- Are some vaccines proving to be more effective than others?
- Are some vaccines proving to be more safe than others?
- Do we need multiple vaccines?
- Does the length of time protection is afforded vary among vaccines?
- Do additional clinical trials need to be conducted before the vaccines should be administered in a wide-spread manner?
The scientific process involves asking a question, gathering data to answer the question and repeating the study until enough data supports a claim. One vaccine trial may successfully reduce the spread of COVID-19, but additional studies are needed to support the results and to make broad recommendations.
If subsequent studies are typical when developing vaccines, even if the results are favorable, can we trust the COVID-19 vaccines that have been recently approved for the public?
Evaluation and introduction of a new medical treatment or therapy (e.g., a vaccine) to the public or a specific population involves critical examination of risks and benefits. We are currently experiencing over 3,000 COVID-19 related deaths in the US each day. Whereas, the usual process of evaluation develops over time through multiple studies, the COVID-19 pandemic presents a unique and challenging context.
Even though the process is moving at a fast pace, sufficient data to demonstrate the safety and efficacy of any vaccine is still a requirement. In addition, the fact that there are multiple vaccines being developed at one time facilitates the benefit of comparison as discussed in the earlier question. We are able to compare safety and efficacy of one COVID-19 vaccine to another.
The scientific process is experimental and constantly evolving, but the research for COVID-19 is being conducted at the highest standard possible considering the context.
Who makes the final approval about what vaccines are introduced into the population?
After the clinical trials are completed, the data and results are sent to the Federal Drug Administration (FDA) for decisions on approval and distribution of the vaccine. Read more about the FDA’s involvement in vaccine development.
Wake Forest School of Medicine is among the many academic institutions across the country dedicating immense time and resources to understanding and treating COVID-19, including researchers who helped author this Q&A. With over 50 active or completed clinical research studies, in addition to basic science research and community projects, much of this research has been completed by cross-disciplinary and inter-departmental teams—demonstrating the important nature of this work. Read about some of the COVID-19 research work, including participation in some of the largest vaccine trials in the country, conducted here.
To learn about the distribution of COVID-19 vaccines from our institution, please visit our resources on our Wake Forest Baptist Health site.
Read more about the faculty who contributed to this article: Mark Espeland, PhD; Nathaniel O’Connell, PhD; David Reboussin, PhD; Joseph Rigdon, PhD; Heather Shappell, PhD; and Jaime Lynn Speiser, MSc, PhD.