non empirical research

Non-Empirical Research

Non-Empirical Research articles focus more on theories, methods and their implications for education research. Non-Empirical Research can include comprehensive reviews and articles that focus on methodology. It should rely on empirical research literature as well but does not need to be essentially data-driven.

The title page should:

present a title that includes, if appropriate, the research design
if a collaboration group should be listed as an author, please list the Group name as an author and include the names of the individual members of the Group in the “Acknowledgements” section in accordance with the instructions below
indicate the corresponding author
Declarations: all manuscripts must contain the following sections under the heading 'Declarations': 1. Availability of data and material 2.Competing interests 3. Funding 4. Authors' contributions 5. Acknowledgements 6. Authors' information (optional) Please see below for details on the information to be included in these sections. If any of the sections are not relevant to your manuscript, please include the heading and write 'Not applicable' for that section. 1. Availability of data and materials All manuscripts must include an ‘Availability of data and materials’ statement. Data availability statements should include information on where data supporting the results reported in the article can be found including, where applicable, hyperlinks to publicly archived datasets analysed or generated during the study. By data we mean the minimal dataset that would be necessary to interpret, replicate and build upon the findings reported in the article. We recognise it is not always possible to share research data publicly, for instance when individual privacy could be compromised, and in such instances data availability should still be stated in the manuscript along with any conditions for access. Data availability statements can take one of the following forms (or a combination of more than one if required for multiple datasets):
The datasets generated and/or analysed during the current study are available in the [NAME] repository, [PERSISTENT WEB LINK TO DATASETS]
The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.
All data generated or analysed during this study are included in this published article [and its supplementary information files].
The datasets generated and/or analysed during the current study are not publicly available due [REASON WHY DATA ARE NOT PUBLIC] but are available from the corresponding author on reasonable request.
Data sharing is not applicable to this article as no datasets were generated or analysed during the current study.
The data that support the findings of this study are available from [third party name] but restrictions apply to the availability of these data, which were used under license for the current study, and so are not publicly available. Data are however available from the authors upon reasonable request and with permission of [third party name].
Not applicable. If your manuscript does not contain any data, please state 'Not applicable' in this section.

More examples of template data availability statements, which include examples of openly available and restricted access datasets, are available here .

SpringerOpen also requires that authors cite any publicly available data on which the conclusions of the paper rely in the manuscript. Data citations should include a persistent identifier (such as a DOI) and should ideally be included in the reference list. Citations of datasets, when they appear in the reference list, should include the minimum information recommended by DataCite and follow journal style. Dataset identifiers including DOIs should be expressed as full URLs. For example:

Hao Z, AghaKouchak A, Nakhjiri N, Farahmand A. Global integrated drought monitoring and prediction system (GIDMaPS) data sets. figshare. 2014. http://dx.doi.org/10.6084/m9.figshare.853801

With the corresponding text in the Availability of data and materials statement:

The datasets generated during and/or analysed during the current study are available in the [NAME] repository, [PERSISTENT WEB LINK TO DATASETS]. [Reference number]

2. Competing interests

All financial and non-financial competing interests must be declared in this section.

See our editorial policies for a full explanation of competing interests. If you are unsure whether you or any of your co-authors have a competing interest please contact the editorial office.

Please use the authors’ initials to refer to each authors' competing interests in this section.

If you do not have any competing interests, please state "The authors declare that they have no competing interests" in this section.

All sources of funding for the research reported should be declared. The role of the funding body in the design of the study and collection, analysis, and interpretation of data and in writing the manuscript should be declared.

4. Authors' contributions

The individual contributions of authors to the manuscript should be specified in this section. Guidance and criteria for authorship can be found in our editorial policies .

Please use initials to refer to each author's contribution in this section, for example: "FC analyzed and interpreted the patient data regarding the hematological disease and the transplant. RH performed the histological examination of the kidney, and was a major contributor in writing the manuscript. All authors read and approved the final manuscript."

5. Acknowledgements

Please acknowledge anyone who contributed towards the article who does not meet the criteria for authorship including anyone who provided professional writing services or materials.

Authors should obtain permission to acknowledge from all those mentioned in the Acknowledgements section.

See our editorial policies for a full explanation of acknowledgements and authorship criteria.

If you do not have anyone to acknowledge, please write "Not applicable" in this section.

Group authorship (for manuscripts involving a collaboration group): if you would like the names of the individual members of a collaboration Group to be searchable through their individual PubMed records, please ensure that the title of the collaboration Group is included on the title page and in the submission system and also include collaborating author names as the last paragraph of the “Acknowledgements” section. Please add authors in the format First Name, Middle initial(s) (optional), Last Name. You can add institution or country information for each author if you wish, but this should be consistent across all authors.

6. Authors' information : This section is optional.

You may choose to use this section to include any relevant information about the author(s) that may aid the reader's interpretation of the article, and understand the standpoint of the author(s). This may include details about the authors' qualifications, current positions they hold at institutions or societies, or any other relevant background information. Please refer to authors using their initials. Note this section should not be used to describe any competing interests

Blinded Manuscript

Abstract The abstract should briefly summarize the aim, findings or purpose of the article. Please minimize the use of abbreviations and do not cite references in the abstract.

See the criteria section for this article type (located at the top of this page) for information on article word limits.

Three to ten keywords representing the main content of the article.

Introduction

The Introduction section should explain the background to the article, its aims, a summary of a search of the existing literature and the issue under discussion.

This should contain the body of the article, and may also be broken into subsections with short, informative headings.

Conclusions

This should state clearly the main conclusions and include an explanation of their relevance or importance to the field.

List of abbreviations

If abbreviations are used in the text they should be defined in the text at first use, and a list of abbreviations should be provided.

Examples of the American Psychological Association (APA) reference style are shown below. For further guidance, see the Publication Manual of the American Psychological Association and the respective web site of the Association ( http://www.apastyle.org/ ).

See our editorial policies for author guidance on good citation practice.

Web links and URLs: All web links and URLs, including links to the authors' own websites, should be given a reference number and included in the reference list rather than within the text of the manuscript. They should be provided in full, including both the title of the site and the URL, as well as the date the site was accessed, in the following format: The Mouse Tumor Biology Database. http://tumor.informatics.jax.org/mtbwi/index.do . Accessed 20 May 2013. If an author or group of authors can clearly be associated with a web link, such as for weblogs, then they should be included in the reference.

Example reference style:

Article within a journal

Harris, M., Karper, E., Stacks, G., Hoffman, D., DeNiro, & R., Cruz, P. (2001). Writing labs and the Hollywood connection. Journal of Film Writing , 44 (3), 213-245.

Article by DOI (with page numbers)

Slifka, M.K., & Whitton, J.L. (2000). Clinical implications of dysregulated cytokine production. Journal of Molecular Medicine , 78 (2), 74-80. doi:10.1007/s001090000086.

Article by DOI (before issue publication and without page numbers)

Kreger, M., Brindis, C.D., Manuel, D.M., & Sassoubre, L. (2007). Lessons learned in systems change initiatives: benchmarks and indicators. American Journal of Community Psychology . doi: 10.1007/s10464-007-9108-14.

Article in electronic journal by DOI (no paginated version)

Kruger, M., Brandis, C.D., Mandel, D.M., & Sassoure, J. (2007). Lessons to be learned in systems change initiatives: benchmarks and indicators. American Journal of Digital Psychology . doi: 10.1007/s10469-007-5108-14.

Complete book

Calfee, R.C., & Valencia, R.R. (1991). APA guide to preparing manuscripts for journal publication. Washington, DC: American Psychological Association.

Book chapter, or an article within a book

O'Neil, J.M., & Egan, J. (1992). Men's and women's gender role journeys: Metaphor for healing, transition, and transformation. In B.R. Wainrib (Ed.), Gender issues across the life cycle (pp. 107-123). New York: Springer .

Online First chapter in a series (without a volume designation but with a DOI)

Saito, Y., & Hyuga, H. (2007). Rate equation approaches to amplification of enantiomeric excess and chiral symmetry breaking. Topics in Current Chemistry . doi:10.1007/128_2006_108.

Complete book, also showing a translated edition [Either edition may be listed first.]

Adorno, T.W. (1966). Negative Dialektik . Frankfurt: Suhrkamp. English edition: Adorno, TW (1973). Negative Dialectics (trans: Ashton, E.B.). London: Routledge.

Online document

Abou-Allaban, Y., Dell, M.L., Greenberg, W., Lomax, J., Peteet, J., Torres, M., & Cowell, V. (2006). Religious/spiritual commitments and psychiatric practice. Resource document. American Psychiatric Association. http://www.psych.org/edu/other_res/lib_archives/archives/200604.pdf. Accessed 25 June 2007.

Online database

German emigrants database (1998). Historisches Museum Bremerhaven. http://www.deutsche-auswanderer-datenbank.de. Accessed 21 June 2007.

Supplementary material/private homepage

Doe, J. (2006). Title of supplementary material. http://www.privatehomepage.com. Accessed 22 Feb 2007.

Doe, J. (1999). Trivial HTTP, RFC2169. ftp://ftp.isi.edu/in-notes/rfc2169.txt. Accessed 12 Feb 2006.

Organization site

ISSN International Centre (2006). The ISSN register. http://www.issn.org. Accessed 20 Feb 2007.

Figures, tables and additional files

See General formatting guidelines for information on how to format figures, tables and additional files.

Submit manuscript

Affiliated with

Innovation and Education is affiliated with the Korea National University of Education.

Innovation and Education

ISSN: 2524-8502

Submission enquiries: Access here and click Contact Us
General enquiries: [email protected]

6.1 Overview of Non-Experimental Research

Learning objectives.

Define non-experimental research, distinguish it clearly from experimental research, and give several examples.
Explain when a researcher might choose to conduct non-experimental research as opposed to experimental research.

What Is Non-Experimental Research?

Non-experimental research is research that lacks the manipulation of an independent variable. Rather than manipulating an independent variable, researchers conducting non-experimental research simply measure variables as they naturally occur (in the lab or real world).

Most researchers in psychology consider the distinction between experimental and non-experimental research to be an extremely important one. This is because although experimental research can provide strong evidence that changes in an independent variable cause differences in a dependent variable, non-experimental research generally cannot. As we will see, however, this inability to make causal conclusions does not mean that non-experimental research is less important than experimental research.

When to Use Non-Experimental Research

As we saw in the last chapter , experimental research is appropriate when the researcher has a specific research question or hypothesis about a causal relationship between two variables—and it is possible, feasible, and ethical to manipulate the independent variable. It stands to reason, therefore, that non-experimental research is appropriate—even necessary—when these conditions are not met. There are many times in which non-experimental research is preferred, including when:

the research question or hypothesis relates to a single variable rather than a statistical relationship between two variables (e.g., How accurate are people’s first impressions?).
the research question pertains to a non-causal statistical relationship between variables (e.g., is there a correlation between verbal intelligence and mathematical intelligence?).
the research question is about a causal relationship, but the independent variable cannot be manipulated or participants cannot be randomly assigned to conditions or orders of conditions for practical or ethical reasons (e.g., does damage to a person’s hippocampus impair the formation of long-term memory traces?).
the research question is broad and exploratory, or is about what it is like to have a particular experience (e.g., what is it like to be a working mother diagnosed with depression?).

Again, the choice between the experimental and non-experimental approaches is generally dictated by the nature of the research question. Recall the three goals of science are to describe, to predict, and to explain. If the goal is to explain and the research question pertains to causal relationships, then the experimental approach is typically preferred. If the goal is to describe or to predict, a non-experimental approach will suffice. But the two approaches can also be used to address the same research question in complementary ways. For example, Similarly, after his original study, Milgram conducted experiments to explore the factors that affect obedience. He manipulated several independent variables, such as the distance between the experimenter and the participant, the participant and the confederate, and the location of the study (Milgram, 1974) [1] .

Types of Non-Experimental Research

Non-experimental research falls into three broad categories: cross-sectional research, correlational research, and observational research.

First, cross-sectional research involves comparing two or more pre-existing groups of people. What makes this approach non-experimental is that there is no manipulation of an independent variable and no random assignment of participants to groups. Imagine, for example, that a researcher administers the Rosenberg Self-Esteem Scale to 50 American college students and 50 Japanese college students. Although this “feels” like a between-subjects experiment, it is a cross-sectional study because the researcher did not manipulate the students’ nationalities. As another example, if we wanted to compare the memory test performance of a group of cannabis users with a group of non-users, this would be considered a cross-sectional study because for ethical and practical reasons we would not be able to randomly assign participants to the cannabis user and non-user groups. Rather we would need to compare these pre-existing groups which could introduce a selection bias (the groups may differ in other ways that affect their responses on the dependent variable). For instance, cannabis users are more likely to use more alcohol and other drugs and these differences may account for differences in the dependent variable across groups, rather than cannabis use per se.

Cross-sectional designs are commonly used by developmental psychologists who study aging and by researchers interested in sex differences. Using this design, developmental psychologists compare groups of people of different ages (e.g., young adults spanning from 18-25 years of age versus older adults spanning 60-75 years of age) on various dependent variables (e.g., memory, depression, life satisfaction). Of course, the primary limitation of using this design to study the effects of aging is that differences between the groups other than age may account for differences in the dependent variable. For instance, differences between the groups may reflect the generation that people come from (a cohort effect) rather than a direct effect of age. For this reason, longitudinal studies in which one group of people is followed as they age offer a superior means of studying the effects of aging. Once again, cross-sectional designs are also commonly used to study sex differences. Since researchers cannot practically or ethically manipulate the sex of their participants they must rely on cross-sectional designs to compare groups of men and women on different outcomes (e.g., verbal ability, substance use, depression). Using these designs researchers have discovered that men are more likely than women to suffer from substance abuse problems while women are more likely than men to suffer from depression. But, using this design it is unclear what is causing these differences. So, using this design it is unclear whether these differences are due to environmental factors like socialization or biological factors like hormones?

When researchers use a participant characteristic to create groups (nationality, cannabis use, age, sex), the independent variable is usually referred to as an experimenter-selected independent variable (as opposed to the experimenter-manipulated independent variables used in experimental research). Figure 6.1 shows data from a hypothetical study on the relationship between whether people make a daily list of things to do (a “to-do list”) and stress. Notice that it is unclear whether this is an experiment or a cross-sectional study because it is unclear whether the independent variable was manipulated by the researcher or simply selected by the researcher. If the researcher randomly assigned some participants to make daily to-do lists and others not to, then the independent variable was experimenter-manipulated and it is a true experiment. If the researcher simply asked participants whether they made daily to-do lists or not, then the independent variable it is experimenter-selected and the study is cross-sectional. The distinction is important because if the study was an experiment, then it could be concluded that making the daily to-do lists reduced participants’ stress. But if it was a cross-sectional study, it could only be concluded that these variables are statistically related. Perhaps being stressed has a negative effect on people’s ability to plan ahead. Or perhaps people who are more conscientious are more likely to make to-do lists and less likely to be stressed. The crucial point is that what defines a study as experimental or cross-sectional l is not the variables being studied, nor whether the variables are quantitative or categorical, nor the type of graph or statistics used to analyze the data. It is how the study is conducted.

Figure 6.1 Results of a Hypothetical Study on Whether People Who Make Daily To-Do Lists Experience Less Stress Than People Who Do Not Make Such Lists

Second, the most common type of non-experimental research conducted in Psychology is correlational research. Correlational research is considered non-experimental because it focuses on the statistical relationship between two variables but does not include the manipulation of an independent variable. More specifically, in correlational research , the researcher measures two continuous variables with little or no attempt to control extraneous variables and then assesses the relationship between them. As an example, a researcher interested in the relationship between self-esteem and school achievement could collect data on students’ self-esteem and their GPAs to see if the two variables are statistically related. Correlational research is very similar to cross-sectional research, and sometimes these terms are used interchangeably. The distinction that will be made in this book is that, rather than comparing two or more pre-existing groups of people as is done with cross-sectional research, correlational research involves correlating two continuous variables (groups are not formed and compared).

Third, observational research is non-experimental because it focuses on making observations of behavior in a natural or laboratory setting without manipulating anything. Milgram’s original obedience study was non-experimental in this way. He was primarily interested in the extent to which participants obeyed the researcher when he told them to shock the confederate and he observed all participants performing the same task under the same conditions. The study by Loftus and Pickrell described at the beginning of this chapter is also a good example of observational research. The variable was whether participants “remembered” having experienced mildly traumatic childhood events (e.g., getting lost in a shopping mall) that they had not actually experienced but that the researchers asked them about repeatedly. In this particular study, nearly a third of the participants “remembered” at least one event. (As with Milgram’s original study, this study inspired several later experiments on the factors that affect false memories.

The types of research we have discussed so far are all quantitative, referring to the fact that the data consist of numbers that are analyzed using statistical techniques. But as you will learn in this chapter, many observational research studies are more qualitative in nature. In qualitative research , the data are usually nonnumerical and therefore cannot be analyzed using statistical techniques. Rosenhan’s observational study of the experience of people in a psychiatric ward was primarily qualitative. The data were the notes taken by the “pseudopatients”—the people pretending to have heard voices—along with their hospital records. Rosenhan’s analysis consists mainly of a written description of the experiences of the pseudopatients, supported by several concrete examples. To illustrate the hospital staff’s tendency to “depersonalize” their patients, he noted, “Upon being admitted, I and other pseudopatients took the initial physical examinations in a semi-public room, where staff members went about their own business as if we were not there” (Rosenhan, 1973, p. 256) [2] . Qualitative data has a separate set of analysis tools depending on the research question. For example, thematic analysis would focus on themes that emerge in the data or conversation analysis would focus on the way the words were said in an interview or focus group.

Internal Validity Revisited

Recall that internal validity is the extent to which the design of a study supports the conclusion that changes in the independent variable caused any observed differences in the dependent variable. Figure 6.2 shows how experimental, quasi-experimental, and non-experimental (correlational) research vary in terms of internal validity. Experimental research tends to be highest in internal validity because the use of manipulation (of the independent variable) and control (of extraneous variables) help to rule out alternative explanations for the observed relationships. If the average score on the dependent variable in an experiment differs across conditions, it is quite likely that the independent variable is responsible for that difference. Non-experimental (correlational) research is lowest in internal validity because these designs fail to use manipulation or control. Quasi-experimental research (which will be described in more detail in a subsequent chapter) is in the middle because it contains some, but not all, of the features of a true experiment. For instance, it may fail to use random assignment to assign participants to groups or fail to use counterbalancing to control for potential order effects. Imagine, for example, that a researcher finds two similar schools, starts an anti-bullying program in one, and then finds fewer bullying incidents in that “treatment school” than in the “control school.” While a comparison is being made with a control condition, the lack of random assignment of children to schools could still mean that students in the treatment school differed from students in the control school in some other way that could explain the difference in bullying (e.g., there may be a selection effect).

Figure 7.1 Internal Validity of Correlational, Quasi-Experimental, and Experimental Studies. Experiments are generally high in internal validity, quasi-experiments lower, and correlational studies lower still.

Figure 6.2 Internal Validity of Correlation, Quasi-Experimental, and Experimental Studies. Experiments are generally high in internal validity, quasi-experiments lower, and correlation studies lower still.

Notice also in Figure 6.2 that there is some overlap in the internal validity of experiments, quasi-experiments, and correlational studies. For example, a poorly designed experiment that includes many confounding variables can be lower in internal validity than a well-designed quasi-experiment with no obvious confounding variables. Internal validity is also only one of several validities that one might consider, as noted in Chapter 5.

Key Takeaways

Non-experimental research is research that lacks the manipulation of an independent variable.
There are two broad types of non-experimental research. Correlational research that focuses on statistical relationships between variables that are measured but not manipulated, and observational research in which participants are observed and their behavior is recorded without the researcher interfering or manipulating any variables.
In general, experimental research is high in internal validity, correlational research is low in internal validity, and quasi-experimental research is in between.
A researcher conducts detailed interviews with unmarried teenage fathers to learn about how they feel and what they think about their role as fathers and summarizes their feelings in a written narrative.
A researcher measures the impulsivity of a large sample of drivers and looks at the statistical relationship between this variable and the number of traffic tickets the drivers have received.
A researcher randomly assigns patients with low back pain either to a treatment involving hypnosis or to a treatment involving exercise. She then measures their level of low back pain after 3 months.
A college instructor gives weekly quizzes to students in one section of his course but no weekly quizzes to students in another section to see whether this has an effect on their test performance.
Milgram, S. (1974). Obedience to authority: An experimental view . New York, NY: Harper & Row. ↵
Rosenhan, D. L. (1973). On being sane in insane places. Science, 179 , 250–258. ↵

Share This Book

Increase Font Size

Want to create or adapt books like this? Learn more about how Pressbooks supports open publishing practices.

Chapter 7: Nonexperimental Research

Overview of Nonexperimental Research

Learning Objectives

Define nonexperimental research, distinguish it clearly from experimental research, and give several examples.
Explain when a researcher might choose to conduct nonexperimental research as opposed to experimental research.

What Is Nonexperimental Research?

Nonexperimental research is research that lacks the manipulation of an independent variable, random assignment of participants to conditions or orders of conditions, or both.

In a sense, it is unfair to define this large and diverse set of approaches collectively by what they are not . But doing so reflects the fact that most researchers in psychology consider the distinction between experimental and nonexperimental research to be an extremely important one. This distinction is because although experimental research can provide strong evidence that changes in an independent variable cause differences in a dependent variable, nonexperimental research generally cannot. As we will see, however, this inability does not mean that nonexperimental research is less important than experimental research or inferior to it in any general sense.

When to Use Nonexperimental Research

As we saw in Chapter 6 , experimental research is appropriate when the researcher has a specific research question or hypothesis about a causal relationship between two variables—and it is possible, feasible, and ethical to manipulate the independent variable and randomly assign participants to conditions or to orders of conditions. It stands to reason, therefore, that nonexperimental research is appropriate—even necessary—when these conditions are not met. There are many ways in which preferring nonexperimental research can be the case.

The research question or hypothesis can be about a single variable rather than a statistical relationship between two variables (e.g., How accurate are people’s first impressions?).
The research question can be about a noncausal statistical relationship between variables (e.g., Is there a correlation between verbal intelligence and mathematical intelligence?).
The research question can be about a causal relationship, but the independent variable cannot be manipulated or participants cannot be randomly assigned to conditions or orders of conditions (e.g., Does damage to a person’s hippocampus impair the formation of long-term memory traces?).
The research question can be broad and exploratory, or it can be about what it is like to have a particular experience (e.g., What is it like to be a working mother diagnosed with depression?).

Again, the choice between the experimental and nonexperimental approaches is generally dictated by the nature of the research question. If it is about a causal relationship and involves an independent variable that can be manipulated, the experimental approach is typically preferred. Otherwise, the nonexperimental approach is preferred. But the two approaches can also be used to address the same research question in complementary ways. For example, nonexperimental studies establishing that there is a relationship between watching violent television and aggressive behaviour have been complemented by experimental studies confirming that the relationship is a causal one (Bushman & Huesmann, 2001) [1] . Similarly, after his original study, Milgram conducted experiments to explore the factors that affect obedience. He manipulated several independent variables, such as the distance between the experimenter and the participant, the participant and the confederate, and the location of the study (Milgram, 1974) [2] .

Types of Nonexperimental Research

Nonexperimental research falls into three broad categories: single-variable research, correlational and quasi-experimental research, and qualitative research. First, research can be nonexperimental because it focuses on a single variable rather than a statistical relationship between two variables. Although there is no widely shared term for this kind of research, we will call it single-variable research . Milgram’s original obedience study was nonexperimental in this way. He was primarily interested in one variable—the extent to which participants obeyed the researcher when he told them to shock the confederate—and he observed all participants performing the same task under the same conditions. The study by Loftus and Pickrell described at the beginning of this chapter is also a good example of single-variable research. The variable was whether participants “remembered” having experienced mildly traumatic childhood events (e.g., getting lost in a shopping mall) that they had not actually experienced but that the research asked them about repeatedly. In this particular study, nearly a third of the participants “remembered” at least one event. (As with Milgram’s original study, this study inspired several later experiments on the factors that affect false memories.)

As these examples make clear, single-variable research can answer interesting and important questions. What it cannot do, however, is answer questions about statistical relationships between variables. This detail is a point that beginning researchers sometimes miss. Imagine, for example, a group of research methods students interested in the relationship between children’s being the victim of bullying and the children’s self-esteem. The first thing that is likely to occur to these researchers is to obtain a sample of middle-school students who have been bullied and then to measure their self-esteem. But this design would be a single-variable study with self-esteem as the only variable. Although it would tell the researchers something about the self-esteem of children who have been bullied, it would not tell them what they really want to know, which is how the self-esteem of children who have been bullied compares with the self-esteem of children who have not. Is it lower? Is it the same? Could it even be higher? To answer this question, their sample would also have to include middle-school students who have not been bullied thereby introducing another variable.

Research can also be nonexperimental because it focuses on a statistical relationship between two variables but does not include the manipulation of an independent variable, random assignment of participants to conditions or orders of conditions, or both. This kind of research takes two basic forms: correlational research and quasi-experimental research. In correlational research , the researcher measures the two variables of interest with little or no attempt to control extraneous variables and then assesses the relationship between them. A research methods student who finds out whether each of several middle-school students has been bullied and then measures each student’s self-esteem is conducting correlational research. In quasi-experimental research , the researcher manipulates an independent variable but does not randomly assign participants to conditions or orders of conditions. For example, a researcher might start an antibullying program (a kind of treatment) at one school and compare the incidence of bullying at that school with the incidence at a similar school that has no antibullying program.

The final way in which research can be nonexperimental is that it can be qualitative. The types of research we have discussed so far are all quantitative, referring to the fact that the data consist of numbers that are analyzed using statistical techniques. In qualitative research , the data are usually nonnumerical and therefore cannot be analyzed using statistical techniques. Rosenhan’s study of the experience of people in a psychiatric ward was primarily qualitative. The data were the notes taken by the “pseudopatients”—the people pretending to have heard voices—along with their hospital records. Rosenhan’s analysis consists mainly of a written description of the experiences of the pseudopatients, supported by several concrete examples. To illustrate the hospital staff’s tendency to “depersonalize” their patients, he noted, “Upon being admitted, I and other pseudopatients took the initial physical examinations in a semipublic room, where staff members went about their own business as if we were not there” (Rosenhan, 1973, p. 256). [3] Qualitative data has a separate set of analysis tools depending on the research question. For example, thematic analysis would focus on themes that emerge in the data or conversation analysis would focus on the way the words were said in an interview or focus group.

Internal Validity Revisited

Recall that internal validity is the extent to which the design of a study supports the conclusion that changes in the independent variable caused any observed differences in the dependent variable. Figure 7.1 shows how experimental, quasi-experimental, and correlational research vary in terms of internal validity. Experimental research tends to be highest because it addresses the directionality and third-variable problems through manipulation and the control of extraneous variables through random assignment. If the average score on the dependent variable in an experiment differs across conditions, it is quite likely that the independent variable is responsible for that difference. Correlational research is lowest because it fails to address either problem. If the average score on the dependent variable differs across levels of the independent variable, it could be that the independent variable is responsible, but there are other interpretations. In some situations, the direction of causality could be reversed. In others, there could be a third variable that is causing differences in both the independent and dependent variables. Quasi-experimental research is in the middle because the manipulation of the independent variable addresses some problems, but the lack of random assignment and experimental control fails to address others. Imagine, for example, that a researcher finds two similar schools, starts an antibullying program in one, and then finds fewer bullying incidents in that “treatment school” than in the “control school.” There is no directionality problem because clearly the number of bullying incidents did not determine which school got the program. However, the lack of random assignment of children to schools could still mean that students in the treatment school differed from students in the control school in some other way that could explain the difference in bullying.

Notice also in Figure 7.1 that there is some overlap in the internal validity of experiments, quasi-experiments, and correlational studies. For example, a poorly designed experiment that includes many confounding variables can be lower in internal validity than a well designed quasi-experiment with no obvious confounding variables. Internal validity is also only one of several validities that one might consider, as noted in Chapter 5.

Key Takeaways

Nonexperimental research is research that lacks the manipulation of an independent variable, control of extraneous variables through random assignment, or both.
There are three broad types of nonexperimental research. Single-variable research focuses on a single variable rather than a relationship between variables. Correlational and quasi-experimental research focus on a statistical relationship but lack manipulation or random assignment. Qualitative research focuses on broader research questions, typically involves collecting large amounts of data from a small number of participants, and analyses the data nonstatistically.
In general, experimental research is high in internal validity, correlational research is low in internal validity, and quasi-experimental research is in between.

Discussion: For each of the following studies, decide which type of research design it is and explain why.

A researcher conducts detailed interviews with unmarried teenage fathers to learn about how they feel and what they think about their role as fathers and summarizes their feelings in a written narrative.
A researcher measures the impulsivity of a large sample of drivers and looks at the statistical relationship between this variable and the number of traffic tickets the drivers have received.
A researcher randomly assigns patients with low back pain either to a treatment involving hypnosis or to a treatment involving exercise. She then measures their level of low back pain after 3 months.
A college instructor gives weekly quizzes to students in one section of his course but no weekly quizzes to students in another section to see whether this has an effect on their test performance.
Bushman, B. J., & Huesmann, L. R. (2001). Effects of televised violence on aggression. In D. Singer & J. Singer (Eds.), Handbook of children and the media (pp. 223–254). Thousand Oaks, CA: Sage. ↵
Milgram, S. (1974). Obedience to authority: An experimental view . New York, NY: Harper & Row. ↵
Rosenhan, D. L. (1973). On being sane in insane places. Science, 179 , 250–258. ↵

Research that lacks the manipulation of an independent variable, random assignment of participants to conditions or orders of conditions, or both.

Research that focuses on a single variable rather than a statistical relationship between two variables.

The researcher measures the two variables of interest with little or no attempt to control extraneous variables and then assesses the relationship between them.

The researcher manipulates an independent variable but does not randomly assign participants to conditions or orders of conditions.

Research Methods in Psychology - 2nd Canadian Edition Copyright © 2015 by Paul C. Price, Rajiv Jhangiani, & I-Chant A. Chiang is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License , except where otherwise noted.

Share This Book

Want to create or adapt books like this? Learn more about how Pressbooks supports open publishing practices.

Non-Experimental Research

28 Overview of Non-Experimental Research

Learning objectives.

Define non-experimental research, distinguish it clearly from experimental research, and give several examples.
Explain when a researcher might choose to conduct non-experimental research as opposed to experimental research.

What Is Non-Experimental Research?

When to Use Non-Experimental Research

the research question or hypothesis relates to a single variable rather than a statistical relationship between two variables (e.g., how accurate are people’s first impressions?).
the research question pertains to a non-causal statistical relationship between variables (e.g., is there a correlation between verbal intelligence and mathematical intelligence?).
the research question is about a causal relationship, but the independent variable cannot be manipulated or participants cannot be randomly assigned to conditions or orders of conditions for practical or ethical reasons (e.g., does damage to a person’s hippocampus impair the formation of long-term memory traces?).
the research question is broad and exploratory, or is about what it is like to have a particular experience (e.g., what is it like to be a working mother diagnosed with depression?).

Again, the choice between the experimental and non-experimental approaches is generally dictated by the nature of the research question. Recall the three goals of science are to describe, to predict, and to explain. If the goal is to explain and the research question pertains to causal relationships, then the experimental approach is typically preferred. If the goal is to describe or to predict, a non-experimental approach is appropriate. But the two approaches can also be used to address the same research question in complementary ways. For example, in Milgram’s original (non-experimental) obedience study, he was primarily interested in one variable—the extent to which participants obeyed the researcher when he told them to shock the confederate—and he observed all participants performing the same task under the same conditions. However, Milgram subsequently conducted experiments to explore the factors that affect obedience. He manipulated several independent variables, such as the distance between the experimenter and the participant, the participant and the confederate, and the location of the study (Milgram, 1974) [1] .

Types of Non-Experimental Research

Non-experimental research falls into two broad categories: correlational research and observational research.

The most common type of non-experimental research conducted in psychology is correlational research. Correlational research is considered non-experimental because it focuses on the statistical relationship between two variables but does not include the manipulation of an independent variable. More specifically, in correlational research , the researcher measures two variables with little or no attempt to control extraneous variables and then assesses the relationship between them. As an example, a researcher interested in the relationship between self-esteem and school achievement could collect data on students’ self-esteem and their GPAs to see if the two variables are statistically related.

Observational research is non-experimental because it focuses on making observations of behavior in a natural or laboratory setting without manipulating anything. Milgram’s original obedience study was non-experimental in this way. He was primarily interested in the extent to which participants obeyed the researcher when he told them to shock the confederate and he observed all participants performing the same task under the same conditions. The study by Loftus and Pickrell described at the beginning of this chapter is also a good example of observational research. The variable was whether participants “remembered” having experienced mildly traumatic childhood events (e.g., getting lost in a shopping mall) that they had not actually experienced but that the researchers asked them about repeatedly. In this particular study, nearly a third of the participants “remembered” at least one event. (As with Milgram’s original study, this study inspired several later experiments on the factors that affect false memories).

Cross-Sectional, Longitudinal, and Cross-Sequential Studies

When psychologists wish to study change over time (for example, when developmental psychologists wish to study aging) they usually take one of three non-experimental approaches: cross-sectional, longitudinal, or cross-sequential. Cross-sectional studies involve comparing two or more pre-existing groups of people (e.g., children at different stages of development). What makes this approach non-experimental is that there is no manipulation of an independent variable and no random assignment of participants to groups. Using this design, developmental psychologists compare groups of people of different ages (e.g., young adults spanning from 18-25 years of age versus older adults spanning 60-75 years of age) on various dependent variables (e.g., memory, depression, life satisfaction). Of course, the primary limitation of using this design to study the effects of aging is that differences between the groups other than age may account for differences in the dependent variable. For instance, differences between the groups may reflect the generation that people come from (a cohort effect ) rather than a direct effect of age. For this reason, longitudinal studies , in which one group of people is followed over time as they age, offer a superior means of studying the effects of aging. However, longitudinal studies are by definition more time consuming and so require a much greater investment on the part of the researcher and the participants. A third approach, known as cross-sequential studies , combines elements of both cross-sectional and longitudinal studies. Rather than measuring differences between people in different age groups or following the same people over a long period of time, researchers adopting this approach choose a smaller period of time during which they follow people in different age groups. For example, they might measure changes over a ten year period among participants who at the start of the study fall into the following age groups: 20 years old, 30 years old, 40 years old, 50 years old, and 60 years old. This design is advantageous because the researcher reaps the immediate benefits of being able to compare the age groups after the first assessment. Further, by following the different age groups over time they can subsequently determine whether the original differences they found across the age groups are due to true age effects or cohort effects.

The types of research we have discussed so far are all quantitative, referring to the fact that the data consist of numbers that are analyzed using statistical techniques. But as you will learn in this chapter, many observational research studies are more qualitative in nature. In qualitative research , the data are usually nonnumerical and therefore cannot be analyzed using statistical techniques. Rosenhan’s observational study of the experience of people in psychiatric wards was primarily qualitative. The data were the notes taken by the “pseudopatients”—the people pretending to have heard voices—along with their hospital records. Rosenhan’s analysis consists mainly of a written description of the experiences of the pseudopatients, supported by several concrete examples. To illustrate the hospital staff’s tendency to “depersonalize” their patients, he noted, “Upon being admitted, I and other pseudopatients took the initial physical examinations in a semi-public room, where staff members went about their own business as if we were not there” (Rosenhan, 1973, p. 256) [2] . Qualitative data has a separate set of analysis tools depending on the research question. For example, thematic analysis would focus on themes that emerge in the data or conversation analysis would focus on the way the words were said in an interview or focus group.

Internal Validity Revisited

Recall that internal validity is the extent to which the design of a study supports the conclusion that changes in the independent variable caused any observed differences in the dependent variable. Figure 6.1 shows how experimental, quasi-experimental, and non-experimental (correlational) research vary in terms of internal validity. Experimental research tends to be highest in internal validity because the use of manipulation (of the independent variable) and control (of extraneous variables) help to rule out alternative explanations for the observed relationships. If the average score on the dependent variable in an experiment differs across conditions, it is quite likely that the independent variable is responsible for that difference. Non-experimental (correlational) research is lowest in internal validity because these designs fail to use manipulation or control. Quasi-experimental research (which will be described in more detail in a subsequent chapter) falls in the middle because it contains some, but not all, of the features of a true experiment. For instance, it may fail to use random assignment to assign participants to groups or fail to use counterbalancing to control for potential order effects. Imagine, for example, that a researcher finds two similar schools, starts an anti-bullying program in one, and then finds fewer bullying incidents in that “treatment school” than in the “control school.” While a comparison is being made with a control condition, the inability to randomly assign children to schools could still mean that students in the treatment school differed from students in the control school in some other way that could explain the difference in bullying (e.g., there may be a selection effect).

Figure 6.1 Internal Validity of Correlational, Quasi-Experimental, and Experimental Studies. Experiments are generally high in internal validity, quasi-experiments lower, and correlational studies lower still.

Notice also in Figure 6.1 that there is some overlap in the internal validity of experiments, quasi-experiments, and correlational (non-experimental) studies. For example, a poorly designed experiment that includes many confounding variables can be lower in internal validity than a well-designed quasi-experiment with no obvious confounding variables. Internal validity is also only one of several validities that one might consider, as noted in Chapter 5.

Milgram, S. (1974). Obedience to authority: An experimental view . New York, NY: Harper & Row. ↵
Rosenhan, D. L. (1973). On being sane in insane places. Science, 179 , 250–258. ↵

A research that lacks the manipulation of an independent variable.

Research that is non-experimental because it focuses on the statistical relationship between two variables but does not include the manipulation of an independent variable.

Research that is non-experimental because it focuses on recording systemic observations of behavior in a natural or laboratory setting without manipulating anything.

Studies that involve comparing two or more pre-existing groups of people (e.g., children at different stages of development).

Differences between the groups may reflect the generation that people come from rather than a direct effect of age.

Studies in which one group of people are followed over time as they age.

Studies in which researchers follow people in different age groups in a smaller period of time.

Research Methods in Psychology Copyright © 2019 by Rajiv S. Jhangiani, I-Chant A. Chiang, Carrie Cuttler, & Dana C. Leighton is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License , except where otherwise noted.

Share This Book

BY 499 - Senior Seminar

Library Catalog(s)
Searching Tips and Source Evaluation
Article Databases
Writing Style - APA
Empirical v. Non-Empirical Research
Poster Design
How Did We Do?

Subject Guide

Biology Articles

Primary Databases

Featuring thousands of full-text journals, this collection of scholarly trade and popular articles offers information on a broad range of important areas including: anthropology, biology, chemistry, ethnic & multicultural studies, law, mathematics, music, psychology, women's studies, and many other fields. Part of the Database Offerings in GALILEO, Georgia’s Virtual Library

ProQuest Research Library™ provides access to more than 5,060 titles —over 3,600 in full text— on a wide range of popular academic subjects. The database features a diversified mix of scholarly journals, trade publications, magazines, and newspapers. Part of the Database Offerings in GALILEO, Georgia’s Virtual Library

The National Science Digital Library (NSDL) was created by the National Science Foundation to provide organized access to high quality resources and tools that support innovations in teaching and learning at all levels of science, technology, engineering, and mathematics education. Part of the database offerings in GALILEO, Georgia's virtual library.

This database contains more than 820 leading full-text journals covering relevant aspects of the scientific and technical community. In addition to full text, Science & Technology Collection™ offers indexing and abstracts for more than 1,750 journals. Topics include aeronautics, astrophysics, biology, chemistry, computer technology, geology, aviation, physics, archaeology, marine sciences and materials science. Part of the Database Offerings in GALILEO, Georgia’s Virtual Library

A leading full-text scientific database covering the life and health sciences.

How to Make a Poster

Designing Conference Posters & Poster Templates

Empirical Versus Non-empirical Research

Empirical research is based on observed and measured phenomena and derives knowledge from actual experience rather than from theory or belief.

How do you know if a study is empirical? Read the subheadings within the article, book, or report and look for a description of the research "methodology." Ask yourself: Could I recreate this study and test these results?

Key characteristics to look for:

Specific research questions to be answered
Definition of the population, behavior, or phenomena being studied
Description of the process used to study this population or phenomena, including selection criteria, controls, and testing instruments (such as surveys)
Author(s) present a new set of findings from original research after conducting an original experiment
Firsthand collection of data

Another hint: some scholarly journals use a specific layout, called the "IMRaD" format, to communicate empirical research findings. Such articles typically have 4 components:

Introduction : sometimes called "literature review" -- what is currently known about the topic -- usually includes a theoretical framework and/or discussion of previous studies
Methodology: sometimes called "research design" -- how to recreate the study -- usually describes the population or variables to be researched, research process, and analytical tools
Results : sometimes called "findings" -- what was learned through the study -- usually appears as statistical data or as substantial quotations from research participants
Discussion : sometimes called "conclusion" or "implications" -- why the study is important -- usually describes how the research results influence professional practices or future studies

Here are some common characteristics of review articles:

Author(s) analyze and summarize existing research
Author(s) did NOT do original research. They are summarizing work of others.
Often focus on a general topic (such as breast cancer treatment) and bring together all relevant, useful articles on that topic in one review article.
Do not contain sections such as Methods (and Materials), Results because they did not do any original research!

Fermentation and quality of yellow pigments from golden brown rice solid culture by a selected Monascus mutant.

<< Previous: Writing Style - APA
Next: Poster Design >>
Last Updated: Feb 26, 2024 2:55 PM
URL: https://libguides.brenau.edu/BY499

Skip to main content
Skip to primary sidebar
Skip to footer
QuestionPro

Solutions Industries Gaming Automotive Sports and events Education Government Travel & Hospitality Financial Services Healthcare Cannabis Technology Use Case NPS+ Communities Audience Contactless surveys Mobile LivePolls Member Experience GDPR Positive People Science 360 Feedback Surveys
Resources Blog eBooks Survey Templates Case Studies Training Help center

Home Investigación de mercado

Non-experimental research: What it is, overview & advantages

Non-experimental research is the type of research that lacks an independent variable. Instead, the researcher observes the context in which the phenomenon occurs and analyzes it to obtain information.

Unlike experimental research , where the variables are held constant, non-experimental research happens during the study when the researcher cannot control, manipulate or alter the subjects but relies on interpretation or observations to conclude.

This means that the method must not rely on correlations, surveys , or case studies and cannot demonstrate an actual cause and effect relationship.

Characteristics of non-experimental research

Some of the essential characteristics of non-experimental research are necessary for the final results. Let’s talk about them to identify the most critical parts of them.

Most studies are based on events that occurred previously and are analyzed later.
In this method, controlled experiments are not performed for reasons such as ethics or morality.
No study samples are created; on the contrary, the samples or participants already exist and develop in their environment.
The researcher does not intervene directly in the environment of the sample.
This method studies the phenomena exactly as they occurred.

Types of non-experimental research

Non-experimental research can take the following forms:

Cross-sectional research : Cross-sectional research is used to observe and analyze the exact time of the research to cover various study groups or samples. This type of research is divided into:

Descriptive: When values are observed where one or more variables are presented.
Causal: It is responsible for explaining the reasons and relationship that exists between variables in a given time.

Longitudinal research: In a longitudinal study , researchers aim to analyze the changes and development of the relationships between variables over time. Longitudinal research can be divided into:

Trend: When they study the changes faced by the study group in general.
Group evolution: When the study group is a smaller sample.
Panel: It is in charge of analyzing individual and group changes to discover the factor that produces them.

LEARN ABOUT: Quasi-experimental Research

When to use non-experimental research

Non-experimental research can be applied in the following ways:

When the research question may be about one variable rather than a statistical relationship about two variables.
There is a non-causal statistical relationship between variables in the research question.
The research question has a causal research relationship, but the independent variable cannot be manipulated.
In exploratory or broad research where a particular experience is confronted.

Advantages and disadvantages

Some advantages of non-experimental research are:

It is very flexible during the research process
The cause of the phenomenon is known, and the effect it has is investigated.
The researcher can define the characteristics of the study group.

Among the disadvantages of non-experimental research are:

The groups are not representative of the entire population.
Errors in the methodology may occur, leading to research biases .

Non-experimental research is based on the observation of phenomena in their natural environment. In this way, they can be studied later to reach a conclusion.

Difference between experimental and non-experimental research

Experimental research involves changing variables and randomly assigning conditions to participants. As it can determine the cause, experimental research designs are used for research in medicine, biology, and social science.

Experimental research designs have strict standards for control and establishing validity. Although they may need many resources, they can lead to very interesting results.

Non-experimental research, on the other hand, is usually descriptive or correlational without any explicit changes done by the researcher. You simply describe the situation as it is, or describe a relationship between variables. Without any control, it is difficult to determine causal effects. The validity remains a concern in this type of research. However, it’s’ more regarding the measurements instead of the effects.

LEARN MORE: Descriptive Research vs Correlational Research

Whether you should choose experimental research or non-experimental research design depends on your goals and resources. If you need any help with how to conduct research and collect relevant data, or have queries regarding the best approach for your research goals, contact us today! You can create an account with our survey software and avail of 88+ features including dashboard and reporting for free.

Create a free account

MORE LIKE THIS

Life@QuestionPro: The Journey of Kristie Lawrence

Jun 7, 2024

We are on the front end of an innovation that can help us better predict how to transform our customer interactions.

How Can I Help You? — Tuesday CX Thoughts

Jun 5, 2024

Why Multilingual 360 Feedback Surveys Provide Better Insights

Jun 3, 2024

Raked Weighting: A Key Tool for Accurate Survey Results

May 31, 2024

Want to create or adapt books like this? Learn more about how Pressbooks supports open publishing practices.

7.1 Overview of Nonexperimental Research

Learning objectives.

Define nonexperimental research, distinguish it clearly from experimental research, and give several examples.
Explain when a researcher might choose to conduct nonexperimental research as opposed to experimental research.

What Is Nonexperimental Research?

Nonexperimental research is research that lacks the manipulation of an independent variable, random assignment of participants to conditions or orders of conditions, or both.

In a sense, it is unfair to define this large and diverse set of approaches collectively by what they are not . But doing so reflects the fact that most researchers in psychology consider the distinction between experimental and nonexperimental research to be an extremely important one. This is because while experimental research can provide strong evidence that changes in an independent variable cause differences in a dependent variable, nonexperimental research generally cannot. As we will see, however, this does not mean that nonexperimental research is less important than experimental research or inferior to it in any general sense.

When to Use Nonexperimental Research

As we saw in Chapter 6 “Experimental Research” , experimental research is appropriate when the researcher has a specific research question or hypothesis about a causal relationship between two variables—and it is possible, feasible, and ethical to manipulate the independent variable and randomly assign participants to conditions or to orders of conditions. It stands to reason, therefore, that nonexperimental research is appropriate—even necessary—when these conditions are not met. There are many ways in which this can be the case.

The research question or hypothesis can be about a single variable rather than a statistical relationship between two variables (e.g., How accurate are people’s first impressions?).
The research question can be about a noncausal statistical relationship between variables (e.g., Is there a correlation between verbal intelligence and mathematical intelligence?).
The research question can be about a causal relationship, but the independent variable cannot be manipulated or participants cannot be randomly assigned to conditions or orders of conditions (e.g., Does damage to a person’s hippocampus impair the formation of long-term memory traces?).
The research question can be broad and exploratory, or it can be about what it is like to have a particular experience (e.g., What is it like to be a working mother diagnosed with depression?).

Again, the choice between the experimental and nonexperimental approaches is generally dictated by the nature of the research question. If it is about a causal relationship and involves an independent variable that can be manipulated, the experimental approach is typically preferred. Otherwise, the nonexperimental approach is preferred. But the two approaches can also be used to address the same research question in complementary ways. For example, nonexperimental studies establishing that there is a relationship between watching violent television and aggressive behavior have been complemented by experimental studies confirming that the relationship is a causal one (Bushman & Huesmann, 2001). Similarly, after his original study, Milgram conducted experiments to explore the factors that affect obedience. He manipulated several independent variables, such as the distance between the experimenter and the participant, the participant and the confederate, and the location of the study (Milgram, 1974).

Types of Nonexperimental Research

As these examples make clear, single-variable research can answer interesting and important questions. What it cannot do, however, is answer questions about statistical relationships between variables. This is a point that beginning researchers sometimes miss. Imagine, for example, a group of research methods students interested in the relationship between children’s being the victim of bullying and the children’s self-esteem. The first thing that is likely to occur to these researchers is to obtain a sample of middle-school students who have been bullied and then to measure their self-esteem. But this would be a single-variable study with self-esteem as the only variable. Although it would tell the researchers something about the self-esteem of children who have been bullied, it would not tell them what they really want to know, which is how the self-esteem of children who have been bullied compares with the self-esteem of children who have not. Is it lower? Is it the same? Could it even be higher? To answer this question, their sample would also have to include middle-school students who have not been bullied.

The final way in which research can be nonexperimental is that it can be qualitative. The types of research we have discussed so far are all quantitative, referring to the fact that the data consist of numbers that are analyzed using statistical techniques. In qualitative research , the data are usually nonnumerical and are analyzed using nonstatistical techniques. Rosenhan’s study of the experience of people in a psychiatric ward was primarily qualitative. The data were the notes taken by the “pseudopatients”—the people pretending to have heard voices—along with their hospital records. Rosenhan’s analysis consists mainly of a written description of the experiences of the pseudopatients, supported by several concrete examples. To illustrate the hospital staff’s tendency to “depersonalize” their patients, he noted, “Upon being admitted, I and other pseudopatients took the initial physical examinations in a semipublic room, where staff members went about their own business as if we were not there” (Rosenhan, 1973, p. 256).

Internal Validity Revisited

Recall that internal validity is the extent to which the design of a study supports the conclusion that changes in the independent variable caused any observed differences in the dependent variable. Figure 7.1 shows how experimental, quasi-experimental, and correlational research vary in terms of internal validity. Experimental research tends to be highest because it addresses the directionality and third-variable problems through manipulation and the control of extraneous variables through random assignment. If the average score on the dependent variable in an experiment differs across conditions, it is quite likely that the independent variable is responsible for that difference. Correlational research is lowest because it fails to address either problem. If the average score on the dependent variable differs across levels of the independent variable, it could be that the independent variable is responsible, but there are other interpretations. In some situations, the direction of causality could be reversed. In others, there could be a third variable that is causing differences in both the independent and dependent variables. Quasi-experimental research is in the middle because the manipulation of the independent variable addresses some problems, but the lack of random assignment and experimental control fails to address others. Imagine, for example, that a researcher finds two similar schools, starts an antibullying program in one, and then finds fewer bullying incidents in that “treatment school” than in the “control school.” There is no directionality problem because clearly the number of bullying incidents did not determine which school got the program. However, the lack of random assignment of children to schools could still mean that students in the treatment school differed from students in the control school in some other way that could explain the difference in bullying.

Experiments are generally high in internal validity, quasi-experiments lower, and correlational studies lower still.

Key Takeaways

Nonexperimental research is research that lacks the manipulation of an independent variable, control of extraneous variables through random assignment, or both.
There are three broad types of nonexperimental research. Single-variable research focuses on a single variable rather than a relationship between variables. Correlational and quasi-experimental research focus on a statistical relationship but lack manipulation or random assignment. Qualitative research focuses on broader research questions, typically involves collecting large amounts of data from a small number of participants, and analyzes the data nonstatistically.
In general, experimental research is high in internal validity, correlational research is low in internal validity, and quasi-experimental research is in between.

Discussion: For each of the following studies, decide which type of research design it is and explain why.

A researcher conducts detailed interviews with unmarried teenage fathers to learn about how they feel and what they think about their role as fathers and summarizes their feelings in a written narrative.
A researcher measures the impulsivity of a large sample of drivers and looks at the statistical relationship between this variable and the number of traffic tickets the drivers have received.
A researcher randomly assigns patients with low back pain either to a treatment involving hypnosis or to a treatment involving exercise. She then measures their level of low back pain after 3 months.
A college instructor gives weekly quizzes to students in one section of his course but no weekly quizzes to students in another section to see whether this has an effect on their test performance.

Bushman, B. J., & Huesmann, L. R. (2001). Effects of televised violence on aggression. In D. Singer & J. Singer (Eds.), Handbook of children and the media (pp. 223–254). Thousand Oaks, CA: Sage.

Milgram, S. (1974). Obedience to authority: An experimental view . New York, NY: Harper & Row.

Rosenhan, D. L. (1973). On being sane in insane places. Science, 179 , 250–258.

Research Methods in Psychology Copyright © 2016 by University of Minnesota is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License , except where otherwise noted.

Summer is here, and so is the sale. Get a yearly plan with up to 65% off today! 🌴🌞

Form Builder
Survey Maker
AI Form Generator
AI Survey Tool
AI Quiz Maker
Store Builder
WordPress Plugin

HubSpot CRM

Google Sheets

Google Analytics

Microsoft Excel

Popular Forms
Job Application Form Template
Rental Application Form Template
Hotel Accommodation Form Template
Online Registration Form Template
Employment Application Form Template
Application Forms
Booking Forms
Consent Forms
Contact Forms
Donation Forms
Customer Satisfaction Surveys
Employee Satisfaction Surveys
Evaluation Surveys
Feedback Surveys
Market Research Surveys
Personality Quiz Template
Geography Quiz Template
Math Quiz Template
Science Quiz Template
Vocabulary Quiz Template

Try without registration Quick Start

Read engaging stories, how-to guides, learn about forms.app features.

Inspirational ready-to-use templates for getting started fast and powerful.

Spot-on guides on how to use forms.app and make the most out of it.

See the technical measures we take and learn how we keep your data safe and secure.

Integrations
Help Center
Sign In Sign Up Free
What is non-experimental research: Definition, types & examples

What is non-experimental research: Definition, types & examples

Defne Çobanoğlu

The experimentation method is very useful for getting information on a specific subject. However, when experimenting is not possible or practical, there is another way of collecting data for those interested. It's a non-experimental way, to say the least.

In this article, we have gathered information on non-experimental research, clearly defined what it is and when one should use it, and listed the types of non-experimental research. We also gave some useful examples to paint a better picture. Let us get started.

What is non-experimental research?

Non-experimental research is a type of research design that is based on observation and measuring instead of experimentation with randomly assigned participants.

What characterizes this research design is the fact that it lacks the manipulation of independent variables . Because of this fact, the non-experimental research is based on naturally occurring conditions, and there is no involvement of external interventions. Therefore, the researchers doing this method must not rely heavily on interviews, surveys , or case studies.

When to use non-experimental research?

An experiment is done when a researcher is investigating the relationship between one or two phenomena and has a theory or hypothesis on the relationship between two variables that are involved. The researcher can carry out an experiment when it is ethical, possible, and feasible to do one.

However, when an experiment can not be done because of a limitation, then they decide to opt for a non-experimental research design . Non-experimental research is considered preferable in some conditions, including:

When the manipulation of the independent variable is not possible because of ethical or practical concerns
When the subjects of an experimental design can not be randomly assigned to treatments.
When the research question is too extensive or it relates to a general experience.
When researchers want to do a starter research before investing in more extensive research.
When the research question is about the statistical relationship between variables , but in a noncausal context.
Characteristics of non-experimental research

Non-experimental research has some characteristics that clearly define the framework of this research method. They provide a clear distinction between experimental design and non-experimental design. Let us see some of them:

Non-experimental research does not involve the manipulation of variables .
The aim of this research type is to explore the factors as they naturally occur .
This method is used when experimentation is not possible because of ethical or practical reasons .
Instead of creating a sample or participant group, the existing groups or natural thresholds are used during the research.
This research method is not about finding causality between two variables.
Most studies are done on past events or historical occurrences to make sense of specific research questions.
Types of non-experimental research

Non-experimental research types

What makes research non-experimental research is the fact that the researcher does not manipulate the factors, does not randomly assign the participants, and observes the existing groups. But this research method can also be divided into different types. These types are:

Correlational research:

In correlation studies, the researcher does not manipulate the variables and is not interested in controlling the extraneous variables. They only observe and assess the relationship between them. For example, a researcher examines students’ study hours every day and their overall academic performance. The positive correlation this between study hours and academic performance suggests a statistical association.

Quasi-experimental research:

In quasi-experimental research, the researcher does not randomly assign the participants into two groups. Because you can not deliberately deprive someone of treatment, the researcher uses natural thresholds or dividing points . For example, examining students from two different high schools with different education methods.

Cross-sectional research:

In cross-sectional research, the researcher studies and compares a portion of a population at the same time . It does not involve random assignment or any outside manipulation. For example, a study on smokers and non-smokers in a specific area.

Observational research:

In observational research, the researcher once again does not manipulate any aspect of the study, and their main focus is observation of the participants . For example, a researcher examining a group of children playing in a playground would be a good example.

Non-experimental research examples

Non-experimental research is a good way of collecting information and exploring relationships between variables. It can be used in numerous fields, from social sciences, economics, psychology, education, and market research. When gathering information using secondary research is not enough and an experiment can not be done, this method can bring out new information.

Non-experimental research example #1

Imagine a researcher who wants to see the connection between mobile phone usage before bedtime and the amount of sleep adults get in a night . They can gather a group of individuals to observe and present them with some questions asking about the details of their day, frequency and duration of phone usage, quality of sleep, etc . And observe them by analyzing the findings.

Non-experimental research example #2

Imagine a researcher who wants to explore the correlation between job satisfaction levels among employees and what are the factors that affect this . The researcher can gather all the information they get about the employees’ ages, sexes, positions in the company, working patterns, demographic information, etc .

The research provides the researcher with all the information to make an analysis to identify correlations and patterns. Then, it is possible for researchers and administrators to make informed predictions.

Frequently asked questions about non-experimental research

When not to use non-experimental research?

There are some situations where non-experimental research is not suitable or the best choice. For example, the aim of non-experimental research is not about finding causality therefore, if the researcher wants to explore the relationship between two variables, then this method is not for them. Also, if the control over the variables is extremely important to the test of a theory, then experimentation is a more appropriate option.

What is the difference between experimental and non-experimental research?

Experimental research is an example of primary research where the researcher takes control of all the variables, randomly assigns the participants into different groups, and studies them in a pre-determined environment to test a hypothesis.

On the contrary, non-experimental research does not intervene in any way and only observes and studies the participants in their natural environments to make sense of a phenomenon

What makes a quasi-experiment a non-experiment?

The same as true experimentation, quasi-experiment research also aims to explore a cause-and-effect relationship between independent and dependent variables. However, in quasi-experimental research, the participants are not randomly selected. They are assigned to groups based on non-random criteria .

Is a survey a non-experimental study?

Yes, as the main purpose of a survey or questionnaire is to collect information from participants without outside interference, it makes the survey a non-experimental study. Surveys are used by researchers when experimentation is not possible because of ethical reasons, but first-hand data is needed

What is non-experimental data?

Non-experimental data is data collected by researchers via using non-experimental methods such as observations, interpretation, and interactions. Non-experimental data could both be qualitative or quantitative, depending on the situation.

Advantages of non-experimental research

Non-experimental research has its positive sides that a researcher should have in mind when going through a study. They can start their research by going through the advantages. These advantages are:

It is used to observe and analyze past events .
This method is more affordable than a true experiment .
As the researcher can adapt the methods during the study, this research type is more flexible than an experimental study.
This method allows the researchers to answer specific questions .

Disadvantages of non-experimental research

Even though non-experimental research has its advantages, it also has some disadvantages a researcher should be mindful of. Here are some of them:

The findings of non-experimental research can not be generalized to the whole population. Therefore, it has low external validity .
This research is used to explore only a single variable .
Non-experimental research designs are prone to researcher bias and may not produce neutral results.
Final words

A non-experimental study differs from an experimental study in that there is no intervention or change of internal or extraneous elements. It is a smart way to collect information without the limitations of experimentation. These limitations could be about ethical or practical problems. When you can not do proper experimentation, your other option is to study existing conditions and groups to draw conclusions. This is a non-experimental design .

In this article, we have gathered information on non-experimental research to shed light on the details of this research method. If you are thinking of doing a study, make sure to have this information in mind. And lastly, do not forget to visit our articles on other research methods and so much more!

Defne is a content writer at forms.app. She is also a translator specializing in literary translation. Defne loves reading, writing, and translating professionally and as a hobby. Her expertise lies in survey research, research methodologies, content writing, and translation.

Form Features
Data Collection

What is RAPID decision making model: Definition & examples

Fatih Serdar Çıtak

Predictive analysis: Definition, techniques & examples

Are customer service surveys effective?

Research Methods

Research Process
Research Design & Method

Qualitative vs. Quantiative

Correlational vs. experimental, empirical vs. non-empirical.

Survey Research
Survey & Interview Data Analysis
Resources for Research
Ethical Considerations in Research

Qualitative Research gathers data about lived experiences, emotions or behaviors, and the meanings individuals attach to them. It assists in enabling researchers to gain a better understanding of complex concepts, social interactions or cultural phenomena. This type of research is useful in the exploration of how or why things have occurred, interpreting events and describing actions.

Quantitative Research gathers numerical data which can be ranked, measured or categorized through statistical analysis. It assists with uncovering patterns or relationships, and for making generalizations. This type of research is useful for finding out how many, how much, how often, or to what extent.


: can be structured, semi-structured or unstructured.	: the same questions asked to large numbers of participants (e.g., Likert scale response) (see book below).
: several participants discussing a topic or set of questions.	: test hypothesis in controlled conditions (see video below).
: can be on-site, in-context, or role play (see video below).	: counting the number of times a phenomenon occurs or coding observed data in order to translate it into numbers.
: analysis of correspondence or reports.	: using numerical data from financial reports or counting word occurrences.
: memories told to a researcher.

Correlational Research cannot determine causal relationships. Instead they examine relationships between variables.

Experimental Research can establish causal relationship and variables can be manipulated.

Empirical Studies are based on evidence. The data is collected through experimentation or observation.

Non-empirical Studies do not require researchers to collect first-hand data.

<< Previous: Research Design & Method
Next: Survey Research >>
Last Updated: Apr 5, 2023 4:19 PM
URL: https://semo.libguides.com/ResearchMethods

Empirical Research: Advantages, Drawbacks and Differences with Non-Empirical Research

Based on the purpose and available resources, researchers conduct empirical or non-empirical research. Researchers employ both of these methods in various fields using qualitative, quantitative, or secondary data. Let's look at the characteristics of empirical research and see how it is different from non-empirical research.

The empirical study is evidence-based research. That is to say, it uses evidence, experiment, or observation to test the hypotheses. It is a systematic collection and analysis of data. Empirical research allows researchers to find new and thorough insights into the issue. Mariam-Webster dictionary defines the word "empirical" as:

"originating in or based on observation or experience"

"relying on experience or observation alone often without due regard for system and theory"

"capable of being verified or disproved by observation or experiment"

Unlike non-empirical research, it does not just rely on theories but also tries to find the reasoning behind those theories in order to prove them. Non-empirical research is based on theories and logic, and researchers don't attempt to test them. Although empirical research mostly depends on primary data, secondary data can also be beneficial for the theory side of the research. The empirical research process includes the following:

Defining the issue
Theory generation and research questions
If available, studying existing theories about the issue
Choosing appropriate data collection methods such as experiment or observation
Data gathering
Data coding , analysis, and evaluation
Data Interpretation and result
Reporting and publishing the findings

Benefits of empirical research

Empirical research aims to find the meaning behind a particular phenomenon. In other words, it seeks answers to how and why something works the way it is.
By identifying the reasons why something happens, it is possible to replicate or prevent similar events.
The flexibility of the research allows the researchers to change certain aspects of the research and adjust them to new goals.
It is more reliable because it represents a real-life experience and not just theories.
Data collected through empirical research may be less biased because the researcher is there during the collection process. In contrast, it is sometimes impossible to verify the accuracy of data in non-empirical research.

Drawbacks of empirical research

It can be time-consuming depending on the research subject.
It is not a cost-effective way of data collection in most cases because of the possible expensive methods of data gathering. Moreover, it may require traveling between multiple locations.
Lack of evidence and research subjects may not yield the desired result. A small sample size prevents generalization because it may not be enough to represent the target audience.
It isn't easy to get information on sensitive topics, and also, researchers may need participants' consent to use the data.

In most scientific fields, acting based solely on theories (or logic) is not enough. Empirical research makes it possible to measure the reliability of the theory before applying it. Researchers sometimes alternate between the two forms of research, as non-empirical research provides them with important information about the phenomenon, while empirical research helps them use that information to test the theory.

MassInitiative

Add custom text here or remove it

What is the difference between empirical and non-empirical research?

Table of Contents

1 What is the difference between empirical and non-empirical research?
2 What makes a research question empirical?
3 How do you know if research is empirical?
4 What is an example of empirical research?
5 What makes a study non-empirical?
6 What are the different types of empirical evidence?
7 What are 2 examples of empirical evidence?
8 Which is the best way to generate research questions?
9 Where do the ideas for research come from?
10 When do students ask questions about empirical research?

Definition: Empirical research is a research approach that makes use of evidence-based data while non-empirical research is a research approach that makes use of theoretical data.

What makes a research question empirical?

Empirical research is research that is based on observation and measurement of phenomena, as directly experienced by the researcher. The data thus gathered may be compared against a theory or hypothesis, but the results are still based on real life experience.

What is a non-empirical research question?

How do you know if research is empirical?

Characteristics of an Empirical Article:

Empirical articles will include charts, graphs, or statistical analysis.
Empirical research articles are usually substantial, maybe from 8-30 pages long.
There is always a bibliography found at the end of the article.

What is an example of empirical research?

An example of an empirical research would be if a researcher was interested in finding out whether listening to happy music promotes prosocial behaviour. An experiment could be conducted where one group of audience is exposed to happy music and the other is not exposed to music at all.

What are some examples of empirical evidence?

Examples of empirical evidence You hear about a new drug called atenolol that slows down the heart and reduces blood pressure. You use a priori reasoning to create a hypothesis that this drug might reduce the risk of a heart attack because it lowers blood pressure.

What makes a study non-empirical?

Non-empirical methods are the opposite, using current events, personal observations, and subjectivity to draw conclusions. Each of these evidence-gathering methods is relevant and acceptable, but when one is discounted over another, the results of the study might not be as valid as it could have been.

What are the different types of empirical evidence?

The two primary types of empirical evidence are qualitative evidence and quantitative evidence.

Qualitative. Qualitative evidence is the type of data that describes non-measurable information.
Quantitative.

What are the types of empirical research?

There are three major types of empirical research:

Quantitative Methods. e.g., numbers, mathematical equations).
Qualitative Methods. e.g., numbers, mathematical equations).
Mixed Methods (a mixture of Quantitative Methods and Qualitative Methods.

What are 2 examples of empirical evidence?

Examples of empirical evidence Imagine that you are a doctor and that you are interested in lowering blood pressure as a way to reduce the probability of having a heart attack. You hear about a new drug called atenolol that slows down the heart and reduces blood pressure.

Which is the best way to generate research questions?

How to turn an idea into a research question?

Where do the ideas for research come from?

When do students ask questions about empirical research.

Privacy Overview

Cookie	Duration	Description
cookielawinfo-checkbox-analytics	11 months	This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Analytics".
cookielawinfo-checkbox-functional	11 months	The cookie is set by GDPR cookie consent to record the user consent for the cookies in the category "Functional".
cookielawinfo-checkbox-necessary	11 months	This cookie is set by GDPR Cookie Consent plugin. The cookies is used to store the user consent for the cookies in the category "Necessary".
cookielawinfo-checkbox-others	11 months	This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Other.
cookielawinfo-checkbox-performance	11 months	This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Performance".
viewed_cookie_policy	11 months	The cookie is set by the GDPR Cookie Consent plugin and is used to store whether or not user has consented to the use of cookies. It does not store any personal data.

Want to create or adapt books like this? Learn more about how Pressbooks supports open publishing practices.

1.6 Reading, understanding and writing up non-empirical articles

SOWK 621: DeCarlo

Reading and Outlining a Non-empirical Journal Article

Explanation

For this course you will have to read a lot of academic journal articles. The goal of this exercise is to build skills on how to extract the information you need from each article you come across as efficiently as possible. This assignment is designed for you to explore in greater detail aspects of the Human Service field that are interesting to you. After conducting a literature search using an academic database (Google Scholar, Academic Search Complete, PubMed, PSYCinfo), choose one article that you find interesting and want to read.

Non-empirical articles do not have a specific structure like empirical articles. Instead, authors organize their articles by topic and subtopic. Non-empirical articles include articles about social theory, history, philosophy, and literature reviews.

Go to the Penn State Fayette Library page and search for a Non-empircal journal article. If you need assistance review the following links

Find Box and Searching
Fayette Library Homepage
University Libraries Homepage, My Account, Ask a Librarian
Using the Catalog or click on University Libraries which have some helpful tutorials

Please create a Word document and submit the following to D2L.

Write out the citation to the article in APA format. (Google Scholar will give you a citation that is correct about 80% of the time, you should double-check it.)
General Idea:
Facts from the Literature
(I usually copy the sentence that the author writes with the internal citation at the end so I remember what the original source is.)
Example: 73 people per year are killed by wombats (Ambrose, 1992).
Facts from the author
Sources of Interest:
From the references, copy all of the citations for any articles you included in the Facts from Other Sources section or that you might find useful in your paper.
Why would someone seek out this source? What questions would they try to answer?
How does this resource build upon, challenge, or relate to other literature on the topic?
Why do you think this is a reputable source from a competent and trustworthy author?

NOTE: In the future, I will refer to these notes as a “Raw Outline.”

Share This Book

Empirical Research

Reference work entry
First Online: 01 January 2020
Cite this reference work entry

Emeka Thaddues Njoku 3

115 Accesses

The term “empirical” entails gathered data based on experience, observations, or experimentation. In empirical research, knowledge is developed from factual experience as opposed to theoretical assumption and usually involved the use of data sources like datasets or fieldwork, but can also be based on observations within a laboratory setting. Testing hypothesis or answering definite questions is a primary feature of empirical research. Empirical research, in other words, involves the process of employing working hypothesis that are tested through experimentation or observation. Hence, empirical research is a method of uncovering empirical evidence.

Through the process of gathering valid empirical data, scientists from a variety of fields, ranging from the social to the natural sciences, have to carefully design their methods. This helps to ensure quality and accuracy of data collection and treatment. However, any error in empirical data collection process could inevitably render such...

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Institutional subscriptions

Bibliography

Bhattacherjee, A. (2012). Social science research: Principles, methods, and practices. Textbooks Collection . Book 3.

Google Scholar

Comte, A., & Bridges, J. H. (Tr.) (1865). A general view of positivism . Trubner and Co. (reissued by Cambridge University Press, 2009).

Dilworth, C. B. (1982). Empirical research in the literature class. English Journal, 71 (3), 95–97.

Article Google Scholar

Heisenberg, W. (1971). Positivism, metaphysics and religion. In R. N. Nanshen (Ed.), Werner Heisenberg – Physics and beyond – Encounters and conversations , World Perspectives. 42. Translator: Arnold J. Pomerans. New York: Harper and Row.

Hossain, F. M. A. (2014). A critical analysis of empiricism. Open Journal of Philosophy, 2014 (4), 225–230.

Kant, I. (1783). Prolegomena to any future metaphysic (trans: Bennett, J.). Early Modern Texts. www.earlymoderntexts.com

Koch, S. (1992). Psychology’s Bridgman vs. Bridgman’s Bridgman: An essay in reconstruction. Theory and Psychology, 2 (3), 261–290.

Matin, A. (1968). An outline of philosophy . Dhaka: Mullick Brothers.

Mcleod, S. (2008). Psychology as science. http://www.simplypsychology.org/science-psychology.html

Popper, K. (1963). Conjectures and refutations: The growth of scientific knowledge . London: Routledge.

Simmel, G. (1908). The problem areas of sociology in Kurt H. Wolf: The sociology of Georg Simmel . London: The Free Press.

Weber, M. (1991). The nature of social action. In W. G. Runciman (Ed.), Weber: Selections in translation . Cambridge: Cambridge University Press.

Download references

Author information

Authors and affiliations.

Department of Political Science, University of Ibadan, Ibadan, Oyo, Nigeria

Emeka Thaddues Njoku

You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Emeka Thaddues Njoku .

Editor information

Editors and affiliations.

University of Connecticut, Storrs, CT, USA

David A. Leeming

Blanton-Peale Institute, New York, NY, USA

Rights and permissions

Reprints and permissions

Copyright information

About this entry

Cite this entry.

Njoku, E.T. (2020). Empirical Research. In: Leeming, D.A. (eds) Encyclopedia of Psychology and Religion. Springer, Cham. https://doi.org/10.1007/978-3-030-24348-7_200051

Download citation

DOI : https://doi.org/10.1007/978-3-030-24348-7_200051

Published : 12 June 2020

Publisher Name : Springer, Cham

Print ISBN : 978-3-030-24347-0

Online ISBN : 978-3-030-24348-7

eBook Packages : Behavioral Science and Psychology Reference Module Humanities and Social Sciences Reference Module Business, Economics and Social Sciences

Share this entry

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Publish with us

Policies and ethics

Find a journal
Track your research

Cambridge Dictionary +Plus

Meaning of non-empirical in English

Your browser doesn't support HTML5 audio

The new system is meant to offer a clean break from the highly theoretical , nonempirical diagnostic practices of the past.
Scholars' preference for empirical or non-empirical methods of research depends on their beliefs regarding the nature of knowledge .
The experiment is supposed to be a non-empirical, intuitive exploration into how different people respond .
abstraction
accepted wisdom
afterthought
anthropocentrism
determinist
non-dogmatic
social Darwinism
supersensible
the domino theory

Translations of non-empirical

Get a quick, free translation!

Word of the Day

at the same time

Worse than or worst of all? How to use the words ‘worse’ and ‘worst’

Learn more with +Plus

Recent and Recommended {{#preferredDictionaries}} {{name}} {{/preferredDictionaries}}
Definitions Clear explanations of natural written and spoken English English Learner’s Dictionary Essential British English Essential American English
Grammar and thesaurus Usage explanations of natural written and spoken English Grammar Thesaurus
Pronunciation British and American pronunciations with audio English Pronunciation
English–Chinese (Simplified) Chinese (Simplified)–English
English–Chinese (Traditional) Chinese (Traditional)–English
English–Dutch Dutch–English
English–French French–English
English–German German–English
English–Indonesian Indonesian–English
English–Italian Italian–English
English–Japanese Japanese–English
English–Norwegian Norwegian–English
English–Polish Polish–English
English–Portuguese Portuguese–English
English–Spanish Spanish–English
English–Swedish Swedish–English
Dictionary +Plus Word Lists
English Adjective
Translations
All translations

To add non-empirical to a word list please sign up or log in.

Add non-empirical to one of your lists below, or create a new one.

Something went wrong.

There was a problem sending your report.

There Is No Pure Empirical Reasoning

As the title says, there is no such thing as pure empirical reasoning.*

[ *Based on: “ There Is No Pure Empirical Reasoning ,” Philosophy and Phenomenological Research 95 (2017): 592-613. ]

1. The Issue

Empiricists think that all substantive knowledge about the world must be justified (directly or indirectly) by observation. This is taken to mean there is no synthetic, a priori knowledge.

By empirical reasoning , I mean a kind of (i) nondeductive reasoning (ii) from observations that (iii) provides adequate justification for its conclusion. The paradigms are induction and scientific reasoning in general.

Empirical reasoning is pure when it does not depend upon any synthetic, a priori inputs (i.e., a priori justification for any synthetic claim).

Empiricists would claim that all empirical reasoning is pure. I claim that no empirical reasoning is pure; empirical reasoning always depends on substantive a priori input. Empiricism is therefore untenable.

2. Empiricism Has No Coherent Account of Empirical Reasons

2.1. the need for background probabilities.

Say you have a hypothesis H and evidence E. Bayes’ Theorem tells us:

P(H|E) = P(H)*P(E|H) / [P(H)*P(E|H) + P(~H)*P(E|~H)]

To determine the probability of the hypothesis in the light of the evidence, you need to first know the prior probability of the hypothesis, P(H), plus the conditional probabilities, P(E|H) and P(E|~H). Note a few things about this:

This is substantive (non-analytic) information. There will in general (except in a measure-zero class of cases) be coherent probability distributions that assign any values between 0 and 1 to each of these probabilities.

This information is not observational. You cannot see a probability with your eyes.

These probabilities cannot, on pain of infinite regress, always be arrived at by empirical reasoning.

So you need substantive, non-empirical information in order to do empirical reasoning.

This argument doesn’t have any unreasonable assumptions. I’m not assuming that probability theory tells us everything about evidential support, nor that there are always perfectly precise probabilities for everything. I’m only assuming that, when a hypothesis is adequately justified by some evidence, there is an objective fact that that hypothesis isn’t improbable on that evidence.

Now for some examples:

Compare two hypotheses about the origin of your sensory experiences:

RWH: You’re a normal person perceiving the real world.

BIVH: You’re a brain in a vat who is being fed a perfect simulation of the real world.

These theories predict exactly the same sensory experiences, so P(E|H) would be the same. Yet obviously you should believe RWH, not BIVH. This can only be because RWH has a higher prior probability. It can’t be that you learned that ~BIVH, or that P(BIVH) is low, empirically , since all your evidence is exactly the way it would be if BIVH were true. It must be that BIVH has a low a priori probability.

Precognition

In 2011, the psychologist Daryl Bem published a paper reporting evidence for a kind of precognition involving backwards causation. He had done nine experiments, in which eight showed statistically significant evidence for precognition.

When I first heard this, my reaction was skeptical, to say the least. I did not then accept precognition, nor did I even withhold judgement. Rather, I continued to disbelieve in precognition. I thought there must be something wrong with the experiments, or that the results had been obtained by luck. (This case illustrates the unreliability of currently accepted statistical methodology—but that is a story for another time.)

That is how rational people in general reacted. But we would not have reacted in that way to other results; e.g., if a study found that people tend to be happier on sunny days, we would have accepted the results at face value. This case illustrates that the rational reaction to some evidence depends upon the prior probability of the hypothesis that the evidence is said to support. Precognition is just so unlikely on its face that this evidence isn’t enough to justify believing in it.

You observe a lot of green emeralds. You then infer that it’s at least likely now that all emeralds are green . However, you do not infer that it’s at all likely that all emeralds are grue , even though that inference would be formally parallel.

[Definition: An object is grue iff (it is first observed before 2025 A.D. and it is green, or it is not observed before 2025 A.D. and it is blue).]

This case illustrates different conditional probabilities: the probability of unobserved emeralds being green , given that observed emeralds have been green, is higher than the probability of unobserved emeralds being grue given that observed emeralds have been grue.

This fact about conditional probabilities is, again, a priori, not empirical. This sort of thing can’t in general be learned by empirical reasoning, because you need this information in order to make any empirical inferences.

2.2. My Argument > Russell’s Argument

Bertrand Russell also defended rationalism by appealing to empirical reasoning. He said that to make inductive inferences, you have to know the correct rules of induction. That knowledge could not itself be arrived at by induction, on pain of circularity. It also can’t be gained by observation. So it must be a priori.

Against Russell’s argument, empiricists could say that you do not need to know the rules of inference in order to gain knowledge by reasoning; to think that you do is to confuse inference rules with premises . Rather, to gain inferential knowledge, you only need to know your premises, and then be disposed to follow the actually correct rules in reasoning from those premises.

Some empiricists have in fact said this, and this is the main response, as far as I know, to Russell’s argument. Some also claim that rule circularity (unlike premise circularity) is okay, so you can use inference to the best explanation in drawing the conclusion that inference to the best explanation is a good form of reasoning.

Well, you may or may not agree with that response. But in any case, my argument avoids it, which makes it better than Russell’s argument. My argument could not be accused of confusing inference rules with premises, because I am not saying that you need to have a priori knowledge about rules of inference. I am saying that you need to have a priori prior probabilities for hypotheses. This is really a lot more like having a priori knowledge of premises than it is like having a priori knowledge of rules of inference.

2.3. Skepticism Is Irrational

One could not plausibly respond by just rejecting empirical reasoning (as Hume did). First, because inductive skepticism is ridiculous. It’s ridiculous to deny that we have more reason to think that water is made of hydrogen and oxygen than we have to think that it’s made of uranium and chlorine.

Second, the success of modern science is perhaps the main thing that motivated empiricism in the first place. It would be irrational then to reject all scientific reasoning just so you can cling to empiricism.

3. Subjective Bayesianism Won’t Save You

Subjective Bayesians think that it’s rationally permissible to start with any coherent set of initial probabilities, and then just update your beliefs by conditionalizing on whatever evidence you get. (To conditionalize, when you receive evidence E, you have to set your new P(H) to what was previously your P(H|E).) On this view, people can have very different degrees of belief, given the same evidence, and yet all be perfectly rational.

Subjective Bayesians sometimes try to make this sound better by appealing to convergence theorems. These show, roughly, that as you get more evidence, the effect of differing prior probabilities tends to wash out. I.e., with enough evidence, people with different priors will still tend to converge on the correct beliefs.

The problem is that there is no amount of evidence that, on the subjective Bayesian view, would make all rational observers converge. No matter how much evidence you have for a theory at any given time, there are still prior probabilities that would result in someone continuing to reject the theory in the light of that evidence. So subjectivists cannot account for the fact that, e.g., it would be definitely irrational, given our current evidence, for someone to believe that the Earth rests on the back of a giant turtle.

The point is illustrated by the following graph, which shows how the posterior probability of a hypothesis is related to its prior probability, for different values of the likelihood ratio, L.

Here, L is defined as P(E|H)/P(E|~H). (You can determine P(H|E) based on just two pieces of information, P(H) and L.) What you see in the graph is that the posterior probability is a (continuous, one-one) function of the prior probability. When L is greater than 1, then the curve is pulled upward, indicating that the posterior probability is greater than the prior, meaning that the hypothesis is supported . When L is <1, the curve is pulled downward, indicating disconfirmation.

But here is the important point: No matter what the value of L is, the graph is always a one-one function from the interval [0,1] onto the interval [0,1]. Thus, if the prior probability is completely unconstrained, then the posterior probability is also completely unconstrained. That is, if P(H) is allowed to be anything between 0 and 1, then P(H|E) can also be anything between 0 and 1.

Gathering more evidence doesn’t change this qualitative fact; gathering more evidence just gives you (typically) a more extreme likelihood ratio. But that still leaves you with the full range of possible posterior probabilities if you have the full range of possible priors going in.

Subjective Bayesians have no way of restricting the range of possible priors (that’s just their view). So they have no way of saying, for example, that it is an objective fact that we have good reason to think water is made of hydrogen and oxygen, or that it is objectively unreasonable, given our current evidence, to think that the Earth rests on the back of a giant turtle.

4. A Rationalist View

Rationalists believe that we have some a priori justification for some substantive (non-analytic) claims. In particular, we have a priori prior probabilities, which enable us to make empirical inferences.

The biggest problem with this view: How do we assign those prior probabilities? In many cases, it just is not at all obvious what they are. E.g., what is the a priori prior probability that water would be composed of hydrogen and oxygen? Or that life would have evolved by natural selection? No one knows how to answer that.

I’m not going to answer that here, either. But I’ll say one thing to make you feel better about not knowing the a priori prior probabilities of things: We don’t need to have perfectly precise a priori probabilities for every proposition. Rather, it’s enough if we can just say that there is a limited range (less than the full range from 0 to 1) of prior probabilities for a hypothesis that are rational. E.g., to do empirical reasoning about the Theory of Evolution, I don’t have to know exactly what the prior probability of Evolution is. I might just be able to say that its prior probability is more than 1 in a trillion, and less than 90%.

How could this be enough? Because in general, if you start with a restricted range of prior probabilities, then as you gather evidence, the range of allowable posterior probabilities in light of that evidence shrinks . The more evidence you collect, the narrower it gets. This is why, even though I have very little idea what the prior probability of the Theory of Evolution was, I have a good idea that its current probability, on my evidence, is well over 90%.

I have another diagram that illustrates the idea:

Suppose that I just know that the prior probability of a hypothesis is between 0.1 and 0.9. But then I collect a lot of evidence for it, so I build up a likelihood ration of 100. In that case, the posterior probability, P(H|E), is constrained to be between 0.917 and 0.999.

In general, if you get enough evidence, then you have a good idea what the posterior probability is, even if you had almost no idea what the prior was.

Liked by Michael Huemer

Ready for more?

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

View all journals
Explore content
About the journal
Publish with us
Sign up for alerts
Perspective
Published: 05 June 2024

Misunderstanding the harms of online misinformation

Ceren Budak ORCID: orcid.org/0000-0002-7767-3217 1 ,
Brendan Nyhan ORCID: orcid.org/0000-0001-7497-1799 2 ,
David M. Rothschild ORCID: orcid.org/0000-0002-7792-1989 3 ,
Emily Thorson ORCID: orcid.org/0000-0002-6514-801X 4 &
Duncan J. Watts ORCID: orcid.org/0000-0001-5005-4961 5

Nature volume 630 , pages 45–53 ( 2024 ) Cite this article

1766 Accesses

374 Altmetric

Metrics details

Communication

The controversy over online misinformation and social media has opened a gap between public discourse and scientific research. Public intellectuals and journalists frequently make sweeping claims about the effects of exposure to false content online that are inconsistent with much of the current empirical evidence. Here we identify three common misperceptions: that average exposure to problematic content is high, that algorithms are largely responsible for this exposure and that social media is a primary cause of broader social problems such as polarization. In our review of behavioural science research on online misinformation, we document a pattern of low exposure to false and inflammatory content that is concentrated among a narrow fringe with strong motivations to seek out such information. In response, we recommend holding platforms accountable for facilitating exposure to false and extreme content in the tails of the distribution, where consumption is highest and the risk of real-world harm is greatest. We also call for increased platform transparency, including collaborations with outside researchers, to better evaluate the effects of online misinformation and the most effective responses to it. Taking these steps is especially important outside the USA and Western Europe, where research and data are scant and harms may be more severe.

This is a preview of subscription content, access via your institution

Access options

Access Nature and 54 other Nature Portfolio journals

Get Nature+, our best-value online-access subscription

24,99 € / 30 days

cancel any time

Subscribe to this journal

Receive 51 print issues and online access

185,98 € per year

only 3,65 € per issue

Buy this article

Purchase on Springer Link
Instant access to full article PDF

Prices may be subject to local taxes which are calculated during checkout

Toolbox of individual-level interventions against online misinformation

Exposure to untrustworthy websites in the 2016 US election

Psychological inoculation protects against the social media infodemic

Myers, S. L. How social media amplifies misinformation more than information. The New York Times , https://www.nytimes.com/2022/10/13/technology/misinformation-integrity-institute-report.html (13 October 2022).

Haidt, J. Why the past 10 years of American life have been uniquely stupid. The Atlantic , https://www.theatlantic.com/magazine/archive/2022/05/social-media-democracy-trust-babel/629369/ (11 April 2022).

Haidt, J. Yes, social media really is undermining democracy. The Atlantic , https://www.theatlantic.com/ideas/archive/2022/07/social-media-harm-facebook-meta-response/670975/ (28 July 2022).

Tufekci, Z. YouTube, the great radicalizer. The New York Times , https://www.nytimes.com/2018/03/10/opinion/sunday/youtube-politics-radical.html (10 March 2018).

Romer, P. A tax that could fix big tech. The New York Times , https://www.nytimes.com/2019/05/06/opinion/tax-facebook-google.html (6 May 2019).

Schnell, M. Clyburn blames polarization on the advent of social media. The Hill , https://thehill.com/homenews/sunday-talk-shows/580440-clyburn-says-polarization-is-at-its-worst-because-the-advent-of/ (7 November 2021).

Robert F. Kennedy Human Rights/AP-NORC Poll (AP/NORC, 2023).

Goeas, E. & Nienaber, B. Battleground Poll 65: Civility in Politics: Frustration Driven by Perception (Tarrance Group, 2019).

Murray, M. Poll: Nearly two-thirds of Americans say social media platforms are tearing us apart. NBC News , https://www.nbcnews.com/politics/meet-the-press/poll-nearly-two-thirds-americans-say-social-media-platforms-are-n1266773 (2021).

Auxier, B. 64% of Americans say social media have a mostly negative effect on the way things are going in the U.S. today. Pew Research Center (2020).

Koomey, J. G. et al. Sorry, wrong number: the use and misuse of numerical facts in analysis and media reporting of energy issues. Annu. Rev. Energy Env. 27 , 119–158 (2002).

Article Google Scholar

Gonon, F., Bezard, E. & Boraud, T. Misrepresentation of neuroscience data might give rise to misleading conclusions in the media: the case of attention deficit hyperactivity disorder. PLoS ONE 6 , e14618 (2011).

Article ADS CAS PubMed PubMed Central Google Scholar

Copenhaver, A., Mitrofan, O. & Ferguson, C. J. For video games, bad news is good news: news reporting of violent video game studies. Cyberpsychol. Behav. Soc. Netw. 20 , 735–739 (2017).

Article PubMed Google Scholar

Bratton, L. et al. The association between exaggeration in health-related science news and academic press releases: a replication study. Wellcome Open Res. 4 , 148 (2019).

Article PubMed PubMed Central Google Scholar

Allcott, H., Braghieri, L., Eichmeyer, S. & Gentzkow, M. The welfare effects of social media. Am. Econ. Rev. 110 , 629–676 (2020).

Braghieri, L., Levy, R. & Makarin, A. Social media and mental health. Am. Econ. Rev. 112 , 3660–3693 (2022).

Guess, A. M., Barberá, P., Munzert, S. & Yang, J. The consequences of online partisan media. Proc. Natl Acad. Sci. USA 118 , e2013464118 (2021).

Article CAS PubMed PubMed Central Google Scholar

Sabatini, F. & Sarracino, F. Online social networks and trust. Soc. Indic. Res. 142 , 229–260 (2019).

Lorenz-Spreen, P., Lewandowsky, S., Sunstein, C. R. & Hertwig, R. How behavioural sciences can promote truth, autonomy and democratic discourse online. Nat. Hum. Behav. 4 , 1102–1109 (2020). This paper provides a review of possible harms from social media .

Lapowsky, I. The mainstream media melted down as fake news festered. Wired , https://www.wired.com/2016/12/2016-mainstream-media-melted-fake-news-festered/ (26 December 2016).

Lalani, F. & Li, C. Why So Much Harmful Content Has Proliferated Online—and What We Can Do about It Technical Report (World Economic Forum, 2020).

Stewart, E. America’s growing fake news problem, in one chart. Vox , https://www.vox.com/policy-and-politics/2020/12/22/22195488/fake-news-social-media-2020 (22 December 2020).

Sanchez, G. R., Middlemass, K. & Rodriguez, A. Misinformation Is Eroding the Public’s Confidence in Democracy (Brookings Institution, 2022).

Bond, S. False Information Is Everywhere. ‘Pre-bunking’ Tries to Head It off Early. NPR , https://www.npr.org/2022/10/28/1132021770/false-information-is-everywhere-pre-bunking-tries-to-head-it-off-ear (National Public Radio, 2022).

Tufekci, Z. Algorithmic harms beyond Facebook and google: emergent challenges of computational agency. Colo. Tech. Law J. 13 , 203 (2015).

Google Scholar

Cohen, J. N. Exploring echo-systems: how algorithms shape immersive media environments. J. Media Lit. Educ. 10 , 139–151 (2018).

Shin, J. & Valente, T. Algorithms and health misinformation: a case study of vaccine books on Amazon. J. Health Commun. 25 , 394–401 (2020).

Ceylan, G., Anderson, I. A. & Wood, W. Sharing of misinformation is habitual, not just lazy or biased. Proc. Natl Acad. Sci. USA 120 , e2216614120 (2023).

Pauwels, L., Brion, F. & De Ruyver, B. Explaining and Understanding the Role of Exposure to New Social Media on Violent Extremism. an Integrative Quantitative and Qualitative Approach (Belgian Science Policy, 2014).

McHugh, B. C., Wisniewski, P., Rosson, M. B. & Carroll, J. M. When social media traumatizes teens: the roles of online risk exposure, coping, and post-traumatic stress. Internet Res. 28 , 1169–1188 (2018).

Soral, W., Liu, J. & Bilewicz, M. Media of contempt: social media consumption predicts normative acceptance of anti-Muslim hate speech and Islamo-prejudice. Int. J. Conf. Violence 14 , 1–13 (2020).

Many believe misinformation is increasing extreme political views and behaviors. AP-NORC https://apnorc.org/projects/many-believe-misinformation-is-increasing-extreme-political-views-an (2022).

Fandos, N., Kang, C. & Isaac, M. Tech executives are contrite about election meddling, but make few promises on Capitol Hill. The New York Times , https://www.nytimes.com/2017/10/31/us/politics/facebook-twitter-google-hearings-congress.html (31 October 2017).

Eady, G., Paskhalis, T., Zilinsky, J., Bonneau, R., Nagler, J. & Tucker, J. A. Exposure to the Russian Internet Research Agency foreign influence campaign on Twitter in the 2016 US election and its relationship to attitudes and voting behavior. Nat. Commun. 14 , 62 (2023). This paper shows that exposure to Russian misinformation on social media in 2016 was a small portion of people’s news diets and not associated with shifting attitudes.

Badawy, A., Addawood, A., Lerman, K. & Ferrara, E. Characterizing the 2016 Russian IRA influence campaign. Soc. Netw. Anal. Min. 9 , 31 (2019). This paper shows that exposure to and amplification of Russian misinformation on social media in 2016 was concentrated among Republicans (who would have been predisposed to support Donald Trump regardless) .

Hosseinmardi, H., Ghasemian, A., Clauset, A., Mobius, M., Rothschild, D. M. & Watts, D. J. Examining the consumption of radical content on YouTube. Proc. Natl Acad. Sci. USA 118 , e2101967118 (2021). This paper shows that extreme content is consumed on YouTube by a small portion of the population who tend to consume similar content elsewhere online and that consumption is largely driven by demand, not algorithms .

Chen, A. Y., Nyhan, B., Reifler, J., Robertson, R. E. & Wilson, C. Subscriptions and external links help drive resentful users to alternative and extremist YouTube channels. Sci. Adv. 9 , eadd8080 (2023). This paper shows that people who consume extremist content on YouTube have highly resentful attitudes and typically find the content through subscriptions and external links, not algorithmic recommendations to non-subscribers .

Munger, K. & Phillips, J. Right-wing YouTube: a supply and demand perspective. Int. J. Press Polit. 27 , 186–219 (2022).

Lasser, J., Aroyehun, S. T., Simchon, A., Carrella, F., Garcia, D. & Lewandowsky, S. Social media sharing of low-quality news sources by political elites. PNAS Nexus 1 , pgac186 (2022).

Muddiman, A., Budak, C., Murray, C., Kim, Y. & Stroud, N. J. Indexing theory during an emerging health crisis: how U.S. TV news indexed elite perspectives and amplified COVID-19 misinformation. Ann. Inte. Commun. Assoc. 46 , 174–204 (2022). This paper shows how mainstream media also spreads misinformation through amplification of misleading statements from elites .

Pereira, F. B. et al. Detecting misinformation: identifying false news spread by political leaders in the Global South. Preprint at OSF , https://doi.org/10.31235/osf.io/hu4qr (2022).

Horwitz, J. & Seetharaman, D. Facebook executives shut down efforts to make the site less divisive. Wall Street Journal , https://www.wsj.com/articles/facebook-knows-it-encourages-division-top-executives-nixed-solutions-11590507499 (26 May 2020).

Hosseinmardi, H., Ghasemian, A., Rivera-Lanas, M., Horta Ribeiro, M., West, R. & Watts, D. J. Causally estimating the effect of YouTube’s recommender system using counterfactual bots. Proc. Natl Acad. Sci. USA 121 , e2313377121 (2024).

Article CAS PubMed Google Scholar

Nyhan, B. et al. Like-minded sources on facebook are prevalent but not polarizing. Nature 620 , 137–144 (2023).

Guess, A. M. et al. How do social media feed algorithms affect attitudes and behavior in an election campaign? Science 381 , 398–404 (2023). This paper shows that algorithms supply less untrustworthy content than reverse chronological feeds .

Article ADS CAS PubMed Google Scholar

Asimovic, N., Nagler, J., Bonneau, R. & Tucker, J. A. Testing the effects of Facebook usage in an ethnically polarized setting. Proc. Natl Acad. Sci. USA 118 , e2022819118 (2021).

Allen, J., Mobius, M., Rothschild, D. M. & Watts, D. J. Research note: Examining potential bias in large-scale censored data. Harv. Kennedy Sch. Misinformation Rev. 2 , https://doi.org/10.37016/mr-2020-74 (2021). This paper shows that engagement metrics such as clicks and shares that are regularly used in popular and academic research do not take into account the fact that fake news is clicked and shared at a higher rate relative to exposure and viewing than non-fake news .

Scheuerman, M. K., Jiang, J. A., Fiesler, C. & Brubaker, J. R. A framework of severity for harmful content online. Proc. ACM Hum. Comput. Interact. 5 , 1–33 (2021).

Vosoughi, S., Roy, D. & Aral, S. The spread of true and false news online. Science 359 , 1146–1151 (2018).

Roy, D. Happy to see the extensive coverage of our science paper on spread of true and false news online, but over-interpretations of the scope of our study prompted me to diagram actual scope (caution, not to scale!). Twitter , https://twitter.com/dkroy/status/974251282071474177 (15 March 2018).

Greenemeier, L. You can’t handle the truth—at least on Twitter. Scientific American , https://www.scientificamerican.com/article/you-cant-handle-the-truth-at-least-on-twitter/ (8 March 2018).

Frankel, S. Deceptively edited video of Biden proliferates on social media. The New York Times , https://www.nytimes.com/2020/11/02/technology/biden-video-edited.html (2 November 2020).

Jiameng P. et al. Deepfake videos in the wild: analysis and detection. In Proc. Web Conference 2021 981–992 (International World Wide Web Conference Committee, 2021).

Widely Viewed Content Report: What People See on Facebook: Q1 2023 Report (Facebook, 2023).

Mayer, J. How Russia helped swing the election for Trump. The New Yorker , https://www.newyorker.com/magazine/2018/10/01/how-russia-helped-to-swing-the-election-for-trump (24 September 2018).

Jamieson, K. H. Cyberwar: How Russian Hackers and Trolls Helped Elect A President: What We Don’t, Can’t, and Do Know (Oxford Univ. Press, 2020).

Solon, O. & Siddiqui, S. Russia-backed Facebook posts ‘reached 126m Americans’ during US election. The Guardian , https://www.theguardian.com/technology/2017/oct/30/facebook-russia-fake-accounts-126-million (30 October 2017).

Watts, D. J. & Rothschild, D. M. Don’t blame the election on fake news. Blame it on the media. Columbia J. Rev. 5 , https://www.cjr.org/analysis/fake-news-media-election-trump.php (2017). This paper explores how seemingly large exposure levels to problematic content actually represent a small proportion of total news exposure .

Jie, Y. Frequency or total number? A comparison of different presentation formats on risk perception during COVID-19. Judgm. Decis. Mak. 17 , 215–236 (2022).

Reyna, V. F. & Brainerd, C. J. Numeracy, ratio bias, and denominator neglect in judgments of risk and probability. Learn. Individ. Differ. 18 , 89–107 (2008). This paper details research into how salient numbers can lead to confusion in judgements of risk and probability, such as denominator neglect in which people fixate on a large numerator and do not consider the appropriate denominator .

Jones, J. Americans: much misinformation, bias, inaccuracy in news. Gallup , https://news.gallup.com/opinion/gallup/235796/americans-misinformation-bias-inaccuracy-news.aspx (2018).

Grinberg, N., Joseph, K., Friedland, L., Swire-Thompson, B. & Lazer, D. Fake news on Twitter during the 2016 US presidential election. Science 363 , 374–378 (2019).

Guess, A. M., Nyhan, B. & Reifler, J. Exposure to untrustworthy websites in the 2016 US election. Nat. Hum. Behav. 4 , 472–480 (2020). This paper shows untrustworthy news exposure was relatively rare in US citizens’ web browsing in 2016 .

Altay, S., Nielsen, R. K. & Fletcher, R. Quantifying the “infodemic”: people turned to trustworthy news outlets during the 2020 coronavirus pandemic. J. Quant. Descr. Digit. Media 2 , 1–30 (2022).

Allen, J., Howland, B., Mobius, M., Rothschild, D. & Watts, D. J. Evaluating the fake news problem at the scale of the information ecosystem. Sci. Adv. 6 , eaay3539 (2020). This paper shows that exposure to fake news is a vanishingly small part of people’s overall news diets when you take television into account .

Article ADS PubMed PubMed Central Google Scholar

Guess, A. M., Nyhan, B., O’Keeffe, Z. & Reifler, J. The sources and correlates of exposure to vaccine-related (mis)information online. Vaccine 38 , 7799–7805 (2020). This paper shows hows how a small portion of the population accounts for the vast majority of exposure to vaccine-sceptical content .

Chong, D. & Druckman, J. N. Framing public opinion in competitive democracies. Am. Polit. Sci. Rev. 101 , 637–655 (2007).

Arendt, F. Toward a dose-response account of media priming. Commun. Res. 42 , 1089–1115 (2015). This paper shows that people may need repeated exposure to information for it to affect their attitudes .

Arceneaux, K., Johnson, M. & Murphy, C. Polarized political communication, oppositional media hostility, and selective exposure. J. Polit. 74 , 174–186 (2012).

Feldman, L. & Hart, P. Broadening exposure to climate change news? How framing and political orientation interact to influence selective exposure. J. Commun. 68 , 503–524 (2018).

Druckman, J. N. Political preference formation: competition, deliberation, and the (ir)relevance of framing effects. Am. Polit. Sci. Rev. 98 , 671–686 (2004).

Bakshy, E., Messing, S. & Adamic, L. A. Exposure to ideologically diverse news and opinion on facebook. Science 348 , 1130–1132 (2015).

Article ADS MathSciNet CAS PubMed Google Scholar

Bozarth, L., Saraf, A. & Budak, C. Higher ground? How groundtruth labeling impacts our understanding of fake news about the 2016 U.S. presidential nominees. In Proc. International AAAI Conference on Web and Social Media Vol. 14, 48–59 (Association for the Advancement of Artificial Intelligence, 2020).

Gerber, A. S., Gimpel, J. G., Green, D. P. & Shaw, D. R. How large and long-lasting are the persuasive effects of televised campaign ads? Results from a randomized field experiment. Am. Polit. Sci. Rev. 105 , 135–150 (2011). This paper shows that the effect of news decays rapidly; news needs repeated exposure for long-term impact .

Hill, S. J., Lo, J., Vavreck, L. & Zaller, J. How quickly we forget: the duration of persuasion effects from mass communication. Polit. Commun. 30 , 521–547 (2013). This paper shows that the effect of persuasive advertising decays rapidly, necessitating repeated exposure for lasting effect .

Larsen, M. V. & Olsen, A. L. Reducing bias in citizens’ perception of crime rates: evidence from a field experiment on burglary prevalence. J. Polit. 82 , 747–752 (2020).

Roose, K. What if Facebook is the real ‘silent majority’? The New York Times , https://www.nytimes.com/2020/08/28/us/elections/what-if-facebook-is-the-real-silent-majority.html (27 August 2020).

Breland, A. A new report shows how Trump keeps buying Facebook ads. Mother Jones , https://www.motherjones.com/politics/2021/07/real-facebook-oversight-board/ (28 July 2021).

Marchal, N., Kollanyi, B., Neudert, L.-M. & Howard, P. N. Junk News during the EU Parliamentary Elections: Lessons from A Seven-language Study of Twitter and Facebook (Univ. Oxford, 2019).

Ellison, N. B., Trieu, P., Schoenebeck, S., Brewer, R. & Israni, A. Why we don’t click: interrogating the relationship between viewing and clicking in social media contexts by exploring the “non-click”. J. Comput. Mediat. Commun. 25 , 402–426 (2020).

Pennycook, G., Epstein, Z., Mosleh, M., Arechar, A. A., Eckles,and, D. & Rand, D. G. Shifting attention to accuracy can reduce misinformation online. Nature 592 , 590–595 (2021).

Ghezae, I. et al. Partisans neither expect nor receive reputational rewards for sharing falsehoods over truth online. Open Science Framework https://osf.io/5jwgd/ (2023).

Guess, A. M. et al. Reshares on social media amplify political news but do not detectably affect beliefs or opinions. Science 381 , 404–408 (2023).

Godel, W. et al. Moderating with the mob: evaluating the efficacy of real-time crowdsourced fact-checking. J. Online Trust Saf. 1 , https://doi.org/10.54501/jots.v1i1.15 (2021).

Rogers, K. Facebook’s algorithm is broken. We collected some suggestion on how to fix it. FiveThirtyEight , https://fivethirtyeight.com/features/facebooks-algorithm-is-broken-we-collected-some-spicy-suggestions-on-how-to-fix-it/ (16 November 2021).

Roose, K. The making of a YouTube radical. The New York Times , https://www.nytimes.com/interactive/2019/06/08/technology/youtube-radical.html (8 June 2019).

Eslami, M. et al. First I “like” it, then I hide it: folk theories of social feeds. In Proc. 2016 CHI Conference on Human Factors in Computing Systems 2371–2382 (Association for Computing Machinery, 2016).

Silva, D. E., Chen, C. & Zhu, Y. Facets of algorithmic literacy: information, experience, and individual factors predict attitudes toward algorithmic systems. New Media Soc. https://doi.org/10.1177/14614448221098042 (2022).

Eckles, D. Algorithmic Transparency and Assessing Effects of Algorithmic Ranking. Testimony before the Senate Subcommittee on Communications, Media, and Broadband , https://www.commerce.senate.gov/services/files/62102355-DC26-4909-BF90-8FB068145F18 (U.S. Senate Committee on Commerce, Science, and Transportation, 2021).

Kantrowitz, A. Facebook removed the news feed algorithm in an experiment. Then it gave up. OneZero , https://onezero.medium.com/facebook-removed-the-news-feed-algorithm-in-an-experiment-then-it-gave-up-25c8cb0a35a3 (25 October 2021).

Ribeiro, M. H., Hosseinmardi, H., West, R. & Watts, D. J. Deplatforming did not decrease parler users’ activity on fringe social media. PNAS Nexus 2 , pgad035 (2023). This paper shows that shutting down Parler just displaced user activity to other fringe social media websites .

Alfano, M., Fard, A. E., Carter, J. A., Clutton, P. & Klein, C. Technologically scaffolded atypical cognition: the case of YouTube’s recommender system. Synthese 199 , 835–858 (2021).

Huszár, F. et al. Algorithmic amplification of politics on Twitter. Proc. Natl Acad. Sci. USA 119 , e2025334119 (2022).

Levy, R. Social media, news consumption, and polarization: evidence from a field experiment. Am. Econ. Rev. 111 , 831–870 (2021).

Cho, J., Ahmed, S., Hilbert, M., Liu, B. & Luu, J. Do search algorithms endanger democracy? An experimental investigation of algorithm effects on political polarization. J. Broadcast. Electron. Media 64 , 150–172 (2020).

Lewandowsky, S., Robertson, R. E. & DiResta, R. Challenges in understanding human-algorithm entanglement during online information consumption. Perspect. Psychol. Sci. https://doi.org/10.1177/17456916231180809 (2023).

Narayanan, A. Understanding Social Media Recommendation Algorithms (Knight First Amendment Institute at Columbia University, 2023).

Finkel, E. J. et al. Political sectarianism in America. Science 370 , 533–536 (2020).

Auxier, B. & Anderson, M. Social Media Use in 2021 (Pew Research Center, 2021).

Frimer, J. A. et al. Incivility is rising among American politicians on Twitter. Soc. Psychol. Personal. Sci. 14 , 259–269 (2023).

Broderick, R. & Darmanin, J. The “yellow vest” riots in France are what happens when Facebook gets involved with local news. Buzzfeed News , https://www.buzzfeednews.com/article/ryanhatesthis/france-paris-yellow-jackets-facebook (2018).

Salzberg, S. De-platform the disinformation dozen. Forbes , https://www.forbes.com/sites/stevensalzberg/2021/07/19/de-platform-the-disinformation-dozen/ (2021).

Karell, D., Linke, A., Holland, E. & Hendrickson, E. “Born for a storm”: hard-right social media and civil unrest. Am. Soc. Rev. 88 , 322–349 (2023).

Smith, N. & Graham, T. Mapping the anti-vaccination movement on Facebook. Inf. Commun. Soc. 22 , 1310–1327 (2019).

Brady, W. J., McLoughlin, K., Doan, T. N. & Crockett, M. J. How social learning amplifies moral outrage expression in online social networks. Sci. Adv. 7 , eabe5641 (2021).

Suhay, E., Bello-Pardo, E. & Maurer, B. The polarizing effects of online partisan criticism: evidence from two experiments. Int. J. Press Polit. 23 , 95–115 (2018).

Arugute, N., Calvo, E. & Ventura, T. Network activated frames: content sharing and perceived polarization in social media. J. Commun. 73 , 14–24 (2023).

Nordbrandt, M. Affective polarization in the digital age: testing the direction of the relationship between social media and users’ feelings for out-group parties. New Media Soc. 25 , 3392–3411 (2023). This paper shows that affective polarization predicts media use, not the other way around .

AFP. Street protests, a French tradition par excellence. The Local https://www.thelocal.fr/20181205/revolutionary-tradition-the-story-behind-frances-street-protests (2018).

Spier, R. E. Perception of risk of vaccine adverse events: a historical perspective. Vaccine 20 , S78–S84 (2001). This article documents the history of untrustworthy information about vaccines, which long predates social media .

Bryant, L. V. The YouTube algorithm and the alt-right filter bubble. Open Inf. Sci. 4 , 85–90 (2020).

Sismeiro, C. & Mahmood, A. Competitive vs. complementary effects in online social networks and news consumption: a natural experiment. Manage. Sci. 64 , 5014–5037 (2018).

Fergusson, L. & Molina, C. Facebook Causes Protests Documento CEDE No. 41 , https://doi.org/10.2139/ssrn.3553514 (2019).

Lu, Y., Wu, J., Tan, Y. & Chen, J. Microblogging replies and opinion polarization: a natural experiment. MIS Q. 46 , 1901–1936 (2022).

Porter, E. & Wood, T. J. The global effectiveness of fact-checking: evidence from simultaneous experiments in Argentina, Nigeria, South Africa, and the United Kingdom. Proc. Natl Acad. Sci. USA 118 , e2104235118 (2021).

Arechar, A. A. et al. Understanding and combatting misinformation across 16 countries on six continents. Nat. Hum. Behav. 7 , 1502–1513 (2023).

Blair, R. A. et al. Interventions to Counter Misinformation: Lessons from the Global North and Applications to the Global South (USAID Development Experience Clearinghouse, 2023).

Haque, M. M. et al. Combating misinformation in Bangladesh: roles and responsibilities as perceived by journalists, fact-checkers, and users. Proc. ACM Hum. Comput. Interact. 4 , 1–32 (2020).

Humprecht, E., Esser, F. & Van Aelst, P. Resilience to online disinformation: a framework for cross-national comparative research. Int. J. Press Polit. 25 , 493–516 (2020).

Gillum, J. & Elliott, J. Sheryl Sandberg and top Facebook execs silenced an enemy of Turkey to prevent a hit to the company’s business. ProPublica , https://www.propublica.org/article/sheryl-sandberg-and-top-facebook-execs-silenced-an-enemy-of-turkey-to-prevent-a-hit-to-their-business (24 February 2021).

Nord M. et al. Democracy Report 2024: Democracy Winning and Losing at the Ballot V-Dem Report (Univ. Gothenburg V-Dem Institute, 2024).

Alba, D. How Duterte used Facebook to fuel the Philippine drug war. Buzzfeed , https://www.buzzfeednews.com/article/daveyalba/facebook-philippines-dutertes-drug-war (4 September 2018).

Zakrzewski, C., De Vynck, G., Masih, N. a& Mahtani, S. How Facebook neglected the rest of the world, fueling hate speech and violence in India. Washington Post , https://www.washingtonpost.com/technology/2021/10/24/india-facebook-misinformation-hate-speech/ (24 October 2021).

Simonite, T. Facebook is everywhere; its moderation is nowhere close. Wired , https://www.wired.com/story/facebooks-global-reach-exceeds-linguistic-grasp/ (21 October 2021).

Cruz, J. C. B. & Cheng, C. Establishing baselines for text classification in low-resource languages. Preprint at https://arxiv.org/abs/2005.02068 (2020). This paper shows one of the challenges that makes content moderation costlier in less resourced countries .

Müller, K. & Schwarz, C. Fanning the flames of hate: social media and hate crime. J. Eur. Econ. Assoc. 19 , 2131–2167 (2021).

Bursztyn, L., Egorov, G., Enikolopov, R. & Petrova, M. Social Media and Xenophobia: Evidence from Russia (National Bureau of Economic Research, 2019).

Lewandowsky, S., Jetter, M. & Ecker, U. K. H. Using the President’s tweets to understand political diversion in the age of social media. Nat. Commun. 11 , 5764 (2020).

Bursztyn, L., Rao, A., Roth, C. P. & Yanagizawa-Drott, D. H. Misinformation During a Pandemic (National Bureau of Economic Research, 2020).

Motta, M. & Stecula, D. Quantifying the effect of Wakefield et al. (1998) on skepticism about MMR vaccine safety in the US. PLoS ONE 16 , e0256395 (2021).

Sanderson, Z., Brown, M. A., Bonneau, R., Nagler, J. & Tucker, J. A. Twitter flagged Donald Trump’s tweets with election misinformation: they continued to spread both on and off the platform. Harv. Kennedy Sch. Misinformation Rev. 2 , https://doi.org/10.37016/mr-2020-77 (2021).

Anhalt-Depies, C., Stenglein, J. L., Zuckerberg, B., Townsend, P. A. & Rissman, A. R. Tradeoffs and tools for data quality, privacy, transparency, and trust in citizen science. Biol. Conserv. 238 , 108195 (2019).

Gerber, N., Gerber, P. & Volkamer, M. Explaining the privacy paradox: a systematic review of literature investigating privacy attitude and behavior. Comput. Secur. 77 , 226–261 (2018). This paper explores the trade-offs between privacy and research .

Isaak, J. & Hanna, M. J. User data privacy: Facebook, Cambridge Analytica, and privacy protection. Computer 51 , 56–59 (2018).

Vogus, C. Independent Researcher Access to Social Media Data: Comparing Legislative Proposals (Center for Democracy and Technology, 2022).

Xie, Y. “Undemocracy”: inequalities in science. Science 344 , 809–810 (2014).

Nielsen, M. W. & Andersen, J. P. Global citation inequality is on the rise. Proc. Natl Acad. Sci. USA 118 , e2012208118 (2021).

King, D. A. The scientific impact of nations. Nature 430 , 311–316 (2004).

Zaugg, I. A., Hossain, A. & Molloy, B. Digitally-disadvantaged languages. Internet Policy Rev. 11 , 1–11 (2022).

Zaugg, I. A. in Digital Inequalities in the Global South (eds Ragnedda, M. & Gladkova, A.) 247–267 (Springer, 2020).

Sablosky, J. Dangerous organizations: Facebook’s content moderation decisions and ethnic visibility in Myanmar. Media Cult. Soc. 43 , 1017–1042 (2021). This paper highlights the challenges of content moderation in the Global South .

Warofka, A. An independent assessment of the human rights impact of Facebook in Myanmar. Facebook Newsroom , https://about.fb.com/news/2018/11/myanmar-hria/ (2018).

Fick, M. & Dave, P. Facebook’s flood of languages leave it struggling to monitor content. Reuters , https://www.reuters.com/article/idUSKCN1RZ0DL/ (23 April 2019).

Newman, N. Executive Summary and Key Findings of the 2020 Report (Reuters Institute for the Study of Journalism, 2020).

Hilbert, M. The bad news is that the digital access divide is here to stay: domestically installed bandwidths among 172 countries for 1986–2014. Telecommun. Policy 40 , 567–581 (2016).

Traynor, I. Internet governance too US-centric, says European commission. The Guardian , https://www.theguardian.com/technology/2014/feb/12/internet-governance-us-european-commission (12 February 2014).

Pennycook, G., Cannon, T. D. & Rand, D. G. Prior exposure increases perceived accuracy of fake news. J. Exp. Psychol. Gen. 147 , 1865–1880 (2018).

Guess, A. M. et al. “Fake news” may have limited effects beyond increasing beliefs in false claims. Kennedy Sch. Misinformation Rev. 1 , https://doi.org/10.37016/mr-2020-004 (2020).

Loomba, S., de Figueiredo, A., Piatek, S. J., de Graaf, K. & Larson, H. J. Measuring the impact of COVID-19 vaccine misinformation on vaccination intent in the UK and USA. Nat. Hum. Behav. 5 , 337–348 (2021).

Lorenz-Spreen, P., Oswald, L., Lewandowsky, S. & Hertwig, R. Digital media and democracy: a systematic review of causal and correlational evidence worldwide. Nat. Hum. Behav. 7 , 74–101 (2023). This paper provides a review of evidence on social media effects .

Donato, K. M., Singh, L., Arab, A., Jacobs, E. & Post, D. Misinformation about COVID-19 and Venezuelan migration: trends in Twitter conversation during a pandemic. Harvard Data Sci. Rev. 4 , https://doi.org/10.1162/99608f92.a4d9a7c7 (2022).

Wieczner, J. Big lies vs. big lawsuits: why Dominion Voting is suing Fox News and a host of Trump allies. Fortune , https://fortune.com/longform/dominion-voting-lawsuits-fox-news-trump-allies-2020-election-libel-conspiracy-theories/ (2 April 2021).

Calma, J. Twitter just closed the book on academic research. The Verge https://www.theverge.com/2023/5/31/23739084/twitter-elon-musk-api-policy-chilling-academic-research (2023).

Edelson, L., Graef, I. & Lancieri, F. Access to Data and Algorithms: for an Effective DMA and DSA Implementation (Centre on Regulation in Europe, 2023).

Download references

Author information

Authors and affiliations.

University of Michigan School of Information, Ann Arbor, MI, USA

Ceren Budak

Department of Government, Dartmouth College, Hanover, NH, USA

Brendan Nyhan

Microsoft Research, New York, NY, USA

David M. Rothschild

Maxwell School of Citizenship and Public Affairs, Syracuse University, Syracuse, NY, USA

Emily Thorson

Department of Computer and Information Science, Annenberg School of Communication, and Operations, Information, and Decisions Department, University of Pennsylvania, Philadelphia, PA, USA

Duncan J. Watts

You can also search for this author in PubMed Google Scholar

Contributions

C.B., B.N., D.M.R., E.T. and D.J.W. wrote and revised the paper. D.M.R. collected the data and prepared Fig. 1 .

Corresponding author

Correspondence to David M. Rothschild .

Ethics declarations

Competing interests.

The authors declare no competing interests, but provide the following information in the interests of transparency and full disclosure. C.B. and D.J.W. previously worked for Microsoft Research and D.M.R. currently works for Microsoft Research. B.N. has received grant funding from Meta. B.N. and E.T. are participants in the US 2020 Facebook and Instagram Election Study as independent academic researchers. D.J.W. has received funding from Google Research. D.M.R. and D.J.W. both previously worked at Yahoo!.

Peer review

Peer review information.

Nature thanks Stephan Lewandowsky, David Rand, Emma Spiro and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Cite this article.

Budak, C., Nyhan, B., Rothschild, D.M. et al. Misunderstanding the harms of online misinformation. Nature 630 , 45–53 (2024). https://doi.org/10.1038/s41586-024-07417-w

Download citation

Received : 13 October 2021

Accepted : 11 April 2024

Published : 05 June 2024

Issue Date : 06 June 2024

DOI : https://doi.org/10.1038/s41586-024-07417-w

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

By submitting a comment you agree to abide by our Terms and Community Guidelines . If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.

Quick links

Explore articles by subject
Guide to authors
Editorial policies

The University of Chicago The Law School

College essays and diversity in the post-affirmative action era, sonja starr’s latest research adds data, legal analysis to discussion about race in college admissions essays.

A woman sitting on a couch with a book on her lap

Editor’s Note: This story is part of an occasional series on research projects currently in the works at the Law School.

The Supreme Court’s decision in June 2023 to bar the use of affirmative action in college admissions raised many questions. One of the most significant is whether universities should consider applicants’ discussion of race in essays. The Court’s decision in Students for Fair Admissions (SFFA) v. Harvard did not require entirely race-blind admissions. Rather, the Court explicitly stated that admissions offices may weigh what students say about how race affected their lives. Yet the Court also warned that this practice may not be used to circumvent the bar on affirmative action.

Many university leaders made statements after SFFA suggesting that they take this passage seriously, and that it potentially points to a strategy for preserving diversity. But it’s not obvious how lower courts will distinguish between consideration of “race-related experience” and consideration of “race qua race.” Sonja Starr, Julius Kreeger Professor of Law & Criminology at the Law School, was intrigued by the implication of that question, calling the key passage of the Court’s opinion the “essay carveout.”

“Where is the line?” she wrote in a forthcoming article, the first of its kind to discuss this issue in depth in the post- SFFA era. “And what other potential legal pitfalls could universities encounter in evaluating essays about race?”

To inform her paper’s legal analysis, Starr conducted empirical analyses of how universities and students have included race in essays, both before and after the Court’s decision. She concluded that large numbers of applicants wrote about race, and that college essay prompts encouraged them to do so, even before SFFA .

Some thought the essay carveout made no sense. Justice Sonia Sotomayor called it “an attempt to put lipstick on a pig” in her dissent. Starr, however, disagrees. She argues that universities are on sound legal footing relying on the essay carveout, so long as they consider race-related experience in an individualized way. In her article, Starr points out reasons the essay carveout makes sense in the context of the Court’s other arguments. However, she points to the potential for future challenges—on both equal protection and First Amendment grounds—and discusses how colleges can survive them.

What the Empirical Research Showed

After SFFA , media outlets suggested that universities would add questions about race or identity in their admissions essays and that students would increasingly focus on that topic. Starr decided to investigate this speculation. She commissioned a professional survey group to recruit a nationally representative sample of recent college applicants. The firm queried 881 people about their essay content, about half of whom applied in 2022-23, before SFFA , and half of whom submitted in 2023-24.

The survey found that more than 60 percent of students in non-white groups wrote about race in at least some of their essays, as did about half of white applicants. But contrary to what the media suggested, there were no substantial changes between the pre-and post- SFFA application cycles.

Starr also reviewed essay prompts that 65 top schools have used over the last four years. She found that diversity and identity questions—as well as questions about overcoming adversity, which, for example, provide opportunities for students to discuss discrimination that they have faced—are common and have increased in frequency both before and after SFFA.

A Personally Inspired Interest

Although Starr has long written about equal protection issues, until about two years ago, she would have characterized educational admissions as a bit outside her wheelhouse. Her research has mostly focused on the criminal justice system, though race is often at the heart of it. In the past, for example, she has assessed the role of race in sentencing, the constitutionality of algorithmic risk assessment instruments in criminal justice, as well as policies to expand employment options for people with criminal records.

But a legal battle around admissions policies at Fairfax County’s Thomas Jefferson High School for Science and Technology—the high school that Starr attended—caught her attention. Starr followed the case closely and predicted that “litigation may soon be an ever-present threat for race-conscious policymaking” in a 2024 Stanford Law Review article on that and other magnet school cases.

“I got really interested in that case partly because of the personal connection,” she said. “But I ended up writing about it as an academic matter, and that got me entrenched in this world of educational admissions questions and their related implications for other areas of equal protection law.”

Implications in Education and Beyond

Starr’s forthcoming paper argues that the essay carveout provides a way for colleges to maintain diversity and stay on the right side of the Court’s decision.

“I believe there’s quite a bit of space that’s open for colleges to pursue in this area without crossing that line,” she said. “I lay out the arguments that colleges can put forth.”

Nevertheless, Starr expects future litigation targeting the essay carveout.

“I think we could see cases filed as soon as this year when the admissions numbers come out,” she said, pointing out that conservative legal organizations, such as the Pacific Legal Foundation, have warned that they’re going to be keeping a close eye on admissions numbers and looking for ways that schools are circumventing SFFA .

Starr envisions her paper being used as a resource for schools that want to obey the law while also maintaining diversity. “The preservation of diversity is not a red flag that something unconstitutional is happening,” she said. “There are lots of perfectly permissible ways that we can expect diversity to be maintained in this post- affirmative action era.”

Starr’s article, “Admissions Essays after SFFA ,” is slated to be published in Indiana Law Journal in early 2025.

school Campus Bookshelves
menu_book Bookshelves
perm_media Learning Objects
login Login
how_to_reg Request Instructor Account
hub Instructor Commons

Margin Size

Download Page (PDF)
Download Full Book (PDF)
Periodic Table
Physics Constants
Scientific Calculator
Reference & Cite
Tools expand_more
Readability

selected template will load here

This action is not available.

6: Nonexperimental Research

Last updated
Save as PDF
Page ID 16087

$ \newcommand{\vecs}[1]{\overset { \scriptstyle \rightharpoonup} {\mathbf{#1}} } $

$ \newcommand{\vecd}[1]{\overset{-\!-\!\rightharpoonup}{\vphantom{a}\smash {#1}}} $

$ \newcommand{\id}{\mathrm{id}}$ $ \newcommand{\Span}{\mathrm{span}}$

( \newcommand{\kernel}{\mathrm{null}\,}\) $ \newcommand{\range}{\mathrm{range}\,}$

$ \newcommand{\RealPart}{\mathrm{Re}}$ $ \newcommand{\ImaginaryPart}{\mathrm{Im}}$

$ \newcommand{\Argument}{\mathrm{Arg}}$ $ \newcommand{\norm}[1]{\| #1 \|}$

$ \newcommand{\inner}[2]{\langle #1, #2 \rangle}$

$ \newcommand{\Span}{\mathrm{span}}$

$ \newcommand{\id}{\mathrm{id}}$

$ \newcommand{\kernel}{\mathrm{null}\,}$

$ \newcommand{\range}{\mathrm{range}\,}$

$ \newcommand{\RealPart}{\mathrm{Re}}$

$ \newcommand{\ImaginaryPart}{\mathrm{Im}}$

$ \newcommand{\Argument}{\mathrm{Arg}}$

$ \newcommand{\norm}[1]{\| #1 \|}$

$ \newcommand{\Span}{\mathrm{span}}$ $ \newcommand{\AA}{\unicode[.8,0]{x212B}}$

$ \newcommand{\vectorA}[1]{\vec{#1}} % arrow$

$ \newcommand{\vectorAt}[1]{\vec{\text{#1}}} % arrow$

$ \newcommand{\vectorB}[1]{\overset { \scriptstyle \rightharpoonup} {\mathbf{#1}} } $

$ \newcommand{\vectorC}[1]{\textbf{#1}} $

$ \newcommand{\vectorD}[1]{\overrightarrow{#1}} $

$ \newcommand{\vectorDt}[1]{\overrightarrow{\text{#1}}} $

$ \newcommand{\vectE}[1]{\overset{-\!-\!\rightharpoonup}{\vphantom{a}\smash{\mathbf {#1}}}} $

In this chapter we look more closely at non-experimental research. We begin with a general definition of, non-experimental research, along with a discussion of when and why non-experimental research is more appropriate than experimental research. We then look separately at three important types of non-experimental research: cross-sectional research, correlational research and observational research.

6.0: Prelude to Nonexperimental Research What do the following classic studies have in common? Stanley Milgram found that about two thirds of his research participants were willing to administer dangerous shocks to another person just because they were told to by an authority figure (Milgram, 1963). Elizabeth Loftus and Jacqueline Pickrell showed that it is relatively easy to “implant” false memories in people by repeatedly asking them about childhood events that did not actually happen to them (Loftus & Pickrell, 1995).
6.1: Overview of Non-Experimental Research Most researchers in psychology consider the distinction between experimental and non-experimental research to be an extremely important one. This is because although experimental research can provide strong evidence that changes in an independent variable cause differences in a dependent variable, non-experimental research generally cannot. As we will see, however, this inability to make causal conclusions does not mean that non-experimental research is less important than experimental research.
6.2: Correlational Research Correlational research is a type of non-experimental research in which the researcher measures two variables and assesses the statistical relationship (i.e., the correlation) between them with little or no effort to control extraneous variables. There are many reasons that researchers interested in statistical relationships between variables would choose to conduct a correlational study rather than an experiment.
6.3: Complex Correlation As we have already seen, researchers conduct correlational studies rather than experiments when they are interested in noncausal relationships or when they are interested in causal relationships but the independent variable cannot be manipulated for practical or ethical reasons. In this section, we look at some approaches to complex correlational research that involve measuring several variables and assessing the relationships among them.
6.4: Qualitative Research Quantitative researchers typically start with a focused research question or hypothesis, collect a small amount of data from a large number of individuals, describe the resulting data using statistical techniques, and draw general conclusions about some large population. Although this method is by far the most common approach to conducting empirical research in psychology, there is an important alternative called qualitative research.
6.5: Observational Research Observational research is used to refer to several different types of non-experimental studies in which behavior is systematically observed and recorded. The goal of observational research is to describe a variable or set of variables. The goal is to obtain a snapshot of specific characteristics of an individual, group, or setting. Observational research is non-experimental because nothing is manipulated or controlled, and as such we cannot arrive at causal conclusions using this approach.

Thumbnail: An example of data produced by data dredging, showing a correlation between the number of letters in a spelling bee's winning word (red curve) and the number of people in the United States killed by venomous spiders (black curve). Image used with permission (CC BY 4.0 International; Tyler Vigen - Spurious Correlations ).

Open access
Published: 03 June 2024

Comparison of two propensity score-based methods for balancing covariates: the overlap weighting and fine stratification methods in real-world claims data

Wen Wan 1 ,
Manoradhan Murugesan 2 ,
Robert S. Nocon 3 ,
Joshua Bolton 4 ,
R. Tamara Konetzka 2 ,
Marshall H. Chin 1 &
Elbert S. Huang 1

BMC Medical Research Methodology volume 24 , Article number: 122 ( 2024 ) Cite this article

15 Accesses

Metrics details

Two propensity score (PS) based balancing covariate methods, the overlap weighting method (OW) and the fine stratification method (FS), produce superb covariate balance. OW has been compared with various weighting methods while FS has been compared with the traditional stratification method and various matching methods. However, no study has yet compared OW and FS. In addition, OW has not yet been evaluated in large claims data with low prevalence exposure and with low frequency outcomes, a context in which optimal use of balancing methods is critical. In the study, we aimed to compare OW and FS using real-world data and simulations with low prevalence exposure and with low frequency outcomes.

We used the Texas State Medicaid claims data on adult beneficiaries with diabetes in 2012 as an empirical example ( N = 42,628). Based on its real-world research question, we estimated an average treatment effect of health center vs. non-health center attendance in the total population. We also performed simulations to evaluate their relative performance. To preserve associations between covariates, we used the plasmode approach to simulate outcomes and/or exposures with N = 4,000. We simulated both homogeneous and heterogeneous treatment effects with various outcome risks (1-30% or observed: 27.75%) and/or exposure prevalence (2.5-30% or observed:10.55%). We used a weighted generalized linear model to estimate the exposure effect and the cluster-robust standard error (SE) method to estimate its SE.

In the empirical example, we found that OW had smaller standardized mean differences in all covariates (range: OW: 0.0–0.02 vs. FS: 0.22–3.26) and Mahalanobis balance distance (MB) (< 0.001 vs. > 0.049) than FS. In simulations, OW also achieved smaller MB (homogeneity: <0.04 vs. > 0.04; heterogeneity: 0.0-0.11 vs. 0.07–0.29), relative bias (homogeneity: 4.04–56.20 vs. 20–61.63; heterogeneity: 7.85–57.6 vs. 15.0-60.4), square root of mean squared error (homogeneity: 0.332–1.308 vs. 0.385–1.365; heterogeneity: 0.263-0.526 vs 0.313-0.620), and coverage probability (homogeneity: 0.0–80.4% vs. 0.0-69.8%; heterogeneity: 0.0-97.6% vs. 0.0-92.8%), than FS, in most cases.

Conclusions

These findings suggest that OW can yield nearly perfect covariate balance and therefore enhance the accuracy of average treatment effect estimation in the total population.

Peer Review reports

Due to infeasibility of running a randomized experiment, observational data are often used to estimate the population health effects of interventions. When estimating plausibly causal effects using observational data, it is necessary to reduce imbalance in the empirical distribution of the pretreatment confounders between the treated and control groups [ 1 ]. Lowering imbalance can reduce the degree of model dependence for the statistical estimation of causal effects [ 1 , 2 , 3 , 4 ], and thus reduces inefficiency and bias [ 1 ]. To achieve balanced covariates, propensity scores (PS) have become a cornerstone in observational studies aimed at estimating causal effects [ 5 , 6 ]. PS are defined as the predicted probability of receiving a particular treatment (or exposure) for the given covariate realizations of a study subject.

In this paper, we study PS-based approaches to estimate the average treatment effect in the total population (ATE). There are three common types of balancing methods via PS: matching, stratifying, and weighting. Among matching methods, the PS matching method (PSM) is the most commonly used in practice [ 1 ]. It is simple and intuitive by reducing the multidimensional covariate space to one dimension. Despite its widespread adoption, a large sample size is required as it discards some subjects who are not matched. In addition, PSM has been shown to increase model “imbalance, inefficiency, model dependence, and bias,” which is not the case with most other matching methods [ 1 ]. Among the stratification methods, the most common one is to stratify subjects into five quintiles of PS. With the stratum boundaries determined by PS distribution in the exposed and the comparison group combined, it eliminates approximately 90% of bias due to measured confounding [ 7 ]. However, when exposure is infrequent, it may result in all exposed subjects being aggregated in one or more extreme strata [ 5 , 8 ]. The fine stratification weights method (FS), a recent method, can solve this issue by increasing number of strata and by determining stratum boundaries based on PS distribution in exposed group only. It has been shown to gain greater efficiency than the traditional one [ 8 ]. Among the weighting methods, inverse probability weighting (IPW) is popular but performs poorly when some subjects have extreme PS [ 9 , 10 , 11 ]. The PS based overlap weighting method (OW), another recent method, overcomes IPW’s extreme weight issue and produces impressive covariate balance [ 12 , 13 ].

OW has been theoretically proven to have small-sample exact balance property [ 12 ]. That is, it leads to exact balance on the mean of every covariate when the PS is estimated by a logistic regression. It is less sensitive to model misspecification compared to the inverse probability weighting method (IPW) in a simulation study [ 14 ]. Despite these features, to our knowledge, OW has only been evaluated by comparing with weighting methods such as IPW and trimmed IPW [ 9 , 10 , 11 , 12 , 13 , 14 ]. Little is known about the relative performance of OW compared with other types of balancing methods including matching and stratification methods [ 15 ]. In addition, OW has not been evaluated in large claims data with low prevalence exposure and/or with low frequency events (i.e., outcomes), a context in which optimal use of balancing methods is critical.

Furthermore, matching on PS is limited by exclusion of subjects without a suitable match leading to a non-representative population and a loss of statistical power [ 16 ]. PSM including 1:1, 1:5, and full matching have less model precision than FS in at least two claims studies [ 8 , 17 ]. Therefore, we aimed to compare OW with FS only, both relatively new and promising methods, using real-world and simulated claims data in settings with infrequent exposure and/or with low prevalence outcomes.

Empirical example

We used a cohort of 42,628 Texas State Medicaid beneficiaries, aged 18–64, diagnosed with type 2 diabetes, who had at least one primary care visit between January 2012 and December 2012. About 10.55% ( n = 4,498) of the patients received the majority of their primary care at federally qualified health centers (FQHCs) (exposure), while the rest (89.45%, n = 38,130) received care at non-FQHCs (control). Researchers analyzed whether or not those patients who had routine primary care at FQHCs had fewer hospitalizations and emergency room visits than the non-FQHC patients. Five continuous and 12 binary covariates were selected based on clinical relevance and previous literature. The empirical example has 10.55% exposure rate which is near rare (typically < 10% considered as rare) and hospitalization quite often is a rare outcome.

The study was reviewed by the University of Chicago Institutional Review Board and determined to be non-human subject research.

Overlap weighting method (OW)

The OW method mimics a randomized trial by assigning appropriate weights to generate a clinically relevant target population – overlapped between groups. That is, a subject in the treatment group receives a weight that is the probability of not receiving the treatment (i.e., 1 – PS), while a subject in the control group receives a weight that is the probability of receiving the treatment (i.e., PS). As a consequence, the two groups have overlapped PS distributions. Those subjects overlapped between the two groups in the PS distribution receive more weight, while those who are only in one non-overlapping tail of the PS distribution receive less. Also OW does not prune any subjects. The target of inference, advantages, and disadvantages of the OW and FS methods are compared in eTable 1 .

PS-based fine stratification method (FS)

The FS method proposed by Desai et al. (2017) [ 8 ] finds matched balancing scores (PS) via stratification with a large number of PS strata (much larger than five in the traditional stratification method), and then assigns appropriate weights to subjects per stratum. It minimizes any loss of exposed subjects that may be relevant especially when treatment exposure is rare, because losing subjects decreases precision of the treatment effect estimates [ 8 ]. The method only excludes subjects whose PS are not in the overlapped PS regions between the two groups. There are two steps for implementation: (1) create equally-sized PS strata by ranking only treated/exposed subjects based on PS values and then assign control/unexposed subjects to these strata; (2) following stratification, in all strata with at least one treated patient and one control patient, weights are calculated (see below).

Regarding the optimum number of strata, Desai et al. stated that it may be difficult to make general recommendations because it may depend on the prevalence of a rare-exposed treatment [ 8 ]. The number of PS strata they used was 10, 50, or 100 and all produced similar bias and precisions in their simulations. In this study, we chose 20 PS strata, their stratification width about 0.05 on average, smaller than the recommended PS width of 0.2 [ 7 ]. . Each stratum had about 225 subjects from the FQHC-exposed group in our empirical example.

Target of inference (estimand) and weights

In the study, since each patient can switch their primary care visits between FQHCs and non-FQHCs, we estimated ATE among all patients [ 6 ]. In literature, there are two existing approaches to assign weights for ATE. One approach is to generate equal total weights between groups, denoted as ‘ATE-equ,’ is based on N total in stratum i/ N total exp in stratum i for the exposed group and N"total in stratum i"/N"total unexp in stratum i" for the unexposed group [ 18 , 19 ]. The other approach, denoted as ‘ATE-unequ,’ is based on ( N total in stratum i/ N total)/( N total exp in stratum i/ N total exp") for the exposed group and ( N total in stratum i/ N total)/( N total unexp in stratum i/ N total unexp) for the unexposed group [ 6 , 20 ]. This alternative approach results in the total weight in one group equivalent to the sample size in that group. The two weighting methods are very similar, except that ATE-unequ has a weight of N“total exp” (N“total unexp”) for the exposed (unexposed) group.

As a weighting method, OW targets the overlap population and its corresponding estimand is referred to as ATE on the overlap population (ATO) [ 12 ]. Zhou et al. (2020) stated that OW was part of a class of balancing weights that target a judiciously chosen subpopulation of interest from which an estimand is closely related to ATE [ 14 ]. Not surprisingly, OW’s total weights are identical between groups, the same as ATE-equ.

Evaluation of performance via the empirical example

In the empirical study for the method evaluation [ 21 ], we used the standardized mean difference between the two groups (SMD), Mahalanobis balance (MB), and final sample size of retaining sample. SMD is a distance measure of balancing criterion for each covariate [ 22 ]. MB is a metric that measures the distance between two group mean vectors of all covariates and is standardized by the sample covariance matrix [ 1 , 17 , 23 ]. Final sample size [ 8 ] is a measure of model precision and can be important for a rare event outcome.

Simulations

After balancing covariates, we determined relative performance of OW and FS for model bias and precision. The degree of covariate imbalance is proportional to bias in the treatment effect [ 24 ], and final sample size is associated with precision. However, due to lack of knowledge of the true FQHC effect in the empirical example, we do not know the real size of model bias and precision, especially, in a setting of infrequent exposure/outcome. Therefore, we conducted simulations.

Instead of using ordinary simulation approaches that do not capture important features that may exist between covariates, we chose the plasmode approach to conduct simulations [ 25 , 26 , 27 ]. Through resampling with replacement from all the observed covariates, plasmode can preserve the associations between covariates with potential complex covariance structures, which are common in healthcare claims databases [ 25 ]. Via a logistic regression model, details were provided in Appendix A (including R code) on how to simulate an outcome and/or an exposure factor. There were two logit models for outcome and exposure, respectively, with two different linear combinations of covariates. We simulated two types of treatment effects: homogeneity and heterogeneity. To simulate a heterogeneous treatment effect, we replaced constant treatment effect with an interaction term between exposure and sex (or age): sex, as an example, represented as a binary heterogeneity factor and age was a continuous one [ 25 ]. Age was standardized first before conducting a heterogeneous treatment effect. The simulation settings can be found in eTable 2.

To examine settings with infrequent outcome and/or occasional exposure, for each type of treatment effect, we simulated four scenarios by varying outcome risks and/or exposure prevalence. Scenarios simulated outcome risks of 1%, 10%, and 30% with the observed exposure prevalence (10.55%) or with 2.5% simulated exposure. We also simulated exposure prevalence of 2.5%, 10%, and 30% with the observed outcome risk (27.75%) or with 1% simulated outcome risk. We set the true FQHC effect to be one as a coefficient to both homogeneous and heterogeneous treatment terms. For each scenario, we simulated 500 datasets, each with the sample size of 4,000. For each simulated dataset, a weighted generalized linear model (GLM) with the log link function in the SAS GENMOD procedure was used to estimate the FQHC effect, i.e., natural logarithms of relative risk ratio [ 8 ]. Due to non-uniform weights included in our GLMs, instead of using the default delta method, we used the cluster-robust standard error method to estimate standard error (SE) of the effect [ 28 , 29 ]. After covariates balanced, adjusting further for covariates is unnecessary because it is unrelated to the treatment independent variable [ 30 ]. That is, a simple difference in means on the balanced data can estimate the causal effect.

Evaluation of performance in simulations

In the simulation study, we used the following criteria to evaluate the methods: mean MB, mean relative bias (rbias), standard deviation (SD) of rbias, square root of mean squared error of bias (rMSE), average SE of the estimated effect, average final sample size, two coverages [ 8 , 17 , 23 ], and significance. Relative bias is the percent relative difference, 100(estimated effect -truth)/truth [ 12 , 13 , 14 ]. The rMSE combines squared bias (not rbias) and its variance. The coverage is a probability of the 95% confidence interval (CI) that covers the true effect (denoted as ‘coverage’) [ 12 , 13 , 31 ]. It can be obtained with two steps: (1) to compute a CI via our weighted GLM and (2) then to calculate a proportion of samples covering the true effect among 500 simulations. In our simulation study, a CI could cover both the non-zero true effect and zero, and statistical significance may be influenced. Therefore, to distinguish from the traditional coverage, we generated another one (‘coverageT’) counting those CIs that cover the true effect but not zero. In some cases where CIs were too narrow to cover the true effect (see results below), significance was defined as a proportion of samples obtaining a significant effect (by a weighted GLM with a two-sided p -value < 0.05). The two coverages and significance are associated with model precision [ 32 ], but more targeted to detect the true treatment effect. Among the criteria, the least useful criterion is SE because it measures variability of effect in a model, not bias, precision, or measures in covariate balance.

Unmatched subjects

Although matching was not involved in the study, via simulations we discovered whether pruning those clearly unmatched subjects has any effects on model bias and precision. The unmatched subjects are those who are available from one group but not from the other group in terms of combination cells of binary covariates.

Summary of all methods

In summary, we used two datasets for performance evaluation: one was the original full dataset (denoted as ‘F’); the other was the dataset (denoted as ‘X’) after deleting those unmatched subjects. We also evaluated the two weighting approaches. Therefore, there were a total of seven methods for comparisons: crude, OW F , OW X , FS F−equ , FS X−equ , FS F−uneq , and FS X−unequ (summary can be found in eTable 3). We used SAS version 9.4 to conduct covariate balancing, and statistical modeling for both empirical and simulation studies (Appendix A for analysis of one simulation), and we used R function (Appendix B) to generate simulation datasets in R version 4.3.0.

Analysis of the empirical dataset

Table 1 shows the evaluation of the seven methods using the real-world data. OW F and OW X were nearly identical and performed the best by reducing all SMD of covariates to zero, and the smallest MB over all covariates, indicating perfect balancing of covariates. The four FS methods consistently performed fairly well over all covariates, with all SMD around zero. Among them, the two FS with equal weights between groups (FS F−equ and FS X−equ ) were closer to each other and achieved better MB than the other two FS with unequal weights (FS F−unequ and FS X−unequ ). The crude method exhibited the worst performance, far more imbalanced.

For the final sample size used for further analysis, OW X excluded 147 subjects (0.345%) who had no matches for combinations of all binary variables. Using the full dataset, FS excluded 21 subjects (< 0.05%) whose PS were in non-overlapped regions. Distributions of PS per group were in eFigure 1.

Analysis of simulated datasets with the homogeneous treatment effect

Tables 2 and 3 showed the simulation evaluation of each method by risk level of outcome and by exposure prevalence, respectively. In most scenarios, OW F and OW X had very similar results and both performed better than the other methods. That is, both OW had small MB, rbias, SD of rbias, rMSE, relatively small SE, and relatively large coverage and coverageT. There were two exceptions. One was that the crude method had smallest SE of estimate. The reason is that model estimation by the crude method is consistent with the simulation method (i.e., two logit models with a constant and additive treatment effect). The second exception was the cases with rare outcome events (1%) and low exposures (2.5% and 10%) (Table 2 ), where the crude method had the smallest rbias, SD of rbias, and rMSE compared to the others. The reason is that both rare events and low exposures resulted in complete separation or quasi-complete separation of data points that caused model estimation to be unstable [ 33 , 34 ]. After removing these simulated samples, both OW had smaller rbias and larger coverage than the others (eTable 4).

Similar to the empirical study, the four FS methods were quite close to each other. The two FS with equal weights had smaller MB than the two FS with unequal weights. However, each pair of FS using the data with the same sample sizes (either full or reduced datasets) had almost the same model estimations. These indicate that the two ATE weighting methods had minor difference in balancing values but almost identical values in model estimation. The two FS using the full datasets generally had better model estimations than the two using the reduced datasets.

In the criteria, there were different change patterns over simulations. As outcome risk (Table 2 ) or exposure prevalence (Table 3 ) increased, the power increased, SE, SD, rMSE, and both coverages decreased, and significance increased. Coverages decreased as outcome risk (or exposure prevalence) increased because smaller SE resulted in a narrower CI of effect that were too narrow to cover the true effect. MB, unrelated to model estimation, remained stable in a method when outcome risk increased, and reduced greatly when exposure prevalence increased. On the other hand, rbias increased when outcome risk increased, and remained similar in a method when exposure increased.

To determine whether the simulation results were due to real differences or Monte Carlo error (MCE), we calculated MCE for both MB and rbias (eTables 5–6). We evaluated the number of simulations needed (Appendix B) and found that 500 simulations were enough for most settings of outcome and exposure.

Analysis of simulated datasets with heterogeneous treatment effect

Tables 4 and 5 and eTables 7 and 8 showed the evaluation results due to sex(age)-dependent treatment effects. Similar to the results with the constant treatment effect, the two OW methods had very similar results and both performed better than the other methods in terms of MB, rbias, rMSE, coverages, and significance. Also the same to the homogeneous cases, there were two exceptions. One was that the crude method had smallest SE of estimate. The other was that FS had smaller rMSE than OW in the case with 1% outcome and 10% or 10.55% exposure. It was also due to the issue of complete separation or quasi-complete separation of data points in a few simulated samples. After removing those samples, the OW methods still performed the best (eTable 9). All change patterns across scenarios were consistent to those in the homogeneous cases.

Both OW and FS methods performed well among PS-based balancing methods for causal inference. To our knowledge, our study is the first to compare OW and FS as the two types of PS-based balancing methods: weighting and stratification. We used a real-world and simulated claims data for their relative performance. We included simulations of rare outcome and/or exposure, not rare in a claims-based observational study. We simulated data for both homogeneous and heterogeneous treatment effects. The OW method obtained nearly perfect covariate balance and performed much better in covariate balance, model bias, and model precision and coverages than FS.

The target of inference (estimand) we focused on was ATE due to the nature of the intervention in the real-world example where the intervention was feasible to treat all eligible patients. The target of inference by OW is a special ATE, called ATO. OW is part of a class of balancing weights that target a judiciously chosen subpopulation from which an estimand is closely related to ATE [ 14 ]. OW produces equal total weights between groups by its definition, i.e., making the two groups overlapped in terms of PS values. For the FS method, we evaluated the two published weighting algorithms for ATE estimation: with and without equal total weights between groups. We found that the ATE-equ performed better than ATE-unequ in terms of covariate balance (SMD and MB) but both algorithms had almost identical model estimation in terms of bias and precision. In its formula, compared with ATE-equ, ATE-unequ includes a group sample size in its numerator to a subject in that group. This additional piece was designed to normalize and stabilize weights by limiting unduly large weights [ 20 ]. However, the additional piece unequaled total weights between groups, reducing covariate balance slightly, but did not affect model bias and precision.

We assume that our study met all the key assumptions for causal inference, including the stable unit treatment value assumption, the consistency assumption, and the positivity assumption [ 14 , 35 ]. However, practical violations of the positivity assumption occur when some subjects almost always (or almost never) receive treatment [ 14 ], for example, those unmatched in combinations of binary covariates. Our study explored if removing those unmatched helped covariate balance and model estimation. This was a minor matter in our case, maybe because the proportion of those removed was very low, about 0.34% of the whole population. Using the reduced data compared to using the full data, the simulation results showed slightly smaller covariate imbalance, but slightly larger model bias and imprecision. That is, although covariate balance is slightly reduced by allowing those clearly unmatched subjects between groups, larger sample size kept model estimation less biased and imprecise, especially with infrequent outcome and/or exposure. In addition, FS further removed some subjects with extreme PS, due to their PS not in the overlapped PS region between groups. However, comparing FS with OW which did not remove any subjects, we confidently state that given balanced PS values between groups, including mismatched subjects does not affect model estimation in settings with infrequent outcome/exposure.

In our weighted GLM analysis, we used the cluster-robust method to estimate SE of the intervention effect. It is inaccurate to use the delta method, the default model-based method, when using matching weights, because it assumes weights are frequency weights rather than probability weights [ 28 ].

In simulation results for both homogeneous and heterogeneous scenarios, we observed that as outcome risk level increased, bias increased. Higher risks and stronger correlations among exposure, outcome, and covariates led to larger bias in effect estimation [ 36 ]. That is, higher confounding, which we did not adjust for in analysis, caused more bias. Among 17 covariates, more than half of them were confounders, i.e., associated with hospitalization rate. As outcome risk increased, these confounders had more confounding effect that resulted in larger bias. Adjusting for those confounders could have improved model precision and accuracy. However, we purposely did not adjust further for them in the modeling stage because in the real-world example, investigators did not know which covariates were real confounders.

In simulation results, we also observed that as exposure prevalence increased, MB values in the crude method decreased. One possible reason is that higher exposure, and stronger correlations between covariates and exposure, resulted in more covariate balance. Furthermore, we found that as rate of outcome and/or exposure increased, coverages decreased and even became zero. That is, when there was larger power, CIs became too narrow to cover the true effect. Their 100% significance rate confirmed the reason.

The choices of our performance criteria were based on the guidance of metrics for covariate balance [ 21 ]. The MB criterion, which considers pairwise correlations between covariates, provides new insights beyond SMD. This is the first study to use MB to evaluate OW. In some simulation settings, coverage probability could be a maximum of 100% because it is different from confidence level [ 32 ]. Our study also solved the issue of some misleading results using the coverage probability as a criterion [ 14 ] by providing two coverages and one significance to replace the traditional one.

Besides OW, the FS method performed relatively well comparing the crude method. The FQHC and non-FQHC groups had significantly overlapped PS distributions, and only < 0.05% subjects were removed due to non-overlapped PS between them. Just as in Desai et al.’s study evaluating FS [ 8 ], after balancing covariates, the PS distributions became perfectly overlapped in the empirical example. This indicates that the number of strata, 20, was sufficient.

Our simulation results for constant treatment effect are consistent with Ripollone et al.’s study which also used simulated claims data [ 17 ]. In the simulated outcome with risk level of 20%, 20% exposure prevalence, and a sample size of 25,000, their FS analysis had 0.054 MB, 0.07–0.08 bias, and 0.178–0.172 rMSE, while ours had 0.047 MB, 0.0183 bias, and 0.025 rMSE, given the sample size of > 40,000 (eTable 10).

In our study, the two study groups were quite similar in that their PS distributions were substantially overlapped. However, when comparator groups are very different, the advantages of OW are actually greatest [ 37 ]. This is because the OW method will add more weight on those overlapped PS regions and fewer weights on those tailed PS regions. Given the same situation, the FS method will remove more subjects from non-overlapped regions which results in more severe bias and probably less model precision due to reduced sample size.

Our study has some limitations. First, the OW method can be used to estimate only ATE on the ATO population, but not average treatment effect on the treated population (ATT). However, two studies showed that when the exposure prevalence is small, ATO approximates ATT [ 12 , 35 ]. Second, due to simulating rare outcome (1%) and exposure (2.5 -10.55%), some simulated samples faced the issue of complete separation or quasi-complete separation of data points that caused model estimation to be unstable. More advanced modeling methods could be used such as Firth’s method [ 34 ] and Bayesian method [ 33 ]. However, this is beyond the goal of the study. Third, our simulation findings may not be generalizable because our simulations were based on one empirical study. However, both OW and FS have been separately evaluated in multiple studies. Fourth, our simulation did not consider misspecifications of a PS model and/or degrees of overlap of PS distributions. However, Zhou et al [ 14 ] conducted simulations for such situations to compare the performances of OW, IPW, and other weighting methods. They found that OW was robust to these situations. One possible reason they pointed out was that the estimand of OW was not defined on the estimated but true PS and OW smoothly down-weights the influence of observations at both end of PS spectrum [ 14 ]. Last, to estimate PS, we used a logistic regression, that is, a logit modeled as a linear combination of covariates. To capture complex dependency patterns between outcome and covariates, a machine learning method such as random forest may provide more accurate and less model dependent estimate of PS [ 38 ]. This will be our future work.

As demonstrated by our analysis with real-world and extensive simulated claims data, the OW method can yield nearly-perfect covariate balance while also retaining all of the sample. Therefore, OW can enhance the accuracy of ATE estimation over FS in most cases. Balancing covariates between treatment and control groups in observational studies can be challenging, especially in settings with infrequent outcomes and exposures. Both OW and FS methods can effectively balance covariates. These two different PS-based methods have been separately evaluated against other methods [ 8 , 9 , 10 , 11 , 12 , 13 , 14 , 17 ] but have never been compared against each other. We found that OW generally led to better covariate balance and model precision. However, in settings with extremely rare outcomes (≤ 1%) and exposures (≤ 10%), OW performed slightly worse than FS in at least one evaluation criterion. Future studies should analyze scenarios with rare outcomes and exposures in more detail. In conclusion, OW could be considered an effective and easy-to-implement method for balancing covariates for ATE estimation in settings with infrequent but not too rare outcomes and exposures.

Data availability

Data are available from the Centers for Medicare & Medicaid Services (CMS) under data use agreement provisions. Per the data use agreement, the relevant limited datasets cannot be made publicly available. For any data request, please contact CMS via the link: https://www.cms.gov/Research-Statistics-Data-and-Systems/Computer-Data-and-Systems/MedicaidDataSourcesGenInfo/MAXGeneralInformation. After obtaining Taxes State Medicaid claims data in 2012, the definitions of study population with diabetes and the diabetes-related hospitalization outcome can be found in the main study published in Knitter et al. (2022, Medical Care) via the link: https://pubmed.ncbi.nlm.nih.gov/36040020/. The authors confirm that we all did not have any special access privileges that others would not have.

Abbreviations

Average treatment effect in the total population

The ATE method to generate equal total weights between groups

The ATE method to generate unequal total weights between groups

Average treatment effect on the overlap population

Average treatment effect on the treated population

95% confidence interval

Probability of the 95% confidence interval that covers the true effect, ignoring whehter zero was covered or not

Probability of the 95% confidence interval (CI) that covers the true effect, but not zero

No balancing method used

Federally qualified health centers

Propensity score-based fine stratification method

The FS method with a full set of data and subjects’ weights assigned by ATE-equ

The FS method with a full set of data and subjects’ weights assigned by ATE-unequ

The FS method with a subset of data and subjects’ weights assigned by ATE-equ

The FS method with a subset of data and subjects’ weights assigned by ATE-unequ

Inverse probability weighting method

Mahalanobis balance

Propensity score-based Overlap weighting method

The OW method with a full set of data

The OW method with a subset of data

Propensity score

Propensity score matching method

Mean relative bias

Square root of mean squared error of bias

Standard deviation of rbias

Standardized mean difference between the two groups

King G, Nielsen R. Why Propensity scores should not be used for matching. Political Anal. 2019;27(4).

Ho DE, Imai K, King G, Stuart EA. Matching as nonparametric preprocessing for reducing Model Dependence in Parametric Causal Inference. Political Anal. 2007;15:199–236.

Article Google Scholar

Imai K, King G, Nall C. The essential role of pair matching in cluster-randomized experiments, with application to the Mexican Universal Health Insurance Evaluation. Stat Sci. 2009;24(1):29–53.

Iacus SM, King G, Porro G. Multivariate Matching methods that are Monotonic Imbalance Bounding. J Am Stat Assoc. 2011;106(493):345–61. https://doi.org/10.1198/jasa.2011.tm09599

Article CAS Google Scholar

Rosenbaum PR, Rubin DB. Reducing Bias in Observational studies using subclassification on the Propensity score. J Am Stat Assoc. 1984;79(387):516–24.

Desai RJ, Franklin JM. Alternative approaches for confounding adjustment in observational studies using weighting based on the propensity score: a primer for practitioners. BMJ. 2019;367:l5657. https://doi.org/10.1136/bmj.l5657 . PubMed PMID: 31645336.

Article PubMed Google Scholar

Austin PC. An introduction to Propensity score methods for reducing the effects of confounding in Observational studies. Multivar Behav Res. 2011;46(3):399–424. doi: 10.1080/00273171.2011.568786. PubMed PMID: 21818162; PubMed Central PMCID: PMCPMC3144483.

Desai RJ, Rothman KJ, Bateman BT, Hernandez-Diaz S, Huybrechts KF. A propensity-score-based Fine Stratification Approach for Confounding Adjustment when exposure is infrequent. Epidemiology. 2017;28(2):249–57. doi: 10.1097/EDE.0000000000000595. PubMed PMID: 27922533; PubMed Central PMCID: PMCPMC5497217.

Article PubMed PubMed Central Google Scholar

Stuart EA. Matching methods for causal inference: a review and a look forward. Stat Sci. 2010;25(1):1–21. Epub 2010/09/28. doi: 10.1214/09-STS313. PubMed PMID: 20871802; PubMed Central PMCID: PMCPMC2943670.

Lee BK, Lessler J, Stuart EA. Weight trimming and propensity score weighting. PLoS ONE. 2011;6(3):e18174. https://doi.org/10.1371/journal.pone.0018174 . PubMed PMID: 21483818; PubMed Central PMCID: PMCPMC3069059.

Article CAS PubMed PubMed Central Google Scholar

Hirano K, Imbens GW. Estimation of Causal effects using propensity score weighting: an application to data on Right Heart catheterization. Health Serv Outcomes Res Methodol Volume. 2001;2:259–78.

Li F, Morgan KL, Zaslavsky AM. Balancing covariates via Propensity score weighting. J Am Stat Assoc. 2018;113(521):390–400.

Li F, Thomas LE, Li F. Addressing Extreme Propensity Scores via the Overlap Weights. Am J Epidemiol. 2019;188(1):250-7. https://doi.org/10.1093/aje/kwy201 . PubMed PMID: 30189042.

Zhou Y, Matsouaka RA, Thomas L. Propensity score weighting under limited overlap and model misspecification. Stat Methods Med Res. 2020;29(12):3721–56. Epub 2020/07/23. doi: 10.1177/0962280220940334. PubMed PMID: 32693715.

Benedetto U, Head SJ, Angelini GD, Blackstone EH. Statistical primer: propensity score matching and its alternatives. Eur J Cardiothorac Surg. 2018;53(6):1112–7. https://doi.org/10.1093/ejcts/ezy167 . PubMed PMID: 29684154.

Chatton A, Borgne FL, Leyrat C, Foucher Y. G-computation and doubly robust standardisation for continuous-time data: a comparison with inverse probability weighting. Stat Methods Med Res. 2022;31(4):706–18. 10.1177/09622802211047345. PubMed PMID: 34861799.

Ripollone JE, Huybrechts KF, Rothman KJ, Ferguson RE, Franklin JM. Evaluating the utility of coarsened exact matching for Pharmacoepidemiology using real and simulated Claims Data. Am J Epidemiol. 2020;189(6):613–22. https://doi.org/10.1093/aje/kwz268 . PubMed PMID: 31845719; PubMed Central PMCID: PMCPMC7368132.

SAS. SAS/STAT 14.3 User’s Guide: The PSMATCH Procedure. SAS. 2017; https://support.sas.com/documentation/onlinedoc/stat/143/psmatch.pdf

Guo S, Fraser MW. Propensity score analysis: statistical methods and applications. Thousand Oaks, CA: Sage; 2015.

Google Scholar

Hong G. Marginal Mean Weighting through Stratification: Adjustment for Selection Bias in Multilevel Data. J Educational Behav Stat. 2010;35(5):499–531.

Franklin JM, Rassen JA, Ackermann D, Bartels DB, Schneeweiss S. Metrics for covariate balance in cohort studies of causal effects. Stat Med. 2014;33(10):1685–99. https://doi.org/10.1002/sim.6058 . PubMed PMID: 24323618.

Austin PC. Using the standardized difference to compare the prevalence of a Binary Variable between two groups in Observational Research. Commun Stat - Simul Comput. 2009;38(6):1228–34.

Ripollone JE, Huybrechts KF, Rothman KJ, Ferguson RE, Franklin JM. Implications of the Propensity score matching Paradox in Pharmacoepidemiology. Am J Epidemiol. 2018;187(9):1951–61. https://doi.org/10.1093/aje/kwy078 . PubMed PMID: 29750409; PubMed Central PMCID: PMCPMC6118075.

Yang S, Starks MA, Hernandez AF, Turner EL, Califf RM, O’Connor CM, et al. Impact of baseline covariate imbalance on bias in treatment effect estimation in cluster randomized trials: race as an example. Contemp Clin Trials. 2020;88:105775. PubMed PMID: 31228563; PubMed Central PMCID: PMCPMC8337048.

Franklin JM, Schneeweiss S, Polinski JM, Rassen JA. Plasmode simulation for the evaluation of pharmacoepidemiologic methods in complex healthcare databases. Comput Stat Data Anal. 2014;72:219–26. https://doi.org/10.1016/j.csda.2013.10.018 . PubMed PMID: 24587587; PubMed Central PMCID: PMCPMC3935334.

Vaughan LK, Divers J, Padilla M, Redden DT, Tiwari HK, Pomp D, et al. The use of plasmodes as a supplement to simulations: a simple example evaluating individual admixture estimation methodologies. Comput Stat Data Anal. 2009;53(5):1755–66. https://doi.org/10.1016/j.csda.2008.02.032 . PubMed PMID: 20161321; PubMed Central PMCID: PMCPMC2678733.

Franklin JM, Eddings W, Glynn RJ, Schneeweiss S. Regularized regression Versus the high-dimensional propensity score for Confounding Adjustment in secondary database analyses. Am J Epidemiol. 2015;182(7):651–9. https://doi.org/10.1093/aje/kwv108 . PubMed PMID: 26233956.

Greifer N, Estimating Effects After M. 2022; https://cran.r-project.org/web/packages/MatchIt/vignettes/estimating-effects.html

Liang K-Y, Zeger S. Longitudinal data analysis using generalized linear models. Biometrika. 1986;73(1):13–22.

Blackwell M, Iacus S, King G. Cem: coarsened exact matching in Stata. Stata J. 2009;9(4):524–46.

Trikalinos T, Hoaglin D, Schmid C, Empirical. and Simulation-Based Comparison of Univariate and Multivariate Meta-Analysis for Binary Outcomes. 2013; https://www.ncbi.nlm.nih.gov/books/NBK132565/table/methods.t9/

Romano JL, Kromrey JD, Hibbard ST. A Monte Carlo Study of eight confidence interval methods for Coefficient Alpha. Educ Psychol Meas. 2010;70(3):376–93.

Rainey C. Dealing with separation in logistic regression models. Political Anal. 2016;24:339–55.

Heinze G, Schemper M. A solution to the problem of separation in logistic regression. Stat Med. 2002;21(16):2409–19. https://doi.org/10.1002/sim.1047 . Epub 2002/09/05.

Matsouaka RA, Zhou Y. A framework for causal inference in the presence of extreme inverse probability weights: the role of overlap weights. Math arXiv: Methodol. 2020; https://arxiv.org/pdf/2011.01388.pdf

Sjoding MW, Luo K, Miller MA, Iwashyna TJ. When do confounding by indication and inadequate risk adjustment bias critical care studies? A simulation study. Crit Care. 2015;19:195. https://doi.org/10.1186/s13054-015-0923-8 . Epub 2015/05/01.

Thomas LE, Li F, Pencina MJ. Overlap weighting: a propensity score method that mimics attributes of a Randomized Clinical Trial. JAMA. 2020;323(23):2417–8. https://doi.org/10.1001/jama.2020.7819 . PubMed PMID: 32369102.

Zhao P, Su X, Ge T, Fan J. Propensity score and proximity matching using random forest. Contemp Clin Trials. 2016;47:85–92. https://doi.org/10.1016/j.cct.2015.12.012 . Epub 2015/12/27.

Download references

Acknowledgements

We would like to thank Neda Laiteerapong, MD, and her team members for inputs on how to identify diabetic adults in Medicaid. Furthermore, we would like to express our gratitude to all reviewers and editors who made excellent comments and helped us strengthen the work greatly.

All authors have no financial/commercial conflicts of interests. This study was funded by Health Resources and Services Administration (HRSA) (HHSH250201300025I) (MPI: Huang and Chin). Drs. Wan, Chin, and Huang were supported in part by the Chicago Center for Diabetes Translation Research (NIDDK P30 DK092949).

Author information

Authors and affiliations.

Section of General Internal Medicine, Department of Medicine, The University of Chicago, 5841 S. Maryland Ave, Chicago, MC, IL, 2007, 60637, USA

Wen Wan, Marshall H. Chin & Elbert S. Huang

Department of Public Health Sciences, Department of Medicine, The University of Chicago, Chicago, IL, USA

Manoradhan Murugesan & R. Tamara Konetzka

Health Systems Science, Kaiser Permanente Bernard J. Tyson School of Medicine, Pasadena, CA, USA

Robert S. Nocon

Department of Information Systems, University of Maryland, Baltimore, MD, USA

Joshua Bolton

You can also search for this author in PubMed Google Scholar

Contributions

WW, MM, RSN, MHC, and ESH had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: All authors.

Acquisition, analysis, or interpretation of data: All authors.

Drafting of the manuscript: WW.

Critical revision of the manuscript for important intellectual content: All authors.

Statistical analysis: WW

Obtained funding: ESH and MHC.

Administrative, technical, or material support: All authors.

Study supervision: All authors.

Corresponding author

Correspondence to Wen Wan .

Ethics declarations

Ethics approval and consent to participate.

The study was reviewed by the University of Chicago Institutional Review Board and determined to be non-human subject research. There was no specific informed consent for the study due to de-identification of the Medicaid data purchased from the Centers for Medicare and Medicaid Services (CMS) under a data use agreement.

Competing interests

The authors declare no competing interests.

Consent for publication

Not applicable.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Wan, W., Murugesan, M., Nocon, R.S. et al. Comparison of two propensity score-based methods for balancing covariates: the overlap weighting and fine stratification methods in real-world claims data. BMC Med Res Methodol 24 , 122 (2024). https://doi.org/10.1186/s12874-024-02228-z

Download citation

Received : 07 December 2023

Accepted : 23 April 2024

Published : 03 June 2024

DOI : https://doi.org/10.1186/s12874-024-02228-z

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Fine stratification method (FS)
Covariate balance
Plasmode simulation method
Propensity score (PS)
Average treatment effect (ATE)

BMC Medical Research Methodology

ISSN: 1471-2288

General enquiries: [email protected]

The independent source for health policy research, polling, and news.

Understanding the Intersection of Medicaid & Work: A Look at What the Data Say

Madeline Guth, Patrick Drake , Robin Rudowitz , and Maiss Mohamed Published: Apr 24, 2023

Issue Brief
Appendix Tables

While data show that the majority of Medicaid enrollees are working, there has been long-standing debate about imposing work requirements in Medicaid. For the first time in the history of the Medicaid program, the Trump Administration approved Section 1115 waivers that included work and reporting requirements as a condition of Medicaid eligibility in some states. However, courts struck down many of these requirements and the Biden Administration withdrew these provisions in all states that had approvals. Arkansas was the only state to implement Medicaid work requirements with consequences for noncompliance.

Work requirements are now back on the agenda as some Congressional Republicans have indicated that they will rely on a budget outline that would require Medicaid enrollees to work, or look for work, in order to receive coverage. In a speech on April 17, Speaker McCarthy emphasized work requirements as part of negotiations to increase the debt limit, and such requirements were included in the Republicans’ proposed debt limit bill released on April 19. In addition, Republican legislators in several states have proposed seeking work requirement waivers. In July 2023, Georgia intends to expand Medicaid eligibility to 100% of the federal poverty level (FPL), with initial and continued enrollment conditioned on meeting work requirements (after a federal judge overturned the Biden Administration’s withdrawal of Georgia’s work requirement).

Experience in Arkansas and earlier estimates of implementing work requirements nationally suggest that many could lose coverage due primarily with barriers in meeting work reporting requirements. An analysis from the Congressional Budget Office (CBO) found that a national Medicaid work requirement would result in 2.2 million adults losing Medicaid coverage per year (and subsequently experiencing increases in medical expenses), and lead to only a very small increase in employment. CBO estimates that this policy would decrease federal spending by $15 billion annually due to the reduction in enrollment. New attention on work and reporting requirements come as millions are at risk of losing coverage due to administrative barriers as states resume routine renewals and disenrollments with the unwinding of the Medicaid continuous enrollment provision that was included in the Families First Coronavirus Response Act (FFCRA) enacted at the start of the COVID-19 pandemic.

To provide context to these debates, this brief explores work status and characteristics of Medicaid enrollees in 2021 to answer three key questions:

What is the work status of Medicaid covered adults?

What do we know about Medicaid adults who are working?

What do we know about the intersection of work and health and the impact of Medicaid work requirements?

These data show that most Medicaid covered adults were either working or faced a barrier to work, leaving just nine percent of enrollees who could be directly targeted by work requirement policies.

What is the work status of Medicaid adults?

Many Medicaid adults who are not working face barriers to moving into employment, such as functional disability. Even if they do not qualify for Medicaid on the basis of a disability through SSI, many adults on Medicaid have high rates of functional disability and serious medical conditions, especially among those not working. Approximately 17% of Medicaid adults have a functional disability, with the highest rates of disability among Medicaid adults not in the labor force (27%) (data not shown). Medicaid adults may also experience mental health conditions that impede their ability to work, with about one in three (30%) non-working Medicaid adults reporting depression. 1

What do we know about the intersection of work and health and the impact of work requirements?

Medicaid can support employment by providing affordable health coverage, and research suggests that the effects of work requirements on health and employment are likely limited. Research shows that being in poor health is associated with increased risk of job loss, while access to affordable health insurance has a positive effect on the ability to obtain and maintain employment. Medicaid coverage helps low-wage workers get care that enables them to remain healthy enough to work. Also, states may launch initiatives , such as voluntary employment referral programs, to support employment for Medicaid enrollees without making employment a condition of eligibility. In focus groups, enrollees report that Medicaid coverage helps them to manage chronic conditions and supports their ability to work jobs that may be physically demanding. However, a review of research on the relationship between work and health found that although there is strong evidence of an association between unemployment and poorer health outcomes, there is limited evidence on the effect of employment on health. Further, research from other public programs, including TANF and SNAP , suggests that work requirements have had little impact on increasing employment. A CBO report finds that earnings associated with employment gains due to TANF and SNAP work requirements were offset by loss of income for those no longer eligible for the programs.

In part due to evidence on the impacts of work requirements, courts and the Biden Administration determined that such requirements do not further Medicaid program objectives. A January 2021 executive order from President Biden directed HHS to review waiver policies that may undermine Medicaid. CMS subsequently withdrew Medicaid work requirement waivers in all states that had approvals. Previously, in 2020 the DC appeals court affirmed that the Trump Administration’s approvals of work requirements in Arkansas and New Hampshire were unlawful because the Secretary failed to consider the impact on coverage; before leaving office, the Trump Administration asked the Supreme Court to reverse these decisions. After the Biden Administration withdrew the Arkansas and New Hampshire work requirements, the Administration asked the Supreme Court to vacate the lower court decisions and dismiss the Arkansas case as moot (as that waiver had expired) and send the New Hampshire case back to HHS (as New Hampshire had not asked the Court to review the case involving its waiver). In April 2022, the Court granted this motion, effectively putting an end to the pending litigation. This dismissal does not preclude a future presidential administration from revisiting work requirements; however, any future work requirements approved would likely face legal challenges.

Looking Ahead

Right now, Georgia is the only state with an approved work requirement waiver, as a Federal District Court judge vacated the Biden Administration’s withdrawal. Once implemented in July 2023, Georgia’s waiver will expand Medicaid eligibility to 100% of the federal poverty level (FPL), with initial and continued enrollment conditioned on meeting work and premium requirements. Section 1115 monitoring and evaluation requirements will require Georgia to track and report the number of enrollees who gain and maintain coverage. As only Arkansas has implemented Medicaid work requirements with consequences for noncompliance, the results of monitoring and evaluation in Georgia will provide further evidence as to the impacts of work requirements—however, Georgia is unique in applying work requirements to a new coverage group rather than to an existing Medicaid population.

Additionally, other states have indicated they may pursue work requirement waivers in the future, and Congressional Republicans have recently discussed a federal Medicaid work requirement tied to approval to raise the debt limit. Although the Biden Administration has said work requirements do not further Medicaid objectives, a future presidential administration could revisit this view and allow state waivers (though any future work requirements approved via waiver could face legal challenges).

Work Requirements
Medicaid's Future

Also of Interest

An Overview of Medicaid Work Requirements: What Happened Under the Trump and Biden Administrations?
Medicaid Waiver Tracker: Approved and Pending Section 1115 Waivers by State
Medicaid Work Requirements are Back on the Agenda

Systematic Review
Open access
Published: 06 June 2024

Non-pharmaceutical interventions in containing COVID-19 pandemic after the roll-out of coronavirus vaccines: a systematic review

Xiaona He 1 , 2 ,
Huiting Chen 1 , 2 ,
Xinyu Zhu 1 , 2 &
Wei Gao 1 , 2

BMC Public Health volume 24 , Article number: 1524 ( 2024 ) Cite this article

19 Accesses

Metrics details

Non-pharmaceutical interventions (NPIs) have been widely utilised to control the COVID-19 pandemic. However, it is unclear what the optimal strategies are for implementing NPIs in the context of coronavirus vaccines. This study aims to systematically identify, describe, and evaluate existing ecological studies on the real-world impact of NPIs in containing COVID-19 pandemic following the roll-out of coronavirus vaccines.

We conducted a comprehensive search of relevant studies from January 1, 2021, to June 4, 2023 in PubMed, Embase, Web of science and MedRxiv. Two authors independently assessed the eligibility of the studies and extracted the data. A risk of bias assessment tool, derived from a bibliometric review of ecological studies, was applied to evaluate the study design, statistical methodology, and the quality of reporting. Data were collected, synthesised and analysed using qualitative and quantitative methods. The results were presented using summary tables and figures, including information on the target countries and regions of the studies, types of NPIs, and the quality of evidence.

The review included a total of 17 studies that examined the real-world impact of NPIs in containing the COVID-19 pandemic after the vaccine roll-out. These studies used five composite indicators that combined multiple NPIs, and examined 14 individual NPIs. The studies had an average quality assessment score of 13 (range: 10–16), indicating moderately high quality. NPIs had a larger impact than vaccination in mitigating the spread of COVID-19 during the early stage of the vaccination implementation and in the context of the Omicron variant. Testing policies, workplace closures, and restrictions on gatherings were the most effective NPIs in containing the COVID-19 pandemic, following the roll-out of vaccines. The impact of NPIs varied across different time frames, countries and regions.

NPIs had a larger contribution to the control of the pandemic as compared to vaccination during the early stage of vaccine implementation and in the context of the omicron variant. The impact of NPIs in containing the COVID-19 pandemic exhibited variability in diverse contexts. Policy- and decision-makers need to focus on the impact of different NPIs in diverse contexts. Further research is needed to understand the policy mechanisms and address potential future challenges.

Peer Review reports

Since the availability of COVID-19 vaccines, governments worldwide have implemented vaccination and non-pharmaceutical interventions (NPIs) such as testing policies, gathering restrictions, facial covering policies, school closures, workplace closures to contain local transmission of COVID-19 [ 1 , 2 ]. The NPIs, also known as public health measures, aim to break infection chains by altering key aspects of our behavior. Extensive research has been dedicated to examining the impact of NPIs in controlling the outbreak of COVID-19 [ 3 , 4 , 5 ].

Before the COVID-19 pandemic, there existed literature in addressing the impact of NPI implementation on influenza pandemic [ 6 ]. However, a key challenge in this topic is the limited evidence regarding the impact of NPIs, which predominantly relies on mathematical modelling with a limited number of empirical studies [ 7 , 8 , 9 ].

Considering the potential harm posed by respiratory infectious disease outbreaks and the high social and economic costs associated with implementing various NPIs, it is essential to conduct research that examines the impact of NPIs in controlling pandemics in real-world settings. Mendez et al. conducted a systematic review and identified that school closures, workplace closures, business and venue shutdowns, and public event restrictions as the most effective measures in controlling the real-world spread of COVID-19 [ 7 ].

However, various countries implemented diverse NPIs at different stages of the pandemic to control the spread of COVID-19, especially after the introduction of coronavirus vaccines. Asian countries consistently enforced strict NPIs throughout the first half of 2021 [ 10 ], while no NPIs were implemented in France after May 2021 [ 11 ]. At the early stage of vaccine roll-out, vaccination coverage in most countries remained relatively low [ 2 ]. As of June 30, 2021, a total of 29.29% of the world’s population had received at least one dose of the vaccine, with significant variations in vaccination coverage across countries [ 2 ]. Despite an increase in vaccination rates in many countries during the latter half of 2021, the number of confirmed new COVID-19 cases remained high worldwide due to the prevalence of the highly transmissible and immune-escape Delta variant in the second half of 2021 [ 12 ], followed by the emergence of the Omicron variant in early 2022 [ 13 ]. Yet, with the increase in COVID-19 vaccination rates, there has been a reduction in mortality and morbidity despite the high level of transmission. This indicates that widespread vaccine coverage has played a positive role in mitigating the health impacts of the disease.

The impact of NPIs in controlling the COVID-19 pandemic after the roll-out of vaccines has also received considerable attention [ 14 ]. Nevertheless, the policy mechanisms underlying their impact, such as determining when to implement stricter lockdown measures or when to ease restrictions, as well as identifying which types of NPIs are more suitable for different stages, remain unclear.

This review focuses on investigating the real-world impact of NPIs in containing the COVID-19 pandemic after vaccine roll-out, in order to search for optimal strategies for implementing NPIs. We summarize the current evidence from the real world on the impact, aiming to deepen the current understanding, fill in the gaps in the topics, and provide evidence for the future.

The reporting of this review was guided by the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement [ 15 ]. See Supplemental file for further details. This review was registered at the international prospective register of systematic reviews (PROSPERO; CRD42023411560).

Data sources and searches

We conducted a comprehensive search of relevant literature in Embase, PubMed, and Web of Science, and preprints on MedRxiv from January 1, 2021 to June 4 2023. Our search was limited to articles written in English. The search terms included NPIs, COVID-19 and vaccination, which were detailed in Table S1 of the supplemental file. We used EndNote (version 20.0) software to process and remove duplicates. In addition, we manually searched for citations and related articles of the included studies using Google Scholar.

Study selection and eligibility criteria

One author (XH) screened eligible studies by reviewing the titles and/or abstracts of searched articles using EndNote (version 20.0). If an article was deemed relevant or if the information provided in the title or abstract was insufficient to make a decision, the full texts were retrieved and examined. For all eligible studies( n = 182), two independent authors (XH and HC) assessed the eligibility criteria for each study by evaluating the full text and determining inclusion or exclusion. Any discrepancies between authors were resolved by discussions with the third reviewer (XZ) and the senior author (WG) to reach a consensus.

In general, we adopted an inclusive approach by retaining all studies that could not be excluded with high confidence. All decisions were documented in a spreadsheet. Studies were included in the review if they: (1) assessed the impact of NPIs during the roll-out of COVID-19 vaccines; (2) evaluated the impact of NPIs and vaccination coverage using real-world data; (3) analyzed the respective/interactive impact of NPIs and vaccination coverage; (4) assessed the impact at least one type of NPIs; (5) measured at least one health outcome; (6) obtained evidence through ecological study. Studies were excluded from the review if they: (1) were based on forecasts or simulations; (2) analyzed the impact of adherence or compliance to NPIs and intention or willingness to vaccination; (3) assessed NPI impact in controlling other diseases; (4) did not directly assess the impact of NPIs.

Quality assessment

To assess the quality of studies, we used a risk of bias assessment tool based on a bibliometric review of ecological studies, as proposed by Dufault et al. [ 16 ]. This tool has been previously used and adapted in recent reviews [ 7 , 17 , 18 ]. The purpose of the risk of bias assessment tool is to critically evaluate study design, statistical methodology and practices, and the quality of reporting. Two independent reviewers (XH and HC) evaluated the risk of bias for each included study. Any discrepancies between the reviewers were resolved by discussion with a third reviewer (XZ) and the senior author (WG) to reach a consensus. The checklist of risk of bias assessment tool was included in the Table S2 of Supplemental File.

Data synthesis and analysis

Characteristics and outcomes of individual studies were extracted, including study authors, year, setting, study design, duration of study, type and/or intensity of NPIs, vaccination coverage, assessment indicators of outcome such as time varying reproduction number (Rt), the number of daily new cases or deaths. The classification and intensity of NPIs were mainly based on information from a global panel database of pandemic policies (Oxford COVID-19 Government Response Tracker) [ 1 ].

In the final stage of the systematic review, we synthesised the findings from all eligible ecological studies( n = 17) to determine the real-world impact of NPIs in containing the COVID-19 pandemic after the vaccine roll-out. Data were collected, synthesized, and analyzed using quantitative and qualitative approaches. Specifically, we collected the impact of NPIs and vaccination reported in each included study, and then summarized and compiled the main results. We have considered not only the impact of individual NPIs but also collected data on the impact measured by a composite indicator of NPIs. This holistic approach is supported by the fact that most countries implemented multiple NPIs as a package to mitigate the spread of COVID-19 during the pandemic. The results were presented using summary tables and figures, including the target countries and regions of the studies, NPIs types, evidence quality.

Summary of literature screening and background

Seventeen ecological studies were included in the review, of which fourteen were published and three were preprints. The PRISMA diagram flow is presented in Fig. 1 . For more information on excluded articles and reasons for their exclusion, please refer to Table S3 in the Supplemental File. These studies encompass research samples from over 88% of countries and regions worldwide, with each study focusing on a different geographical scope. Table 1 provides a breakdown of the studies: eight evaluated the impact of NPIs in containing the COVID-19 pandemic on a global scale [ 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 ], three focused on Europe [ 27 , 28 , 29 ], two on the United States [ 30 , 31 ], one on Asia [ 10 ], and one each on India [ 32 ], France [ 11 ] and Korea [ 33 ].

PRISMA flow diagram for the selection of studies

The seventeen studies examined the impact of NPIs on the COVID-19 pandemic during different periods. Eight studies evaluated their impact during the early stage of vaccine roll-out (before July 2021), five during the later stage (the second half of 2021), and the remaining four during the Omicron stage (contain 2022 period).

In terms of quality assessment, the seventeen studies received a moderately high score, averaging 13 (range: 10–16) out of a maximum score of 17. This score reflects the strength of the evidence. The primary sources of risk of bias were the quality of reporting, validity of regression, control of covariates, and internal validity of the methodology. Detailed assessment records can be found in Table S4. Studies used different statistical methods and outcomes, and conducted sensitivity analyses. Only a few studies considered the impact of seasonality.

Study characteristics

Researchers examined various types of NPIs in the seventeen identified studies. Nine studies evaluated the overall impact of a composite indicator of NPIs (Table 2 ), while twelve studies specifically assessed the impact of individual NPIs (Table 3 ). Bollyky et al., Paireau et al., Li et al., and Hongjian Wang et al. analyzed both the impact of the composite indicator of NPIs and individual NPIs. More detailed information including the data sources and the analytical methods used in the studies can be found in Supplemental File (Table S5 ).

The composite indicator of NPIs in this review primarily included five types: stringency index, policy mandates, social distancing policy index, lockdown, and combination of four NPIs (school closure, workplace closure, restrictions on mass gatherings, and stay-at-home requirements). The specific names for these measures varied depending on their sources. The data on NPIs in these studies mainly came from OxCGRT, with additional sources including The Yale State and Local COVID-19 restriction database, governmental websites, and others. These composite indicators were calculated by combining multiple containment and closure measures, representing the overall intensity of various containment and closure policies to some extent. The individual NPIs included containment and closure measures, as well as health systems indicators. Containment and closure measures primarily encompassed restrictions on gatherings, school closures, workplace closures, and stay-at-home requirements. The evaluated health systems indicators included testing policy, facial coverings, contact tracing, public information campaigns, and vaccine mandates.

The studies on vaccination included data on the administration of the first dose, full vaccination, and booster doses according to the vaccination protocol. Two studies did not explicitly specify the doses. The outcome assessed mainly included cases, deaths, Rt, and others.

The impact of composite indicator of NPIs for containing the COVID-19 pandemic after the roll-out of COVID-19 vaccines

As shown in Table 4 , even after the introduction of vaccines, the implementation of containment and closure measures continued to have a impact on curtailing the spread of COVID-19. Some studies also compared the impact of NPIs during that period with the impact of vaccination coverage in mitigating the transmission of COVID-19 within populations. Specifically, NPIs were found to have a larger impact than vaccination in mitigating the spread of COVID-19 during early stage of the vaccination implementation. In the latter half of 2021, the impact of NPIs had relatively diminished compared to the earlier stages. During the Omicron stage, the measures implemented to control the spread of COVID-19 had a larger impact than vaccination coverage. More detailed information is provided in Table S6.

In addition to assessing the impact of combinations of containment and closure measures in controlling the spread of COVID-19(cases and Rt), NPIs were also considered in reducing the number of COVID-19 related deaths. Bollyky et al. ‘s study did not find evidence supporting the impact of NPIs in reducing COVID-19 related deaths, while other studies (Zhou et al., Hale et al. and Caixia Wang et al.) suggested that NPIs could reduced the number of deaths.

The relative impact of individual NPIs for containing the COVID-19 pandemic after the roll-out of COVID-19 vaccines

Twelve studies have evaluated the individual impacts of NPIs on controlling the COVID-19 pandemic after the roll-out of COVID-19 vaccines. A total of 14 individual NPIs were assessed in this review. Among them, testing policies, facial coverings, and school closures were mentioned most frequently. Additionally, workplace closures, restrictions on gatherings, stay-at-home requirements, and restrictions on international travel were also highlighted. Figure 2 demonstrates these findings.

The relative impact of individual NPIs for containing the COVID-19 pandemic after the roll-out of COVID-19 vaccines. The Y-axis represents the count of assessments for this NPIs. The colors of the stacked bar represent the impact of assessed NPIs in containing the COVID-19 pandemic following the roll-out of vaccines. The purple color indicates that a study considers the NPI to be the most effective measure. The blue color signifies limited impact or a lack of association with containing the COVID-19 pandemic, according to the study. The brown color represents a negative correlation between the NPI and containing the spread of COVID-19

Moreover, there is consistent evidence suggesting that testing policies (4/6), workplace closures (3/5), and restrictions on gatherings (3/5) may be the most effective containing the COVID-19 pandemic following the roll-out of vaccines.

We categorised the included studies according to the target countries and regions, as shown in Table 5 . The types of NPIs evaluated and the impact NPIs identified varied across different geographical locations. Based on global studies, testing policy, restrictions on gathering size, workplace closure, school closure, stay-at-home requirements, and restrictions on international travel were found to be relatively effective. In Asian studies, restrictions on gathering size and the closure of public transport were considered as effective measures. In studies conducted in the United States, only vaccination mandates were deemed effective. Data from France indicated that curfews were effective. Face covering mandates were associated with a decrease in COVID-19 incidence in European countries. Testing was effective in India during the vaccination stage.

Summary of the main findings

The types of NPIs evaluated and the impact of NPIs identified varied across different periods and geographical locations. Overall, our research shows that NPIs continued to be effective in controlling the spread of COVID-19 even after the roll-out of vaccines. Our previous research work also supports this conclusion [ 34 ]. The most frequently evaluated NPIs included testing policies, facial coverings, and school closures, followed by workplace closures, restrictions on gatherings, stay-at-home requirements, and restrictions on international travel.

The overall impact of a composite indicator of NPIs varied depending on the period, with factors such as the intensity of implementation, compliance, increasing vaccine coverage, and the emergence of VOCs playing a role. NPIs remained important for mitigating the pandemic in the early stage of the vaccination when coverage was low [ 11 , 27 , 29 ]. However, as vaccine coverage increased, their marginal effects were surpassed by vaccination [ 27 , 33 ]. In Omicron stage, measures were more effective in controlling the spread of COVID-19 than vaccination coverage due to the high immune evasion capability of the Omicron variant [ 13 , 26 ]. It is important to note that NPIs and vaccinations work through different mechanisms to combat the pandemic [ 35 ]. NPIs physically reduce population contact and transmission of the virus, while vaccinations reduce susceptible populations by enhancing immunity. Overall, a combination of both containment and closure measures and vaccination is recommended to contain COVID-19 after the vaccine has been introduced [ 11 , 21 , 26 , 27 , 29 , 33 ].

The types of the evaluated NPIs, as well as the effective NPIs, varied across different target countries and regions. Factors such as differences in government effectiveness [ 22 ], public awareness and behavioral responses to the prevention and control measures [ 21 ], and economic disparities among different countries and regions may affect the impact of various NPIs [ 36 ].

Testing policies is a central pillar of public health response to global health emergencies. In the included studies, testing policies were primarily evaluated during the Omicron period. Nesteruk [ 23 ], Nesteruk et al. [ 24 ], Wang et al. [ 26 ], and Shin et al. [ 32 ] found that strengthening testing could reduce the number of COVID-19 infections. Faster and decentralised nucleic acid testing technology may has the potential to be implemented on a larger scale in the community, help control the pandemic [ 37 ], reduce the need for strict control measures, and accelerate the recovery of social and economic activities. In addition, Shao et al. also found that large-scale SARS-CoV-2 rapid antigen testing alleviated the Omicron outbreak in China [ 38 ]. Moreover, it is crucial to ensure accessibility to COVID-19 tests (e.g., availability and familarity with COVID-19 tests), bolster public confidence in governmental control measures, and increase understanding of and perceived susceptibility to COVID-19 [ 39 , 40 , 41 , 42 ]. These efforts collectively help reduce barriers to testing, improve public willingness, and ultimately encourage individuals to participate in testing voluntarily.

Facial coverings are a form of personal protective equipment used to shield the face from various external hazards like splashes, droplets, and aerosols. Among the summarized evidence, only one study [ 28 ] found that the incidence of COVID-19 was significantly higher after the relaxation of face covering mandates. Other studies found no association between the implementation of facial covering policies and a reduction in COVID-19 cases [ 10 , 19 , 22 , 30 , 31 ]. Facial covering policies do not represent the actual use of masks for preventing infections but rather serve as public health measures. The impact of facial covering policies depends on compliance with the policies, proper mask usage, and the duration of mask-wearing. Bollyky et al. found no evidence that implementing facial covering policies reduced the number of COVID-19 infections, but they did observe an association between mask use and lower rates of COVID-19 infection [ 30 ]. Furthermore, it is important to note that a limitation of the mask evidence is the absence of standardized regulation or reporting regarding the type of masks employed. This variability in mask types may impact the assessment and comparison of mask impacts across studies.

The impact of school closures in reducing COVID-19 infections appears to be controversial. Liang et al. [ 22 ] and Li et al. [ 21 ] argued that school closures were associated with mitigating the spread of COVID-19, while Paireau et al. [ 11 ] and Ge et al. [ 19 ] suggested that their impact was limited. Conversely, Huy et al. [ 10 ] discovered that the policy of school closure had the opposite effect on the reduction of infection rate. A previous study found that the policy had a potential for effectively reducing influenza transmission [ 43 ]. However, the optimum strategy of the policy of school closures remains unclear, whether in controlling the spread of influenza or COVID-19 pandemic.

Liang et al. [ 22 ], Li et al. [ 21 ], and Ge et al [ 19 ]. considered workplace closure as an effective intervention in containing the spread of COVID-19, after the roll-out of coronavirus vaccines. Modeling studies estimated that implementing only workplace social distancing measures could reduce the median cumulative incidence of influenza in the general population by 23% from 2000 to 2017 [ 44 ].

Gathering restrictions are primarily implemented to curb the spread of infectious diseases by reducing interpersonal contact, which can occur through various transmission pathways such as droplets, direct contact, and aerosols [ 45 ]. A previous study indicated that restrictions on gathering had the greatest contribution (37.60%) to suppressing influenza transmission during the 2019–2020 influenza season [ 46 ]. Different levels of gathering restrictions have shown varying impact. According to the categorization by OxCGRT, the levels of restrictions on gatherings range from strictest to the weakest, including limitations on gatherings of 10 or fewer people, 11–100 people, 101–1000 people, and gatherings with 1000 or more individuals [ 1 ]. Studies by Huy et al. suggested that limiting gatherings to 10 or fewer people was most strongly correlated with a decrease in COVID-19 case numbers [ 10 ]. A similar finding was also supported by research conducted by Liang et al [ 22 ].

Stay-at-home orders [ 21 ], restrictions on international travel [ 19 ], public transport closures [ 10 ], vaccine mandates [ 30 ], and curfews [ 11 ] have been identified as effective measures in controlling the spread of COVID-19 according to a minority of included studies after the introduction of vaccines. Additionally, there is no evidence to suggest that restrictions on internal movement [ 10 , 19 , 21 ], public information campaigns [ 10 , 22 ], and contact tracing [ 10 , 22 ] were associated with a reduction in the transmission of COVID-19. Considering the number and heterogeneity of existing evidences, further research is needed to identify the impact and mechanisms of the implementation of these NPIs in controlling the spread of COVID-19.

There is controversy surrounding whether NPIs can effectively reduce COVID-19 deaths. Studies have shown that NPIs do not directly reduce the number of COVID-19 deaths [ 47 ]. However, research conducted by Hale et al. [ 20 ] and Wang et al. [ 25 ] found that, a higher stringency index was associated with a lower average daily death toll. It is possible that NPIs indirectly reduce the number of deaths by mitigating the spread of COVID-19. Nevertheless, these studies lack analysis or explanation regarding the specific indirect impacts.

The research on NPIs’ impact in reducing the transmission of infectious diseases, especially respiratory ones like SARS, influenza, and COVID-19, has always received attention. However, our understanding of the impact of these measures in controlling respiratory infectious diseases is still not comprehensive enough, even in the context of the ongoing COVID-19 pandemic, particularly since the introduction of vaccines.

Strengths and limitations

This study has several strengths. Firstly, we conducted a systematic and comprehensive search across various databases to investigate the real-world impact of NPIs in containing the COVID-19 pandemic post-vaccine roll-out. Secondly, we employed a risk of bias assessment tool to critically assess the potential biases in the included studies. Thirdly, we summarized and analyzed the available evidence using quantitative and qualitative approaches, presenting the findings in tables and figures. Nonetheless, our study also has limitations. Firstly, we included three preprints that had not been peer-reviewed, although we did evaluate their risk of bias. Secondly, due to variations in study design, analytical methodologies, and outcome measures, we were unable to perform a meta-analysis and provide numerical estimates of impact. Thirdly, these studies primarily focused on the impact of NPIs during the first half of 2021. The available evidence be limited for conducting comparative analyses of the impact of NPIs at different stages of the epidemic curves or in communities utilizing different types of vaccines. Additionally, studies that evaluated the impact of NPIs included the post-vaccine rollout period but without considered vaccine coverage were excluded, despite the studies may provide insightful evidences. However, it is difficult to draw inferences about the impacts of NPIs after the vaccine rollout from these studies. Lastly, the evidence derived from the included studies was limited as they relied on retrospective and observational data, which cannot establish a causal relationship between NPIs and outcomes due to potential confounding variables.

In conclusion, the understanding of the impact of NPIs in mitigating the pandemic post-vaccination is inadequate. NPIs had a larger contribution to the control of the pandemic as compared to vaccination during the early stage of vaccine implementation and in the context of the omicron variant. It is recommended to tailor NPIs based on factors like vaccination rates and variants with strong immune evasion, instead of lifting them suddenly, during early phases of vaccine roll-out in future pandemics. Various studies showed NPIs had varying impacts on curbing the COVID-19 pandemic. Policy- and decision-makers need to focus on the impact of different NPIs in diverse contexts, to determine when to ease or reinforce restrictions. It is essential to comprehend the policy mechanisms of these intervention measures in controlling the spread of COVID-19 and other respiratory infectious diseases, such as influenza.

Data availability

All data were collected from publicly available literatures, and all data generated or analyzed during this study are included in this article and its supplemental files.

Abbreviations

Non-pharmaceutical interventions

Coronavirus disease 2019

Time varying reproduction number

Hale T, Angrist N, Goldszmidt R, Kira B, Petherick A, Phillips T, et al. A global panel database of pandemic policies (Oxford COVID-19 Government Response Tracker). Nat Hum Behav. 2021;5(4):529–38.

Article PubMed Google Scholar

Mathieu E, Ritchie H, Ortiz-Ospina E, Roser M, Hasell J, Appel C, et al. A global database of COVID-19 vaccinations. Nat Hum Behav. 2021;5(7):947–53.

Hsiang S, Allen D, Annan-Phan S, Bell K, Bolliger I, Chong T, et al. The effect of large-scale anti-contagion policies on the COVID-19 pandemic. Nature. 2020;584(7820):262–7.

Article CAS PubMed Google Scholar

Islam N, Sharp SJ, Chowell G, Shabnam S, Kawachi I, Lacey B et al. Physical distancing interventions and incidence of coronavirus disease 2019: natural experiment in 149 countries. BMJ. 2020;370.

Liu Y, Morgenstern C, Kelly J, Lowe R, Jit M. The impact of non-pharmaceutical interventions on SARS-CoV-2 transmission across 130 countries and territories. BMC Med. 2021;19(1):1–12.

Article Google Scholar

Bootsma MC, Ferguson NM. The effect of public health measures on the 1918 influenza pandemic in US cities. Proceedings of the National Academy of Sciences. 2007;104(18):7588-93.

Mendez-Brito A, El Bcheraoui C, Pozo-Martin F. Systematic review of empirical studies comparing the effectiveness of non-pharmaceutical interventions against COVID-19. J Infect. 2021;83(3):281–93.

Article CAS PubMed PubMed Central Google Scholar

Perra N. Non-pharmaceutical interventions during the COVID-19 pandemic: a review. Phys Rep. 2021;913:1–52.

Verelst F, Willem L, Beutels P. Behavioural change models for infectious disease transmission: a systematic review (2010–2015). J Royal Soc Interface. 2016;13(125):20160820.

Huy LD, Nguyen NTH, Phuc PT, Huang C-C. The effects of non-pharmaceutical interventions on COVID-19 epidemic growth rate during pre-and post-vaccination period in Asian countries. Int J Environ Res Public Health. 2022;19(3):1139.

Paireau J, Charpignon M-L, Larrieu S, Calba C, Hozé N, Boëlle P-Y, et al. Impact of non-pharmaceutical interventions, weather, vaccination, and variants on COVID-19 transmission across departments in France. BMC Infect Dis. 2023;23(1):1–12.

Tian D, Sun Y, Zhou J, Ye Q. The global epidemic of the SARS-CoV-2 delta variant, key spike mutations and immune escape. Front Immunol. 2021;12:751778.

Shrestha LB, Foster C, Rawlinson W, Tedla N, Bull RA. Evolution of the SARS-CoV‐2 omicron variants BA. 1 to BA. 5: implications for immune escape and transmission. Rev Med Virol. 2022;32(5):e2381.

Zhang Y, Quigley A, Wang Q, MacIntyre CR. Non-pharmaceutical interventions during the roll out of covid-19 vaccines. BMJ. 2021;375.

Moher D, Shamseer L, Clarke M, Ghersi D, Liberati A, Petticrew M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst Reviews. 2015;4(1):1–9.

Dufault B, Klar N. The quality of modern cross-sectional ecologic studies: a bibliometric review. Am J Epidemiol. 2011;174(10):1101–7.

Betran AP, Torloni MR, Zhang J, Ye J, Mikolajczyk R, Deneux-Tharaux C, et al. What is the optimal rate of caesarean section at population level? A systematic review of ecologic studies. Reproductive Health. 2015;12(1):1–10.

Ford N, Holmer HK, Chou R, Villeneuve PJ, Baller A, Van Kerkhove M et al. Mask use in community settings in the context of COVID-19: a systematic review of ecological data. EClinicalMedicine. 2021;38.

Ge Y, Zhang W, Liu H, Ruktanonchai CW, Hu M, Wu X et al. Effects of worldwide interventions and vaccination on COVID-19 between waves and countries. Preprint. 2021.

Hale T, Angrist N, Hale AJ, Kira B, Majumdar S, Petherick A, et al. Government responses and COVID-19 deaths: global evidence across multiple pandemic waves. PLoS ONE. 2021;16(7):e0253116.

Li H, Wang L, Zhang M, Lu Y, Wang W. Effects of vaccination and non-pharmaceutical interventions and their lag times on the COVID-19 pandemic: comparison of eight countries. PLoS Negl Trop Dis. 2022;16(1):e0010101.

Liang L-L, Kao C-T, Ho HJ, Wu C-Y. COVID-19 case doubling time associated with non-pharmaceutical interventions and vaccination: a global experience. J Global Health. 2021;11.

Nesteruk I. Vaccination and testing as a means of ending the COVID-19 pandemic: comparative and statistical analysis. MedRxiv. 2022:2022.06. 16.22276531.

Nesteruk I, Rodionov O. Omicron waves of the COVID-19 pandemic and Effi Cacy of vaccinations and Testing. J ISSN. 2022;2766:2276.

Google Scholar

Wang C, Li H. Variation in global policy responses to COVID-19: a bidirectional analysis. Int J Environ Res Public Health. 2023;20(5):4252.

Article PubMed PubMed Central Google Scholar

Wang H, Lan Y. The global dynamic transmissibility of COVID-19 and its influencing factors: an analysis of control measures from 176 countries. BMC Public Health. 2023;23(1):404.

Ge Y, Zhang W-B, Wu X, Ruktanonchai CW, Liu H, Wang J, et al. Untangling the changing impact of non-pharmaceutical interventions and vaccination on European COVID-19 trajectories. Nat Commun. 2022;13(1):3106.

Kim S, Oh J, Tak S. Association between face covering policies and the incidence of coronavirus disease 2019 in European countries. Osong Public Health Res Perspect. 2023;14(1):31–9.

Zhou F, Hu T-J, Zhang X-Y, Lai K, Chen J-H, Zhou X-H. The association of intensity and duration of non-pharmacological interventions and implementation of vaccination with COVID-19 infection, death, and excess mortality: natural experiment in 22 European countries. J Infect Public Health. 2022;15(5):499–507.

Bollyky TJ, Castro E, Aravkin AY, Bhangdia K, Dalos J, Hulland EN, et al. Assessing COVID-19 pandemic policies and behaviours and their economic and educational trade-offs across US states from Jan 1, 2020, to July 31, 2022: an observational analysis. Lancet. 2023;401(10385):1341–60.

Ertem Z, Nelson RE, Schechter-Perkins EM, Al-Amery A, Zhang X, Branch-Elliman W. Condition-Dependent and dynamic impacts of indoor masking policies for COVID-19 mitigation: a Nationwide, interrupted Time-Series Analysis. Clin Infect Dis. 2023:ciad115.

Shin J, Khuong QL, Abbas K, Oh J. Impact assessment of mobility restrictions, testing, and vaccination on the COVID-19 pandemic in India. medRxiv. 2022;2022(03):24–22272864.

Kim K, Kim S, Lee D, Park C-Y. Impacts of social distancing policy and vaccination during the COVID-19 pandemic in the Republic of Korea. J Economic Dynamics Control. 2023;150:104642.

He X, Liu H, Zeng F, Gao W. Factors influencing the trajectory of COVID-19 evolution: a longitudinal study of 12 Asian countries. medRxiv. 2023:2023.10. 20.23297319.

Doroshenko A. The combined effect of vaccination and nonpharmaceutical public health interventions—ending the COVID-19 pandemic. JAMA Netw Open. 2021;4(6):e2111675–e.

Alessandro C, Ferrone L, Squarcina M, Are. COVID-19 Containment Measures Equally Effective in Different World Regions? DISEI: Università degli Studi di Firenze; 2020.

Wang X, Kong D, Guo M, Wang L, Gu C, Dai C, et al. Rapid SARS-CoV-2 nucleic acid testing and pooled assay by tetrahedral DNA nanostructure transistor. Nano Lett. 2021;21(22):9450–7.

Shao Z, Ma L, Bai Y, Tan Q, Liu XF, Liu S et al. Impact of mass rapid antigen testing for SARS-CoV-2 to mitigate Omicron outbreaks in China. J Travel Med. 2022;29(8).

Embrett M, Sim SM, Caldwell HA, Boulos L, Yu Z, Agarwal G, et al. Barriers to and strategies to address COVID-19 testing hesitancy: a rapid scoping review. BMC Public Health. 2022;22(1):750.

Lin L, Song Y, Wang Q, Pu J, Sun FY, Zhang Y, et al. Public attitudes and factors of COVID-19 testing hesitancy in the United Kingdom and China: comparative infodemiology study. JMİR Infodemiology. 2021;1(1):e26895.

Song S, Zang S, Gong L, Xu C, Lin L, Francis MR, et al. Willingness and uptake of the COVID-19 testing and vaccination in urban China during the low-risk period: a cross-sectional study. BMC Public Health. 2022;22(1):556.

Xin M, Lau JT-f, Lau MM. Multi-dimensional factors related to participation in a population-wide mass COVID-19 testing program among Hong Kong adults: a population-based randomized survey. Soc Sci Med. 2022;294:114692.

Jackson C, Vynnycky E, Hawker J, Olowokure B, Mangtani P. School closures and influenza: systematic review of epidemiological studies. BMJ open. 2013;3(2):e002149.

Ahmed F, Zviedrite N, Uzicanin A. Effectiveness of workplace social distancing measures in reducing influenza transmission: a systematic review. BMC Public Health. 2018;18(1):1–13.

Wang CC, Prather KA, Sznitman J, Jimenez JL, Lakdawala SS, Tufekci Z, et al. Airborne transmission of respiratory viruses. Science. 2021;373(6558):eabd9149.

Ishola DA, Phin N. Could influenza transmission be reduced by restricting mass gatherings? Towards an evidence-based policy framework. J Epidemiol Global Health. 2011;1(1):33–60.

Mader S, Rüttenauer T. The effects of non-pharmaceutical interventions on COVID-19 mortality: a generalized synthetic control approach across 169 countries. Front Public Health. 2022;10:820642.

Download references

Acknowledgements

Not applicable.

Senior Talent Startup Fund of Nanchang University.

Author information

Authors and affiliations.

Department of Epidemiology and Health Statistics, School of Public Health, Jiangxi Medical College, Nanchang University, Nanchang, China

Xiaona He, Huiting Chen, Xinyu Zhu & Wei Gao

Jiangxi Provincial Key Laboratory of Preventive Medicine and Public Health, Nanchang University, No. 461, Bayi Ave,, Nanchang, 330006, PR China

You can also search for this author in PubMed Google Scholar

Contributions

WG conceived the study and devised the methodology. XH performed the literature search, literature screening, data extraction, management and analysis. HC and XZ reviewed the literature and conducted the collection and curation of data. XH drafted the manuscript. WG directed the study, and critically revised the manuscript. All authors had full access to all the data in the study and verified the data. WG had final responsibility for the decision to submit for publication.

Corresponding author

Correspondence to Wei Gao .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Rights and permissions.

Reprints and permissions

About this article

Cite this article.

He, X., Chen, H., Zhu, X. et al. Non-pharmaceutical interventions in containing COVID-19 pandemic after the roll-out of coronavirus vaccines: a systematic review. BMC Public Health 24 , 1524 (2024). https://doi.org/10.1186/s12889-024-18980-2

Download citation

Received : 10 December 2023

Accepted : 28 May 2024

Published : 06 June 2024

DOI : https://doi.org/10.1186/s12889-024-18980-2

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Optimal strategies
Vaccination
Real-world impact

BMC Public Health

ISSN: 1471-2458

General enquiries: [email protected]

Integrating Spatial and Non-Spatial Dimensions to Evaluate Access to Rural Primary Healthcare Service: A Case Study of Songzi, China

Yang, Taohua
Luo, Weicong
Tian, Lingling
Li, Jinpeng

Access to rural primary healthcare services has been broadly studied in the past few decades. However, most earlier studies that focused on examining access to rural healthcare services have conventionally treated spatial and non-spatial access as separate factors. This research aims to measure access to primary healthcare services in rural areas with the consideration of both spatial and non-spatial dimensions. The methodology of study is threefold. First, the Gaussian two-step floating catchment area (G-2SFCA) method was adopted to measure spatial access to primary healthcare services. Then, a questionnaire survey was conducted to investigate non-spatial access factors, including demographic condition, patient's household income, healthcare insurance, education level, and patient satisfaction level with the services. After that, a comprehensive evaluation index system was employed to integrate both spatial and non-spatial access. The empirical study showed a remarkable disparity in spatial access to primary healthcare services. In total, 78 villages with 185,137 local people had a "low" or "very low" level of spatial access to both clinics and hospitals. For the non-spatial dimension, the results depicted that Songzi had significant inequalities in socioeconomic status (e.g., income, education) and patient satisfaction level for medical service. When integrating both spatial and non-spatial factors, the disadvantaged areas were mainly located in the eastern and middle parts. In addition, this study found that comprehensively considering the spatial and non-spatial access had a significant impact on results in healthcare access. In conclusion, this study calls for policymakers to pay more attention to primary healthcare inequalities within rural areas. The spatial and non-spatial access should be considered comprehensively when the long-term rural medical support policy is designated.

spatial access;
non-spatial factors;
primary healthcare;
rural areas;

IMAGES

What Are Some Examples Of Non Experimental Research
Empirical and non-empirical research
Non-Empirical Poster Template
Research methodology -empirical vs non-empirical
Guidance for writing a non empirical research project
Non Empirical Dissertation Examples

VIDEO

Nonparametric Methods: Nominal-Level Hypothesis
Lec 38: Empirical Usability Evaluation
Theoretical Research and Indian Knowledge Tradition/भारतीय ज्ञान-परंपरा पर आधारित सैधांतिक शोध
Empirical research methods
Research
Empirical Labs Distressor

COMMENTS

(PDF) Empirical and Non-Empirical Methods
The dividing line between empirical and non -empirical methods is marked by scholars' approach to knowledge. gain (i.e., epistemology). Empirical methods typically involve syste matic collection ...
Non-Empirical Research
Learn how to prepare a manuscript for non-empirical research articles that focus on theories, methods and implications for education research. Find out the requirements for title page, declarations, abstract, keywords, introduction and more.
6.1 Overview of Non-Experimental Research
Non-experimental research is research that lacks the manipulation of an independent variable and measures variables as they naturally occur. Learn when to use non-experimental research, such as cross-sectional, correlational, and observational studies, and how they differ from experimental research.
Overview of Nonexperimental Research
Nonexperimental research is research that lacks the manipulation of an independent variable, random assignment of participants, or both. It can be single-variable, correlational, quasi-experimental, or qualitative. Learn the differences, advantages, and disadvantages of each type.
6.2: Overview of Non-Experimental Research
Non-experimental research is research that lacks the manipulation of an independent variable. Rather than manipulating an independent variable, researchers conducting non-experimental research simply measure variables as they naturally occur (in the lab or real world). Most researchers in psychology consider the distinction between experimental ...
6.1: Overview of Non-Experimental Research
Non-experimental research is research that lacks the manipulation of an independent variable. Rather than manipulating an independent variable, researchers conducting non-experimental research simply measure variables as they naturally occur (in the lab or real world). Most researchers in psychology consider the distinction between experimental ...
Empirical and Nonempirical Methods
The dividing line between empirical and nonempirical methods is marked by scholars' approach to knowledge gain (i.e., epistemology). Empirical methods typically involve systematic collection and analysis of data (i.e., observation and evidence). They are used primarily in quantitative research involving original collection of data, but also in ...
6: Non-Experimental Research
Although this method is by far the most common approach to conducting empirical research in psychology, there is an important alternative called qualitative research. 6.6: Observational Research Observational research is used to refer to several different types of non-experimental studies in which behavior is systematically observed and recorded.
Overview of Non-Experimental Research
Learn what non-experimental research is, when to use it, and how it differs from experimental research. Explore examples of correlational, observational, and cross-sectional, longitudinal, and cross-sequential studies.
Empirical v. Non-Empirical Research
Empirical Versus Non-empirical Research. Empirical research is based on observed and measured phenomena and derives knowledge from actual experience rather than from theory or belief. How do you know if a study is empirical? Read the subheadings within the article, book, or report and look for a description of the research "methodology."
Non-experimental research: What it is, Types & Tips
Non-experimental research is the type of research that lacks an independent variable. Instead, the researcher observes the context in which the phenomenon occurs and analyzes it to obtain information. Unlike experimental research, where the variables are held constant, non-experimental research happens during the study when the researcher ...
7.1 Overview of Nonexperimental Research
Nonexperimental research is research that lacks the manipulation of an independent variable, random assignment of participants, or both. It can be single-variable, correlational, quasi-experimental, or qualitative. Learn the differences, advantages, and limitations of each type.
What is non-experimental research: Definition, types & examples
Non-experimental research is a type of research design that is based on observation and measuring instead of experimentation with randomly assigned participants. What characterizes this research design is the fact that it lacks the manipulation of independent variables. Because of this fact, the non-experimental research is based on naturally ...
Non-empirical Values
Nonempirical values serve to delineate specific distinctions of scientific knowledge beyond empirical adequacy. Such values express requirements of significance and confirmation. The former are influential on the choice of problems and the pursuit of theories, the latter contribute to assessing the bearing of evidence on theory. Nonempirical ...
Types of Research
Observations: counting the number of times a phenomenon occurs or coding observed data in order to translate it into numbers. Document Analysis: analysis of correspondence or reports. Document Screening: using numerical data from financial reports or counting word occurrences. Oral History or Life Stories: memories told to a researcher.
Empirical Research: Advantages, Drawbacks and Differences with Non
Non-empirical research is based on theories and logic, and researchers don't attempt to test them. Although empirical research mostly depends on primary data, secondary data can also be beneficial for the theory side of the research. The empirical research process includes the following:
What is the difference between empirical and non-empirical research
Learn the difference between empirical and non-empirical research, how to identify them, and what types of evidence they use. Find out how to generate research questions and where to get ideas for them.
Non-Empirical Papers
Learn how to write a non-empirical paper based on literature review, theory evaluation, or research proposal. Find out how to determine the purpose, choose the topic, and analyze the sources of your paper.
1.6 Reading, understanding and writing up non-empirical articles
Non-empirical articles do not have a specific structure like empirical articles. Instead, authors organize their articles by topic and subtopic. Non-empirical articles include articles about social theory, history, philosophy, and literature reviews. Go to the Penn State Fayette Library page and search for a Non-empircal journal article.
Empirical Research
The term "empirical" entails gathered data based on experience, observations, or experimentation. In empirical research, knowledge is developed from factual experience as opposed to theoretical assumption and usually involved the use of data sources like datasets or fieldwork, but can also be based on observations within a laboratory setting.
PDF Outline for Non-Emipircal Research
Participants who were granted permission in advance of the Institute to conduct non-empirical projects must use this form as a guide. The outline is designed specifically for a. deeper, more extended analysis of empirical and other data in the literature. The final scholarly paper will be a combination of what was done to prepare the paper and ...
NON-EMPIRICAL definition
NON-EMPIRICAL meaning: 1. based on theory rather than on what is experienced or seen: 2. based on theory rather than on…. Learn more.
There Is No Pure Empirical Reasoning
As the title says, there is no such thing as pure empirical reasoning.* [ *Based on: "There Is No Pure Empirical Reasoning," Philosophy and Phenomenological Research 95 (2017): 592-613. ] 1. The Issue Empiricists think that all substantive knowledge about the world must be justified (directly or indirectly) by observation. This is taken to mean there is no…
Misunderstanding the harms of online misinformation
The controversy over online misinformation and social media has opened a gap between public discourse and scientific research. Public intellectuals and journalists frequently make sweeping claims ...
College Essays and Diversity in the Post-Affirmative Action Era
Editor's Note: This story is part of an occasional series on research projects currently in the works at the Law School. The Supreme Court's decision in June 2023 to bar the use of affirmative action in college admissions raised many questions. One of the most significant is whether universities should consider applicants' discussion of race in essays. The Court's decision in Students ...
6: Nonexperimental Research
Although this method is by far the most common approach to conducting empirical research in psychology, there is an important alternative called qualitative research. 6.5: Observational Research Observational research is used to refer to several different types of non-experimental studies in which behavior is systematically observed and recorded.
Comparison of two propensity score-based methods for balancing
The empirical example has 10.55% exposure rate which is near rare (typically < 10% considered as rare) and hospitalization quite often is a rare outcome. The study was reviewed by the University of Chicago Institutional Review Board and determined to be non-human subject research. Overlap weighting method (OW)
Understanding the Intersection of Medicaid & Work: A Look at What the
However, working adults may be ineligible for Medicaid in non-expansion states where the median eligibility limit for parents as of January 2023 is 37% of the FPL (and ranges from 16% in Texas to ...
Non-pharmaceutical interventions in containing COVID-19 pandemic after
Non-pharmaceutical interventions (NPIs) have been widely utilised to control the COVID-19 pandemic. However, it is unclear what the optimal strategies are for implementing NPIs in the context of coronavirus vaccines. This study aims to systematically identify, describe, and evaluate existing ecological studies on the real-world impact of NPIs in containing COVID-19 pandemic following the roll ...
Integrating Spatial and Non-Spatial Dimensions to Evaluate Access to
After that, a comprehensive evaluation index system was employed to integrate both spatial and non-spatial access. The empirical study showed a remarkable disparity in spatial access to primary healthcare services. In total, 78 villages with 185,137 local people had a "low" or "very low" level of spatial access to both clinics and hospitals.