define empirical evidence in education

Penn State University Libraries

Empirical research in the social sciences and education.

• What is Empirical Research and How to Read It
• Finding Empirical Research in Library Databases
• Designing Empirical Research
• Ethics, Cultural Responsiveness, and Anti-Racism in Research
• Citing, Writing, and Presenting Your Work

Contact the Librarian at your campus for more help!

Introduction: What is Empirical Research?

Empirical research is based on observed and measured phenomena and derives knowledge from actual experience rather than from theory or belief. 

How do you know if a study is empirical? Read the subheadings within the article, book, or report and look for a description of the research "methodology."  Ask yourself: Could I recreate this study and test these results?

Key characteristics to look for:

• Specific research questions to be answered
• Definition of the population, behavior, or   phenomena being studied
• Description of the process used to study this population or phenomena, including selection criteria, controls, and testing instruments (such as surveys)

Another hint: some scholarly journals use a specific layout, called the "IMRaD" format, to communicate empirical research findings. Such articles typically have 4 components:

• Introduction : sometimes called "literature review" -- what is currently known about the topic -- usually includes a theoretical framework and/or discussion of previous studies
• Methodology: sometimes called "research design" -- how to recreate the study -- usually describes the population, research process, and analytical tools used in the present study
• Results : sometimes called "findings" -- what was learned through the study -- usually appears as statistical data or as substantial quotations from research participants
• Discussion : sometimes called "conclusion" or "implications" -- why the study is important -- usually describes how the research results influence professional practices or future studies

Reading and Evaluating Scholarly Materials

Reading research can be a challenge. However, the tutorials and videos below can help. They explain what scholarly articles look like, how to read them, and how to evaluate them:

• CRAAP Checklist A frequently-used checklist that helps you examine the currency, relevance, authority, accuracy, and purpose of an information source.
• IF I APPLY A newer model of evaluating sources which encourages you to think about your own biases as a reader, as well as concerns about the item you are reading.
• Credo Video: How to Read Scholarly Materials (4 min.)
• Credo Tutorial: How to Read Scholarly Materials
• Credo Tutorial: Evaluating Information
• Credo Video: Evaluating Statistics (4 min.)
• Next: Finding Empirical Research in Library Databases >>
• Last Updated: Feb 18, 2024 8:33 PM
• URL: https://guides.libraries.psu.edu/emp

• Games & Quizzes
• History & Society
• Science & Tech
• Biographies
• Animals & Nature
• Geography & Travel
• Arts & Culture
• On This Day
• One Good Fact
• New Articles
• Lifestyles & Social Issues
• Philosophy & Religion
• Politics, Law & Government
• World History
• Health & Medicine
• Browse Biographies
• Birds, Reptiles & Other Vertebrates
• Bugs, Mollusks & Other Invertebrates
• Environment
• Fossils & Geologic Time
• Entertainment & Pop Culture
• Sports & Recreation
• Visual Arts
• Demystified
• Image Galleries
• Infographics
• Top Questions
• Britannica Kids
• Saving Earth
• Space Next 50
• Student Center

• When did science begin?
• Where was science invented?

empirical evidence

Our editors will review what you’ve submitted and determine whether to revise the article.

• National Center for Biotechnology Information - PubMed Central - Expert opinion vs. empirical evidence
• LiveScience - Empirical evidence: A definition
• Academia - Empirical Evidence

empirical evidence , information gathered directly or indirectly through observation or experimentation that may be used to confirm or disconfirm a scientific theory or to help justify, or establish as reasonable, a person’s belief in a given proposition. A belief may be said to be justified if there is sufficient evidence to make holding the belief reasonable.

The concept of evidence is the basis of philosophical evidentialism, an epistemological thesis according to which a person is justified in believing a given proposition p if and only if the person’s evidence for p is proper or sufficient. In this context , the Scottish Enlightenment philosopher David Hume (1711–76) famously asserted that the “wise man…proportions his belief to the evidence.” In a similar vein, the American astronomer Carl Sagan popularized the statement, “Extraordinary claims require extraordinary evidence.”

Foundationalists , however, defend the view that certain basic, or foundational, beliefs are either inherently justified or justified by something other than another belief (e.g., a sensation or perception) and that all other beliefs may be justified only if they are directly or indirectly supported by at least one foundational belief (that is, only if they are either supported by at least one foundational belief or supported by other beliefs that are themselves supported by at least one foundational belief). The most influential foundationalist of the modern period was the French philosopher and mathematician René Descartes (1596–1650), who attempted to establish a foundation for justified beliefs regarding an external world in his intuition that, for as long as he is thinking, he exists (“I think, therefore I am”; see cogito, ergo sum ). A traditional argument in favour of foundationalism asserts that no other account of inferential justification—the act of justifying a given belief by inferring it from another belief that itself is justified—is possible. Thus, assume that one belief, Belief 1, is justified by another belief, Belief 2. How is Belief 2 justified? It cannot be justified by Belief 1, because the inference from Belief 2 to Belief 1 would then be circular and invalid. It cannot be justified by a third nonfoundational Belief 3, because the same question would then apply to that belief, leading to an infinite regress. And one cannot simply assume that Belief 2 is not justified, for then Belief 1 would not be justified through the inference from Belief 2. Accordingly, there must be some beliefs whose justification does not depend on other beliefs, and those justified beliefs must function as a foundation for the inferential justification of other beliefs.

Empirical evidence can be quantitative or qualitative. Typically, numerical quantitative evidence can be represented visually by means of diagrams, graphs, or charts, reflecting the use of statistical or mathematical data and the researcher’s neutral noninteractive role. It can be obtained by methods such as experiments, surveys, correlational research (to study the relationship between variables), cross-sectional research (to compare different groups), causal-comparative research (to explore cause-effect relationships), and longitudinal studies (to test a subject during a given time period).

Qualitative evidence, on the other hand, can foster a deeper understanding of behaviour and related factors and is not typically expressed by using numbers. Often subjective and resulting from interaction between the researcher and participants, it can stem from the use of methods such as interviews (based on verbal interaction), observation (informing ethnographic research design), textual analysis (involving the description and interpretation of texts), focus groups (planned group discussions), and case studies (in-depth analyses of individuals or groups).

Empirical evidence is subject to assessments of its validity. Validity can be internal, involving the soundness of an experiment’s design and execution and the accuracy of subsequent data analysis , or external, involving generalizability to other research contexts ( see ecological validity ).

Empirical evidence: A definition

Empirical evidence is information that is acquired by observation or experimentation.

The scientific method

Types of empirical research, identifying empirical evidence, empirical law vs. scientific law, empirical, anecdotal and logical evidence, additional resources and reading, bibliography.

Empirical evidence is information acquired by observation or experimentation. Scientists record and analyze this data. The process is a central part of the scientific method , leading to the proving or disproving of a hypothesis and our better understanding of the world as a result.

Empirical evidence might be obtained through experiments that seek to provide a measurable or observable reaction, trials that repeat an experiment to test its efficacy (such as a drug trial, for instance) or other forms of data gathering against which a hypothesis can be tested and reliably measured. 

"If a statement is about something that is itself observable, then the empirical testing can be direct. We just have a look to see if it is true. For example, the statement, 'The litmus paper is pink', is subject to direct empirical testing," wrote Peter Kosso in " A Summary of Scientific Method " (Springer, 2011).

"Science is most interesting and most useful to us when it is describing the unobservable things like atoms , germs , black holes , gravity , the process of evolution as it happened in the past, and so on," wrote Kosso. Scientific theories , meaning theories about nature that are unobservable, cannot be proven by direct empirical testing, but they can be tested indirectly, according to Kosso. "The nature of this indirect evidence, and the logical relation between evidence and theory, are the crux of scientific method," wrote Kosso.

The scientific method begins with scientists forming questions, or hypotheses , and then acquiring the knowledge through observations and experiments to either support or disprove a specific theory. "Empirical" means "based on observation or experience," according to the Merriam-Webster Dictionary . Empirical research is the process of finding empirical evidence. Empirical data is the information that comes from the research.

Before any pieces of empirical data are collected, scientists carefully design their research methods to ensure the accuracy, quality and integrity of the data. If there are flaws in the way that empirical data is collected, the research will not be considered valid.

The scientific method often involves lab experiments that are repeated over and over, and these experiments result in quantitative data in the form of numbers and statistics. However, that is not the only process used for gathering information to support or refute a theory. 

This methodology mostly applies to the natural sciences. "The role of empirical experimentation and observation is negligible in mathematics compared to natural sciences such as psychology, biology or physics," wrote Mark Chang, an adjunct professor at Boston University, in " Principles of Scientific Methods " (Chapman and Hall, 2017).

"Empirical evidence includes measurements or data collected through direct observation or experimentation," said Jaime Tanner, a professor of biology at Marlboro College in Vermont. There are two research methods used to gather empirical measurements and data: qualitative and quantitative.

Qualitative research, often used in the social sciences, examines the reasons behind human behavior, according to the National Center for Biotechnology Information (NCBI) . It involves data that can be found using the human senses . This type of research is often done in the beginning of an experiment. "When combined with quantitative measures, qualitative study can give a better understanding of health related issues," wrote Dr. Sanjay Kalra for NCBI.

Quantitative research involves methods that are used to collect numerical data and analyze it using statistical methods, ."Quantitative research methods emphasize objective measurements and the statistical, mathematical, or numerical analysis of data collected through polls, questionnaires, and surveys, or by manipulating pre-existing statistical data using computational techniques," according to the LeTourneau University . This type of research is often used at the end of an experiment to refine and test the previous research.

Identifying empirical evidence in another researcher's experiments can sometimes be difficult. According to the Pennsylvania State University Libraries , there are some things one can look for when determining if evidence is empirical:

• Can the experiment be recreated and tested?
• Does the experiment have a statement about the methodology, tools and controls used?
• Is there a definition of the group or phenomena being studied?

The objective of science is that all empirical data that has been gathered through observation, experience and experimentation is without bias. The strength of any scientific research depends on the ability to gather and analyze empirical data in the most unbiased and controlled fashion possible. 

However, in the 1960s, scientific historian and philosopher Thomas Kuhn promoted the idea that scientists can be influenced by prior beliefs and experiences, according to the Center for the Study of Language and Information . 

— Amazing Black scientists

— Marie Curie: Facts and biography

— What is multiverse theory?

"Missing observations or incomplete data can also cause bias in data analysis, especially when the missing mechanism is not random," wrote Chang.

Because scientists are human and prone to error, empirical data is often gathered by multiple scientists who independently replicate experiments. This also guards against scientists who unconsciously, or in rare cases consciously, veer from the prescribed research parameters, which could skew the results.

The recording of empirical data is also crucial to the scientific method, as science can only be advanced if data is shared and analyzed. Peer review of empirical data is essential to protect against bad science, according to the University of California .

Empirical laws and scientific laws are often the same thing. "Laws are descriptions — often mathematical descriptions — of natural phenomenon," Peter Coppinger, associate professor of biology and biomedical engineering at the Rose-Hulman Institute of Technology, told Live Science. 

Empirical laws are scientific laws that can be proven or disproved using observations or experiments, according to the Merriam-Webster Dictionary . So, as long as a scientific law can be tested using experiments or observations, it is considered an empirical law.

Empirical, anecdotal and logical evidence should not be confused. They are separate types of evidence that can be used to try to prove or disprove and idea or claim.

Logical evidence is used proven or disprove an idea using logic. Deductive reasoning may be used to come to a conclusion to provide logical evidence. For example, "All men are mortal. Harold is a man. Therefore, Harold is mortal."

Anecdotal evidence consists of stories that have been experienced by a person that are told to prove or disprove a point. For example, many people have told stories about their alien abductions to prove that aliens exist. Often, a person's anecdotal evidence cannot be proven or disproven. 

There are some things in nature that science is still working to build evidence for, such as the hunt to explain consciousness .

Meanwhile, in other scientific fields, efforts are still being made to improve research methods, such as the plan by some psychologists to fix the science of psychology .

" A Summary of Scientific Method " by Peter Kosso (Springer, 2011)

"Empirical" Merriam-Webster Dictionary

" Principles of Scientific Methods " by Mark Chang (Chapman and Hall, 2017)

"Qualitative research" by Dr. Sanjay Kalra National Center for Biotechnology Information (NCBI)

"Quantitative Research and Analysis: Quantitative Methods Overview" LeTourneau University

"Empirical Research in the Social Sciences and Education" Pennsylvania State University Libraries

"Thomas Kuhn" Center for the Study of Language and Information

"Misconceptions about science" University of California

Sign up for the Live Science daily newsletter now

Get the world’s most fascinating discoveries delivered straight to your inbox.

30,000 years of history reveals that hard times boost human societies' resilience

'We're meeting people where they are': Graphic novels can help boost diversity in STEM, says MIT's Ritu Raman

32 haunting shipwrecks from the ancient world

Most Popular

• 2 100-foot 'walking tree' in New Zealand looks like an Ent from Lord of the Rings — and is the lone survivor of a lost forest
• 3 Neanderthals and humans interbred 47,000 years ago for nearly 7,000 years, research suggests
• 4 Viking Age 'treasure' discovered by metal detectorist on Isle of Man
• 5 James Webb telescope finds carbon at the dawn of the universe, challenging our understanding of when life could have emerged
• 2 James Webb telescope finds carbon at the dawn of the universe, challenging our understanding of when life could have emerged
• 3 Shigir Idol: World's oldest wood sculpture has mysterious carved faces and once stood 17 feet tall
• 4 What could aliens look like?
• 5 Neanderthals and humans interbred 47,000 years ago for nearly 7,000 years, research suggests

Empirical Research in Education

Assumptions and Problems

Cite this chapter

665 Accesses

In the previous chapters we have reviewed the history of social science research and introduced some of the basic principles on which empirical research, or LPE, in education is based. In this chapter we turn our attention toward identifying how the principles of logical positivism, when applied to education research, are ineffective for strengthening the discipline.

In the following sections, we address some of the most problematic assumptions involved in carrying out empirical research in education and grapple with several related problems. Some of the major assumptions in social science research that promote positivistic or scientific principles in educational research include the following claims that we deconstruct within the course of our discussion:

Educational researchers, like physical scientists, are detached from their objects of study in that their personal preferences and biases are excluded from their subject matter, observations, and attending analyses.

Investigations of educational phenomena can be conducted in a value-neutral fashion, with the researcher eliminating all personal bias and preconceptions and employing language that expresses objectivity. In other words, there is objectivity and conceptual clarity in describing the studied phenomena within genuine scientific inquiry.

Educational research, like the physical sciences is nomothetic – that is, it is possible to extrapolate from educational research data laws that apply generally across numerous classroom and schooling contexts. In education, this assumption is particularly crucial since the search for the holy grail of some universal, but of course entirely illusive, instructional design drives much of the empirical investigation within the field. Two researchers working in different contexts who employ the same experimental method ought to arrive at the same conclusion. As we demonstrate in this chapter, within education this outcome is simply not the case.

We will demonstrate that each of these scientific principles, or assumptions, is fundamentally flawed when applied to educational research. Hence, education research is once again unable to meet the minimal standards of meaningful scientific inquiry. Later in this chapter we will also discuss the conceptual confusions that impact negatively on education. Finally, we examine how an implicit commitment to the direct reference theory of language, and the related search for conceptual certainty, leads to ontological errors about certain education concepts and how these errors affect student academic experience.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Compact, lightweight edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info
• Durable hardcover edition

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Unable to display preview.  Download preview PDF.

Rights and permissions

Reprints and permissions

Copyright information

© 2007 Springer

About this chapter

(2007). Empirical Research in Education. In: Scientism and Education. Springer, Dordrecht. https://doi.org/10.1007/978-1-4020-6678-8_4

Download citation

DOI : https://doi.org/10.1007/978-1-4020-6678-8_4

Publisher Name : Springer, Dordrecht

Print ISBN : 978-1-4020-6677-1

Online ISBN : 978-1-4020-6678-8

eBook Packages : Humanities, Social Sciences and Law Education (R0)

Share this chapter

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

• Tools and Resources
• Customer Services
• Original Language Spotlight
• Alternative and Non-formal Education 
• Cognition, Emotion, and Learning
• Curriculum and Pedagogy
• Education and Society
• Education, Change, and Development
• Education, Cultures, and Ethnicities
• Education, Gender, and Sexualities
• Education, Health, and Social Services
• Educational Administration and Leadership
• Educational History
• Educational Politics and Policy
• Educational Purposes and Ideals
• Educational Systems
• Educational Theories and Philosophies
• Globalization, Economics, and Education
• Languages and Literacies
• Professional Learning and Development
• Research and Assessment Methods
• Technology and Education
• Share This Facebook LinkedIn Twitter

Article contents

Evidence-based educational practice.

• Tone Kvernbekk Tone Kvernbekk University of Oslo
• https://doi.org/10.1093/acrefore/9780190264093.013.187
• Published online: 19 December 2017

Evidence-based practice (EBP) is a buzzword in contemporary professional debates, for example, in education, medicine, psychiatry, and social policy. It is known as the “what works” agenda, and its focus is on the use of the best available evidence to bring about desirable results or prevent undesirable ones. We immediately see here that EBP is practical in nature, that evidence is thought to play a central role, and also that EBP is deeply causal: we intervene into an already existing practice in order to produce an output or to improve the output. If our intervention brings the results we want, we say that it “works.”

How should we understand the causal nature of EBP? Causality is a highly contentious issue in education, and many writers want to banish it altogether. But causation denotes a dynamic relation between factors and is indispensable if one wants to be able to plan the attainment of goals and results. A nuanced and reasonable understanding of causality is therefore necessary to EBP, and this we find in the INUS-condition approach.

The nature and function of evidence is much discussed. The evidence in question is supplied by research, as a response to both political and practical demands that educational research should contribute to practice. In general, evidence speaks to the truth value of claims. In the case of EBP, the evidence emanates from randomized controlled trials (RCTs) and presumably speaks to the truth value of claims such as “if we do X, it will lead to result Y.” But what does research evidence really tell us? It is argued here that a positive RCT result will tell you that X worked where the RCT was conducted and that an RCT does not yield general results.

Causality and evidence come together in the practitioner perspective. Here we shift from finding causes to using them to bring about desirable results. This puts contextual matters at center stage: will X work in this particular context? It is argued that much heterogeneous contextual evidence is required to make X relevant for new contexts. If EBP is to be a success, research evidence and contextual evidence must be brought together.

• effectiveness
• INUS conditions
• practitioner perspective

Introduction

Evidence-based practice, hereafter EBP, is generally known as the “what works” agenda. This is an apt phrase, pointing as it does to central practical issues: how to attain goals and produce desirable results, and how we know what works. Obviously, this goes to the heart of much (but not all) of the everyday activity that practitioners engage in. The “what works” agenda is meant to narrow the gap between research and practice and be an area in which research can make itself directly useful to practice. David Hargreaves, one of the instigators of the EBP debate in education, has stated that the point of evidence-based research is to gather evidence about what works in what circumstances (Hargreaves, 1996a , 1996b ). Teachers, Hargreaves said, want to know what works; only secondarily are they interested in understanding the why of classroom events. The kind of research we talk about is meant to be relevant not only for teachers but also for policymakers, school developers, and headmasters. Its purpose is to improve practice, which largely comes down to improving student achievement. Hargreaves’s work was supported by, for example, Robert Slavin, who stated that education research not only can address questions about “what works” but also must do so (Slavin, 2004 ).

All the same, despite the fact that EBP, at least at the outset, seems to speak directly to the needs of practitioners, it has met with much criticism. It is difficult to characterize both EBP and the debate about it, but let me suggest that the debate branches off in different but interrelated directions. We may roughly identify two: what educational research can and should contribute to practice and what EBP entails for the nature of educational practice and the teaching profession. There is ample space here for different definitions, different perspectives, different opinions, as well as for some general unclarity and confusions. To some extent, advocates and critics bring different vocabularies to the debate, and to some extent, they employ the same vocabulary but take very different stances. Overall in the EBP conceptual landscape we find such concepts as relevance, effectiveness, generality, causality, systematic reviews, randomized controlled trials (RCTs), what works, accountability, competences, outcomes, measurement, practical judgment, professional experience, situatedness, democracy, appropriateness, ends, and means as constitutive of ends or as instrumental to the achievement of ends. Out of this tangle we shall carefully extract and examine a selection of themes, assumptions, and problems. These mainly concern the causal nature of EBP, the function of evidence, and EBP from the practitioner point of view.

Definition, History, and Context

The term “evidence-based” originates in medicine—evidence-based medicine—and was coined in 1991 by a group of doctors at McMaster University in Hamilton, Ontario. Originally, it denoted a method for teaching medicine at the bedside. It has long since outgrown the hospital bedside and has become a buzzword in many contemporary professions and professional debates, not only in education, but also leadership, psychiatry, and policymaking. The term EBP can be defined in different ways, broadly or more narrowly. We shall here adopt a parsimonious, minimal definition, which says that EBP involves the use of the best available evidence to bring about desirable outcomes, or conversely, to prevent undesirable outcomes (Kvernbekk, 2016 ). That is to say, we intervene to bring about results, and this practice should be guided by evidence of how well it works. This minimal definition does not specify what kinds of evidence are allowed, what “based” should mean, what practice is, or how we should understand the causality that is inevitably involved in bringing about and preventing results. Minimal definitions are eminently useful because they are broad in their phenomenal range and thus allow differing versions of the phenomenon in question to fall under the concept.

We live in an age which insists that practices and policies of all kinds be based on research. Researchers thus face political demands for better research bases to underpin, inform and guide policy and practice, and practitioners face political demands to make use of research to produce desirable results or improve results already produced. Although the term EBP is fairly recent, the idea that research should be used to guide and improve practice is by no means new. To illustrate, in 1933 , the School Commission of Norwegian Teacher Unions (Lærerorganisasjonenes skolenevnd, 1933 ) declared that progress in schooling can only happen through empirical studies, notably, by different kinds of experiments and trials. Examples of problems the commission thought research should solve are (a) in which grade the teaching of a second language should start and (b) what the best form of differentiation is. The accumulated evidence should form the basis for policy, the unions argued. Thus, the idea that pedagogy should be based on systematic research is not entirely new. What is new is the magnitude and influence of the EBP movement and other, related trends, such as large-scale international comparative studies (e.g., the Progress in International Reading Literacy Study, PIRLS, and the Programme for International Student Assessment, PISA). Schooling is generally considered successful when the predetermined outcomes have been achieved, and education worldwide therefore makes excessive requirements of assessment, measurement, testing, and documentation. EBP generally belongs in this big picture, with its emphasis on knowing what works in order to maximize the probability of attaining the goal. What is also new, and quite unprecedented, is the growth of organizations such as the What Works Clearinghouses, set up all around the world. The WWCs collect, review, synthesize, and report on studies of educational interventions. Their main functions are, first, to provide hierarchies that rank evidence. The hierarchies may differ in their details, but they all rank RCTs, meta-analyses, and systematic reviews on top and professional judgment near the bottom (see, e.g., Oancea & Pring, 2008 ). Second, they provide guides that offer advice about how to choose a method of instruction that is backed by good evidence; and third, they serve as a warehouse, where a practitioner might find methods that are indeed backed by good evidence (Cartwright & Hardie, 2012 ).

Educationists today seem to have a somewhat ambiguous relationship to research and what it can do for practice. Some, such as Robert Slavin ( 2002 ), a highly influential educational researcher and a defender of EBP, think that education is on the brink of a scientific revolution. Slavin has argued that over time, rigorous research will yield the same step-by-step, irreversible progress in education that medicine has enjoyed because all interventions would be subjected to strict standards of evaluation before being recommended for general use. Central to this optimism is the RCT. Other educationists, such as Gert Biesta ( 2007 , 2010 ), also a highly influential figure in the field and a critic of EBP, are wary of according such weight to research and to the advice guides and practical guidelines of the WWCs for fear that this might seriously restrict, or out and out replace, the experience and professional judgment of practitioners. And there matters stand: EBP is a huge domain with many different topics, issues, and problems, where advocates and critics have criss-crossing perspectives, assumptions, and value stances.

The Causal Nature of Evidence-Based Practice

As the slogan “what works” suggests, EBP is practical in nature. By the same token, EBP is also deeply causal. Works is a causal term, as are intervention, effectiveness , bring about , influence , and prevent . In EBP we intervene into an already existing practice in order to change its outcomes in what we judge to be a more desirable direction. To say that something (an intervention) works is roughly to say that doing it yields the outcomes we want. If we get other results or no results at all, we say that it does not work. To put it crudely, we do X, and if it leads to some desirable outcome Y, we judge that X works. It is the ambition of EBP to provide knowledge of how intervention X can be used to bring about or produce Y (or improvements in Y) and to back this up by solid evidence—for example, how implementing a reading-instruction program can improve the reading skills of slow or delayed readers, or how a schoolwide behavioral support program can serve to enhance students’ social skills and prevent future problem behavior. For convenience, I adopt the convention of calling the cause (intervention, input) X and the effect (result, outcome, output) Y. This is on the explicit understanding that both X and Y can be highly complex in their own right, and that the convention, as will become clear, is a simplification.

There can be no doubt that EBP is causal. However, the whole issue of causality is highly contentious in education. Many educationists and philosophers of education have over the years dismissed the idea that education is or can be causal or have causal elements. In EBP, too, this controversy runs deep. By and large, advocates of EBP seem to take for granted that causality in the social and human realm simply exists, but they tend not to provide any analysis of it. RCTs are preferred because they allow causal inferences to be made with a high degree of certainty. As Slavin ( 2002 ) put it, “The experiment is the design of choice for studies that seek to make causal conclusions, and particularly for evaluations of educational innovations” (p. 18). In contrast, critics often make much of the causal of nature of EBP, since for many of them this is reason to reject EBP altogether. Biesta is a case in point. For him and many others, education is a moral and social practice and therefore non causal. According to Biesta ( 2010 ):

The most important argument against the idea that education is a causal process lies in the fact that education is not a process of physical interaction but a process of symbolic or symbolically mediated interaction. (p. 34)

Since education is noncausal and EBP is causal, on this line of reasoning, it follows that EBP must be rejected—it fundamentally mistakes the nature of education.

Such wholesale dismissals rest on certain assumptions about the nature of causality, for example, that it is deterministic, positivist, and physical and that it essentially belongs in the natural sciences. Biesta, for example, clearly assumes that causality requires a physical process. But since the mid-1900s our understanding of causality has witnessed dramatic developments; arguably the most important of which is its reformulation in probabilistic terms, thus making it compatible with indeterminism. A quick survey of the field reveals that causality is a highly varied thing. The concept is used in different ways in different contexts, and not all uses are compatible. There are several competing theories, all with counterexamples. As Nancy Cartwright ( 2007b ) has pointed out, “There is no single interesting characterizing feature of causation; hence no off-the-shelf or one-size-fits-all method for finding out about it, no ‘gold standard’ for judging causal relations” (p. 2).

The approach to causality taken here is twofold. First, there should be room for causality in education; we just have to be very careful how we think about it. Causality is an important ingredient in education because it denotes a dynamic relationship between factors of various kinds. Causes make their effects happen; they make a difference to the effect. Causality implies change and how it can be brought about, and this is something that surely lies at the heart of education. Ordinary educational talk is replete with causal verbs, for example, enhance, improve, reduce, increase, encourage, motivate, influence, affect, intervene, bring about, prevent, enable, contribute. The short version of the causal nature of education, and so EBP, is therefore that EBP is causal because it concerns the bringing about of desirable results (or the preventing of undesirable results). We have a causal connection between an action or an intervention and its effect, between X and Y. The longer version of the causal nature of EBP takes into account the many forms of causality: direct, indirect, necessary, sufficient, probable, deterministic, general, actual, potential, singular, strong, weak, robust, fragile, chains, multiple causes, two-way connections, side-effects, and so on. What is important is that we adopt an understanding of causality that fits the nature of EBP and does not do violence to the matter at hand. That leads me to my second point: the suggestion that in EBP causes are best understood as INUS conditions.

The understanding of causes as INUS conditions was pioneered by the philosopher John Mackie ( 1975 ). He placed his account within what is known as the regularity theory of causality. Regularity theory is largely the legacy of David Hume, and it describes causality as the constant conjunction of two entities (cause and effect, input and output). Like many others, Mackie took (some version of) regularity theory to be the common view of causality. Regularities are generally expressed in terms of necessity and sufficiency. In a causal law, the cause would be held to be both necessary and sufficient for the occurrence of the effect; the cause would produce its effect every time; and the relation would be constant. This is the starting point of Mackie’s brilliant refinement of the regularity view. Suppose, he said, that a fire has broken out in a house, and that the experts conclude that it was caused by an electrical short circuit. How should we understand this claim? The short circuit is not necessary, since many other events could have caused the fire. Nor is it sufficient, since short circuits may happen without causing a fire. But if the short circuit is neither necessary nor sufficient, then what do we mean by saying that it caused the fire? What we mean, Mackie ( 1975 ) suggests, is that the short circuit is an INUS condition: “an insufficient but necessary part of a condition which is itself unnecessary but sufficient for the result” (p. 16), INUS being an acronym formed of the initial letters of the italicized words. The main point is that a short circuit does not cause a fire all by itself; it requires the presence of oxygen and combustible material and the absence of a working sprinkler. On this approach, therefore, a cause is a complex set of conditions, of which some may be positive (present), and some may be negative (absent). In this constellation of factors, the event that is the focus of the definition (the insufficient but necessary factor) is the one that is salient to us. When we speak of an event causing another, we tend to let this factor represent the whole complex constellation.

In EBP, our own intervention X (strategy, method of instruction) is the factor we focus on, the factor that is salient to us, is within our control, and receives our attention. I propose that we understand any intervention we implement as an INUS condition. Then it immediately transpires that X not only does not bring about Y alone, but also that it cannot do so.

Before inquiring further into interventions as INUS -conditions, we should briefly characterize causality in education more broadly. Most causal theories, but not all of them, understand causal connections in terms of probability—that is, causing is making more likely. This means that causes sometimes make their effects happen, and sometimes not. A basic understanding of causality as indeterministic is vitally important in education, for two reasons. First, because the world is diverse, it is to some extent unpredictable, and planning for results is by no means straightforward. Second, because we can here clear up a fundamental misunderstanding about causality in education: causality is not deterministic and the effect is therefore not necessitated by the cause. The most common phrase in causal theory seems to be that causes make a difference for the effect (Schaffer, 2007 ). We must be flexible in our thinking here. One factor can make a difference for another factor in a great variety of ways: prevent it, contribute to it, enhance it as part of a causal chain, hinder it via one path and increase it via another, delay it, or produce undesirable side effects, and so on. This is not just conceptual hair-splitting; it has great practical import. Educational researchers may tell us that X causes Y, but what a practitioner can do with that knowledge differs radically if X is a potential cause, a disabler, a sufficient cause, or the absence of a hindrance.

Interventions as INUS Conditions

Human affairs, including education, are complex, and it stands to reason that a given outcome will have several sources and causes. While one of the factors in a causal constellation is salient to us, the others jointly enable X to have an effect. This enabling role is eminently generalizable and crucial to understanding how interventions bring about their effects. As Mackie’s example suggests, enablers may also be absences—that is vital to note, since absences normally go under our radar.

The term “intervention” deserves brief mention. To some it seems to denote a form of practice that is interested only (or mainly) in producing measurable changes on selected output variables. It is not obvious that there is a clear conception of intervention in EBP, but we should refrain from imposing heavy restrictions on it. I thus propose to employ the broad understanding suggested by Peter Menzies and Huw Price ( 1993 )—namely, interventions as a natural part of human agency. We all have the ability to intervene in the world and influence it; that is, to act as agents. Educational interventions may thus take many forms and encompass actions, strategies, programs and methods of instruction. Most interventions will be composites consisting of many different activities, and some, for instance, schoolwide behavioral programs, are meant to run for a considerable length of time.

When practitioners consider implementing an intervention X, the INUS approach encourages them to also consider what the enabling conditions are and how they might allow X to produce Y (or to contribute to its production). Our general knowledge of house fires and how they start prompts us to look at factors such as oxygen, materials, and fire extinguishers. In other cases, we might not know what the enabling conditions are. Suppose a teacher observes that some of his first graders are reading delayed. What to do? The teacher may decide to implement what we might call “Hatcher’s method” (Hatcher et al., 2006 ). This “method” focuses on letter knowledge, single-word reading, and phoneme awareness and lasts for two consecutive 10-week periods. Hatcher and colleagues’ study showed that about 75% of the children who received it made significant progress. So should our teacher now simply implement the method and expect the results with his own students to be (approximately) the same? As any teacher knows, what worked in one context might not work in another context. What we can infer from the fact that the method, X, worked where the data were collected is that a sufficient set of support factors were present to enable X to work. That is, Hatcher’s method serves as an INUS condition in a larger constellation of factors that together are sufficient for a positive result for a good many of the individuals in the study population. Do we know what the enabling factors are—the factors that correspond to presence of oxygen and inflammable material and absence of sprinkler in Mackie’s example? Not necessarily. General educational knowledge may tell us something, but enablers are also contextual. Examples of possible enablers include student motivation, parental support (important if the method requires homework), adequate materials, a separate room, and sufficient time. Maybe the program requires a teacher’s assistant? The enablers are factors that X requires to bring about or improve Y; if they are missing, X might not be able to do its work.

Understanding X as an INUS condition adds quite a lot of complexity to the simple X–Y picture and may thus alleviate at least some of the EBP critics’ fear that EBP is inherently reductionist and oversimplified. EBP is at heart causal, but that does not entail a deterministic, simplistic or physical understanding. Rather, I have argued, to do justice to EBP in education its causal nature must be understood to be both complex and sophisticated. We should also note here that X can enter into different constellations. The enablers in one context need not be the same as the enablers in another context. In fact, we should expect them to be different, simply because contexts are different.

Evidence and Its Uses

Evidence is an epistemological concept. In its immediate surroundings we find such concepts as justification, support, hypotheses, reasons, grounds, truth, confirmation, disconfirmation, falsification, and others. It is often unclear what people take evidence and its function to be. In epistemology, evidence is that which serves to confirm or disconfirm a hypothesis (claim, belief, theory; Achinstein, 2001 ; Kelly, 2008 ). The basic function of evidence is thus summed up in the word “support”: evidence is something that stands in a relation of support (confirmation, disconfirmation) to a claim or hypothesis, and provides us with good reason to believe that a claim is true (or false). The question of what can count as evidence is the question of what kind of stuff can enter into such evidential relations with a claim. This question is controversial in EBP and usually amounts to criticism of evidence hierarchies. The standard criticisms are that such hierarchies unduly privilege certain forms of knowledge and research design (Oancea & Pring, 2008 ), undervalue the contribution of other research perspectives (Pawson, 2012 ), and undervalue professional experience and judgment (Hammersley, 1997 , 2004 ). It is, however, not of much use to discuss evidence in and of itself—we must look at what we want evidence for . Evidence is that which can perform a support function, including all sorts of data, facts, personal experiences, and even physical traces and objects. In murder mysteries, bloody footprints, knives, and witness observations count as evidence, for or against the hypothesis that the butler did it. In everyday life, a face covered in ice cream is evidence of who ate the dessert before dinner.

There are three important things to keep in mind concerning evidence. First, in principle, many different entities can play the role of evidence and enter into an evidentiary relation with a claim (hypothesis, belief). Second, what counts as evidence in each case has everything to do with the type of claim we are interested in. If we want evidence that something is possible, observation of one single instance is sufficient evidence. If we want evidence for a general claim, we at least need enough data to judge that the hypothesis has good inductive support. If we want to bolster the normative conclusion that means M1 serves end E better than means M2, we have to adduce a range of evidences and reasons, from causal connections to ethical considerations (Hitchcock, 2011 ). If we want to back up our hypothesis that the butler is guilty of stealing Lady Markham’s necklace, we have to take into consideration such diverse pieces of evidence as fingerprints, reconstructed timelines, witness observations and alibis. Third, evidence comes in different degrees of trustworthiness, which is why evidence must be evaluated—bad evidence cannot be used to support a hypothesis and does not speak to its truth value; weak evidence can support a hypothesis and speak to its truth value, but only weakly.

The goal in EBP is to find evidence for a causal claim. Here we meet with a problem, because causal claims come in many different shapes: for example, “X leads to Y,” “doing X sometimes leads to Y and sometimes to G,” “X contributes moderately to Y” and “given Z, X will make a difference to Y.” On the INUS approach the hypothesis is that X, in conjunction with a suitable set of support factors, in all likelihood will lead to Y (or will contribute positively to Y, or make a difference to the bringing about of Y). The reason why RCTs are preferred is precisely that we are dealing with causal claims. Provided that the RCT design satisfies all requirements, it controls for confounders, and makes it possible to distinguish correlations from causal connections and to draw causal inferences with a high degree of confidence. In RCTs we compare two groups, the study group and the control group. Random assignment is supposed to ensure that the groups have the same distribution of causal and other factors, save one—namely, the intervention X (but do note that the value of randomization has recently been problematized, most notably by John Worrall ( 2007 ). The standard result from an RCT is a treatment effect, expressed in terms of an effect size. An effect size is a statistical measure denoting average effect in the treatment group minus average effect in the control group (to simplify). We tend to assume that any difference between the groups requires a causal explanation. Since other factors and confounders are (assumed to be) evenly distributed and thus controlled for, we infer that the treatment, whatever it is, is the cause of the difference. Thus, the evidence-ranking schemes seem to have some justification, despite Cartwright’s insistence that there is no gold standard for drawing causal inferences. We want evidence for causal claims, and RCTs yield highly trustworthy evidence and, hence, give us good reason to believe the causal hypothesis. In most cases the causal hypothesis is of the form “if we do X it will lead to Y.”

Effectiveness

Effectiveness is much sought after in EBP. For example, Philip Davies ( 2004 ) describes the role of the Campbell Collaboration as helping both policymakers and practitioners make good decisions by providing systematic reviews of the effectiveness of social and behavioral interventions in education. The US Department of Education’s Identifying and Implementing Educational Practices Supported by Rigorous Evidence: A User Friendly Guide ( 2003 ) provides an example of how evidence, evidence hierarchies, effectiveness, and “what works” are tied together. The aim of the guide is to provide practitioners with the tools to distinguish practices that are supported by rigorous evidence from practices that are not. “Rigorous evidence” is here identical to RCT evidence, and the guide devotes an entire chapter to RCTs and why they yield strong evidence for the effectiveness of some intervention. Thus:

The intervention should be demonstrated effective, through well-designed randomized controlled trials, in more than one site of implementation;

These sites should be typical school or community settings, such as public school classrooms taught by regular teachers; and

The trials should demonstrate the intervention’s effectiveness in school settings similar to yours, before you can be confident that it will work in your schools/classrooms (p. 17).

Effectiveness is clearly at the heart of EBP, but what does it really mean? “Effectiveness” is a complex multidimensional concept containing causal, normative, and conceptual dimensions, all of which have different sides to them. Probabilistic causality comes in two main versions, one concerning causal strength and one concerning causal frequency or tendency (Kvernbekk, 2016 ). One common interpretation says that effectiveness concerns the relation between input and output—that is, the degree to which an intervention works. Effect sizes would seem to fall into this category, expressing as they do the magnitude of the effect and thereby the strength of the cause.

But a large effect size is not the only thing we want; we also want the cause to make its effect happen regularly across different contexts. In other words, we are interested in frequency . A cause may not produce its effect every time but often enough to be of interest. If we are to be able to plan for results, X must produce its effect regularly. Reproducibility of desirable results thus depends crucially of the tendency of the cause to produce its effect wherever and whenever it appears. Hence, the term “effectiveness” signals generality. In passing, the same generality hides in the term “works”—if an intervention works, it can be relied on to produce its desired results wherever it is implemented. The issue of scope also belongs to this generality picture: for which groups do we think our causal claim holds? All students of a certain kind, for example, first graders who are responsive to extra word and phoneme training? Some first graders somewhere in the world? All first graders everywhere?

The normative dimension of “what works,” or effectiveness, is equally important, also because it demonstrates so well that effectiveness is a judgment we make. We sometimes gauge effectiveness by the relation between desired output and actual output; that is, if the correlation between the two is judged to be sufficiently high, we conclude that the method of instruction in question is effective. In such cases, the result (actual or desired) is central to our judgment, even if the focus of EBP undeniably lies on the means, and not on the goals. In a similar vein, to conclude that X works, you must judge the output to be satisfactory (enough), and that again depends on which success criteria you adopt (Morrison, 2001 ). Next, we have to consider the temporal dimension: how long must an effect linger for us to judge that X works? Three weeks? Two months? One year? Indefinitely? Finally, there is a conceptual dimension to judgments of effectiveness: the judgment of how well X works also depends on how the target is defined. For example, an assessment of the effectiveness of reading-instruction methods depends on what it means to say that students can read. Vague target articulations give much leeway for judgments of whether the target (Y) is attained, which, in turn, opens the possibility that many different Xs are judged to lead to Y.

Given the different dimensions of the term “effectiveness,” we should not wonder that effectiveness claims often equivocate on whether they mean effectiveness in terms of strength or frequency or perhaps both. The intended scope is often unclear, the target may be imprecise and the success criteria too broad or too narrow or left implicit altogether. However, since reproducibility of results is vitally important in EBP, it stands to reason that generality—external validity—should be of the greatest interest. All the strategies Dean, Hubbell, Pitler, and Stone ( 2012 ) discuss in their book about classroom instruction that works are explicitly general, for example, that providing feedback on homework assignments will benefit students and help enhance their achievements. This is generality in the frequency and (large) scope sense. It is future oriented: we expect interventions to produce much the same results in the future as they did in the past, and this makes planning possible.

The evidence for general causal claims is thought to emanate from RCTs, so let us turn again to RCTs to see whether they supply us with evidence that can support such claims. It would seem that we largely assume that they do. The Department of Education’s guide, as we have seen, presupposes that two RCTs are sufficient to demonstrate general effectiveness. Keith Morrison ( 2001 ) thinks that advocates of EBP simply assume that RCTs ensure generalizability, which is, of course, exactly what one wants in EBP—if results are generalizable, we may assume that the effect travels to other target populations so that results are reproducible and we can plan for their attainment. But does RCT evidence tell us that a cause holds widely? No, Cartwright ( 2007a ) argued, RCTs require strong premises, and strong premises do not hold widely. Because of design restrictions, RCT results hold formally for the study group (the sample) and only for that group, she insists. Methods that are strong on internal validity are correspondingly weak on external validity. RCTs establish efficacy, not effectiveness. We tend to assume without question, Cartwright argues, that efficacy is evidence for effectiveness. But we should not take this for granted—either it presumes that the effect depends exclusively on the intervention and not on who receives it, or it relies on presumed commonalities between the study group and the target group. This is a matter of concern to EBP and its advocates, because if Cartwright is correct, RCT evidence does not tell us what we think it tells us. Multiple RCTs will not solve this problem; the weakness of enumerative induction—inferences from single instances to a general conclusion—is well known. So how then can we ground our expectation that results are reproducible and can be planned for?

The Practitioner Perspective

EBP, as it is mostly discussed, is researcher centered. The typical advice guides, such as that of the What Works Clearinghouse, tend to focus on the finding of causes and the quality of the evidence produced. Claims and interventions should be rigorously tested by stringent methods such as RCTs and ranked accordingly. The narrowness of the kind of evidence thus admitted (or preferred) is pointed out by many critics, but it is of equal importance that the kind of claims RCT evidence is evidence for is also rather narrow. Shifting the focus from research to practice significantly changes the game. And bring in the practitioners we must—EBP is eminently practical in nature, concerning as it does the production of desirable results. Putting practice center stage means shifting from finding causes and assessing the quality of research evidence to using causes to produce change. In research we can control for confounders and keep variables fixed. In practice we can do no such thing; hence the significant change of the game.

The claim a practitioner wants evidence for is not the same claim that a researcher wants evidence for. The researcher wants evidence for a causal hypothesis, which we have seen can be of many different kinds, for example, the contribution of X to Y. The practitioner wants evidence for a different kind of claim—namely, whether X will contribute positively to Y for his students, in his context. This is the practitioner’s problem: the evidence that research provides, rigorous as it may be, does not tell him whether a proposed intervention will work here , for this particular target group. Something more is required.

Fidelity is a demand for faithfulness in implementation: if you are to implement an intervention that is backed by, say, two solid RCTs, you should do it exactly as it was done where the evidence was collected. The minimal definition of EBP adopted here leaves it open whether fidelity should be included or not, but there can be no doubt that both advocates and critics take it that it is—making fidelity one of the most controversial issues in EBP. The advocate argument centers on quality of implementation (e.g., Arnesen, Ogden, & Sørlie, 2006 ). It basically says that if X is implemented differently than is prescribed by researchers or program developers, we can no longer know exactly what it is that works. If unfaithfully implemented, the intervention might not produce the expected results, and the program developers cannot be held responsible for the results that do obtain. Failure to obtain the expected results is to be blamed on unsystematic or unfaithful implementation of a program, the argument goes. Note that the results are described as expected.

The critics, on the other hand, interpret fidelity as an attempt to curb the judgment and practical knowledge of the teachers; perhaps even as an attempt to replace professional judgment with research evidence. Biesta ( 2007 ), for example, argues that in the EBP framework the only thing that remains for practitioners to do is to follow rules for action. These rules are thought to be somehow directly derived from the evidence. Biesta is by no means the only EBP critic to voice this criticism; we find the same view in Bridges, Smeyers, and Smith ( 2008 ):

The evidence-based policy movement seems almost to presuppose an algorithm which will generate policy decisions: If A is what you want to achieve and if research shows R1, R2 and R3 to be the case, and if furthermore research shows that doing P is positively correlated with A, then it follows that P is what you need to do. So provided you have your educational/political goals sorted out, all you need to do is slot in the appropriate research findings—the right information—to extract your policy. (p. 9)

No consideration of the concrete situation is deemed necessary, and professional judgment therefore becomes practically superfluous. Many critics of EBP make the same point: teaching should not be a matter of following rules, but a matter of making judgments. If fidelity implies following highly scripted lessons to the letter, the critics have a good point. If fidelity means being faithful to higher level principles, such as “provide feedback on home assignments,” it becomes more open and it is no longer clear exactly what one is supposed to be faithful to, since feedback can be given in a number of ways. We should also note here that EBP advocates, for example, David Hargreaves ( 1996b ), emphatically insist that evidence should enhance professional judgment, not replace it. Let us also note briefly the usage of the term “evidence,” since it deviates from the epistemological usage of the term. Biesta (and other critics) picture evidence as something from which rules for action can be inferred. But evidence is (quantitative) data that speak to the truth value of a causal hypothesis, not something from which you derive rules for action. Indeed, the word “based” in evidence-based practice is misleading—practice is not based on the RCT evidence; it is based on the hypothesis (supposedly) supported by the evidence. Remember that the role of evidence can be summed up as support . Evidence surely can enhance judgment, although EBP advocates tend to be rather hazy about how this is supposed to happen, especially if they also endorse the principle of fidelity.

Contextual Matters

If we hold that causes are easily exportable and can be relied on to produce their effect across a variety of different contexts, we rely on a number of assumptions about causality and about contexts. For example, we must assume that the causal X–Y relation is somehow basic, that it simply holds in and of itself. This assumption is easy to form; if we have conducted an RCT (or several, and pooled the results in a meta-analysis) and found a relation between an intervention and an effect of a decent magnitude, chances are that we conclude that this relation simply exists. Causal relations that hold in and of themselves naturally also hold widely; they are stable, and the cause can be relied on as sufficient to bring about its effect most of the time, in most contexts, if not all. This is a very powerful set of assumptions indeed—it underpins the belief that desirable results are reproducible and can be planned for, which is exactly what not only EBP wants but what practical pedagogy wants and what everyday life in general runs on.

The second set of assumptions concerns context. The US Department of Education guide ( 2003 ) advises that RCTs should demonstrate the intervention’s effectiveness in school settings similar to yours, before you can be confident that it will work for you. The guide provides no information about what features should be similar or how similar those features should be; still, a common enough assumption is hinted at here: if two contexts are (sufficiently) similar (on the right kind of features) the cause that worked in one will also work in the other. But as all teachers know, students are different, teachers are different, parents are different, headmasters are different, and school cultures are different. The problem faced by EBP is how deep these differences are and what they imply for the exportability of interventions.

On the view taken here, causal relations are not general, not basic, and therefore do not hold in and of themselves. Causal relations are context dependent, and contexts should be expected to be different, just as people are different. This view poses problems for the practitioner, because it means that an intervention that is shown by an RCT to work somewhere (or in many somewheres ) cannot simply be assumed to work here . Using causes in practice to bring about desirable changes is very different from finding them, and context is all-important (Cartwright, 2012 ).

All interventions are inserted into an already existing practice, and all practices are highly complex causal/social systems with many factors, causes, effects, persons, beliefs, values, interactions and relations. This system already produces an output Y; we are just not happy with it and wish to improve it. Suppose that most of our first graders do learn to read, but that some are reading delayed. We wish to change that, so we consider whether to implement Hatcher’s method. We intervene by changing the cause that we hold to be (mainly) responsible for Y—namely, X—or we implement a brand-new X. But when we implement X or change it from x i to x j (shifting from one method of reading instruction to another), we generally thereby also change other factors in the system (context, practice), not just the ones causally downstream from X. We might (inadvertently) have changed both A, B, and C—all of which may have an effect on Y. Some of these contextual changes might reinforce the effect of X; others might counteract it. For example, in selecting the group of reading-delayed children for special treatment, we might find that we change the interactional patterns in the class, and that we change the attitudes of parents toward their children’s education and toward the teacher or the school. With the changes to A, B, and C, we are no longer in system g but in system h . The probability of Y might thereby change; it might increase or it might decrease. Hence, insofar as EBP focuses exclusively on the X–Y relation, natural as this is, it tells only half the story. If we take the context into account, it transpires that if X is going to be an efficacious strategy for changing (bringing about, enhancing, improving, preventing, reducing) Y, then it is not the relation between X and Y that matters the most. What matters instead is that the probability of Y given X-in-conjunction-with-system is higher than the probability of Y given not-X-in-conjunction-with-system. But what do we need to know to make such judgments?

Relevance and Evidence

On the understanding of EBP advanced here, fidelity is misguided. It rests on causal assumptions that are at least problematic; it fails to distinguish between finding causes and using causes; and it fails to pay proper attention to contextual matters.

What, then, should a practitioner look for when trying to make a decision about whether to implement X or not? X has worked somewhere ; that has been established by RCTs. But when is the fact that X has worked somewhere relevant to a judgment that X will also work here ? If the world is diverse, we cannot simply export a causal connection, insert it into a different context, and expect it to work there. The practitioner will need to gather a lot of heterogeneous evidence, put it together, and make an astute all-things-considered judgment about the likelihood that X will bring about the desired results here were it to be implemented. The success of EBP depends not only on rigorous research evidence but also on the steps taken to use an intervention to bring about desirable changes in a context where the intervention is as yet untried.

What are the things to be considered for an all-things-considered decision about implementing X? First, the practitioner already knows that X has worked somewhere ; the RCT evidence tells him or her that. Thus, we do know that X played a positive causal role for many of the individuals in the study group (but not necessarily all of them; effect sizes are aggregate results and thus compatible with negative results for some individuals).

Second, the practitioner must think about how the intervention might work if it were implemented. RCTs run on an input–output logic and do not tell us anything about how the cause is thought to bring about its effect. But a practitioner needs to ask whether X can play a positive causal role in his or her context, and then the question to ask is how , rather than what .

Third, given our understanding of causes as INUS conditions, the practitioner will have to map the contextual factors that are necessary for X to be able to do its work and bring about Y. What are the enabling factors? If they are not present, can they be easily procured? Do they outweigh any disabling factors that may be present? It is important to remember that enablers may be absences of hindrances. Despite their adherence to the principle of fidelity, Arnesen, Ogden, and Sørlie ( 2006 ) acknowledge the importance of context for bringing about Y. For example, they point out that there must be no staff conflicts if the behavioural program is to work. Such conflicts would be a contextual disabler, and their absence is necessary. If you wish to implement Hatcher’s method, you have to look at your students and decide whether you think this will suit them, whether they are motivated, and how they might interact with the method and the materials. As David Olson ( 2004 ) points out, the effect of an intervention depends on how it is “taken” or understood by the learner. But vital contextual factors also include mundane things such as availability of adequate materials, whether the parents will support and help if the method requires homework, whether you have a suitable classroom and sufficient extra time, whether a teacher assistant is available, and so on. Hatcher’s method is the INUS condition, the salient factor, but it requires a contextual support team to be able to do its work.

Fourth, the practitioner needs to have some idea of how the context might change as a result of implementing X. Will it change the interactions among the students? Create jealousy? Take resources meant for other activities? The stability of the system into which an intervention is inserted is generally of vital importance for our chances of success. If the system is shifting and unstable X may never be able to make its effect happen. The practitioner must therefore know what the stabilizing factors are and how to control them (assuming they are within his or her control).

In sum, the INUS approach to causality and the all-important role of contextual factors and the target group members themselves in bringing about results strongly suggest that fidelity is misguided. The intervention is not solely responsible for the result; one has to take both the target group (whatever the scope) and contextual factors into consideration. On the other hand, similarity of contexts loses its significance because an intervention that worked somewhere can be made to be relevant here—there is no reason to assume that one needs exactly the same contextual support factors. The enablers that made X work there need not the same enablers that will make X work here . What is important is that the practitioner carefully considers how X can be made to work in his or her context.

EBP is a complex enterprise. The seemingly simple question of using the best available evidence to bring about desirable results and prevent undesirable ones branches out in different directions to involve problems concerning what educational research can and should contribute to practice, the nature of teaching, what kind of knowledge teachers need, what education should be all about, how we judge what works, the role of context and the exportability of interventions, what we think causality is, and so on. We thus meet both ontological, epistemological, and normative questions.

It is important to distinguish between the evidence and the claim which it is evidence for . Evidence serves to support (confirm, disconfirm) a claim, and strictly speaking practice is based on claims, not on evidence. Research evidence (as well as everyday types of evidence) should always be evaluated for its trustworthiness, its relevance, and its scope.

EBP as it is generally discussed emphasizes research at the expense of practice. The demands of rigor made on research evidence are very high. There is a growing literature on implementation and a growing understanding of the importance of quality of implementation, but insofar as this focuses on fidelity, it is misguided. Fidelity fails to take into account the diversity of the world and the importance of the context into which an intervention is to be inserted. It is argued here that implementation centers on the matter of whether an intervention will work here and that a reasonable answer to that question requires much local, heterogeneous evidence. The local evidence concerning target group and context must be provided by the practitioner. The research evidence tells only part of the story.

If EBP is to be a success, the research story and the local-practice story must be brought together, and this is the practitioner’s job. The researcher does not know what is relevant in the concrete context faced by the practitioner; that is for the practitioner to decide.

EBP thus demands much knowledge, good thinking, and astute judgments by practitioners.

As a recommendation for future research, I would suggest inquiries into how the research story and the contextual story come together; how practitioners understand the causal systems they work within, how they understand effectiveness, and how they adapt or translate generalized guidelines into concrete local practice.

• Achinstein, P. (2001). The book of evidence . Oxford: Oxford University Press.
• Arnesen, A. , Ogden, T. , & Sørlie, M.-A. (2006). Positiv atferd og støttende læringsmiljø i skolen . Oslo: Universitetsforlaget.
• Biesta, G. (2007). Why “what works” won’t work: Evidence-based practice and the democratic deficit in educational research. Educational Theory , 57 , 1–22.
• Biesta, G. (2010). Good education in an age of measurement: Ethics, politics, democracy . Boulder, CO: Paradigm.
• Bridges, D. , Smeyers, P. , & Smith, R. (2008). Educational research and the practical judgment of policy makers. Journal of Philosophy of Education , 42 (Suppl. 1), 5–11.
• Cartwright, N. (2007a). Are RCTs the gold standard? BioSocieties , 2 , 11–20.
• Cartwright, N. (2007b). Hunting causes and using them: Approaches in philosophy and e conomics . Cambridge, U.K.: Cambridge University Press.
• Cartwright, N. (2012). Will this policy work for you? Predicting effectiveness better: How philosophy helps. Philosophy of Science , 79 , 973–989.
• Cartwright, N. , & Hardie, J. (2012). Evidence-based policy: A practical guide to doing it better . Oxford: Oxford University Press.
• Davies, P. (2004). Systematic reviews and the Campbell Collaboration. In G. Thomas & R. Pring (Eds.), Evidence-based practice in education (pp. 21–33). Maidenhead, U.K.: Open University Press.
• Dean, C. B. , Hubbell, E. R. , Pitler, H. , & Stone, Bj. (2012). Classroom instruction that works: Research-based strategies for increasing student achievement (2d ed.). Denver, CO: Mid-continent Research for Education and Learning.
• Hammersley, M. (1997). Educational research and teaching: A response to David Hargreaves’ TTA lecture. British Educational Research Journal , 23 , 141–161.
• Hammersley, M. (2004). Some questions about evidence-based practice in education. In G. Thomas & R. Pring (Eds.), Evidence-based practice in education (pp. 133–149). Maidenhead, U.K.: Open University Press.
• Hargreaves, D. (1996a). Educational research and evidence-based educational practice: A response to critics. Research Intelligence , 58 , 12–16.
• Hargreaves, D. (1996b). Teaching as a research-based profession: Possibilities and prospects. Teacher Training Agency Annual Lecture, London. Retrieved from https://eppi.ioe.ac.uk/cms/Portals/0/PDF%20reviews%20and%20summaries/TTA%20Hargreaves%20lecture.pdf .
• Hatcher, P. , Hulme, C. , Miles, J. N. , Caroll, J. M. , Hatcher, J. , Gibbs, S. , . . . Snowling, M. J. (2006). Efficacy of small group reading intervention for readers with reading delay: A randomised controlled trial. Journal of Child Psychology and Psychiatry , 47 (8), 820–827.
• Hitchcock, D. (2011). Instrumental rationality. In P. McBurney , I. Rahwan , & S. Parsons (Eds.), Argumentation in multi-agent systems: Proceedings of the 7th international ArgMAS Workshop (pp. 1–11). New York: Springer.
• Kelly, T. (2008). Evidence. In E. Zalta (Ed.), Stanford encyclopedia of philosophy . Retrieved from http://plato.stanford.edu/entries/evidence/ .
• Kvernbekk, T. (2016). Evidence-based practice in education: Functions of evidence and causal presuppositions . London: Routledge.
• Lærerorganisasjonenes skolenevnd . (1933). Innstilling . Oslo: O. Fredr. Arnesens Bok- og Akcidenstrykkeri.
• Mackie, J. L. (1975). Causes and conditions. In E. Sosa (Ed.), Causation and conditionals (pp. 15–38). Oxford: Oxford University Press.
• Menzies, P. , & Price, H. (1993). Causation as a secondary quality. British Journal for the Philosophy of Science , 44 , 187–203.
• Morrison, K. (2001). Randomised controlled trials for evidence-based education: Some problems in judging “what works.” Evaluation and Research in Education , 15 (2), 69–83.
• Oancea, A. , & Pring, R. (2008). The importance of being thorough: On systematic accumulation of “what works” in education research. Journal of Philosophy of Education , 42 (Suppl. 1), 15–39.
• Olson, D. R. (2004). The triumph of hope over experience in the search for “what works”: A response to Slavin. Educational Researcher , 33 , 24–26.
• Pawson, R. (2012). Evidence-based policy: A realist perspective . Los Angeles: SAGE.
• Schaffer, J. (2007). The metaphysics of causation. In E. Zalta (Ed.), Stanford encyclopedia of philosophy . Retrieved from http://plato.stanford.edu/entries/causation-metaphysics/ .
• Slavin, R. E. (2002). Evidence-based education policies: Transforming educational practice and research. Educational Researcher , 31 , 15–21.
• Slavin, R. E. (2004). Education research can and must address “what works” questions. Educational Researcher , 33 , 27–28.
• U.S. Department of Education . (2003). Identifying and implementing educational practices supported by rigorous evidence: A user friendly guide . Washington, DC: Coalition for Evidence-Based Policy. Retrieved from http://www2.ed.gov/rschstat/research/pubs/rigorousevid/rigorousevid/pdf .
• Worrall, J. (2007). Why there’s no cause to randomize. British Journal for the Philosophy of Science , 58 , 451–488.

Bibliography

• Biesta, G. (2010). Why “what works” still won’t work: From evidence-based education to value-based education. Studies in Philosophy and Education , 29 , 491–503.
• Bridges, D. , & Watts, M. (2008). Educational research and policy: Epistemological considerations. Journal of Philosophy of Education , 42 (Suppl. 1), 41–62.
• Cartwright, N. (2013). Knowing what we are talking about: Why evidence doesn’t always travel. Evidence and Policy , 9 , 97–112.
• Cartwright, N. , & Munro, E. (2010): The limitations of randomized controlled trials in predicting effectiveness. Journal of Evaluation in Clinical Practice , 16 , 260–266.
• Cartwright, N. , & Stegenga, J. (2011). A theory of evidence for evidence-based policy. Proceedings of the British Academy , 171 , 289–319.
• Hammersley, M. (Ed.). (2007). Educational research and evidence-based practice . Los Angeles: SAGE.
• Hattie, J. (2009). Visible learning: A synthesis of over 800 meta-analyses relating to achievement . London: Routledge.
• Hattie, J. (2012). Visible learning for teachers: Maximizing impact on learning . London: Routledge.
• Kvernbekk, T. (2013). Evidence-based practice: On the functions of evidence in practical reasoning. Studier i Pædagogisk Filosofi , 2 (2), 19–33.
• Pearl, J. (2009). Causality: Models, reasoning, and inference . Cambridge, U.K.: Cambridge University Press.
• Phillips, D. C. (2007). Adding complexity: Philosophical perspectives on the relationship between evidence and policy. In P. Moss (Ed.), Evidence and decision making . Yearbook of the National Society for the Study of Education, 106 (pp. 376–402). Malden, MA: Blackwell.
• Psillos, S. (2009). Regularity theories. In H. Beebee , C. Hitchcock , & P. Menzies (Eds.), The Oxford handbook of causation (pp. 131–157). Oxford: Oxford University Press.
• Reiss, J. (2009). Causation in the social sciences: Evidence, inference, and purpose. Philosophy of the Social Sciences , 39 , 20–40.
• Rosenfield, S. , & Berninger, V. (Eds.). (2009). Implementing evidence-based academic interventions in school settings . Oxford: Oxford University Press.
• Sanderson, I. (2003). Is it “what works” that matters? Evaluation and evidence-based policy- making. Research Papers in Education , 18 , 331–345.
• Sloman, S. (2005). Causal models: How people think about the world and its alternatives . Oxford: Oxford University Press.
• Smeyers, P. , & Depaepe, M. (Eds.). (2006). Educational research: Why “what works” doesn’t work . Dordrecht, The Netherlands: Springer.
• Thomas, G. , & Pring, R. (Eds.). (2004). Evidence-based practice in education . Maidenhead, U.K.: Open University Press.
• Williamson, J. (2009). Probabilistic theories. In H. Beebee , C. Hitchcock , & P. Menzies (Eds.), The Oxford handbook of causation (pp. 185–212). Oxford: Oxford University Press.
• Woodward, J. (2003). Making things happen: A theory of causal explanation . Oxford: Oxford University Press.

Related Articles

• Rethinking Curriculum and Teaching
• Accountabilities in Schools and School Systems

Printed from Oxford Research Encyclopedias, Education. Under the terms of the licence agreement, an individual user may print out a single article for personal use (for details see Privacy Policy and Legal Notice).

date: 09 June 2024

• Cookie Policy
• Privacy Policy
• Legal Notice
• Accessibility
• [81.177.182.136]
• 81.177.182.136

Character limit 500 /500

• Research Guides

Doing Research in Education

• Empirical Research
• Case Studies
• Statistics and Reports
• Citing Your Sources: APA

What are Empirical Studies?

The  APA Dictionary of Psychology  defines empirical as: "derived from or denoting experimentation or systematic observations as the basis for conclusion or determination, as opposed to speculative, theoretical, or exclusively reason-based approaches" (p. 327).

Therefore, an empirical study is one based on "facts, systematic observation, or experiment, rather than theory or general philosophical principle" ( APA Databases Methodology Field Values ).

Finding Empirical Studies in PsycInfo

It is easy to find empirical studies in PsycInfo .

On the Advanced Search screen, scroll down to near the bottom of the screen and choose "Empirical Study" from the Methodology limit box:

If you have already done a search, look for the filter for Methodology on the left side of the search results screen:

Using PsycInfo to Find Peer-Reviewed, Empirical Studies

For additional PsycInfo search tips, see this YouTube playlist

• << Previous: Home
• Next: Case Studies >>
• Last Updated: Mar 13, 2023 4:38 PM
• URL: https://libguides.williams.edu/education-research

What is Empirical Evidence?

Published Jan 5, 2018

Empirical evidence is information that researchers generate to help uncover answers to questions that can have significant implications for our society.

Take seatbelts. Prior to their invention, people were killed or maimed in what today we would think of as minor traffic accidents. So smart engineers put their heads together to try to do something about it.

Let’s try tying people down! Let’s change what the steering wheel is made of! Let’s put an exploding bag of air in the steering wheel! (Imagine how crazy that sounded in a pitch meeting.) These all seem like reasonable ideas (well except that exploding airbag one), so how do we know which one we should do?

The answer is to generate and weigh empirical evidence.

Theory vs. Empirical Evidence

One might have a theory about how something will play out, but what one observes or experiences can be different from what a theory might predict. People want to know the effectiveness of all sorts of things, which means they have to test them.

Social scientists produce empirical evidence in a variety of ways to test theories and measure the ability of A to produce an expected result: B.

Usually, researchers collect data through direct or indirect observation, and they analyze these data to answer empirical questions (questions that can be answered through observation).

Let’s look at our car safety example. Engineers and scientists equipped cars with various safety devices in various configurations, then smashed them into walls, poles and other cars and recorded what happened. Over time, they were able to figure out what types of safety devices worked and which ones didn’t. As it turns out, that whole airbag thing wasn’t so crazy after all.

They didn’t get everything right immediately. For instance, early seatbelts weren’t retractable. Some airbags shot pieces of metal into passengers . But, in fits and in starts, auto safety got better, and even though people are driving more and more miles, fewer and fewer are dying on the road .

How Gathering Empirical Evidence in Social Science is Different

Testing the effects of, say, a public policy on a group of people puts us in the territory of social science.

For instance, education research is not the same as automotive research because children (people) aren’t cars (objects). Education, though, can be made better by attempting new things, gathering data on those efforts, rigorously analyzing that data and then weighing all available empirical evidence to see if those new things accomplish what we hope they do.

Unfortunately, the “rigorously analyzing” bit is often missing from education research. In the labs of automobile engineers, great care is taken to only change one bit of design (a variable) at a time so that each test isolates the individual factor that is making a car more or less safe. OK, for this test, let’s just change the material of the steering wheel and keep everything else the same, so we’ll know if it is the wheel that is hurting people.

Comparing Apples with Apples

In social science and especially in education, trying to isolate variables is challenging, but possible, if researchers can make “apples-to-apples” comparisons.

The best way to get an apples-to-apples comparison is to perform something called a randomized control trial (RCT). You might have heard about these in relation to the testing of medicine. Drug testing uses RCTs all the time.

In an educational RCT, students are divided into two groups by a randomized lottery and half of the students receive whatever the educational “treatment” is (a new reading program, a change in approach to discipline, a school voucher, etc.) while the other does not. Researchers compare the results of those two groups and estimate the “treatment” effect. This approach gives us confidence that the observed effect is caused by the intervention and no other factors.

RCTs are not always possible. Sometimes researchers can get close by using random events that separate kids into two groups, such as school district boundaries that are created by rivers or creeks that split a community more or less by chance or birthday cutoffs for preschool that place a child born on August 31st in one grade but one born September 1st in another even though there is basically no difference between them. Depending on the exact nature of the event, these can be known as “regression discontinuity” or “instrumental variable” analyses, and they can be useful tools to estimate the effects of a program.

Researchers can also follow individual children that receive a treatment if they have data from before and after to see how that child’s educational trajectory changes over time. These are known as “fixed effects” analyses.

All three of these—randomized control trials, regression discontinuity analyses and fixed effects analyses —have their drawbacks.

Very few outside events are truly random. If, as regression discontinuity analysis often does, researchers only look at children just above or just below the cutoff, or, as fixed effects analysis often does, researchers look at only those children who switch from one school to another, those children might not be representative of the population. How would an intervention affect kids who are not close to a cutoff or border? Or kids who do not switch schools?

In the SlideShare below, we present empirical evidence based on rigorous research on private school choice programs as an example of how we, as academics and researchers ourselves, identify and characterize the high-quality empirical evidence in a given area of study.

[Slideshare no longer available]

A Couple Considerations

It’s a lot to wade through, so before you do, we’d like to offer two notes.

First, it is always important to understand the tradeoffs between internal and external validity.

Internal validity refers to how well a study is conducted—it gives us confidence that the effects we observe can be attributed to the intervention or program, not other factors.

For example, when the federal government wanted to know if Washington, D.C.’s school voucher program increased students’ reading and math test scores, researchers took the 2,308 students who applied for the program and randomly assigned 1,387 to get vouchers and 921 not to . They then followed the two groups over time, and when they analyzed the results, they could reasonably conclude that any differences were due to the offer of a voucher, because that is the only thing that was different between the two groups and they were different only because of random chance. This study had high internal validity.

External validity refers to the extent that we can generalize the findings from a study to other settings.

Let’s think about that same study. The D.C. program was unique. The amount of money that students receive, the regulations that participating schools had to agree to, the size of the program, its politically precarious situation and numerous other factors were different in that program than in others, not to mention the fact that Washington, D.C. is not representative of the United States as a whole demographically, politically or in really any way we can possibly imagine. As a result, we have to be cautious when we try to generalize the findings. The study has lower external validity.

To combat issues around lower external validity, researchers can collect and analyze empirical evidence on program design to understand its impact. We can also look at multiple studies to see how similar interventions affect students in different settings.

Second, the respect and use of research does not endorse technocracy. Research and expertise is incredibly useful. When you get on an airplane or head into surgery, you want the person who is doing the work to be an expert. Empirical evidence can help us know more about the world and be better at what we do. But we should also exercise restraint and humility by recognizing the limits of social science.

Public policy involves weighing tradeoffs that social science cannot do for us. Social science can tell us that a program increases reading scores but also increases anxiety and depression in children. Should that program be allowed to continue? Ultimately, that comes down to human judgment and values. That should never be forgotten.

With that, we hope you found this article helpful. Please feel free to reach out with any questions by emailing [email protected] or posting your question in the comments section below.

Michael Q. McShane

Director of national research, edchoice.

Dr. Michael McShane is Director of National Research at EdChoice. He is the author, editor, co-author, or co-editor eleven books on education policy, including his most recent Hybrid Homeschooling: A Guide to the Future of Education (Rowman and Littlefield, 2021) He is currently an opinion contributor to Forbes, and his analyses and commentary have been published widely in the media, including in USA Today, The Washington Post, and the Wall Street Journal. He has also been featured in education-specific outlets such as Teachers College Record, Education Week, Phi Delta Kappan, and Education Next. In addition to authoring numerous white papers, McShane has had academic work published in Education Finance and Policy, The Handbook of Education Politics and Policy, and the Journal of School Choice. A former high school teacher, he earned a Ph.D. in education policy from the University of Arkansas, an M.Ed. from the University of Notre Dame, and a B.A. in English from St. Louis University.

Will Flanders

Education research director, wisconsin institute for law & liberty.

Will Flanders is the education research director at the Wisconsin Institute for Law & Liberty. He holds a Ph.D. in Political Science with a specialization in American Politics and Public Policy, an M.S. in Political Science and an M.S. in Applied American Politics and Policy from Florida State University.

Higher Education and Student Protests: How K–12 Parents View Campus Activism

John Kristof

Updated June 5, 2024

Poll Suggests 10% of School Parents are Microschooling Their Kids

Updated May 15, 2024

RECEIVE EDUCATIONAL CHOICE UPDATES STRAIGHT TO YOUR INBOX

• First Name *
• Last Name *
• © 2024 EdChoice
• Privacy Policy
• Terms of Use

Open Education Sociology Dictionary

empirical evidence

Table of Contents

Definition of Empirical Evidence

( noun ) Data gained through observation or experimentation .

Example of Empirical Evidence

• Ethnographic observations of a social scene.

Empirical Evidence Pronunciation

Pronunciation Usage Guide

Syllabification : em·pir·i·cal ev·i·dence

Audio Pronunciation

Phonetic Spelling

• American English – /im-pIR-i-kuhl Ev-uh-duhns/
• British English – /im-pIr-i-kuhl E-vi-duhns/

International Phonetic Alphabet

• American English – /ˌɛmˈpɪrɪkəl ˈɛvədəns/
• British English – /ɛmˈpɪrɪkəl ˈɛvɪdəns/

Usage Notes

• Empirical evidence is discovered through empirical research .
• empirical data
• empirical knowledge
• sense experience

Related Quotation

• “For essentialists, race , sex , sexual orientation, disability , and social class identify significant, empirically verifiable differences among people . From the essentialist perspective, each of the these exist apart from any social processes; they are objective categories of real differences among people ” (Rosenblum and Travis 2012:3).

Related Video

Additional Information

• Qualitative Research Resources – Books, Journals, and Helpful Links
• Quantitative Research Resources – Books, Journals, and Helpful Links
• Word origin of “empirical” and “evidence” – Online Etymology Dictionary: etymonline.com

Related Terms

• control group
• correlation
• experimentation
• field research

Rosenblum, Karen Elaine, and Toni-Michelle Travis. 2012. The Meaning of Difference: American Constructions of Race, Sex and Gender, Social Class, Sexual Orientation, and Disability . 6th ed. New York: McGraw-Hill.

Works Consulted

Griffiths, Heather, Nathan Keirns, Eric Strayer, Susan Cody-Rydzewski, Gail Scaramuzzo, Tommy Sadler, Sally Vyain, Jeff Bry, Faye Jones. 2016. Introduction to Sociology 2e . Houston, TX: OpenStax.

Macionis, John. 2012.  Sociology . 14th ed. Boston: Pearson.

Macionis, John, and Kenneth Plummer. 2012.  Sociology: A Global Introduction . 4th ed. Harlow, England: Pearson Education.

Merriam-Webster. (N.d.) Merriam-Webster Dictionary . ( http://www.merriam-webster.com/ ).

Oxford University Press. (N.d.) Oxford Dictionaries . ( https://www.oxforddictionaries.com/ ).

Random House Webster’s College Dictionary . 1997. New York: Random House.

Scott, John, and Gordon Marshall. 2005.  A Dictionary of Sociology . New York: Oxford University Press.

Thorpe, Christopher, Chris Yuill, Mitchell Hobbs, Sarah Tomley, and Marcus Weeks. 2015. The Sociology Book: Big Ideas Simply Explained . London: Dorling Kindersley.

Turner, Bryan S., ed. 2006. The Cambridge Dictionary of Sociology . Cambridge: Cambridge University Press.

Wikipedia contributors. (N.d.) Wikipedia, The Free Encyclopedia . Wikimedia Foundation. ( https://en.wikipedia.org/ ).

Cite the Definition of Empirical Evidence

ASA – American Sociological Association (5th edition)

Bell, Kenton, ed. 2013. “empirical evidence.” In Open Education Sociology Dictionary . Retrieved June 9, 2024 ( https://sociologydictionary.org/empirical-evidence/ ).

APA – American Psychological Association (6th edition)

empirical evidence. (2013). In K. Bell (Ed.), Open education sociology dictionary . Retrieved from https://sociologydictionary.org/empirical-evidence/

Chicago/Turabian: Author-Date – Chicago Manual of Style (16th edition)

Bell, Kenton, ed. 2013. “empirical evidence.” In Open Education Sociology Dictionary . Accessed June 9, 2024. https://sociologydictionary.org/empirical-evidence/ .

MLA – Modern Language Association (7th edition)

“empirical evidence.” Open Education Sociology Dictionary . Ed. Kenton Bell. 2013. Web. 9 Jun. 2024. < https://sociologydictionary.org/empirical-evidence/ >.

• My Favorites

You have successfully logged in but...

... your login credentials do not authorize you to access this content in the selected format. Access to this content in this format requires a current subscription or a prior purchase. Please select the WEB or READ option instead (if available). Or consider purchasing the publication.

Evidence in Education

Linking Research and Policy

Centre for educational research and innovation.

Education policies and systems in all OECD countries are coming under increasing pressure to show greater accountability and effectiveness and it is crucial that educational policy decisions are made based on the best evidence possible. This book brings together international experts on evidence-informed policy in education from a wide range of OECD countries. The report looks at the issues facing educational policy makers, researchers, and stakeholders – teachers, media, parents – in using evidence to best effect. It focuses on the challenge of effective brokering between policy makers and researchers, offers specific examples of major policy-related research, and presents perspectives from several senior politicians. This book provides a fresh outlook on key issues facing policy makers, researchers and school leaders today

12 Jun 2007 182 pages English

https://doi.org/10.1787/9789264033672-en 9789264033672 (PDF)

Author(s): OECD

• Click to access:
• Click to download PDF - 2.68MB PDF
• Click to Read online and share READ

Cite this content as:

Related content.

• New Challenges for Educational Research

See other content on

• Open access
• Published: 30 May 2024

Enhancing AI competence in health management: students’ experiences with ChatGPT as a learning Tool

• Lior Naamati-Schneider 1  

BMC Medical Education volume  24 , Article number:  598 ( 2024 ) Cite this article

396 Accesses

Metrics details

The healthcare industry has had to adapt to significant shifts caused by technological advancements, demographic changes, economic pressures, and political dynamics. These factors are reshaping the complex ecosystem in which healthcare organizations operate and have forced them to modify their operations in response to the rapidly evolving landscape. The increase in automation and the growing importance of digital and virtual environments are the key drivers necessitating this change. In the healthcare sector in particular, processes of change, including the incorporation of artificial intelligent language models like ChatGPT into daily life, necessitate a reevaluation of digital literacy skills.

This study proposes a novel pedagogical framework that integrates problem-based learning with the use of ChatGPT for undergraduate healthcare management students, while qualitatively exploring the students’ experiences with this technology through a thematic analysis of the reflective journals of 65 students.

Through the data analysis, the researcher identified five main categories: (1) Use of Literacy Skills; (2) User Experiences with ChatGPT; (3) ChatGPT Information Credibility; (4) Challenges and Barriers when Working with ChatGPT; (5) Mastering ChatGPT-Prompting Competencies . The findings show that incorporating digital tools, and particularly ChatGPT, in medical education has a positive impact on students’ digital literacy and on AI Literacy skills.

Conclusions

The results underscore the evolving nature of these skills in an AI-integrated educational environment and offer valuable insights into students’ perceptions and experiences. The study contributes to the broader discourse about the need for updated AI literacy skills in medical education from the early stages of education.

Peer Review reports

Introduction

In recent years, the healthcare sector has undergone significant shifts in both local and global contexts. These shifts are primarily attributed to demographic, technological, economic, and political factors. These changes have had a profound impact on the healthcare ecosystem, requiring organizations to adapt their operations and strategies to this evolving landscape [ 1 , 2 ]. In response, healthcare organizations have had to modify their behavior to adapt to this ever-changing reality [ 3 ]. Among the factors that have most significantly affected the healthcare system are technological advancements, automation, and the rise of digital and virtual environments. The impact of these factors gained momentum in December 2019, primarily due to the COVID-19 pandemic. Technological advances, particularly the rise of artificial intelligence (AI) and digital tools, have been central to this transformation, with the COVID-19 pandemic accelerating the need for healthcare systems to adapt and innovate [ 3 , 4 , 5 , 6 , 7 , 8 ]. The integration of AI in healthcare, including the deployment of chatbots like ChatGPT that utilize the Generative Pre-trained Transformer (GPT)—a type of large language model (LLM)—underscores a shift toward digital and AI literacy in medical education and practice. [ 9 , 10 ].

The adoption of AI in healthcare, highlighted by the use of systems like ChatGPT, marks a pivotal shift towards greater digital and AI literacy in medical education and practice [ 9 , 10 , 11 , 12 ]. This reflects the healthcare sector’s broader move towards technological innovation, aiming to enhance patient care and revolutionize healthcare professional training. Incorporating AI, such as ChatGPT, into educational frameworks prepares students for the complexities of modern healthcare, demonstrating AI’s potential to transform both healthcare delivery and professional skill development [ 11 , 12 ].

In the rapidly evolving landscape of AI, where technological developments are occurring at an accelerated pace, there is a significant need for comprehensive research to navigate this ever-changing landscape. In particular, research into the impact of AI on healthcare is still limited, highlighting the urgent need for more focused studies on the implications for medical education and the effective training of healthcare professionals in the use of AI technologies [ 13 , 14 ]. The emergence of LLMs, such as GPT, and their applications in educational frameworks, including chatbots like ChatGPT, has increased the urgency of reassessing the skills required, with a particular focus on digital literacy. This reassessment is essential to determine the continued relevance of these skills or whether a fundamental refocusing is required. Such a re-examination is essential to ensure that the healthcare workforce is adequately prepared for the challenges and opportunities presented by the integration of AI into healthcare practice [ 11 ].

Studies [ 15 , 16 , 17 , 18 ] have identified a significant gap in understanding how digital literacy skills—such as accessing, analyzing, evaluating, and creating digital content—play a role in effectively leveraging LLMs like GPT and their applications, including chatbots such as ChatGPT, within educational frameworks. Furthermore, the successful integration of ChatGPT into educational settings may potentially lessen the reliance on traditional digital literacy skills, prompting a reevaluation of their ongoing relevance [ 19 , 20 ]. This gap underscores the need for more research into the critical role that digital literacy skills hold in the efficient use of technologies like ChatGPT for educational aims, as highlighted by recent literature [ 15 , 17 , 18 ]. ChatGPT’s access to accurate medical information could reduce the need for individual data analysis skills [ 21 , 22 ]. Yet, concerns persist among researchers that its content generation might hinder critical thinking development, including source evaluation and idea generation [ 23 , 24 ].

This qualitative study introduces a pedagogical framework that synergizes problem-based learning with the application of ChatGPT among undergraduate healthcare management students. It aims to qualitatively examine their interactions with this technology, focusing on the transition from traditional digital literacy towards a more advanced AI literacy. This evolution in educational focus is poised to revolutionize the requisite competencies for navigating the dynamic healthcare sector of today.

The rationale behind focusing on ChatGPT stems from its notable accessibility, user-friendly design, and versatility as a comprehensive tool in healthcare settings. Its capability to simulate human-like dialogues positions it as a prime resource for educational initiatives, thereby enriching the pedagogical domain of healthcare management and clinical practices. The unrestricted access to ChatGPT, along with its wide-ranging utility in executing diverse healthcare operations, underscores its capacity to significantly contribute to and spearhead innovation within healthcare education and practices. The selection of ChatGPT, attributed to its approachability and adaptability, marks a strategic endeavor to investigate the impact of artificial intelligence amidst the shifting paradigms of healthcare requirements. Yet, despite the widespread integration of ChatGPT in healthcare, research into the long-term effects and the necessary adaptation of skills and methods remains lacking. [ 11 , 12 ].

Literature review

Ai tools in medical settings.

AI involves creating systems that mimic human cognitive functions such as perception, speech recognition, and decision-making through machine learning. It excels in analyzing data, identifying patterns, and making predictions, offering improvements over traditional data processing. AI’s applications span multiple sectors, including healthcare, at various levels from individual to global [ 25 , 26 ]. The integration of AI into healthcare enhances diagnostic, treatment, and patient care, offering advanced decision-making and predictions [ 9 , 10 , 25 , 27 ].AI technologies enhance clinical decision-making, diagnosis, and treatment by analyzing patient data through machine learning for informed decisions, offering 24/7 support via AI chatbots, and enabling remote monitoring with AI-powered devices like wearable sensors [ 9 , 28 ].

AI facilitates remote patient monitoring, minimizing in-person healthcare visits [ 29 ]. It improves service personalization, with AI assistants managing appointments and reminders, and chatbots streamlining insurance claims, easing provider workloads [ 9 ]. AI automates routine administrative tasks, freeing providers to concentrate on patient care. It streamlines operations, cuts bureaucracy, and analyzes data to improve healthcare management and predict service demand, allowing for better resource allocation. AI’s analysis of patient feedback further aids in enhancing service delivery [ 10 ]. AI integration can transform patient-caregiver dynamics, enhancing diagnosis, treatment, and self-management of health conditions [ 30 ]. While AI integration in healthcare promises significant advancements, it presents challenges, including data management issues and the need for specialized skills.

Sallam [ 14 ] highlights ChatGPT’s potential advantages in healthcare, including enhancing clinical workflows, diagnostics, and personalized medicine. However, challenges such as ethical dilemmas, interpretability issues, and content accuracy must be tackled. In healthcare education, although ChatGPT holds promise for customized learning and creating lifelike clinical scenarios, concerns about bias, plagiarism, and content quality persist. Addressing these concerns necessitates preparing healthcare professionals and students through education and training to navigate the complexities of AI. Additionally, extensive research in these domains is essential [ 6 , 9 , 14 , 31 , 32 ].

Teaching with AI and about AI: advancing education in the digital age

To be able to utilize AI tools effectively and integrate them seamlessly into their everyday work, healthcare professionals need early exposure to AI tools in their education to boost their proficiency and confidence, understanding both their potential and limitations [ 9 , 32 , 33 ]. York et al. [ 32 ] explored medical professionals’ attitudes towards AI in radiology, revealing a positive outlook on AI’s healthcare benefits but also highlighting a notable gap in AI knowledge. This emphasizes the need for enhanced AI training in medical education.

According to Sallam [ 14 ], ChatGPT and other models based on lLLMs have significantly improved healthcare education. They customize responses to student inquiries, curate relevant educational material, and tailor content to individual learning styles. For instance, ChatGPT generates personalized quiz questions, suggests resources to fill knowledge gaps, and adjusts explanations to suit diverse learning preferences. Moreover, it simplifies complex medical concepts, employs analogies and examples for clarity, and offers supplementary materials to enhance comprehension.

Breeding et al. [ 11 ] argued that in medical education, ChatGPT should be viewed as a supplementary tool rather than a substitute for traditional sources. While it offers clear and organized information, medical students still perceive evidence-based sources as more comprehensive. Eysenbach [ 33 ] engaged in a series of dialogues with ChatGPT to explore its integration into medical education. ChatGPT demonstrated proficiency in various tasks, such as grading essays, providing feedback, creating virtual patient scenarios, enhancing medical textbooks, summarizing research articles, and explaining key findings. Nevertheless, it also demonstrated a tendency to produce erroneous responses and fabricated data, including references. Such inaccuracies have the potential to generate student misconceptions, spread misinformation, and cause a decline in critical thinking skills [ 33 ]. Han et al. [ 34 ] conducted a comprehensive examination of ChatGPT’s effectiveness as a pedagogical tool in medical education, focusing on the chatbot’s interaction with delineated educational objectives and tasks. Their findings suggest that while ChatGPT is capable of providing elementary data and explanations, it is not impervious to constraints and sometimes provides incorrect or partial information. The study stresses active learning and analytical reasoning in medical education, emphasizing the importance of understanding basic sciences and the need for expert oversight to ensure AI-generated information accuracy [ 34 ].

Das et al. [ 35 ] evaluated ChatGPT’s efficacy in medical education, focusing on microbiology questions at different difficulty levels. They found that ChatGPT could answer basic and complex microbiology queries with roughly 80% accuracy, indicating its potential as an automated educational tool in medicine. The study underscores the importance of ongoing improvements in training language models to enhance their effectiveness for academic use [ 35 , 36 ].AI implementation in healthcare must be carefully managed to maximize benefits and minimize risks [ 11 , 12 , 35 , 36 ]. With the rapid development of digital technologies and AI tools, particularly in healthcare, students need appropriate resources to use these technologies effectively [ 37 ]. Digital literacy is essential in the 21st century, including skills for interacting with digital content [ 16 , 18 ]. Hence, medical literacy skills should start early in the education of healthcare students.

Digital literacy and eHealth literacy skills

Digital literacy skills encompass a collection of essential abilities necessary for using digital technologies effectively in accessing and retrieving information [ 38 ]. These skills are often viewed as foundational digital literacies that are critical for full participation in the digital era [ 39 ]. The European Commission emphasizes the importance of digital literacy for employability and citizenship. They advocate for policies and programs to enhance digital skills across all segments of society. The EU aims for 70% of adults to have basic digital skills by 2025, focusing on analytical, evaluative, and critical thinking abilities crucial for assessing digital information’s quality and credibility [ 40 ]. Individuals need these skills to discern biases and misinformation in various media formats [ 16 , 17 , 41 ] and evaluate the credibility of online sources [ 42 ]. Critical thinking is crucial for distinguishing between accurate information and misinformation [ 43 ], while data literacy is essential for interpreting data and detecting misleading statistics [ 44 ]. These competencies are fundamental for navigating today’s complex digital information landscape.

eHealth literacy, which incorporates the digital skills needed to access and utilize medical information from digital platforms [ 45 ], is gaining recognition as an integral component of overall health literacy. Enhanced online medical literacy is vital for healthcare professionals and administrators [ 46 ] to adapt to changing demands and improve care management within evolving healthcare paradigms [ 47 ]. Additionally, acquisition of digital competencies has been identified as a valuable strategy that healthcare providers and managers can use to manage the psychological effects of heightened workloads and uncertainty, such as the fear, stress, and anxiety emerging from the COVID-19 pandemic [ 48 ]. These skills enable individuals to use AI as both an independent tool and a supplementary aid in decision-making. However, addressing challenges like bias and academic integrity is crucial when integrating AI into medical education [ 32 , 33 , 49 ]. Critical thinking skills are essential for analyzing digital information, identifying inconsistencies, and evaluating arguments. In today’s era of misinformation, users must verify the accuracy of online content and distinguish between reliable sources and hoaxes [ 43 ]. Data literacy skills are also crucial for interpreting data accurately, detecting misleading statistics, and making informed decisions based on credible sources in the digital age [ 44 ].

Research on digital literacy emphasizes the importance of analytical and evaluative skills. Morgan et al. [ 17 ] found that higher education students struggle most with evaluating digital content for bias and quality. They excel in social literacy skills like communication. This highlights the need to prioritize adaptability in digital literacy, integrating industry-relevant experiences into education to ensure students can navigate and critically assess digital information for real-world applications.

Indeed, since the introduction of ChatGPT in 2022, it has been beneficial in various educational contexts. Nevertheless, concerns have been raised about potential inaccuracies and misinformation that may affect student learning and critical thinking [ 20 ]. Moreover, the potential redundancy of certain digital skills as a result of ChatGPT’s capabilities has also sparked discussions on changing educational objectives [ 19 , 21 , 22 ]. The development of ChatGPT may replace some digital skills as it takes over tasks previously expected of students. Researchers [ 21 , 22 ] argue that it is constantly improving its ability to access accurate medical information, providing reliable advice and treatment options from reputable sources. This ability may render the need for individuals to be adept at information retrieval and evaluation redundant. In other words, ChatGPT’s growing proficiency in tasks such as translation, text summarization, and sentiment analysis, and its ability to generate content like movies [ 23 ] may potentially lead to the underdevelopment of critical thinking skills, including the ability to evaluate source quality and reliability, formulate informed judgments, and generate creative and original ideas [ 24 ]. Indeed, the integration of AI into the healthcare sector raises critical questions about the nature and scope of the digital skills required in the future [ 19 , 20 ].

As AI advances, essential digital competencies may need reassessment to keep pace with technology. This requires forward-thinking digital literacy initiatives, particularly in healthcare education and practice. Proactively addressing the potential impact of AI on human interactions with digital healthcare technologies is critical. This will ensure that healthcare professionals and students are skilled in current digital practices, and prepared for the evolving role of AI in the sector. Despite the swift integration of AI tools in healthcare, and applications like ChatGPT, research on their long-term impacts, effects on users, and the necessary adaptation of skills and methodologies in the ever-evolving learning environment remains insufficient [ 11 , 12 , 15 , 17 , 18 , 49 , 50 , 51 , 52 , 53 , 54 , 55 , 56 , 57 , 58 ].

This study aims to address the intersection of AI adoption in healthcare and its implications for medical education, specifically focusing on the skills required by healthcare professionals. With the rapid incorporation of AI, into healthcare settings, there is an urgent need to reassess the digital literacy skills traditionally emphasized in medical education. This reassessment prompts questions about the ongoing relevance of these skills as AI technologies continue to evolve and expand their role in healthcare [ 13 , 15 , 16 , 17 , 18 , 19 , 20 ].

Research questions

Given the context, this study aims to explore the following qualitative research questions:

How does a pedagogical framework integrating problem-based learning with ChatGPT affect healthcare management undergraduates’ digital literacy skills?

What are students’ experiences with the combined use of problem-based learning and ChatGPT in their healthcare management education?

How do students perceive the shift towards AI-relevant skills as a result of engaging with this integrated pedagogical approach?

Methodology

Methodological approach.

The present research adopts the case study methodology, which entails in-depth empirical research of a current phenomenon within its real-world context [ 50 ]. This approach involves collecting data on human activities within a defined time and space setting, thereby facilitating an understanding of the various processes occurring within the research case. In qualitative research, and particularly in case study research, themes are formulated from the participants’ narratives, thus allowing for the development of arguments or generalizations derived deductively from participants’ statements [ 51 ]. By focusing on our research questions and using a methodological framework that emphasizes depth and context, the study aims to shed light on the transformative impact of AI on medical education and the development of the skills required for future healthcare professionals.

The research was conducted and analyzed by the researcher, who has a PhD in Healthcare Management and over 15 years of experience in qualitative analysis. Her expertise ensures a deep understanding of the study’s qualitative data. Throughout the research, she engaged in continuous reflexive practices to evaluate how her subjectivity and context influenced the study. This included reflecting on her assumptions, considering power dynamics with participants, aligning research paradigms and methods, and understanding the research context [ 59 ].

Participants and research population

The study involved 89 third-year undergraduate students enrolled in a Health System Management degree program, specifically participating in a course on Service Quality in the Healthcare System during the 2023 academic year. The researcher, serving as the lecturer for this course, integrated writing reflective journals into the curriculum as part of the learning process. Following the course’s conclusion and after grades were distributed, the researcher asked students, in adherence to ethical guidelines, if they consented to have their reflective journals analyzed for research purposes, as outlined in the data collection section. Only students who completed all components of the intervention plan outlined for the class were considered potential participants in the research population.

From this group, qualitative data was extracted from the reflective journals of 65 students who consented to participate. The demographic breakdown of this participant subset included 80% females, with an average age of 24.26 years (Standard Deviation = 3.80).

Data collection

Throughout the course, participants were required to keep a reflective journal documenting their learning journey, to be submitted at the end of the semester. The aim of writing the journal was to capture their personal perceptions of their learning experience. They were encouraged to articulate various challenges, obstacles, and positive and negative aspects they encountered [ 52 ]. Specifically, they were asked to describe the main challenges they faced and the obstacles they overcame, and to provide an introspective account of their experiences. The practice of writing a personal journal not only served as a tool for reflection but also helped them adopt a comprehensive perspective on their educational process [ 53 ].

The credibility of the reflective journal prompts was assured by grounding their development in an extensive literature review and expert consultations within the field of healthcare education. This process ensured that the prompts accurately reflected the constructs of interest, facilitating consistent and meaningful student reflections. Content validity was emphasized to ensure the journal prompts were aligned with the study’s objectives and relevant to students’ experiences in healthcare management education. Refinement of these prompts to effectively meet research objectives was facilitated through expert input. A detailed coding scheme was developed, featuring definitions and categories reflecting the study’s aims and insights from the journals. The coding was applied to a subset of journals by the researcher to ensure credibility.

The data were collected from the reflective journals in accordance with the intervention plan outlined in the Instructional Method section. The study carefully complied with several ethical guidelines for research with human subjects. The nature and purpose of the research were fully explained to the students, with particular emphasis on the use of reflective journals to evaluate the intervention plan. The students gave their informed consent and signed consent forms. To ensure confidentiality, participants were informed that all names would be replaced by pseudonyms and all identifying details would be removed from the final research report. They were also explicitly told that the journal entries would be processed anonymously. The research was approved by the college’s Ethics Committee.

Instructional method procedure (intervention plan)

The focus of this study is a required course titled Introducing Quality into the Health System, which had formerly been taught using traditional frontal teaching methods. The study examines the transformation of this course into a course taught using ChatGPT-mediated online guided learning. This innovative learning approach provides learners a comprehensive experience that entails self-directed learning. The approach emphasizes problem-based learning and focuses on identifying ethical dilemmas and analyzing them within organizational contexts. The intervention plan was strategically organized into five primary stages. Each stage comprised a series of carefully constructed steps that were specifically designed to build upon the knowledge and skills acquired in the previous stages, thus ensuring a coherent and cumulative educational progression. Figure  1 summarizes the instructional method.

Initial Familiarization with ChatGPT

At the beginning of the course, students were introduced to ChatGPT to develop their understanding and proficiency with the tool. This involved providing them detailed instructions on effective usage and encouraging them to engage in interactive dialogues with ChatGPT. The aim was to foster a sense of familiarity and ease, thereby facilitating an informal, hands-on learning experience.

Exploratory Analysis of a Dilemma using ChatGPT

In this exploratory stage, students began to examine the topic of hospital accreditation. Through interactions with ChatGPT, they were introduced to the pros and cons of the accreditation process and to the dilemmas posed by following the accreditation guidelines. The issue of accreditation is central to the discourse on how to improve healthcare quality, but it is also fraught with challenges, such as staff shortages and funding issues. Hospitals have had to make significant changes to meet accreditation standards, leading to debates about possible abolition of the accreditation system. While accreditation is crucial for quality control, its associated costs, particularly those related to inspections and the need for additional staff, pose significant challenges. Without proportional funding, compulsory accreditation has placed financial pressures on hospitals, creating a complex dynamic for both the Ministry of Health and healthcare institutions as they navigate the accreditation process.

To explore the topic of accreditation in depth, students were instructed to develop a series of questions to input to ChatGPT aimed at extracting detailed information about the accreditation dilemma. Students engaged with ChatGPT by posing questions and critically analyzing the answers from three perspectives: organizational, healthcare worker, and patient/customer. They iteratively refined their queries to increase precision until they achieved a comprehensive understanding. Following guidelines, they condensed and reorganized the information into a structured paragraph, incorporating the core dilemmas and arguments from each perspective. To meet objectives, students demonstrated digital media skills, including locating and sharing relevant materials, analyzing ChatGPT responses, verifying sources, and assessing content credibility.

Synthesis and Documentation of Concepts Emerging through ChatGPT Interaction

In the third stage, students were required to submit a comprehensive list detailing new concepts, themes, and sub-themes that emerged from their learning experience with ChatGPT. Their submitted list was not limited to the final results, but also included documentation of all stages of their work, including their initial set of questions, their subsequent refinement of these questions, and the process of their development throughout the learning journey. In addition, they were required to provide a final section summarizing the culmination of their exploration and learning process with ChatGPT. This comprehensive approach was designed to demonstrate the students’ engagement and progression with the tool and to highlight their ability to develop their inquiries and synthesize information effectively.

Analytical Structuring of Learning Outcomes

In the fourth stage, students attempted to refine the learning outcomes they had previously generated. Following the established guidelines, their main objective was to identify and highlight the pros and cons of the various arguments related to the dilemmas they had studied, making sure to consider them from different perspectives. The challenge was to present their arguments in a coherent and logical order, for example by comparing budgetary considerations with quality considerations. They were also expected to support each argument with scientific evidence, thereby aligning their analysis with academic accuracy and empirical research. This stage was crucial in developing their ability to critically evaluate and articulate complex issues, particularly in the field of healthcare.

Final project: Integrative Analysis and multidimensional presentation

In the final stage, students developed and presented a final project, building upon their prior work to explore a comprehensive research question or delve into a specific aspect of their study. This included presenting organizational and managerial viewpoints. The choice of format and tools for their project and presentation—ranging from e-posters and slides to video clips, using familiar technologies like PowerPoint and ThingLink—was left to the students. This method fostered diversity and empowered students by allowing them to select their preferred presentation technique. Moreover, the project featured a peer review phase where students critiqued each other’s work through insightful questions and suggestions, enhancing the discussion. This interactive element aimed to bolster critical thinking and collaborative learning.

Summary of instructional method

Reflective Journaling: documenting the Learning Journey

Throughout the semester, students kept a reflective journal, which they submitted at the end of the course. The primary aim of this journal was to document their personal learning experiences. The journal provided a window on their challenges, difficulties and successes they encountered, all viewed through the lens of their own perceptions and experiences.

Data analysis

The present research employed a deductive-inductive method for categorical analysis of the dataset. Integration of these deductive and inductive approaches was essential to facilitate investigation of predefined categories that are grounded in extant literature and theoretical frameworks, as well as to permit the discovery of novel categories that surfaced during the analysis process [ 51 ]. Initially, the deductive stage was conducted, focusing on predefined categories derived from existing literature and theoretical frameworks. Following this, the inductive stage allowed for the identification and development of novel categories based on the data analysis. The inclusion of episodes, thoughts, and feelings expressed by the students in this study serves to reinforce the reliability of the identified themes. The analysis of the reflective journals began with in-depth reading to identify initial themes from students’ narratives. Inductive coding facilitated the identification and development of themes by the researcher, rather than merely allowing them to ‘emerge.’ This active interpretation and organization of the data by the researcher led to a compilation of key insights. After ensuring the reliability and validity of these findings through careful review, the researcher then organized the codes into themes and sub-themes, ensuring they accurately reflected the data and provided a clear narrative of the students’ experiences.

The findings

The researcher’s analysis of the reflective journals actively uncovered five main categories: (1) Use of Literacy Skills; (2) User Experiences with ChatGPT; (3) ChatGPT Information Credibility; (4) Challenges and Barriers when Working with ChatGPT; (5) Mastering ChatGPT Prompting Competencies. Table  1 summarizes the identified categories and subcategories. To further clarify each category, the table includes representative quotations from the data for illustrative purposes. Throughout the manuscript, pseudonyms have been used with quotations. This approach ensures confidentiality and anonymity for all participants.

Use of literacy skills

The category comprising the use of literacy skills, the code refers to instances where participants relate literacy skills such as reading comprehension, searching evaluation of Information, etc., in their interactions with ChatGPT.

It includes three subcategories: Search Strategies and Access to Data in ChatGPT Use; Data Analysis Enhancement with ChatGPT ; and Evaluation of Information in ChatGPT Interactions Search Strategies and Access to Data in ChatGPT Use.

In the reflective journals, the students consistently expressed their high regard for the efficiency and ease of searching for and accessing information through ChatGPT. The chat interface significantly improved the process of retrieving information by removing the necessity to navigate through multiple websites or sources, thereby making the material more accessible. Furthermore, the interface’s user-friendly and accessible content format played a crucial role in significantly enhancing students’ understanding of the material. Shir wrote: The chat was super easy and helpful in making the dilemma clearer for me. It put all the info I needed in one spot, and everything was explained in a way that was simple to understand.

The analysis of the student journals underscored the remarkable proficiency of ChatGPT in rapidly and effortlessly providing information for various tasks. This technology alleviated the necessity for students to delve into multiple sources, offering a direct approach for understanding concepts, interpreting implications, and compiling data for complex issues. ChatGPT’s swift and handy information retrieval supported autonomous learning on the topic. As an accessible and user-friendly tool, it saved considerable time. Moreover, its accessibility and constant availability helped in tailoring learning experiences to fit the learner’s schedule, independent of external factors or intermediaries. ChatGPT’s use of simple, everyday language, coupled with its capacity to deconstruct and elucidate complex concepts, rendered it exceedingly approachable and beneficial for information searches and for enhancing the accessibility of educational content. Lihi also acknowledged the efficacy of ChatGPT in facilitating the rapid acquisition and expansion of her conceptual knowledge. She underscored that the ChatGPT tool obviated the need to consult multiple databases and websites for extracting conceptual information: ChatGPT is really fast and easy to use when you need info on lots of different things. It’s great for finding technical stuff, explaining problems, understanding things better, and getting new ideas on the spot. You don’t even have to go looking for more sources – it’s all right there.

Data synthesis and analysis enhancement with ChatGPT

Analysis of the reflective journals indicates that students found the synthesis, editing, and analysis of content facilitated by ChatGPT to be extremely beneficial. The tool significantly reduced the technical complexity of gathering and synthesizing information from different sources, tasks that had previously been their responsibility. As a result, they were spared the need for synthesizing, editing, and analyzing the raw data, with ChatGPT efficiently performing these functions on their behalf. Meir wrote: ChatGPT really helped us out. It gave us a full picture of the whole process, including the good and bad parts, and how to handle them. We didn’t even need to look at any other info sources at that point .

Evaluation of information in ChatGPT Interaction

The streamlined data collection procedures enabled the students to engage in more advanced learning processes, such as distinguishing between facts and assumptions, differentiating critical from non-critical information, and developing arguments as they advanced to more complex stages. The students observed that although ChatGPT presented data objectively, it did not offer explicit arguments, thus requiring them to actively interpret and formulate their own positions regarding the dilemma and identify the foundational principles for their principal arguments. For example, Miri’s reflections highlighted her need to formulate and develop a stance on the dilemma, which compelled her to engage in critical assessment of the situation:

ChatGPT didn’t really point out which arguments were more important or less important. It kind of listed them all the same way, which made me decide for myself what to focus on. I had to pick the arguments I thought were key and then find evidence to back them up.

Furthermore, the students were asked to support their arguments with evidence from the academic literature, necessitating a thorough evaluation and critical analysis of the information. This process led them to make informed decisions and formulate solutions. In their reflective journals, students documented a cautious approach, emphasizing the need not to simply accept information as it is presented. Instead, they highlighted the importance of thoroughly evaluating the information’s accuracy. Amir similarly addressed this issue, noting his necessity to independently navigate the “thinking part” and acquire the skills to construct strong arguments or effectively employ academic resources: The chat didn’t really help me figure out what’s important and what’s not when I write. It also didn’t teach me how to make strong arguments or how to use academic stuff to back up my points.

User experiences with ChatGPT

This category refers to the qualitative data related to participants’ overall experiences, perceptions, and attitudes towards interacting with ChatGPT. The theme of user experiences is divided into three sub-themes: Time Efficiency using ChatGPT; Accessibility and Availability of ChatGPT; and User-Friendly Dynamics . Overall, analysis of the students’ reflective journals reveals broad agreement about ChatGPT’s user-friendliness and ease of use. Many students noted the chatbot’s intuitive interface and straightforward functionality, which made it accessible to those who may not be tech-savvy. This consensus highlights the effectiveness of ChatGPT as a tool that simplifies information acquisition and supports learning without the typical complexities associated with advanced technological tools.

Time efficiency using ChatGPT

In this sub-category, analysis of the student journals revealed the major time-saving benefits of using ChatGPT for various tasks. ChatGPT successfully eliminated the need for students to sift through numerous sources of information. By providing a straightforward way to understand a concept, grasp its implications, and gather information on complex dilemmas, ChatGPT demonstrated its efficiency in saving students’ time. Riad mentioned the significant time efficiency gained from using the tool, highlighting how it saved him considerable time: You can find out a lot about all sorts of things really quickly. The chat gives you detailed breakdowns and explanations, sorting everything into different arguments and topics; it saves you a lot of time.

Ali also referred to this point: I was not very familiar with the details of accreditation, including its benefits and challenges, but within minutes I was able to grasp its essence and understand the importance of the whole process.

The time efficiency extended not only to data retrieval and collection but also encompassed information synthesis, significantly reducing the amount of time usually required for comprehensive and coherent processing and reformulating of acquired data. Mai observed that the time saved was also because she didn’t need to search for data across multiple sources and combine it together:

The amount of time I save is insane. If I had to search for this stuff on the internet instead of using the chat, it would take me way longer to find an answer. And even after finding it, I’d have to summarize what I found and then rephrase it in my own words, which takes so much time.

Accessibility and availability of ChatGPT

A majority of the students noted that the tool’s immediate accessibility and availability significantly facilitated the personalization of learning approaches. This customization seamlessly interfaced with the unique scheduling needs of each learner, offering flexibility that in traditional learning settings is typically constrained by external factors or intermediaries. Hana highlighted ChatGPT’s anytime, anywhere accessibility through a simple interface, enabling quick and comprehensive responses without the wait for expert assistance: ChatGPT is available to use anytime, anywhere using a simple and convenient interface. This would allow you to get a quick and comprehensive response at any time of the day, without having to wait around for people or experts to help you out.

Lina similarly noted: It’s pretty great how available it is (as long as it’s not too busy…). Any question I have, I get an answer. It saved me a lot of Google searches and reading articles and stuff. I get a quick and clear answer to everything I ask and it’s all super fast.

ChatGPT Information credibility

This category involves instances where participants discuss the credibility, reliability, and trustworthiness of the information provided by ChatGPT. Analysis of the reflective journals showed that interaction with ChatGPT facilitated students’ ability to acquire fundamental knowledge, which could then be expanded upon through subsequent inquiries and verification. Nevertheless, as students proceeded in their tasks, particularly those that required articulating arguments and substantiating their stances on complex dilemmas, they acknowledged the limitations of relying solely on ChatGPT. These limitations focused primarily on concerns about the tool’s credibility in providing sufficiently authoritative information. In this regard, Ofri appreciated ChatGPT’s quick access to information but expressed concerns over its credibility and occasional inaccuracies, leading to unexpected disappointment:

I have found that ChatGPT has a lot of good points. It can quickly give you a lot of information on so many topics and you can really use that information. But I have also learned that this tool has its drawbacks. It is not always right, and it certainly doesn’t always give you things that are based on solid academic facts. Sometimes ChatGPT just makes things up. To be honest, realizing this was a bit of a shock to me.

Students also noted that they were often faced with an overwhelming amount of information, some of which was irrelevant or incorrect, requiring them to evaluate the information and determine its quality. Dalia noted that while ChatGPT provided extensive information initially, aiding in learning about the topic, it also required discernment to distinguish between accurate and less relevant information: In the first stage, the chat gave us a lot of information, which was great because it helped us learn more about the topic. But at the same time, we had to decide which information was really important and accurate and which wasn’t.

Students’ understanding of the limitations of relying solely on the information provided to justify arguments and articulate positions in dilemmas motivated them to examine and assess its reliability. They did so by asking specific questions and consulting established academic references. From the students’ point of view, this careful research and critical evaluation process not only provided them with the opportunity to refine their powers of critical thinking and analysis, it also equipped them with the capacity to critically evaluate the credibility of the information presented. Lina wrote:

I attempted to back up the info I found with academic sources, but then I figured out that the chat isn’t always reliable…. I went through each article that I got results from…to check where is it from, and whether the author actually existed or was just made up… After that, I did another check with other databases. This whole process made me super cautious and thorough in checking everything.

The students expressed unanimous agreement that the need to assess the information provided by the chat forced them to be critical and use evaluation skills. Not only was this a skill they needed to be able to put to good use. It also constituted a challenge in using ChatGPT, as Limor stated that, contrary to reducing critical thinking, proper use of ChatGPT can enhance it by prompting users to reconsider and verify information, despite the challenge:

It might seem that using ChatGPT would make you think less because, well, it’s like chatting to a robot. But actually, if you use it properly and really get into it, it adds a lot to your knowledge and makes you think more broadly and deeper. This is because it makes you think about things over and over again, and double-check the information… it wasn’t easy.

Challenges and barriers in Working with ChatGPT

This category encompasses the various obstacles, difficulties, and limitations encountered by participants while using ChatGPT, including technical issues, comprehension challenges, and frustration. The analysis suggests that despite the students’ widespread agreement on the advantages of using ChatGPT, such as its ease of use, constant availability, and user-friendliness, its accompanying challenges should also be considered. Among these challenges are hesitation in adopting new, cutting-edge technology, difficulties in learning how to use the tool, and language barriers. The language issue was particularly significant, as ChatGPT operates mainly in English, which is not the first language of many of the students. Shir faced difficulties with English translation but viewed it as an opportunity to improve language skills, eventually becoming more comfortable with the chat and reducing reliance on outside translation help:

One big problem I had was writing in English and then translating it to express what I wanted to say. But I decided to take it on as a challenge and use it as a chance to improve my reading and writing in English. Since we didn’t have to use English much, at first it felt like it took forever to understand or read stuff. But gradually, we got the hang of the chat and didn’t need as much help with translating from outside sources.

Some students noted that they also faced some technical issues, revealing the downside of depending exclusively on online tools for studying. For many students, this was their first time using AI including applications like ChatGPT that are built on large language models. As they continued to use it, however, they became more accustomed to it. Ali found initially accessing the GPT chat difficult and, despite its ease of use, experienced issues with site access due to high traffic and occasional freezing, hindering continuous use:

When I first tried the GPT chat for my task, it was a bit tough to get onto the site. But after a while, I noticed that even though the chat is easy to use, it’s got its problems. Sometimes, you can’t even get into the chat because too many people are trying to use it at the same time, and other times, it just freezes up, and you can’t keep using it.

Mastering ChatGPT-Prompting competency

This category involves instances where participants demonstrate proficiency in formulating effective prompts and questions to elicit accurate and relevant responses from ChatGPT. Analysis of the reflective journals revealed that this theme posed a notable challenge for the students, primarily due to their unfamiliarity with the tool. Indeed, they needed to learn how to use the chat effectively to elicit the correct responses and achieve their desired outcomes. Additionally, they encountered challenges in ensuring accuracy and setting the right parameters to establish a reliable and precise database. Despite these obstacles, the students recognized that their efforts to achieve accuracy and their practice of asking repetitive questions were instrumental in developing higher-order thinking skills and being able to organize and manage the required information proficiently. Liya related to this challenge by noted that dealing with inaccurate responses from the model involves clarifying questions with more details, considering alternative answers, and emphasizing the importance of verifying the information received:

Sometimes the model may give you wrong information or answers… to cope with getting answers that are not accurate, you should make your question clearer and add more details. Also think about using different choices of answers. And it is really important to always check the answers you’re getting.

Analysis of the reflective journals showed that systematic demonstration of these activities, along with comprehensive detailing of early learning stages and the cumulative nature of the tasks, provided students the chance to assess and revisit each step retrospectively. This reflective review allowed them to seek explanations for any aspects that were unclear, ask more questions and craft more targeted prompts, and gain a deeper understanding of the entire process. Rim, for example, explained: The chat lets us get information in a series, like being able to ask another question to get a better understanding or clear up something from the first questions we asked. This helped us keep track of everything by linking all our questions together.

Nir noted that the need to aim for accuracy by repeatedly refining the questions really helped in dealing with the assigned tasks effectively:

From my experience with ChatGPT, I have learned that if you want good answers, you have to be really clear about what you are asking. You need to know what you want to achieve with the chat. It is best to give specific instructions to obtain the exact info you need. Also, you should think carefully about the answers you get, making sure the facts are right, and using your own thinking to make wise decisions.

This qualitative study examined the process of introducing and using a pedagogical framework that integrates problem-based learning with the use of ChatGPT among undergraduate healthcare management students. The study also provided a qualitative exploration of their experiences using this technology and assessed how the use of ChatGPT can shift the focus from traditional digital literacy skills to advanced AI literacy skills. It demonstrated how the use of the ChatGPT platform can be managed to encourage the development of critical thinking and evaluation skills through active student engagement. These skills are considered critical for learning and working with AI platforms.

The analysis of students’ reflective journals indicated a perception of the platform as user-friendly. Minichiello et al. [ 54 ] expand the definition of “user experience” beyond mere interaction with user interfaces to include design, information presentation, technological features, and factors related to emotion, personal connection, and experience. Students described their experience with the platform positively, citing it as an incentive for ongoing engagement.

The analysis also showed that the platform’s efficiency was significantly influenced by its high availability and accessibility, which were key factors in its attractiveness to users. This attractiveness was further enhanced by its ease of use. A critical aspect of the platform’s effectiveness was its efficiency in providing key materials in a timely manner, drastically reducing the time required to retrieve information. Users particularly appreciated this aspect of the platform as it streamlined their access to information and significantly improved their learning efficiency. The platform’s ability to deliver relevant information quickly and efficiently was instrumental in its positive reception. In an academic environment where efficient time management and quick access to educational materials are essential, the platform’s ability to meet these needs effectively constituted a notable advantage.

However, students noted initial difficulties and obstacles in utilizing ChatGPT, primarily related to data credibility. These challenges, highlighted in the qualitative data, necessitated the application of critical thinking and conducting various checks to verify the information received. This concern over the credibility of information from AI tools aligns with observations by Mohamad-Hani et al. [ 55 ], who reported similar credibility issues with ChatGPT data among healthcare professionals.

Another significant challenge for the students focused on how to retrieve relevant and accurate information. To this end, they had to refine their question formulation to extract the most relevant and accurate data from the tool. Such challenges have increasingly become a focus of academic attention due to the emerging recognition of the importance of developing prompting skills for effective interaction with platforms such as ChatGPT and other AI tools [ 19 , 20 ].

In terms of digital literacy skills, the findings of this study suggest that basic literacy skills such as locating, retrieving, synthesizing, and summarizing information may become less important as AI systems improve. Yet students still must be trained to evaluate and think critically about AI tools and what they can accomplish, especially since AI technologies like ChatGPT are not always completely trustworthy. Therefore, students need to learn how to evaluate the information these tools provide. These findings also offer some support for the notion that while digital literacy is undeniably recognized as crucial for the 21st century, especially in the healthcare arena [ 36 , 45 ], the definition of digital literacy is changing as technological tools develop. For decades, education focused on developing basic skills. Over time, however, there was a shift toward the cultivation of more complex skills involving information evaluation, synthesis, and assessment [ 56 , 57 ]. Yet as AI continues to penetrate everyday life, there has been a noticeable evolution in the forms of literacy required.

This evolution marks a transition from traditional data digital literacy, which emphasizes a basic understanding and processing of information, to AI digital literacy, which goes beyond mere data consumption to include using digital tools skillfully, understanding the nature of digital content, and effectively navigating the complex digital landscape. This shift reflects the changing demands of a technology-driven society, in which digital literacy is becoming increasingly essential for both personal and professional development [ 58 ]. As AI becomes integrated into different dimensions of work and daily life, especially in the healthcare industry, AI digital literacy will continue to evolve to meet the new demands. This will require a different set of skills, including prompting skills that allow users to better interact with AI tools [ 19 , 20 ].

These results highlight the importance of rethinking the educational use of AI tools such as ChatGPT, potentially leading to changes in future learning curricula. Without the ability to use digital tools, students are liable to fall behind when it comes to adapting to new technologies, thus limiting their ability to learn key skills. Therefore, AI tools must be taught and used in a way that supports students’ holistic learning. These findings align with those of other researchers who focus on the use of the AI platform in education [ 40 , 42 , 43 ]. Such an approach will ensure that students are prepared for the evolving challenges and opportunities of our increasingly digital world. This is especially important in the medical education field, as AI is increasingly being used in different ways to improve the accuracy of disease diagnosis, treatment strategies, and prediction of patient outcomes [ 9 , 10 , 25 , 27 ].

Given that AI technology is still developing and is anticipated to advance and become more widely used [ 21 , 22 ], the need to adapt and acquire new literacy skills is growing. As AI evolves, reliance on traditional basic skills may decline over time, underscoring the importance of learning how to effectively utilize and interact with emerging technologies. Learning to engage with AI tools such as ChatGPT from an early stage in their education can greatly enhance students’ learning experiences. This early exposure will not only provide them with a deeper understanding of these tools. It will also boost their motivation to learn how to use them more effectively, thus highlighting the importance of training students to handle such technologies proficiently. Equally important is the need to guide students through these learning processes to ensure they acquire the necessary skills and knowledge to navigate and utilize AI tools successfully in their educational journey [ 11 ].

Limitations and future research directions

This study utilized a pedagogical framework that integrates problem-based learning with the use of ChatGPT. While the researcher focused on the pedagogical aspect, future research is warranted to compare this digitally supported activity to a non-digital equivalent and examine the impact on students’ literacy and skills. Such a comparison would make it possible to assess what the digital instrument contributes to skill development and to identify any challenges encountered.

The use of this tool across different teaching methods could also be explored to determine whether it is particularly effective for certain types of tasks or requirements. The current study focused on health management. Implementation of this teaching approach in other academic areas should be examined to assess its effectiveness in acquiring competencies in different arenas. The findings of this study highlight the need for further research into the use of AI in learning environments that focus on goal-oriented pedagogy. Such research can help in developing educational strategies that promote the skills essential for lifelong learning.

Conclusions and recommendations

In conclusion, revisiting the research questions in the context of our findings highlights the transformative potential of integrating ChatGPT with problem-based learning in healthcare management education. This study underscores how such integration not only shifts the focus from traditional digital literacy to advanced AI literacy skills but also enhances critical thinking and evaluation capabilities among students. These competencies are indispensable as AI continues to reshape the landscape of healthcare and medical education. AI is emerging as a transformative force that will fundamentally change the global landscape. Although we are still in the early stages of integrating and understanding AI capabilities, its potential to shape our future is clear. Adapting to this digital transformation, especially in healthcare, is crucial [ 4 , 6 ].

Integrating AI into healthcare systems poses significant challenges and raises many unanswered questions [ 9 , 10 ]. These issues require careful consideration and strategic planning to maximize benefits while addressing implementation complexities. The extent and impact of these transformations on the health system and its workforce remain uncertain. However, it is crucial to prepare for these changes at both individual and organizational levels. Educational institutions must update their teaching methods to meet digital demands, recognizing the critical role of educators in developing effective support strategies.

To enable healthcare professionals to integrate AI tools effectively, these tools should be introduced early in education, such as during undergraduate studies or initial professional training [ 9 , 32 , 33 ]. Hands-on experience allows learners to build confidence and understand the tools’ limitations. Additionally, AI tools and especially LLMs such as GPT and their applications, including platforms like ChatGPT, can serve as user-friendly and efficient learning aids, as demonstrated in this research. In addition, researchers should strive to develop innovative pedagogical methods for integrating these tools into different curricula, as exemplified here by the effective use of dilemma-based learning enhanced by ChatGPT. These studies should focus on determining which skills will become redundant and on highlighting essential competencies needed for AI literacy, including prompting, evaluation skills, and critical thinking, all of which are essential for effectively integrating AI and LLMs into medical education and daily practice. Participants in such studies have noted that the acquisition of such skills, particularly in the area of effective prompting, significantly improves the quality of AI responses. Similar to learning a new language, learning to use AI requires precise phrasing and an in-depth understanding of context. Not only will AI skills improve student engagement and comprehension, they will also encourage critical thinking, leading to better educational outcomes. Students who formulate well-structured search queries obtain more accurate responses from AI, which are critical to improving healthcare and learning outcomes.

It is therefore imperative that academia and higher education institutions, including medical education institutions, adopt methods for effectively guiding and training students in using AI. This approach is essential to address the evolving global educational landscape and to embrace the shift in roles. Educators should move from being primarily providers of knowledge to being facilitators of cultural understanding and skill development. Such a shift is essential to promote the transformative evolution of the role of educators in the modern educational context.

Availability of data and materials

Data are available upon request from the Corresponding author.

Counte MA, Howard SW, Chang L, Aaronson W. Global advances in value-based payment and their implications for global health management education, development, and practice. Front Public Health. 2019;6:379. https://doi.org/10.3389/fpubh.2018.00379 .

Article   Google Scholar  

WHO. World Health Organisation. Universal health coverage. 2019. Retrieved from https://www.who.int/news-room/fact-sheets/detail/universal-health-coverage-(uhc) . June 2022.

Naamati Schneider N. Public-private: unequal competition: Israeli public hospitals vs. the private healthcare system following government reforms. Int J Organ Anal. 2021. https://doi.org/10.1108/IJOA-06-2020-2237 .

Deshpande N, Arora VM, Vollbrecht H, Meltzer DO, Press V. eHealth literacy and patient portal use and attitudes: cross-sectional observational study. JMIR Hum Factors. 2023;10:e40105. https://doi.org/10.2196/40105

Naamati Schneider L. Strategic management as an adaptation to changes in the ecosystems of public hospitals in Israel. Isr J Health Policy Res. 2020;9:65. https://doi.org/10.1186/s13584-020-00424-y .

Naamati-Schneider L. The effect of digitalization on service orientation and service perception among Israeli healthcare professionals: a qualitative study. Digit Health. 2023;9. https://doi.org/10.1177/20552076231191892 .

Naamati-Schneider L, Salvatore FP. Sustainability for healthcare organisations and systems: cultivating strategy and governance processes for a better future. In: Vrontis D, Thrassou A, Efthymiou L, Weber Y, Shams SMR, Tsoukatos E, editors. Business for sustainability. 2023; Volume I. pp.227–249. Palgrave studies in cross-disciplinary business research, in association with EuroMed Academy of Business. Cham: Palgrave Macmillan. https://doi.org/10.1007/978-3-031-37361-9_11 .

Naamati-Schneider L, Zaks O. Public hospitals in crisis: managerial and strategic adaptation. In: Vrontis D, Thrassou A, Weber Y, Shams SMR, Tsoukatos E, Efthymiou L, editors. Business under crisis: volume II. Palgrave Macmillan; 2022. p. 43–64. https://doi.org/10.1007/978-3-030-76575-0_3 .

Ali O, Abdelbaki W, Shrestha A, Elbasi E, Alryalat MAA, Dwivedi YK. A systematic literature review of artificial intelligence in the healthcare sector: benefits, challenges, methodologies, and functionalities. J Innov Knowl. 2023;8(1):100333.  https://doi.org/10.1016/j.jik.2023.100333

Manne R, Kantheti SC. Application of artificial intelligence in healthcare: chances and challenges. Curr J Appl Sci Technol. 2021;40(6):78–89. https://doi.org/10.9734/cjast/2021/v40i631320 .

Breeding T, Martinez B, Patel H, et al. The utilization of ChatGPT in reshaping future medical education and learning perspectives: a curse or a blessing? Am Surg. 2024;90(4):560–6. https://doi.org/10.1177/00031348231180950 .

Boscardin CK, Gin B, Golde PB, Hauer KE. ChatGPT and generative artificial intelligence for medical education: potential impact and opportunity. Acad Med. 2024;99(1):22–7. https://doi.org/10.1097/ACM.0000000000005439 .

Tinmaz H, Fanea-Ivanovici M, Baber H. A snapshot of digital literacy. Libr Hi Tech News. 2023;40(1):20–3. https://doi.org/10.1108/LHTN-12-2021-0095 .

Sallam M. ChatGPT utility in healthcare education, research, and practice: systematic review on the promising perspectives and valid concerns. Healthc. 2023;11(6):887. https://doi.org/10.3390/healthcare11060887 .

Buchholz BA, DeHart J, Moorman G. Digital citizenship during a global pandemic: moving beyond digital literacy. J Adolesc Adult Lit. 2020;64(1):11–7. https://doi.org/10.1002/jaal.1076 .

Kuatbekov A, Vershitskaya E, Kosareva I, Ananishnev V. E-learning as a basis for the development of media competences in students. J Inform Sci. 2023;49(4):1111–25.

Morgan A, Sibson R, Jackson D. Digital demand and digital deficit: conceptualising digital literacy and gauging proficiency among higher education students. J High Educ Policy Manag. 2022;44(3):258–75. https://doi.org/10.1080/1360080X.2022.2030275 .

van Laar E, van Deursen AJ, van Dijk JA. Developing policy aimed at 21st-century digital skills for the creative industries: an interview study with founders and managing directors. J Educ Work. 2022;35(2):195–209. https://doi.org/10.1080/13639080.2022.2036710 .

Eager B, Brunton R. Prompting higher education towards AI-augmented teaching and learning practice. J Univ Teach Learn Pract. 2023;20(5):02. https://doi.org/10.53761/1.20.5.02 .

Kasneci E, Seßler K, Küchemann S, Bannert M, Dementieva D, Fischer F, Kasneci G. ChatGPT for good? On opportunities and challenges of large language models for education. Learn Indiv Differ. 2023;103:102274. https://doi.org/10.1016/j.lindif.2023.102274 .

He N, Yan Y, Wu Z, Cheng Y, Liu F, Li X, Zhai S. ChatGPT-4 significantly surpasses GPT-3.5 in drug information queries. J Telemed Telecare. 2023;2023:1–3. https://doi.org/10.1177/1357633X231181922 .

Javaid M, Haleem A, Singh RP. ChatGPT for healthcare services: an emerging stage for an innovative perspective. BenchCouncil Trans Benchmarks Stand Eval. 2023;3(1):100105. https://doi.org/10.1016/j.tbench.2023.100105 .

Haleem A, Javaid M, Singh RP. An era of ChatGPT as a significant futuristic support tool: a study on features, abilities, and challenges. BenchCouncil Trans Benchmarks Stand Eval. 2022;2(4):100089.  https://doi.org/10.1016/j.tbench.2023.100089

Fuchs K. Exploring the opportunities and challenges of NLP models in higher education: is ChatGPT a blessing or a curse? Front Educ. 2023;8:8. https://doi.org/10.3389/feduc.2023.1166682 .

Jiang F, Jiang Y, Zhi H, Dong Y, Li H, Ma S, Wang Y. Artificial intelligence in healthcare: past, present and future. Stroke Vasc Neurol. 2017;2(4):e000101. https://doi.org/10.1136/svn-2017-000101 .

Tekkeşin Aİ. Artificial intelligence in healthcare. Anatol J Cardiol. 2019;22(1):8–9. https://doi.org/10.14744/AnatolJCardiol.2019.28 .

Jimma BL. Artificial intelligence in healthcare: a bibliometric analysis. Telemat Inform Rep. 2023:100041.‏ https://doi.org/10.1016/j.teler.2023.10004 .

Yu KH, Beam AL, Kohane IS. Artificial intelligence in healthcare. Nat Biomed Eng. 2018;2(10):719–31.

Rong G, Mendez A, Assi EB, Zhao B, Sawan M. Artificial intelligence in healthcare: review and prediction case studies. Engineering. 2020;6(3):291–301. https://doi.org/10.1016/j.eng.2019.08.015 .

Sqalli MT, Al-Thani D. AI-supported health coaching model for patients with chronic diseases. In: 2019 16th International Symposium on Wireless Communication Systems (ISWCS). IEEE; 2019. p. 452–456. https://doi.org/10.1109/ISWCS.2019.8877113 .

Matheny ME, Whicher D, Thadaney Israni S. Artificial intelligence in health care: a report from the national academy of medicine. JAMA. 2020;323(6):509–10. https://doi.org/10.1001/jama.2019.21579 .

York TJ, Raj S, Ashdown T, Jones G. Clinician and computer: a study on doctors’ perceptions of artificial intelligence in skeletal radiography. BMC Med Educ. 2023;23(1):1–10. https://doi.org/10.1186/s12909-022-03976-6 .

Eysenbach G. The role of ChatGPT, generative language models, and artificial intelligence in medical education: a conversation with ChatGPT and a call for papers. JMIR Med Educ. 2023;9:e46885. https://doi.org/10.2196/46885

Han Z, Battaglia F, Udaiyar A, Fooks A, Terlecky SR. An explorative assessment of ChatGPT as an aid in medical education: use it with caution. medRxiv. 2023.‏ https://doi.org/10.1101/2023.02.13.23285879 .

Das D, Kumar N, Longjam LA, Sinha R, Roy AD, Mondal H, Gupta P. Assessing the capability of ChatGPT in answering first-and second-order knowledge questions on microbiology as per competency-based medical education curriculum. Cureus. 2023;15(3):e36034. https://doi.org/10.7759/cureus.36034 .

Thurzo A, Strunga M, Urban R, Surovková J, Afrashtehfar KI. Impact of artificial intelligence on dental education: a review and guide for curriculum update. Educ Sci. 2023;13:150. https://doi.org/10.3390/educsci13020150 .

Veikkolainen P, Tuovinen T, Jarva E, Tuomikoski AM, Männistö M, Pääkkönen J, et al. eHealth competence building for future doctors and nurses–attitudes and capabilities. Int J Med Inform. 2023;169:104912. https://doi.org/10.1016/j.ijmedinf.2022.104912 .

Baro EE, Obaro OG, Aduba ED. An assessment of digital literacy skills and knowledge-based competencies among librarians working in university libraries in Africa. Digit Libr Perspect. 2019;35(3/4):172–92. https://doi.org/10.1108/DLP-04-2019-0013 .

Falloon G. From digital literacy to digital competence: the teacher digital competency (TDC) framework. Educ Tech Res Dev. 2020;68:2449–72. https://doi.org/10.1007/s11423-020-09767-4 .

Polizzi G. Digital literacy and the national curriculum for England: learning from how the experts engage with and evaluate online content. Comput Educ. 2020;152:103859.  https://doi.org/10.1016/j.compedu.2020.103859

Tynes BM, Stewart A, Hamilton M, Willis HA. From google searches to russian disinformation: adolescent critical race digital literacy needs and skills. Int J Multicult Educ. 2021;23(1):110–30. https://doi.org/10.18251/ijme.v23i1.2463 .

Ahmed ST, Roche T. Making the connection: examining the relationship between undergraduate students’ digital literacy and academic success in an English medium instruction (EMI) university. Educ Info Technol. 2021;26(4):4601–20. https://doi.org/10.1007/s10639-021-10443-0 .

Musi E, Aloumpi M, Carmi E, Yates S, O'Halloran K. Developing fake news immunity: fallacies as misinformation triggers during the pandemic. Online J Commun Media Technol. 2022;12(3). https://doi.org/10.30935/ojcmt%2F12083 .

Pangrazio L, Sefton-Green J. The social utility of ‘data literacy.’ Learn Media Technol. 2020;45(2):208–20. https://doi.org/10.1080/17439884.2020.1707223 .

Mehoudar O. Health literacy, and eHealth literacy: access to health information as a key to equal opportunity in society. Kidum Briut Isr. 2014;5:25–34 (Hebrew).

Google Scholar  

Alt D, Naamati-Schneider L, Meirovich A. Future problem-solving practiced during COVID-19: implications for health management students’ e-health literacy identity. Front Psychol. 2022;13:829243. https://doi.org/10.3389/fpsyg.2022.829243 .

Kaper MS, Reijneveld SA, van Es FD, de Zeeuw J, Almansa J, Koot J, de Winter AF. Effectiveness of a comprehensive health literacy consultation skills training for undergraduate medical students: a randomized controlled trial. Int J Environ Res Public Health. 2019;17(1):81. https://doi.org/10.3390/ijerph17010081 .

Nguyen HT, Do BN, Pham KM, Kim GB, Dam H, Nguyen TT, Nguyen T, Nguyen YH, Sørensen K, Pleasant A, Duong TV. Fear of COVID-19 scale-associations of its scores with health literacy and health-related behaviors among medical students. Int J Environ Res Public Health. 2020;17(11):4164. https://doi.org/10.3390/ijerph17114164 .

TerKonda SP, Fish EM. Artificial intelligence viewed through the lens of state regulation. Intell Based Med. 2023:100088. https://doi.org/10.1016/j.ibmed.2023.100088 .

Yin RK. Case study research design and methods. 5th ed. California: Sage; 2014.

Creswell JW. Educational research: planning, conducting, and evaluating quantitative and qualitative research. 4th ed. Nebraska-Lincoln: Merrill; 2012.

Tripto J, Ben-Zvi Assaraf O, Snapir Z, Amit M. The ‘What is a system’ reflection interview as a knowledge integration activity for high school students’ understanding of complex systems in human biology. Int J Sci Educ. 2016;38(4):564–95. https://doi.org/10.1080/09500693.2016.1150620 .

Zohar A, Barzilai S. A review of research on metacognition in science education: current and future directions. Stud Sci Educ. 2013;49:121–69. https://doi.org/10.1080/03057267.2013.84726 .

Minichiello A, Hood JR, Harkness DS. Bringing user experience design to bear on STEM education: a narrative literature review. J STEM Educ Res. 2018;1:7–33. https://doi.org/10.1007/s41979-018-0005-3 .

Mohamad-Hani T, Aljamaan F, Malki KH, Alhasan K, Altamimi I, Aljarbou R, Al-Eyadhy A. ChatGPT and the future of digital health: a study on healthcare workers’ perceptions and expectations. Healthcare. 2023;11(13):1812. https://doi.org/10.3390/healthcare11131812 .

Alt D, Raichel N. Enhancing perceived digital literacy skills and creative self-concept through gamified learning environments: insights from a longitudinal study. Int J Educ Res. 2020;101:101561. https://doi.org/10.1016/j.ijer.2020.101561 .

UNESCO. Digital skills critical for job and social inclusion. 2018. Retrieved from https://en.unesco.org/news/digital-skills-critical-jobs-and-social-inclusion .

Kreinsen M, Schulz S. Towards the triad of digital literacy, data literacy, and AI literacy in teacher education: a discussion in light of the accessibility of novel generative AI. A position paper. 2023. https://doi.org/10.1177/0163443710393382 .

Olmos-Vega FM, Stalmeijer RE, Varpio L, Kahlke R. A practical guide to reflexivity in qualitative research: AMEE Guide No. 149. Med Teach. 2023;45(3):241–51.

Download references

Acknowledgements

Conflict of interest.

The author confirms that there are no known conflicts of interest associated with this publication, and there has been no financial support for this work that could have influenced its outcome.

Author information

Authors and affiliations.

Health Systems Management Department, Hadassah Academic College, Jerusalem, Israel

Lior Naamati-Schneider

You can also search for this author in PubMed   Google Scholar

Contributions

L.N.S.: Conceptualization, data curation, methodology, writing- original draft preparation writing- reviewing and editing. The author read and approved the final manuscript.

Corresponding author

Correspondence to Lior Naamati-Schneider .

Ethics declarations

Ethics approval and consent to participate.

Informed consent was obtained from all participants involved in the study. The research was approved by the Hadassah Academic College’s Ethics Committee.

Consent for publication

Competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Naamati-Schneider, L. Enhancing AI competence in health management: students’ experiences with ChatGPT as a learning Tool. BMC Med Educ 24 , 598 (2024). https://doi.org/10.1186/s12909-024-05595-9

Download citation

Received : 23 February 2024

Accepted : 23 May 2024

Published : 30 May 2024

DOI : https://doi.org/10.1186/s12909-024-05595-9

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• AI digital literacy
• ChatGPT in education
• Healthcare management education

BMC Medical Education

ISSN: 1472-6920

• Submission enquiries: [email protected]
• General enquiries: [email protected]

Search the United Nations

• What Is Climate Change
• Myth Busters
• Renewable Energy
• Finance & Justice
• Initiatives
• Sustainable Development Goals
• Paris Agreement
• Climate Ambition Summit 2023
• Climate Conferences
• Press Material
• Communications Tips

What Is Climate Change?

Climate change refers to long-term shifts in temperatures and weather patterns. Such shifts can be natural, due to changes in the sun’s activity or large volcanic eruptions. But since the 1800s, human activities have been the main driver of climate change , primarily due to the burning of fossil fuels like coal, oil and gas.

Burning fossil fuels generates greenhouse gas emissions that act like a blanket wrapped around the Earth, trapping the sun’s heat and raising temperatures.

The main greenhouse gases that are causing climate change include carbon dioxide and methane. These come from using gasoline for driving a car or coal for heating a building, for example. Clearing land and cutting down forests can also release carbon dioxide. Agriculture, oil and gas operations are major sources of methane emissions. Energy, industry, transport, buildings, agriculture and land use are among the main sectors  causing greenhouse gases.

Humans are responsible for global warming

Climate scientists have showed that humans are responsible for virtually all global heating over the last 200 years. Human activities like the ones mentioned above are causing greenhouse gases that are warming the world faster than at any time in at least the last two thousand years.

The average temperature of the Earth’s surface is now about 1.2°C warmer than it was in the late 1800s (before the industrial revolution) and warmer than at any time in the last 100,000 years. The last decade (2011-2020) was the warmest on record , and each of the last four decades has been warmer than any previous decade since 1850.

Many people think climate change mainly means warmer temperatures. But temperature rise is only the beginning of the story. Because the Earth is a system, where everything is connected, changes in one area can influence changes in all others.

The consequences of climate change now include, among others, intense droughts, water scarcity, severe fires, rising sea levels, flooding, melting polar ice, catastrophic storms and declining biodiversity.

People are experiencing climate change in diverse ways

Climate change can affect our health , ability to grow food, housing, safety and work. Some of us are already more vulnerable to climate impacts, such as people living in small island nations and other developing countries. Conditions like sea-level rise and saltwater intrusion have advanced to the point where whole communities have had to relocate, and protracted droughts are putting people at risk of famine. In the future, the number of people displaced by weather-related events is expected to rise.

Every increase in global warming matters

In a series of UN reports , thousands of scientists and government reviewers agreed that limiting global temperature rise to no more than 1.5°C would help us avoid the worst climate impacts and maintain a livable climate. Yet policies currently in place point to a 3°C temperature rise by the end of the century.

The emissions that cause climate change come from every part of the world and affect everyone, but some countries produce much more than others .The seven biggest emitters alone (China, the United States of America, India, the European Union, Indonesia, the Russian Federation, and Brazil) accounted for about half of all global greenhouse gas emissions in 2020.

Everyone must take climate action, but people and countries creating more of the problem have a greater responsibility to act first.

We face a huge challenge but already know many solutions

Many climate change solutions can deliver economic benefits while improving our lives and protecting the environment. We also have global frameworks and agreements to guide progress, such as the Sustainable Development Goals , the UN Framework Convention on Climate Change and the Paris Agreement . Three broad categories of action are: cutting emissions, adapting to climate impacts and financing required adjustments.

Switching energy systems from fossil fuels to renewables like solar or wind will reduce the emissions driving climate change. But we have to act now. While a growing number of countries is committing to net zero emissions by 2050, emissions must be cut in half by 2030 to keep warming below 1.5°C. Achieving this means huge declines in the use of coal, oil and gas: over two-thirds of today’s proven reserves of fossil fuels need to be kept in the ground by 2050 in order to prevent catastrophic levels of climate change.

Adapting to climate consequences protects people, homes, businesses, livelihoods, infrastructure and natural ecosystems. It covers current impacts and those likely in the future. Adaptation will be required everywhere, but must be prioritized now for the most vulnerable people with the fewest resources to cope with climate hazards. The rate of return can be high. Early warning systems for disasters, for instance, save lives and property, and can deliver benefits up to 10 times the initial cost.

We can pay the bill now, or pay dearly in the future

Climate action requires significant financial investments by governments and businesses. But climate inaction is vastly more expensive. One critical step is for industrialized countries to fulfil their commitment to provide $100 billion a year to developing countries so they can adapt and move towards greener economies.

To get familiar with some of the more technical terms used in connection with climate change, consult the Climate Dictionary .

Learn more about…

The facts on climate and energy

Climate change is a hot topic – with myths and falsehoods circulating widely. Find some essential facts here .

The science

See the latest climate reports from the United Nations as well as climate action facts .

Causes and Effects

Fossil fuels are by far the largest contributor to the greenhouse gas emissions that cause climate change, which poses many risks to all forms of life on Earth. Learn more .

From the Secretary-General

Read the UN Chief’s latest statements on climate action.

What is net zero? Why is it important? Our  net-zero page  explains why we need steep emissions cuts now and what efforts are underway.

Renewable energy – powering a safer future

What is renewable energy and why does it matter? Learn more about why the shift to renewables is our only hope for a brighter and safer world.

How will the world foot the bill? We explain the issues and the value of financing climate action.

What is climate adaptation? Why is it so important for every country? Find out how we can protect lives and livelihoods as the climate changes.

Climate Issues

Learn more about how climate change impacts are felt across different sectors and ecosystems.

Why women are key to climate action

Women and girls are on the frontlines of the climate crisis and uniquely situated to drive action. Find out why it’s time to invest in women.

Facts and figures

• What is climate change?
• Causes and effects
• Myth busters

Cutting emissions

• Explaining net zero
• High-level expert group on net zero
• Checklists for credibility of net-zero pledges
• Greenwashing
• What you can do

Clean energy

• Renewable energy – key to a safer future
• What is renewable energy
• Five ways to speed up the energy transition
• Why invest in renewable energy
• Clean energy stories
• A just transition

Adapting to climate change

• Climate adaptation
• Early warnings for all
• Youth voices

Financing climate action

• Finance and justice
• Loss and damage
• $100 billion commitment
• Why finance climate action
• Biodiversity
• Human Security

International cooperation

• What are Nationally Determined Contributions
• Acceleration Agenda
• Climate Ambition Summit
• Climate conferences (COPs)
• Youth Advisory Group
• Action initiatives
• Secretary-General’s speeches
• Press material
• Fact sheets
• Communications tips

• Share full article

Advertisement

Subscriber-only Newsletter

Are American Workers’ Wages Really Lagging Productivity?

By Peter Coy

Opinion Writer

It’s become conventional wisdom in some circles that sometime in the 1970s, the link between pay and productivity broke. Workers’ productivity kept rising, but their inflation-adjusted pay did not. “The divergence between the two trends suggests that there may be forces suppressing the pay of workers relative to their productivity,” President Biden’s Council of Economic Advisers wrote in 2022.

The reality is more complicated. There is a problem in the labor market — a big one — but it’s not about employers winning against workers. It’s more about some workers winning big while most don’t. In short, it’s a problem of inequality, not subjugation by management.

I’m building this newsletter on the following two charts, which appear to contradict each another. The first, from the center-left Economic Policy Institute, shows a huge gap opening between pay growth and productivity growth. The second, from the center-right American Enterprise Institute, shows no such gap.

Both charts are correct in the sense that they are accurately plotting real data. The main source of the discrepancy is that they define pay and productivity differently, in ways that create very different impressions of economic reality. That, anyway, is what I figured out after speaking with scholars from both organizations: Josh Bivens, the chief economist at the Economic Policy Institute, and Scott Winship, a senior fellow at the American Enterprise Institute, who has a new report , “Understanding Trends in Worker Pay Over the Past Fifty Years.”

We are having trouble retrieving the article content.

Please enable JavaScript in your browser settings.

Thank you for your patience while we verify access. If you are in Reader mode please exit and  log into  your Times account, or  subscribe  for all of The Times.

Thank you for your patience while we verify access.

Already a subscriber?  Log in .

Want all of The Times?  Subscribe .

Information

• Author Services

Initiatives

You are accessing a machine-readable page. In order to be human-readable, please install an RSS reader.

All articles published by MDPI are made immediately available worldwide under an open access license. No special permission is required to reuse all or part of the article published by MDPI, including figures and tables. For articles published under an open access Creative Common CC BY license, any part of the article may be reused without permission provided that the original article is clearly cited. For more information, please refer to https://www.mdpi.com/openaccess .

Feature papers represent the most advanced research with significant potential for high impact in the field. A Feature Paper should be a substantial original Article that involves several techniques or approaches, provides an outlook for future research directions and describes possible research applications.

Feature papers are submitted upon individual invitation or recommendation by the scientific editors and must receive positive feedback from the reviewers.

Editor’s Choice articles are based on recommendations by the scientific editors of MDPI journals from around the world. Editors select a small number of articles recently published in the journal that they believe will be particularly interesting to readers, or important in the respective research area. The aim is to provide a snapshot of some of the most exciting work published in the various research areas of the journal.

Original Submission Date Received: .

• Active Journals
• Find a Journal
• Proceedings Series
• For Authors
• For Reviewers
• For Editors
• For Librarians
• For Publishers
• For Societies
• For Conference Organizers
• Open Access Policy
• Institutional Open Access Program
• Special Issues Guidelines
• Editorial Process
• Research and Publication Ethics
• Article Processing Charges
• Testimonials
• Preprints.org
• SciProfiles
• Encyclopedia

Article Menu

• Subscribe SciFeed
• Recommended Articles
• Google Scholar
• on Google Scholar
• Table of Contents

Find support for a specific problem in the support section of our website.

Please let us know what you think of our products and services.

Visit our dedicated information section to learn more about MDPI.

JSmol Viewer

Can migrant workers returning home for entrepreneurship increase agricultural labor productivity: evidence from a quasi-natural experiment in china.

1. Introduction

2. policy context and theoretical analysis, 2.1. policy context, 2.2. theoretical analysis, 3. data and methods, 3.1. data source, 3.2. variable definition and descriptive statistical analysis, 3.3. model setup, 3.3.1. benchmark regression model, 3.3.2. mechanism test model, 4. result analysis, 4.1. baseline regression results, 4.2. mechanism test, 4.3. robustness tests, 4.3.1. parallel trend test, 4.3.2. placebo test, 4.3.3. other robustness tests, 4.4. heterogeneity analysis, 4.4.1. regional differences, 4.4.2. topographic differences, 4.4.3. agricultural resource differences, 5. discussions, 6. conclusions and policy recommendations, author contributions, institutional review board statement, informed consent statement, data availability statement, conflicts of interest.

• Li, Z.; Wen, T.; Wei, H.; Du, Z.; Li, C.; Jin, W. Speeding up the Modernization of Agriculture and Rural Areas: An ln-depth interpretation by Authoritative Experts of the Spirit of China’s No.1 Central Document. Chin. Rural. Econ. 2023 , 4 , 2–20. (In Chinese) [ Google Scholar ]
• Carter, C.A.; Chen, J.; Chu, B. Agricultural productivity growth in China: Farm level versus aggregate measurement. China Econ. Rev. 2003 , 14 , 53–71. [ Google Scholar ] [ CrossRef ]
• McCullough, E.B. Labor productivity and employment gaps in Sub-Saharan Africa. Food Policy 2017 , 67 , 133–152. [ Google Scholar ] [ CrossRef ]
• Blanco, C.; Raurich, X. Agricultural composition and labour productivity. J. Dev. Econ. 2022 , 158 , 102934. [ Google Scholar ] [ CrossRef ]
• McErlean, S.; Wu, Z. Regional agricultural labor productivity convergence in China. Food Policy 2003 , 28 , 237–252. [ Google Scholar ] [ CrossRef ]
• Liu, S.; Ma, S.; Yin, L.; Zhu, J. Land titling, human capital misallocation, and agricultural productivity in China. J. Dev. Econ. 2023 , 165 , 103165. [ Google Scholar ] [ CrossRef ]
• Wu, B.; Liu, L. Social capital for rural revitalization in China: A critical evaluation on the government’s new countryside programme in Chengdu. Land Use Policy 2020 , 91 , 104268. [ Google Scholar ] [ CrossRef ]
• Giannakis, E.; Bruggeman, A. Exploring the labour productivity of agricultural systems across European regions: A multilevel approach. Land Use Policy 2018 , 77 , 94–106. [ Google Scholar ] [ CrossRef ]
• Lachaud, M.A.; Bravo-Ureta, B.E. A Bayesian statistical analysis of return to agricultural R&D investment in Latin America: Implications for food security. Technol. Soc. 2022 , 70 , 102054. [ Google Scholar ] [ CrossRef ]
• Rahman, M.M.; Mamun, S.A.K. The effects of telephone infrastructure on farmers’ agricultural outputs in China. Inf. Econ. Policy 2017 , 41 , 88–95. [ Google Scholar ] [ CrossRef ]
• Shamdasani, Y.S. Rural road infrastructure & agricultural production: Evidence from India. J. Dev. Econ. 2021 , 152 , 102686. [ Google Scholar ] [ CrossRef ]
• Ma, W.; Liu, T.; Li, W.; Yang, H. The role of agricultural machinery in improving green grain productivity in China: Towards trans-regional operation and low-carbon practices. Heliyon 2023 , 9 , e20279. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Li, F.; Feng, S.; Lu, H.; Qu, F.; D’Haese, M. How do non-farm employment and agricultural mechanization impact on large-scale farming? A spatial panel data analysis from Jiangsu Province, China. Land Use Policy 2021 , 107 , 105517. [ Google Scholar ] [ CrossRef ]
• Qian, L.; Lu, H.; Gao, Q.; Lu, H. Household-owned farm machinery vs. outsourced machinery services: The impact of agricultural mechanization on the land leasing behavior of relatively large-scale farmers in China. Land Use Policy 2022 , 115 , 106008. [ Google Scholar ] [ CrossRef ]
• Ren, J.; Gao, T.; Shi, X.; Chen, X.; Mu, K. The impact and heterogeneity analysis of digital financial inclusion on non-farm employment of rural labor. Chin. J. Popul. Resour. Environ. 2023 , 21 , 103–110. [ Google Scholar ] [ CrossRef ]
• Zhang, J.; Mishra, A.K.; Zhu, P.; Li, X. Land rental market and agricultural labor productivity in rural China: A mediation analysis. World Dev. 2020 , 135 , 105089. [ Google Scholar ] [ CrossRef ]
• Li, X.; Liu, J.; Huo, X. Impacts of tenure security and market-oriented allocation of farmland on agricultural productivity: Evidence from China’s apple growers. Land Use Policy 2021 , 102 , 105233. [ Google Scholar ] [ CrossRef ]
• Tran, D.; Vu, H.T.; Goto, D. Agricultural land consolidation, labor allocation and land productivity: A case study of plot exchange policy in Vietnam. Econ. Anal. Policy 2021 , 73 , 455–473. [ Google Scholar ] [ CrossRef ]
• Zhu, Z. Agri-tourism integration, factor allocation and agricultural labor productivity. J. Manag. 2022 , 35 , 29–43. (In Chinese) [ Google Scholar ]
• Zhang, H.; Zhang, J.; Song, J. Analysis of the threshold effect of agricultural industrial agglomeration and industrial structure upgrading on sustainable agricultural development in China. J. Clean. Prod. 2022 , 341 , 130818. [ Google Scholar ] [ CrossRef ]
• Wang, Y.; Zhao, Z.; Xu, M.; Tan, Z.; Han, J.; Zhang, L.; Chen, S. Agriculture–Tourism Integration’s Impact on Agricultural Green Productivity in China. Agriculture 2023 , 13 , 1941. [ Google Scholar ] [ CrossRef ]
• Dong, X.; Liang, Y.; Zhang, J. Fertility responses to the relaxation of migration restrictions: Evidence from the Hukou reform in China. China Econ. Rev. 2023 , 81 , 102040. [ Google Scholar ] [ CrossRef ]
• Wang, T.; Liu, J.; Zhu, H.; Jiang, Y. The Impact of Risk Aversion and Migrant Work Experience on Farmers’ Entrepreneurship: Evidence from China. Agriculture 2024 , 14 , 209. [ Google Scholar ] [ CrossRef ]
• Wang, X.; Cui, C.; Zhao, Y. Jobbing and homing: Employment transfer and rural growth. Manag. World 2003 , 7 , 99–109. (In Chinese) [ Google Scholar ]
• Huang, Z.; Song, W.; Ye, C.; Hu, W. The county economic growth effect of government support for migrant workers to return home for entrepreneurship. Chin. Rural. Econ. 2022 , 1 , 24–43. (In Chinese) [ Google Scholar ]
• Zang, Y.; Hu, S.; Zhou, B.B.; Lv, L.; Sui, X. Entrepreneurship and the formation mechanism of Taobao Villages: Implications for sustainable development in rural areas. J. Rural. Stud. 2023 , 100 , 103030. [ Google Scholar ] [ CrossRef ]
• Tang, L.; Chen, X.; Zhao, Y. Can returning to the countryside to start a business raise the income of rural residents: An examination based on the pilot policy of returning to rural areas for entrepreneurship. J. Agrotech. Econ. 2023 , 1–18. (In Chinese) [ Google Scholar ] [ CrossRef ]
• Wang, Y.; Liu, L. How Can migrant workers returning home for entrepreneurship promote common prosperity of farmers in rural areas? Chin. Rural. Econ. 2023 , 9 , 44–62. (In Chinese) [ Google Scholar ]
• Deller, S.; Kures, M.; Conroy, T. Rural entrepreneurship and migration. J. Rural. Stud. 2019 , 66 , 30–42. [ Google Scholar ] [ CrossRef ]
• Ito, J. Inter-regional difference of agricultural productivity in China: Distinction between biochemical and machinery technology. China Econ. Rev. 2010 , 21 , 394–410. [ Google Scholar ] [ CrossRef ]
• Schultz, T.W. Transforming Traditional Agriculture ; Yale University Press: New Haven, CT, USA, 1964. [ Google Scholar ]
• Du, J.; Xie, J.; Liu, B. Agricultural industry agglomeration and agricultural labor productivity in China: An empirical study based on data from 275 cities. J. Financ. Econ. 2020 , 46 , 49–63. (In Chinese) [ Google Scholar ]
• Tan, C.; Yang, J.; Sun, J. How does rural e-commerce improve agricultural labor productivity under the background of digital economy: A quasi-natural experiment based on e-commerce entering rural comprehensive demonstration county. World Agric. 2022 , 8 , 77–89. (In Chinese) [ Google Scholar ]
• Hao, W.; Hu, X.; Wang, J.; Zhang, Z.; Shi, Z.; Zhou, H. The impact of farmland fragmentation in China on agricultural productivity. J. Clean. Prod. 2023 , 425 , 138962. [ Google Scholar ] [ CrossRef ]
• Li, J.; Lin, R. Returning entrepreneurship on rural revitalization: Based on the difference test of the pilot policy of returning entrepreneurship. Chin. J. Agric. Resour. Reg. Plan. 2024 , 45 , 150–162. (In Chinese) [ Google Scholar ]
• Li, B.; Zhao, J.; Jin, Q. How to promote the upgrading of county Industrial structure by returning home for entrepreneurship: A quasi-natural experiment based on the pilot policy. J. Huazhong Agric. Univ. Soc. Sci. Ed. 2023 , 3 , 34–43. (In Chinese) [ Google Scholar ]
• Chen, X.; Lu, J.; Wang, K.; Wang, H.; Jiang, Y. An empirical test of the impact of agri-tourism integration on agricultural labor productivity: A case study of national demonstration counties of leisure agriculture. Chin. J. Agric. Resour. Reg. Plan. 2023 , 44 , 155–163. (In Chinese) [ Google Scholar ]

Share and Cite

Shen, L.; Wang, F. Can Migrant Workers Returning Home for Entrepreneurship Increase Agricultural Labor Productivity: Evidence from a Quasi-Natural Experiment in China. Agriculture 2024 , 14 , 905. https://doi.org/10.3390/agriculture14060905

Shen L, Wang F. Can Migrant Workers Returning Home for Entrepreneurship Increase Agricultural Labor Productivity: Evidence from a Quasi-Natural Experiment in China. Agriculture . 2024; 14(6):905. https://doi.org/10.3390/agriculture14060905

Shen, Lulin, and Fang Wang. 2024. "Can Migrant Workers Returning Home for Entrepreneurship Increase Agricultural Labor Productivity: Evidence from a Quasi-Natural Experiment in China" Agriculture 14, no. 6: 905. https://doi.org/10.3390/agriculture14060905

Article Metrics

Article access statistics, further information, mdpi initiatives, follow mdpi.

Subscribe to receive issue release notifications and newsletters from MDPI journals