define experimental psychologists

Experimental Psychology Studies Humans and Animals

Understanding Experimental Psychology

Our personalities, and to some degree our life experiences, are defined by the way we behave. But what influences the way we behave in the first place? How does our behavior shape our experiences throughout our lives? 

Experimental psychologists are interested in exploring theoretical questions, often by creating a hypothesis and then setting out to prove or disprove it through experimentation. They study a wide range of behavioral topics among humans and animals, including sensation, perception, attention, memory, cognition and emotion.

Experimental Psychology Applied

Experimental psychologists use scientific methods to collect data and perform research. Often, their work builds, one study at a time, to a larger finding or conclusion. Some researchers have devoted their entire career to answering one complex research question. 

These psychologists work in a variety of settings, including universities, research centers, government agencies and private businesses. The focus of their research is as varied as the settings in which they work. Often, personal interest and educational background will influence the research questions they choose to explore. 

In a sense, all psychologists can be considered experimental psychologists since research is the foundation of the discipline, and many psychologists split their professional focus among research, patient care, teaching or program administration. Experimental psychologists, however, often devote their full attention to research — its design, execution, analysis and dissemination. 

Those focusing their careers specifically on experimental psychology contribute work across subfields . For example, they use scientific research to provide insights that improve teaching and learning, create safer workplaces and transportation systems, improve substance abuse treatment programs and promote healthy child development.

Related books

Deliberate Practice in Interpersonal Psychotherapy

APA Handbook of Health Psychology

Methodological Issues and Strategies, 5e

Essentials of Constructivist Critical Incident ...

How to Mix Methods

Psychology subfields

• History & Society
• Science & Tech
• Biographies
• Animals & Nature
• Geography & Travel
• Arts & Culture
• Games & Quizzes
• On This Day
• One Good Fact
• New Articles
• Lifestyles & Social Issues
• Philosophy & Religion
• Politics, Law & Government
• World History
• Health & Medicine
• Browse Biographies
• Birds, Reptiles & Other Vertebrates
• Bugs, Mollusks & Other Invertebrates
• Environment
• Fossils & Geologic Time
• Entertainment & Pop Culture
• Sports & Recreation
• Visual Arts
• Demystified
• Image Galleries
• Infographics
• Top Questions
• Britannica Kids
• Saving Earth
• Space Next 50
• Student Center

experimental psychology

Our editors will review what you’ve submitted and determine whether to revise the article.

• American Psychological Association - Understanding Experimental Psychology

experimental psychology , a method of studying psychological phenomena and processes. The experimental method in psychology attempts to account for the activities of animals (including humans) and the functional organization of mental processes by manipulating variables that may give rise to behaviour; it is primarily concerned with discovering laws that describe manipulable relationships. The term generally connotes all areas of psychology that use the experimental method.

These areas include the study of sensation and perception , learning and memory , motivation , and biological psychology . There are experimental branches in many other areas, however, including child psychology , clinical psychology , educational psychology , and social psychology . Usually the experimental psychologist deals with normal, intact organisms; in biological psychology, however, studies are often conducted with organisms modified by surgery, radiation, drug treatment, or long-standing deprivations of various kinds or with organisms that naturally present organic abnormalities or emotional disorders. See also psychophysics .

B.A. in Psychology

What Is Experimental Psychology?

The science of psychology spans several fields. There are dozens of disciplines in psychology, including abnormal psychology, cognitive psychology and social psychology.

One way to view these fields is to separate them into two types: applied vs. experimental psychology. These groups describe virtually any type of work in psychology.

The following sections explore what experimental psychology is and some examples of what it covers.

Experimental psychology seeks to explore and better understand behavior through empirical research methods. This work allows findings to be employed in real-world applications (applied psychology) across fields such as clinical psychology, educational psychology, forensic psychology, sports psychology, and social psychology. Experimental psychology is able to shed light on people’s personalities and life experiences by examining what the way people behave and how behavior is shaped throughout life, along with other theoretical questions. The field looks at a wide range of behavioral topics including sensation, perception, attention, memory, cognition, and emotion, according to the  American Psychological Association  (APA).

Research is the focus of experimental psychology. Using scientific methods to collect data and perform research, experimental psychology focuses on certain questions, and, one study at a time, reveals information that contributes to larger findings or a conclusion. Due to the breadth and depth of certain areas of study, researchers can spend their entire careers looking at a complex research question.

Experimental Psychology in Action

The APA  writes about  one experimental psychologist, Robert McCann, who is now retired after 19 years working at NASA. During his time at NASA, his work focused on the user experience — on land and in space — where he applied his expertise to cockpit system displays, navigation systems, and safety displays used by astronauts in NASA spacecraft. McCann’s knowledge of human information processing allowed him to help NASA design shuttle displays that can increase the safety of shuttle missions. He looked at human limitations of attention and display processing to gauge what people can reliably see and correctly interpret on an instrument panel. McCann played a key role in helping determining the features of cockpit displays without overloading the pilot or taxing their attention span.

“One of the purposes of the display was to alert the astronauts to the presence of a failure that interrupted power in a specific region,” McCann said, “The most obvious way to depict this interruption was to simply remove (or dim) the white line(s) connecting the affected components. Basic research on visual attention has shown that humans do not notice the removal of a display feature very easily when the display is highly cluttered. We are much better at noticing a feature or object that is suddenly added to a display.” McCann utilized his knowledge in experimental psychology to research and develop this very important development for NASA. 

Valve Corporation

Another experimental psychologist, Mike Ambinder, uses his expertise to help design video games. He is a senior experimental psychologist at Valve Corporation, a video game developer and developer of the software distribution platform Steam. Ambinder told  Orlando Weekly  that his career working on gaming hits such as Portal 2 and Left 4 Dead “epitomizes the intersection between scientific innovation and electronic entertainment.” His career started when he gave a presentation to Valve on applying psychology to game design; this occurred while he was finishing his PhD in experimental design. “I’m very lucky to have landed at a company where freedom and autonomy and analytical decision-making are prized,” he said. “I realized how fortunate I was to work for a company that would encourage someone with a background in psychology to see what they could contribute in a field where they had no prior experience.” 

Ambinder spends his time on data analysis, hardware research, play-testing methodologies, and on any aspect of games where knowledge of human behavior could be useful. Ambinder described Valve’s process for refining a product as straightforward. “We come up with a game design (our hypothesis), and we place it in front of people external to the company (our play-test or experiment). We gather their feedback, and then iterate and improve the design (refining the theory). It’s essentially the scientific method applied to game design, and the end result is the consequence of many hours of applying this process.” To gather play-test data, Ambinder is engaged in the newer field of biofeedback technology, which can quantify gamers’ enjoyment. His research looks at unobtrusive measurements of facial expressions that can achieve such goals. Ambinder is also examining eye-tracking as a next-generation input method.

Pursue Your Career Goals in Psychology

Develop a greater understanding of psychology concepts and applications with Concordia St. Paul’s  online bachelor’s in psychology . Enjoy small class sizes with a personal learning environment geared toward your success, and learn from knowledgeable faculty who have industry experience. 

• Bipolar Disorder
• Therapy Center
• When To See a Therapist
• Types of Therapy
• Best Online Therapy
• Best Couples Therapy
• Best Family Therapy
• Managing Stress
• Sleep and Dreaming
• Understanding Emotions
• Self-Improvement
• Healthy Relationships
• Student Resources
• Personality Types
• Sweepstakes
• Guided Meditations
• Verywell Mind Insights
• 2024 Verywell Mind 25
• Mental Health in the Classroom
• Editorial Process
• Meet Our Review Board
• Crisis Support

How the Experimental Method Works in Psychology

sturti/Getty Images

The Experimental Process

Types of experiments, potential pitfalls of the experimental method.

The experimental method is a type of research procedure that involves manipulating variables to determine if there is a cause-and-effect relationship. The results obtained through the experimental method are useful but do not prove with 100% certainty that a singular cause always creates a specific effect. Instead, they show the probability that a cause will or will not lead to a particular effect.

At a Glance

While there are many different research techniques available, the experimental method allows researchers to look at cause-and-effect relationships. Using the experimental method, researchers randomly assign participants to a control or experimental group and manipulate levels of an independent variable. If changes in the independent variable lead to changes in the dependent variable, it indicates there is likely a causal relationship between them.

What Is the Experimental Method in Psychology?

The experimental method involves manipulating one variable to determine if this causes changes in another variable. This method relies on controlled research methods and random assignment of study subjects to test a hypothesis.

For example, researchers may want to learn how different visual patterns may impact our perception. Or they might wonder whether certain actions can improve memory . Experiments are conducted on many behavioral topics, including:

The scientific method forms the basis of the experimental method. This is a process used to determine the relationship between two variables—in this case, to explain human behavior .

Positivism is also important in the experimental method. It refers to factual knowledge that is obtained through observation, which is considered to be trustworthy.

When using the experimental method, researchers first identify and define key variables. Then they formulate a hypothesis, manipulate the variables, and collect data on the results. Unrelated or irrelevant variables are carefully controlled to minimize the potential impact on the experiment outcome.

History of the Experimental Method

The idea of using experiments to better understand human psychology began toward the end of the nineteenth century. Wilhelm Wundt established the first formal laboratory in 1879.

Wundt is often called the father of experimental psychology. He believed that experiments could help explain how psychology works, and used this approach to study consciousness .

Wundt coined the term "physiological psychology." This is a hybrid of physiology and psychology, or how the body affects the brain.

Other early contributors to the development and evolution of experimental psychology as we know it today include:

• Gustav Fechner (1801-1887), who helped develop procedures for measuring sensations according to the size of the stimulus
• Hermann von Helmholtz (1821-1894), who analyzed philosophical assumptions through research in an attempt to arrive at scientific conclusions
• Franz Brentano (1838-1917), who called for a combination of first-person and third-person research methods when studying psychology
• Georg Elias Müller (1850-1934), who performed an early experiment on attitude which involved the sensory discrimination of weights and revealed how anticipation can affect this discrimination

Key Terms to Know

To understand how the experimental method works, it is important to know some key terms.

Dependent Variable

The dependent variable is the effect that the experimenter is measuring. If a researcher was investigating how sleep influences test scores, for example, the test scores would be the dependent variable.

Independent Variable

The independent variable is the variable that the experimenter manipulates. In the previous example, the amount of sleep an individual gets would be the independent variable.

A hypothesis is a tentative statement or a guess about the possible relationship between two or more variables. In looking at how sleep influences test scores, the researcher might hypothesize that people who get more sleep will perform better on a math test the following day. The purpose of the experiment, then, is to either support or reject this hypothesis.

Operational definitions are necessary when performing an experiment. When we say that something is an independent or dependent variable, we must have a very clear and specific definition of the meaning and scope of that variable.

Extraneous Variables

Extraneous variables are other variables that may also affect the outcome of an experiment. Types of extraneous variables include participant variables, situational variables, demand characteristics, and experimenter effects. In some cases, researchers can take steps to control for extraneous variables.

Demand Characteristics

Demand characteristics are subtle hints that indicate what an experimenter is hoping to find in a psychology experiment. This can sometimes cause participants to alter their behavior, which can affect the results of the experiment.

Intervening Variables

Intervening variables are factors that can affect the relationship between two other variables. 

Confounding Variables

Confounding variables are variables that can affect the dependent variable, but that experimenters cannot control for. Confounding variables can make it difficult to determine if the effect was due to changes in the independent variable or if the confounding variable may have played a role.

Psychologists, like other scientists, use the scientific method when conducting an experiment. The scientific method is a set of procedures and principles that guide how scientists develop research questions, collect data, and come to conclusions.

The five basic steps of the experimental process are:

• Identifying a problem to study
• Devising the research protocol
• Conducting the experiment
• Analyzing the data collected
• Sharing the findings (usually in writing or via presentation)

Most psychology students are expected to use the experimental method at some point in their academic careers. Learning how to conduct an experiment is important to understanding how psychologists prove and disprove theories in this field.

There are a few different types of experiments that researchers might use when studying psychology. Each has pros and cons depending on the participants being studied, the hypothesis, and the resources available to conduct the research.

Lab Experiments

Lab experiments are common in psychology because they allow experimenters more control over the variables. These experiments can also be easier for other researchers to replicate. The drawback of this research type is that what takes place in a lab is not always what takes place in the real world.

Field Experiments

Sometimes researchers opt to conduct their experiments in the field. For example, a social psychologist interested in researching prosocial behavior might have a person pretend to faint and observe how long it takes onlookers to respond.

This type of experiment can be a great way to see behavioral responses in realistic settings. But it is more difficult for researchers to control the many variables existing in these settings that could potentially influence the experiment's results.

Quasi-Experiments

While lab experiments are known as true experiments, researchers can also utilize a quasi-experiment. Quasi-experiments are often referred to as natural experiments because the researchers do not have true control over the independent variable.

A researcher looking at personality differences and birth order, for example, is not able to manipulate the independent variable in the situation (personality traits). Participants also cannot be randomly assigned because they naturally fall into pre-existing groups based on their birth order.

So why would a researcher use a quasi-experiment? This is a good choice in situations where scientists are interested in studying phenomena in natural, real-world settings. It's also beneficial if there are limits on research funds or time.

Field experiments can be either quasi-experiments or true experiments.

Examples of the Experimental Method in Use

The experimental method can provide insight into human thoughts and behaviors, Researchers use experiments to study many aspects of psychology.

A 2019 study investigated whether splitting attention between electronic devices and classroom lectures had an effect on college students' learning abilities. It found that dividing attention between these two mediums did not affect lecture comprehension. However, it did impact long-term retention of the lecture information, which affected students' exam performance.

An experiment used participants' eye movements and electroencephalogram (EEG) data to better understand cognitive processing differences between experts and novices. It found that experts had higher power in their theta brain waves than novices, suggesting that they also had a higher cognitive load.

A study looked at whether chatting online with a computer via a chatbot changed the positive effects of emotional disclosure often received when talking with an actual human. It found that the effects were the same in both cases.

One experimental study evaluated whether exercise timing impacts information recall. It found that engaging in exercise prior to performing a memory task helped improve participants' short-term memory abilities.

Sometimes researchers use the experimental method to get a bigger-picture view of psychological behaviors and impacts. For example, one 2018 study examined several lab experiments to learn more about the impact of various environmental factors on building occupant perceptions.

A 2020 study set out to determine the role that sensation-seeking plays in political violence. This research found that sensation-seeking individuals have a higher propensity for engaging in political violence. It also found that providing access to a more peaceful, yet still exciting political group helps reduce this effect.

While the experimental method can be a valuable tool for learning more about psychology and its impacts, it also comes with a few pitfalls.

Experiments may produce artificial results, which are difficult to apply to real-world situations. Similarly, researcher bias can impact the data collected. Results may not be able to be reproduced, meaning the results have low reliability .

Since humans are unpredictable and their behavior can be subjective, it can be hard to measure responses in an experiment. In addition, political pressure may alter the results. The subjects may not be a good representation of the population, or groups used may not be comparable.

And finally, since researchers are human too, results may be degraded due to human error.

What This Means For You

Every psychological research method has its pros and cons. The experimental method can help establish cause and effect, and it's also beneficial when research funds are limited or time is of the essence.

At the same time, it's essential to be aware of this method's pitfalls, such as how biases can affect the results or the potential for low reliability. Keeping these in mind can help you review and assess research studies more accurately, giving you a better idea of whether the results can be trusted or have limitations.

Colorado State University. Experimental and quasi-experimental research .

American Psychological Association. Experimental psychology studies human and animals .

Mayrhofer R, Kuhbandner C, Lindner C. The practice of experimental psychology: An inevitably postmodern endeavor . Front Psychol . 2021;11:612805. doi:10.3389/fpsyg.2020.612805

Mandler G. A History of Modern Experimental Psychology .

Stanford University. Wilhelm Maximilian Wundt . Stanford Encyclopedia of Philosophy.

Britannica. Gustav Fechner .

Britannica. Hermann von Helmholtz .

Meyer A, Hackert B, Weger U. Franz Brentano and the beginning of experimental psychology: implications for the study of psychological phenomena today . Psychol Res . 2018;82:245-254. doi:10.1007/s00426-016-0825-7

Britannica. Georg Elias Müller .

McCambridge J, de Bruin M, Witton J.  The effects of demand characteristics on research participant behaviours in non-laboratory settings: A systematic review .  PLoS ONE . 2012;7(6):e39116. doi:10.1371/journal.pone.0039116

Laboratory experiments . In: The Sage Encyclopedia of Communication Research Methods. Allen M, ed. SAGE Publications, Inc. doi:10.4135/9781483381411.n287

Schweizer M, Braun B, Milstone A. Research methods in healthcare epidemiology and antimicrobial stewardship — quasi-experimental designs . Infect Control Hosp Epidemiol . 2016;37(10):1135-1140. doi:10.1017/ice.2016.117

Glass A, Kang M. Dividing attention in the classroom reduces exam performance . Educ Psychol . 2019;39(3):395-408. doi:10.1080/01443410.2018.1489046

Keskin M, Ooms K, Dogru AO, De Maeyer P. Exploring the cognitive load of expert and novice map users using EEG and eye tracking . ISPRS Int J Geo-Inf . 2020;9(7):429. doi:10.3390.ijgi9070429

Ho A, Hancock J, Miner A. Psychological, relational, and emotional effects of self-disclosure after conversations with a chatbot . J Commun . 2018;68(4):712-733. doi:10.1093/joc/jqy026

Haynes IV J, Frith E, Sng E, Loprinzi P. Experimental effects of acute exercise on episodic memory function: Considerations for the timing of exercise . Psychol Rep . 2018;122(5):1744-1754. doi:10.1177/0033294118786688

Torresin S, Pernigotto G, Cappelletti F, Gasparella A. Combined effects of environmental factors on human perception and objective performance: A review of experimental laboratory works . Indoor Air . 2018;28(4):525-538. doi:10.1111/ina.12457

Schumpe BM, Belanger JJ, Moyano M, Nisa CF. The role of sensation seeking in political violence: An extension of the significance quest theory . J Personal Social Psychol . 2020;118(4):743-761. doi:10.1037/pspp0000223

By Kendra Cherry, MSEd Kendra Cherry, MS, is a psychosocial rehabilitation specialist, psychology educator, and author of the "Everything Psychology Book."

Experimental Psychologist: Role, Responsibilities & Education

Written by Sarah Walsh

Clinical PsyD — Rutgers University | Clinical Psychologist

Experimental psychologists are those who conduct rigorous research studies to explore and uncover the underlying mechanisms of human behavior. Their systematic investigations provide invaluable insights into various aspects of human cognition, perception, emotion, and motivation. Experimental psychologists employ scientific methods and carefully designed experiments to gather data, analyze results, and draw meaningful conclusions. Their contributions profoundly impacted diverse areas such as cognitive psychology, developmental psychology, social psychology, and neuroscience.

This article aims to provide a comprehensive understanding of the role, responsibilities, and education required to become an experimental psychologist in the United States.

Understanding Experimental Psychology

Experimental psychology is a branch of psychology that emphasizes the scientific study of behavior and mental processes. It seeks to uncover human behavior’s underlying mechanisms and causes through systematic observation, measurement, and experimentation. By employing rigorous research methodologies and statistical analyses, experimental psychologists aim to establish causal relationships and make evidence-based conclusions.

The roots of experimental psychology can be traced back to the late 19th century when psychologists such as Wilhelm Wundt and William James pioneered the use of laboratory experiments to study human behavior. They advocated for the scientific approach in psychology and emphasized the importance of systematic observation and measurement. 

Over the years, experimental psychology has evolved and diversified, incorporating advancements in technology, statistics, and interdisciplinary collaborations. Today, experimental psychologists utilize various research methods, including controlled experiments, surveys, observations, and neuroimaging techniques, to investigate various facets of human cognition and behavior.

Role of an Experimental Psychologist

1. conducting research studies.

Central to the role of experimental psychologists is the design and execution of research studies. They formulate research questions, develop hypotheses, and design experiments that allow them to collect relevant data. This may involve selecting appropriate participant samples, designing experimental conditions, and employing measurement tools to assess behavior, cognition, or physiological responses. Experimental psychologists carefully control variables and employ statistical analyses to derive meaningful insights from their data, contributing to the scientific knowledge base.

2. Designing and Implementing Experiments

Experimental psychologists are responsible for designing and implementing scientifically rigorous and ethically sound experiments. They carefully plan every aspect of the experiment, including selecting appropriate research designs, manipulating independent variables, and controlling confounding factors. They also consider ethical guidelines to ensure participants’ well-being and informed consent throughout the study. By employing systematic and controlled experimental designs, experimental psychologists can draw reliable and valid conclusions from their research.

3. Analyzing Data and Drawing Conclusions

Once the data is collected, experimental psychologists utilize various statistical methods and data analysis techniques to make sense of the information gathered. They employ statistical software to analyze the data and interpret the results objectively. This involves running statistical tests, examining effect sizes, and assessing the significance of findings. Experimental psychologists critically evaluate the data to determine the implications and draw meaningful conclusions based on the evidence. They consider the study’s limitations and discuss the implications of their findings within the context of existing research and theories.

4. Reporting Findings and Publishing Research

An essential responsibility of experimental psychologists is to communicate their findings to the scientific community and the broader public. They prepare research reports, academic papers, and presentations that effectively communicate their study design, methodology, results, and conclusions. Experimental psychologists often publish their research in scientific journals, which contributes to advancing knowledge in the field. By disseminating their findings, they foster collaboration and encourage further exploration and replication of their work. Additionally, experimental psychologists may present their research at conferences, seminars, and workshops, promoting dialogue and knowledge exchange among professionals in the field.

Responsibilities of an Experimental Psychologist

1. academic research and teaching.

Experimental psychologists often engage in academic research, conducting studies to contribute to the scientific understanding of human behavior. They may secure research grants, collaborate with colleagues, and publish their findings in scholarly journals. Additionally, many experimental psychologists are involved in teaching at universities and colleges, sharing their expertise with students and mentoring aspiring psychologists.

2. Collaborating with Other Professionals

Experimental psychologists frequently collaborate with professionals from various disciplines, including other psychologists, neuroscientists, statisticians, and social scientists. Such collaborations allow for a multidisciplinary research approach, facilitating a deeper understanding of complex psychological phenomena and their underlying mechanisms. By working in interdisciplinary teams, experimental psychologists can integrate diverse perspectives and methodologies into their research, leading to comprehensive and impactful findings.

3. Ethical Considerations

Experimental psychologists are committed to upholding ethical standards in their research practices. They must obtain informed consent from participants, protect their privacy and confidentiality, and ensure the well-being and safety of individuals involved in their studies. Experimental psychologists adhere to professional, ethical guidelines and institutional review board protocols to ensure the ethical conduct of their research. They are responsible for addressing any potential ethical concerns that may arise during the research process.

4. Professional Development and Continuing Education

As the field of psychology constantly evolves, experimental psychologists engage in ongoing professional development and continuing education. They stay updated on the latest research, methodologies, and advancements in their area of specialization. This may involve attending conferences, workshops, and seminars and actively participating in professional organizations and networks. Experimental psychologists also engage in professional supervision and seek opportunities for collaboration and mentorship to enhance their skills and expand their knowledge base.

How to Become an Experimental Psychologist?

To become an experimental psychologist in the United States, aspiring individuals typically begin their journey by obtaining a bachelor’s degree in psychology or a related field. Undergraduate programs provide a foundation in core psychological principles, research methods, statistics, and critical thinking skills. Students may be able to participate in research projects or gain practical experience through internships.

After completing an undergraduate degree, aspiring experimental psychologists pursue advanced education at the graduate level. This typically involves earning a Master’s in Experimental Psychology and then proceeding to a doctoral program in experimental psychology or a related discipline. Doctoral programs offer specialized coursework, research training, and opportunities for independent research under the guidance of experienced faculty mentors. Graduates may choose to specialize in areas such as cognitive psychology, social psychology, developmental psychology, or neuroscience.

While licensing requirements vary by state, many experimental psychologists pursue licensure to practice independently or in applied settings. Licensing typically involves meeting specific educational and experiential requirements, passing a licensing exam, and fulfilling ongoing continuing education obligations. Additionally, some experimental psychologists may pursue certification from professional organizations, which can demonstrate their expertise and commitment to ethical and professional standards.

Experimental psychologists benefit from joining professional associations that cater to their specific interests and areas of specialization. Organizations such as the American Psychological Association (APA) and the Society for Experimental Psychology and Cognitive Science (SEPCS) provide valuable resources, networking opportunities, and access to the latest research in the field. Membership in these associations can enhance professional development, offer mentorship opportunities, and facilitate collaboration with colleagues.

Subfields of Experimental Psychology

1. cognitive psychology.

Cognitive psychology focuses on the study of mental processes, including attention, perception, memory, language, and problem-solving. Experimental psychologists in this subfield investigate how individuals acquire, process, store, and retrieve information, contributing to our understanding of human cognition and its underlying mechanisms.

2. Developmental Psychology

Developmental psychology explores the changes in human behavior and cognitive processes across the lifespan. Experimental psychologists in this subfield study various aspects of development, including social, cognitive, emotional, and physiological changes, shedding light on the factors that influence human growth and maturation.

3. Social Psychology

Social psychology examines how social interactions and the social environment influence individuals’ thoughts, feelings, and behaviors. Experimental psychologists in this subfield investigate social cognition, group dynamics, attitudes, persuasion, and intergroup relations, contributing to our understanding of the complexities of human social behavior.

4. Psychobiology and Neuroscience

Psychobiology and neuroscience involve studying the relationship between the brain, behavior, and mental processes. Experimental psychologists in this subfield employ neuroimaging techniques, physiological measures, and other research methods to investigate the neural underpinnings of psychological phenomena, providing insights into the biological basis of human behavior.

5. Other Specializations

Experimental psychology encompasses various other specialized areas of study, such as sensation and perception, emotion and motivation, personality, and psychopharmacology. Experimental psychologists may choose to specialize in these subfields or pursue interdisciplinary research that spans multiple areas, contributing to the richness and diversity of the field.

Career Opportunities for Experimental Psychologists

• Academic Positions : Experimental psychologists often pursue careers in academia, where they can engage in research, teaching, and mentoring. They may secure faculty positions at universities or colleges, conduct research studies, publish scholarly articles, and educate the next generation of psychologists.
• Research Institutions and Laboratories : Experimental psychologists can find opportunities in research institutions and laboratories in academic and non-academic settings. These positions involve conducting research studies, collaborating with interdisciplinary teams, and contributing to scientific advancements in various domains of experimental psychology.
• Private Sector Opportunities : The private sector also offers employment opportunities for experimental psychologists. They may work in research and development departments of corporations, consulting firms, or technology companies, where they apply their research expertise to areas such as user experience, product design, market research, and human factors.
• Government Agencies and Non-Profit Organizations : Experimental psychologists may contribute their expertise to government agencies and non-profit organizations. They can work in research divisions of governmental bodies, such as the National Institutes of Health (NIH) or the Centers for Disease Control and Prevention (CDC). Non-profit organizations may employ experimental psychologists to research social issues, mental health, or program evaluation.

Key Takeaways

• Experimental psychologists play a crucial role in advancing our understanding of human behavior through rigorous research and experimentation.
• They design and implement experiments, analyze data, and draw meaningful conclusions that contribute to the scientific knowledge base in psychology.
• The responsibilities of experimental psychologists include conducting research studies, collaborating with other professionals, addressing ethical considerations, and engaging in ongoing professional development.
• Education and training pathways for aspiring experimental psychologists typically involve obtaining an undergraduate degree in psychology, pursuing graduate education, and potentially obtaining licensure or certification.
• Career opportunities for experimental psychologists exist in academia, research institutions, the private sector, government agencies, and non-profit organizations, providing diverse avenues for applying their expertise.

© 2023 Psychology Masters Programs

Privacy Policy | Terms of Condition

Experimental Psychology

Definition:

Experimental psychology is a subfield of psychology that focuses on scientific investigation and research methods to study human behavior and mental processes. It involves conducting controlled experiments to examine hypotheses and gather empirical data.

Subfields of Experimental Psychology:

Sensory processes:.

Sensory processes in experimental psychology involve understanding how humans perceive and process information through their senses, such as vision, hearing, taste, smell, and touch.

Learning and Memory:

This subfield explores how individuals acquire and retain knowledge and skills, including the study of different types of memory, learning strategies, and factors that influence memory processes.

Cognitive Psychology:

Cognitive psychology examines mental processes, including attention, perception, problem-solving, decision-making, language, and thinking. It investigates how individuals process information, solve problems, and make decisions.

Developmental Psychology:

Developmental psychology focuses on the study of human development across the lifespan, from infancy to old age. It investigates how individuals change physically, cognitively, and emotionally as they grow and mature.

Social Psychology:

Social psychology studies how individuals’ thoughts, feelings, and behaviors are influenced by social interactions and social environments. It examines topics such as conformity, persuasion, group dynamics, and intergroup relations.

Personality Psychology:

Personality psychology aims to understand individual differences in behavior, thoughts, and emotions. It investigates various personality traits, their development, and how they influence behavior and well-being.

Psychopathology:

This subfield focuses on the study of mental disorders, their causes, symptoms, and treatments. Psychopathology research is often conducted using experimental methods to examine the effectiveness of therapeutic interventions.

Psychopharmacology:

Psychopharmacology involves studying the effects of drugs on behavior, cognition, and emotions. It examines how different medications impact mental processes and aims to develop effective pharmacological treatments for psychological disorders.

Neuropsychology:

Neuropsychology investigates the relationship between brain function and behavior. It examines how brain damage, genetics, and neurological disorders affect cognitive abilities, emotions, and behavior.

Experimental Method In Psychology

Saul McLeod, PhD

Editor-in-Chief for Simply Psychology

BSc (Hons) Psychology, MRes, PhD, University of Manchester

Saul McLeod, PhD., is a qualified psychology teacher with over 18 years of experience in further and higher education. He has been published in peer-reviewed journals, including the Journal of Clinical Psychology.

Learn about our Editorial Process

Olivia Guy-Evans, MSc

Associate Editor for Simply Psychology

BSc (Hons) Psychology, MSc Psychology of Education

Olivia Guy-Evans is a writer and associate editor for Simply Psychology. She has previously worked in healthcare and educational sectors.

On This Page:

The experimental method involves the manipulation of variables to establish cause-and-effect relationships. The key features are controlled methods and the random allocation of participants into controlled and experimental groups .

What is an Experiment?

An experiment is an investigation in which a hypothesis is scientifically tested. An independent variable (the cause) is manipulated in an experiment, and the dependent variable (the effect) is measured; any extraneous variables are controlled.

An advantage is that experiments should be objective. The researcher’s views and opinions should not affect a study’s results. This is good as it makes the data more valid  and less biased.

There are three types of experiments you need to know:

1. Lab Experiment

A laboratory experiment in psychology is a research method in which the experimenter manipulates one or more independent variables and measures the effects on the dependent variable under controlled conditions.

A laboratory experiment is conducted under highly controlled conditions (not necessarily a laboratory) where accurate measurements are possible.

The researcher uses a standardized procedure to determine where the experiment will take place, at what time, with which participants, and in what circumstances.

Participants are randomly allocated to each independent variable group.

Examples are Milgram’s experiment on obedience and  Loftus and Palmer’s car crash study .

• Strength : It is easier to replicate (i.e., copy) a laboratory experiment. This is because a standardized procedure is used.
• Strength : They allow for precise control of extraneous and independent variables. This allows a cause-and-effect relationship to be established.
• Limitation : The artificiality of the setting may produce unnatural behavior that does not reflect real life, i.e., low ecological validity. This means it would not be possible to generalize the findings to a real-life setting.
• Limitation : Demand characteristics or experimenter effects may bias the results and become confounding variables .

2. Field Experiment

A field experiment is a research method in psychology that takes place in a natural, real-world setting. It is similar to a laboratory experiment in that the experimenter manipulates one or more independent variables and measures the effects on the dependent variable.

However, in a field experiment, the participants are unaware they are being studied, and the experimenter has less control over the extraneous variables .

Field experiments are often used to study social phenomena, such as altruism, obedience, and persuasion. They are also used to test the effectiveness of interventions in real-world settings, such as educational programs and public health campaigns.

An example is Holfing’s hospital study on obedience .

• Strength : behavior in a field experiment is more likely to reflect real life because of its natural setting, i.e., higher ecological validity than a lab experiment.
• Strength : Demand characteristics are less likely to affect the results, as participants may not know they are being studied. This occurs when the study is covert.
• Limitation : There is less control over extraneous variables that might bias the results. This makes it difficult for another researcher to replicate the study in exactly the same way.

3. Natural Experiment

A natural experiment in psychology is a research method in which the experimenter observes the effects of a naturally occurring event or situation on the dependent variable without manipulating any variables.

Natural experiments are conducted in the day (i.e., real life) environment of the participants, but here, the experimenter has no control over the independent variable as it occurs naturally in real life.

Natural experiments are often used to study psychological phenomena that would be difficult or unethical to study in a laboratory setting, such as the effects of natural disasters, policy changes, or social movements.

For example, Hodges and Tizard’s attachment research (1989) compared the long-term development of children who have been adopted, fostered, or returned to their mothers with a control group of children who had spent all their lives in their biological families.

Here is a fictional example of a natural experiment in psychology:

Researchers might compare academic achievement rates among students born before and after a major policy change that increased funding for education.

In this case, the independent variable is the timing of the policy change, and the dependent variable is academic achievement. The researchers would not be able to manipulate the independent variable, but they could observe its effects on the dependent variable.

• Strength : behavior in a natural experiment is more likely to reflect real life because of its natural setting, i.e., very high ecological validity.
• Strength : Demand characteristics are less likely to affect the results, as participants may not know they are being studied.
• Strength : It can be used in situations in which it would be ethically unacceptable to manipulate the independent variable, e.g., researching stress .
• Limitation : They may be more expensive and time-consuming than lab experiments.
• Limitation : There is no control over extraneous variables that might bias the results. This makes it difficult for another researcher to replicate the study in exactly the same way.

Key Terminology

Ecological validity.

The degree to which an investigation represents real-life experiences.

Experimenter effects

These are the ways that the experimenter can accidentally influence the participant through their appearance or behavior.

Demand characteristics

The clues in an experiment lead the participants to think they know what the researcher is looking for (e.g., the experimenter’s body language).

Independent variable (IV)

The variable the experimenter manipulates (i.e., changes) is assumed to have a direct effect on the dependent variable.

Dependent variable (DV)

Variable the experimenter measures. This is the outcome (i.e., the result) of a study.

Extraneous variables (EV)

All variables which are not independent variables but could affect the results (DV) of the experiment. EVs should be controlled where possible.

Confounding variables

Variable(s) that have affected the results (DV), apart from the IV. A confounding variable could be an extraneous variable that has not been controlled.

Random Allocation

Randomly allocating participants to independent variable conditions means that all participants should have an equal chance of participating in each condition.

The principle of random allocation is to avoid bias in how the experiment is carried out and limit the effects of participant variables.

Order effects

Changes in participants’ performance due to their repeating the same or similar test more than once. Examples of order effects include:

(i) practice effect: an improvement in performance on a task due to repetition, for example, because of familiarity with the task;

(ii) fatigue effect: a decrease in performance of a task due to repetition, for example, because of boredom or tiredness.

Experimental Psychology: 10 Examples & Definition

Dave Cornell (PhD)

Dr. Cornell has worked in education for more than 20 years. His work has involved designing teacher certification for Trinity College in London and in-service training for state governments in the United States. He has trained kindergarten teachers in 8 countries and helped businessmen and women open baby centers and kindergartens in 3 countries.

Learn about our Editorial Process

Chris Drew (PhD)

This article was peer-reviewed and edited by Chris Drew (PhD). The review process on Helpful Professor involves having a PhD level expert fact check, edit, and contribute to articles. Reviewers ensure all content reflects expert academic consensus and is backed up with reference to academic studies. Dr. Drew has published over 20 academic articles in scholarly journals. He is the former editor of the Journal of Learning Development in Higher Education and holds a PhD in Education from ACU.

Experimental psychology refers to studying psychological phenomena using scientific methods. Originally, the primary scientific method involved manipulating one variable and observing systematic changes in another variable.

Today, psychologists utilize several types of scientific methodologies.

Experimental psychology examines a wide range of psychological phenomena, including: memory, sensation and perception, cognitive processes, motivation, emotion, developmental processes, in addition to the neurophysiological concomitants of each of these subjects.

Studies are conducted on both animal and human participants, and must comply with stringent requirements and controls regarding the ethical treatment of both.

Definition of Experimental Psychology

Experimental psychology is a branch of psychology that utilizes scientific methods to investigate the mind and behavior.

It involves the systematic and controlled study of human and animal behavior through observation and experimentation .

Experimental psychologists design and conduct experiments to understand cognitive processes, perception, learning, memory, emotion, and many other aspects of psychology. They often manipulate variables ( independent variables ) to see how this affects behavior or mental processes (dependent variables).

The findings from experimental psychology research are often used to better understand human behavior and can be applied in a range of contexts, such as education, health, business, and more.

Experimental Psychology Examples

1. The Puzzle Box Studies (Thorndike, 1898) Placing different cats in a box that can only be escaped by pulling a cord, and then taking detailed notes on how long it took for them to escape allowed Edward Thorndike to derive the Law of Effect: actions followed by positive consequences are more likely to occur again, and actions followed by negative consequences are less likely to occur again (Thorndike, 1898).

2. Reinforcement Schedules (Skinner, 1956) By placing rats in a Skinner Box and changing when and how often the rats are rewarded for pressing a lever, it is possible to identify how each schedule results in different behavior patterns (Skinner, 1956). This led to a wide range of theoretical ideas around how rewards and consequences can shape the behaviors of both animals and humans.

3. Observational Learning (Bandura, 1980) Some children watch a video of an adult punching and kicking a Bobo doll. Other children watch a video in which the adult plays nicely with the doll. By carefully observing the children’s behavior later when in a room with a Bobo doll, researchers can determine if television violence affects children’s behavior (Bandura, 1980).

4. The Fallibility of Memory (Loftus & Palmer, 1974) A group of participants watch the same video of two cars having an accident. Two weeks later, some are asked to estimate the rate of speed the cars were going when they “smashed” into each other. Some participants are asked to estimate the rate of speed the cars were going when they “bumped” into each other. Changing the phrasing of the question changes the memory of the eyewitness.

5. Intrinsic Motivation in the Classroom (Dweck, 1990) To investigate the role of autonomy on intrinsic motivation, half of the students are told they are “free to choose” which tasks to complete. The other half of the students are told they “must choose” some of the tasks. Researchers then carefully observe how long the students engage in the tasks and later ask them some questions about if they enjoyed doing the tasks or not.

6. Systematic Desensitization (Wolpe, 1958) A clinical psychologist carefully documents his treatment of a patient’s social phobia with progressive relaxation. At first, the patient is trained to monitor, tense, and relax various muscle groups while viewing photos of parties. Weeks later, they approach a stranger to ask for directions, initiate a conversation on a crowded bus, and attend a small social gathering. The therapist’s notes are transcribed into a scientific report and published in a peer-reviewed journal.

7. Study of Remembering (Bartlett, 1932) Bartlett’s work is a seminal study in the field of memory, where he used the concept of “schema” to describe an organized pattern of thought or behavior. He conducted a series of experiments using folk tales to show that memory recall is influenced by cultural schemas and personal experiences.

8. Study of Obedience (Milgram, 1963) This famous study explored the conflict between obedience to authority and personal conscience. Milgram found that a majority of participants were willing to administer what they believed were harmful electric shocks to a stranger when instructed by an authority figure, highlighting the power of authority and situational factors in driving behavior.

9. Pavlov’s Dog Study (Pavlov, 1927) Ivan Pavlov, a Russian physiologist, conducted a series of experiments that became a cornerstone in the field of experimental psychology. Pavlov noticed that dogs would salivate when they saw food. He then began to ring a bell each time he presented the food to the dogs. After a while, the dogs began to salivate merely at the sound of the bell. This experiment demonstrated the principle of “classical conditioning.”

10, Piaget’s Stages of Development (Piaget, 1958) Jean Piaget proposed a theory of cognitive development in children that consists of four distinct stages: the sensorimotor stage (birth to 2 years), where children learn about the world through their senses and motor activities, through to the the formal operational stage (12 years and beyond), where abstract reasoning and hypothetical thinking develop. Piaget’s theory is an example of experimental psychology as it was developed through systematic observation and experimentation on children’s problem-solving behaviors .

Types of Research Methodologies in Experimental Psychology 

Researchers utilize several different types of research methodologies since the early days of Wundt (1832-1920).

1. The Experiment

The experiment involves the researcher manipulating the level of one variable, called the Independent Variable (IV), and then observing changes in another variable, called the Dependent Variable (DV).

The researcher is interested in determining if the IV causes changes in the DV. For example, does television violence make children more aggressive?

So, some children in the study, called research participants, will watch a show with TV violence, called the treatment group. Others will watch a show with no TV violence, called the control group.

So, there are two levels of the IV: violence and no violence. Next, children will be observed to see if they act more aggressively. This is the DV.

If TV violence makes children more aggressive, then the children that watched the violent show will me more aggressive than the children that watched the non-violent show.

A key requirement of the experiment is random assignment . Each research participant is assigned to one of the two groups in a way that makes it a completely random process. This means that each group will have a mix of children: different personality types, diverse family backgrounds, and range of intelligence levels.

2. The Longitudinal Study

A longitudinal study involves selecting a sample of participants and then following them for years, or decades, periodically collecting data on the variables of interest.

For example, a researcher might be interested in determining if parenting style affects academic performance of children. Parenting style is called the predictor variable , and academic performance is called the outcome variable .

Researchers will begin by randomly selecting a group of children to be in the study. Then, they will identify the type of parenting practices used when the children are 4 and 5 years old.

A few years later, perhaps when the children are 8 and 9, the researchers will collect data on their grades. This process can be repeated over the next 10 years, including through college.

If parenting style has an effect on academic performance, then the researchers will see a connection between the predictor variable and outcome variable.

Children raised with parenting style X will have higher grades than children raised with parenting style Y.

3. The Case Study

The case study is an in-depth study of one individual. This is a research methodology often used early in the examination of a psychological phenomenon or therapeutic treatment.

For example, in the early days of treating phobias, a clinical psychologist may try teaching one of their patients how to relax every time they see the object that creates so much fear and anxiety, such as a large spider.

The therapist would take very detailed notes on how the teaching process was implemented and the reactions of the patient. When the treatment had been completed, those notes would be written in a scientific form and submitted for publication in a scientific journal for other therapists to learn from.

There are several other types of methodologies available which vary different aspects of the three described above. The researcher will select a methodology that is most appropriate to the phenomenon they want to examine.

They also must take into account various practical considerations such as how much time and resources are needed to complete the study. Conducting research always costs money.

People and equipment are needed to carry-out every study, so researchers often try to obtain funding from their university or a government agency. 

Origins and Key Developments in Experimental Psychology

Wilhelm Maximilian Wundt (1832-1920) is considered one of the fathers of modern psychology. He was a physiologist and philosopher and helped establish psychology as a distinct discipline (Khaleefa, 1999).  

In 1879 he established the world’s first psychology research lab at the University of Leipzig. This is considered a key milestone for establishing psychology as a scientific discipline. In addition to being the first person to use the term “psychologist,” to describe himself, he also founded the discipline’s first scientific journal Philosphische Studien in 1883.

Another notable figure in the development of experimental psychology is Ernest Weber . Trained as a physician, Weber studied sensation and perception and created the first quantitative law in psychology.

The equation denotes how judgments of sensory differences are relative to previous levels of sensation, referred to as the just-noticeable difference (jnd). This is known today as Weber’s Law (Hergenhahn, 2009).    

Gustav Fechner , one of Weber’s students, published the first book on experimental psychology in 1860, titled Elemente der Psychophysik. His worked centered on the measurement of psychophysical facets of sensation and perception, with many of his methods still in use today.    

The first American textbook on experimental psychology was Elements of Physiological Psychology, published in 1887 by George Trumball Ladd .

Ladd also established a psychology lab at Yale University, while Stanley Hall and Charles Sanders continued Wundt’s work at a lab at Johns Hopkins University.

In the late 1800s, Charles Pierce’s contribution to experimental psychology is especially noteworthy because he invented the concept of random assignment (Stigler, 1992; Dehue, 1997).

Go Deeper: 15 Random Assignment Examples

This procedure ensures that each participant has an equal chance of being placed in any of the experimental groups (e.g., treatment or control group). This eliminates the influence of confounding factors related to inherent characteristics of the participants.

Random assignment is a fundamental criterion for a study to be considered a valid experiment.

From there, experimental psychology flourished in the 20th century as a science and transformed into an approach utilized in cognitive psychology, developmental psychology, and social psychology .

Today, the term experimental psychology refers to the study of a wide range of phenomena and involves methodologies not limited to the manipulation of variables.

The Scientific Process and Experimental Psychology

The one thing that makes psychology a science and distinguishes it from its roots in philosophy is the reliance upon the scientific process to answer questions. This makes psychology a science was the main goal of its earliest founders such as Wilhelm Wundt.

There are numerous steps in the scientific process, outlined in the graphic below.

1. Observation

First, the scientist observes an interesting phenomenon that sparks a question. For example, are the memories of eyewitnesses really reliable, or are they subject to bias or unintentional manipulation?

2. Hypothesize

Next, this question is converted into a testable hypothesis. For instance: the words used to question a witness can influence what they think they remember.

3. Devise a Study

Then the researcher(s) select a methodology that will allow them to test that hypothesis. In this case, the researchers choose the experiment, which will involve randomly assigning some participants to different conditions.

In one condition, participants are asked a question that implies a certain memory (treatment group), while other participants are asked a question which is phrased neutrally and does not imply a certain memory (control group).

The researchers then write a proposal that describes in detail the procedures they want to use, how participants will be selected, and the safeguards they will employ to ensure the rights of the participants.

That proposal is submitted to an Institutional Review Board (IRB). The IRB is comprised of a panel of researchers, community representatives, and other professionals that are responsible for reviewing all studies involving human participants.

4. Conduct the Study

If the IRB accepts the proposal, then the researchers may begin collecting data. After the data has been collected, it is analyzed using a software program such as SPSS.

Those analyses will either support or reject the hypothesis. That is, either the participants’ memories were affected by the wording of the question, or not.

5. Publish the study

Finally, the researchers write a paper detailing their procedures and results of the statistical analyses. That paper is then submitted to a scientific journal.

The lead editor of that journal will then send copies of the paper to 3-5 experts in that subject. Each of those experts will read the paper and basically try to find as many things wrong with it as possible. Because they are experts, they are very good at this task.

After reading those critiques, most likely, the editor will send the paper back to the researchers and require that they respond to the criticisms, collect more data, or reject the paper outright.

In some cases, the study was so well-done that the criticisms were minimal and the editor accepts the paper. It then gets published in the scientific journal several months later.

That entire process can easily take 2 years, usually more. But, the findings of that study went through a very rigorous process. This means that we can have substantial confidence that the conclusions of the study are valid.

Experimental psychology refers to utilizing a scientific process to investigate psychological phenomenon.

There are a variety of methods employed today. They are used to study a wide range of subjects, including memory, cognitive processes, emotions and the neurophysiological basis of each.

The history of psychology as a science began in the 1800s primarily in Germany. As interest grew, the field expanded to the United States where several influential research labs were established.

As more methodologies were developed, the field of psychology as a science evolved into a prolific scientific discipline that has provided invaluable insights into human behavior.

Bartlett, F. C., & Bartlett, F. C. (1995).  Remembering: A study in experimental and social psychology . Cambridge university press.

Dehue, T. (1997). Deception, efficiency, and random groups: Psychology and the gradual origination of the random group design. Isis , 88 (4), 653-673.

Ebbinghaus, H. (2013). Memory: A contribution to experimental psychology.  Annals of neurosciences ,  20 (4), 155.

Hergenhahn, B. R. (2009). An introduction to the history of psychology. Belmont. CA: Wadsworth Cengage Learning .

Khaleefa, O. (1999). Who is the founder of psychophysics and experimental psychology? American Journal of Islam and Society , 16 (2), 1-26.

Loftus, E. F., & Palmer, J. C. (1974).  Reconstruction of auto-mobile destruction : An example of the interaction between language and memory.  Journal of Verbal Learning and Verbal behavior , 13, 585-589.

Pavlov, I.P. (1927). Conditioned reflexes . Dover, New York.

Piaget, J. (1959).  The language and thought of the child  (Vol. 5). Psychology Press.

Piaget, J., Fraisse, P., & Reuchlin, M. (2014). Experimental psychology its scope and method: Volume I (Psychology Revivals): History and method . Psychology Press.

Skinner, B. F. (1956). A case history in scientlfic method. American Psychologist, 11 , 221-233

Stigler, S. M. (1992). A historical view of statistical concepts in psychology and educational research. American Journal of Education , 101 (1), 60-70.

Thorndike, E. L. (1898). Animal intelligence: An experimental study of the associative processes in animals. Psychological Review Monograph Supplement 2 .

Wolpe, J. (1958). Psychotherapy by reciprocal inhibition. Stanford, CA: Stanford University Press.

Appendix: Images reproduced as Text

Definition: Experimental psychology is a branch of psychology that focuses on conducting systematic and controlled experiments to study human behavior and cognition.

Overview: Experimental psychology aims to gather empirical evidence and explore cause-and-effect relationships between variables. Experimental psychologists utilize various research methods, including laboratory experiments, surveys, and observations, to investigate topics such as perception, memory, learning, motivation, and social behavior .

Example: The Pavlov’s Dog experimental psychology experiment used scientific methods to develop a theory about how learning and association occur in animals. The same concepts were subsequently used in the study of humans, wherein psychology-based ideas about learning were developed. Pavlov’s use of the empirical evidence was foundational to the study’s success.

Experimental Psychology Milestones:

1890: William James publishes “The Principles of Psychology”, a foundational text in the field of psychology.

1896: Lightner Witmer opens the first psychological clinic at the University of Pennsylvania, marking the beginning of clinical psychology.

1913: John B. Watson publishes “Psychology as the Behaviorist Views It”, marking the beginning of Behaviorism.

1920: Hermann Rorschach introduces the Rorschach inkblot test.

1938: B.F. Skinner introduces the concept of operant conditioning .

1967: Ulric Neisser publishes “Cognitive Psychology” , marking the beginning of the cognitive revolution.

1980: The third edition of the Diagnostic and Statistical Manual of Mental Disorders (DSM-III) is published, introducing a new classification system for mental disorders.

The Scientific Process

• Observe an interesting phenomenon
• Formulate testable hypothesis
• Select methodology and design study
• Submit research proposal to IRB
• Collect and analyzed data; write paper
• Submit paper for critical reviews

• Dave Cornell (PhD) https://helpfulprofessor.com/author/dave-cornell-phd/ 23 Achieved Status Examples
• Dave Cornell (PhD) https://helpfulprofessor.com/author/dave-cornell-phd/ 25 Defense Mechanisms Examples
• Dave Cornell (PhD) https://helpfulprofessor.com/author/dave-cornell-phd/ 15 Theory of Planned Behavior Examples
• Dave Cornell (PhD) https://helpfulprofessor.com/author/dave-cornell-phd/ 18 Adaptive Behavior Examples

• Chris Drew (PhD) https://helpfulprofessor.com/author/chris-drew-phd-2/ 23 Achieved Status Examples
• Chris Drew (PhD) https://helpfulprofessor.com/author/chris-drew-phd-2/ 15 Ableism Examples
• Chris Drew (PhD) https://helpfulprofessor.com/author/chris-drew-phd-2/ 25 Defense Mechanisms Examples
• Chris Drew (PhD) https://helpfulprofessor.com/author/chris-drew-phd-2/ 15 Theory of Planned Behavior Examples

Leave a Comment Cancel Reply

Your email address will not be published. Required fields are marked *

Want to create or adapt books like this? Learn more about how Pressbooks supports open publishing practices.

The Science of Psychology

5 Experimental and Clinical Psychologists

Learning objectives.

• Define the clinical practice of psychology and distinguish it from experimental psychology.
• Explain how science is relevant to clinical practice.
• Define the concept of an empirically supported treatment and give some examples.

Who Conducts Scientific Research in Psychology?

Experimental psychologists.

Scientific research in psychology is generally conducted by people with doctoral degrees (usually the  doctor of philosophy [Ph.D.] ) and master’s degrees in psychology and related fields, often supported by research assistants with bachelor’s degrees or other relevant training. Some of them work for government agencies (e.g., doing research on the impact of public policies), national associations (e.g., the American Psychological Association), non-profit organizations (e.g., National Alliance on Mental Illness), or in the private sector (e.g., in product marketing and development; organizational behavior). However, the majority of them are college and university faculty, who often collaborate with their graduate and undergraduate students. Although some researchers are trained and licensed as clinicians for mental health work—especially those who conduct research in clinical psychology—the majority are not. Instead, they have expertise in one or more of the many other subfields of psychology: behavioral neuroscience, cognitive psychology, developmental psychology, personality psychology, social psychology, and so on. Doctoral-level researchers might be employed to conduct research full-time or, like many college and university faculty members, to conduct research in addition to teaching classes and serving their institution and community in other ways.

Of course, people also conduct research in psychology because they enjoy the intellectual and technical challenges involved and the satisfaction of contributing to scientific knowledge of human behavior. You might find that you enjoy the process too. If so, your college or university might offer opportunities to get involved in ongoing research as either a research assistant or a participant. Of course, you might find that you do not enjoy the process of conducting scientific research in psychology. But at least you will have a better understanding of where scientific knowledge in psychology comes from, an appreciation of its strengths and limitations, and an awareness of how it can be applied to solve practical problems in psychology and everyday life.

Scientific Psychology Blogs

A fun and easy way to follow current scientific research in psychology is to read any of the many excellent blogs devoted to summarizing and commenting on new findings. Among them are the following:

Research Digest, http://digest.bps.org.uk/ Talk Psych, http://www.talkpsych.com/ Brain Blogger, http://brainblogger.com/ Mind Hacks, http://mindhacks.com/ PsyBlog, http://www.spring.org.uk

You can also browse to http://www.researchblogging.org , select psychology as your topic, and read entries from a wide variety of blogs.

Clinical Psychologists

Psychology is the scientific study of behavior and mental processes. But it is also the application of scientific research to “help people, organizations, and communities function better” (American Psychological Association, 2011) [1] . By far the most common and widely known application is the clinical practice of psychology — the diagnosis and treatment of psychological disorders and related problems. Let us use the term  clinical practice  broadly to refer to the activities of clinical and counseling psychologists, school psychologists, marriage and family therapists, licensed clinical social workers, and others who work with people individually or in small groups to identify and help address their psychological problems. It is important to consider the relationship between scientific research and clinical practice because many students are especially interested in clinical practice, perhaps even as a career.

The main point is that psychological disorders and other behavioral problems are part of the natural world. This means that questions about their nature, causes, and consequences are empirically testable and therefore subject to scientific study. As with other questions about human behavior, we cannot rely on our intuition or common sense for detailed and accurate answers. Consider, for example, that dozens of popular books and thousands of websites claim that adult children of alcoholics have a distinct personality profile, including low self-esteem, feelings of powerlessness, and difficulties with intimacy. Although this sounds plausible, scientific research has demonstrated that adult children of alcoholics are no more likely to have these problems than anybody else (Lilienfeld et al., 2010) [2] . Similarly, questions about whether a particular psychotherapy is effective are empirically testable questions that can be answered by scientific research. If a new psychotherapy is an effective treatment for depression, then systematic observation should reveal that depressed people who receive this psychotherapy improve more than a similar group of depressed people who do not receive this psychotherapy (or who receive some alternative treatment). Treatments that have been shown to work in this way are called empirically supported treatments .

Empirically Supported Treatments

An empirically supported treatment is one that has been studied scientifically and shown to result in greater improvement than no treatment, a placebo, or some alternative treatment. These include many forms of psychotherapy, which can be as effective as standard drug therapies. Among the forms of psychotherapy with strong empirical support are the following:

• Acceptance and committment therapy (ACT) . for depression, mixed anxiety disorders, psychosis, chronic pain, and obsessive-compulsive disorder.
• Behavioral couples therapy. For alcohol use disorders.
• Cognitive behavioral therapy (CBT). For many disorders including eating disorders, depression, anxiety disorders, etc.
• Exposure therapy. For post-traumatic stress disorder and phobias.
• Exposure therapy with response prevention.  For obsessive-compulsive disorder.
• Family-based treatment. For eating disorders.

For a more complete list, see the following website, which is maintained by Division 12 of the American Psychological Association, the Society for Clinical Psychology: http://www.div12.org/psychological-treatments

Many in the clinical psychology community have argued that their field has not paid enough attention to scientific research—for example, by failing to use empirically supported treatments—and have suggested a variety of changes in the way clinicians are trained and treatments are evaluated and put into practice. Others believe that these claims are exaggerated and the suggested changes are unnecessary (Norcross, Beutler, & Levant, 2005) [3] . On both sides of the debate, however, there is agreement that a scientific approach to clinical psychology is essential if the goal is to diagnose and treat psychological problems based on detailed and accurate knowledge about those problems and the most effective treatments for them. So not only is it important for scientific research in clinical psychology to continue, but it is also important for clinicians who never conduct a scientific study themselves to be scientifically literate so that they can read and evaluate new research and make treatment decisions based on the best available evidence.

• American Psychological Association. (2011). About APA . Retrieved from http://www.apa.org/about ↵
• Lilienfeld, S. O., Lynn, S. J., Ruscio, J., & Beyerstein, B. L. (2010). 50 great myths of popular psychology . Malden, MA: Wiley-Blackwell. ↵
• Norcross, J. C., Beutler, L. E., & Levant, R. F. (Eds.). (2005). Evidence-based practices in mental health: Debate and dialogue on the fundamental questions . Washington, DC: American Psychological Association. ↵

An academic degree earned through intensive study of a particular discipline and the completion of a set of research studies that contribute new knowledge to the academic literature.

The diagnosis and treatment of psychological disorders and related problems.

A treatment that that has been shown through systematic observation to lead to better outcomes when compared to no-treatment or placebo control groups.

Research Methods in Psychology Copyright © 2019 by Rajiv S. Jhangiani, I-Chant A. Chiang, Carrie Cuttler, & Dana C. Leighton is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License , except where otherwise noted.

Share This Book

• Psychology Today

What is Experimental Psychology?

Experimental psychology is an interesting subdiscipline of psychology.

On the one hand, it refers to an approach to studying human behavior – the standardized methods and techniques used to collect and analyze data.  On the other hand, experimental psychology is a unique branch, an applied field of psychology that explores theoretical questions about human behavior.

Featured Programs

So, while virtually all psychologists can engage in experimental psychology in one form or another, there are also professionals who spend their entire careers conducting applied experimental research in the field of psychology. This is what deems professionals in this field experimental psychologists.

In this guide, we’ll explore how experimental psychology developed and review some of the methods that are used in conducting studies of behavior. We’ll also discuss types of experiments, survey a few types of psychological experiments, and go over career-related information for experimental psychologists as well.

Let’s get started!

What is The History of Experimental Psychology?

Cognitive psychology is one of the most fascinating fields today. Questions about the nature of human behavior and the connection of the body and mind go back to classic philosophers like Plato and Aristotle. Likewise, the debate over nature vs nurture raged long before experimental psychologists came along in their formal and modern form.

But the centuries of philosophizing about why people behave in the manners in which they behave sparked the interest of scientific researchers to study human behavior in an empirical manner. If the principles of the scientific methods could be used for the study of behavior, perhaps scientists could provide definitive answers to the age-old questions surrounding human behavior.

This is precisely why the experimental psychologist was born.

The first experimental psychology lab in the world was founded in Leipzig, Germany, in 1879 by Wilhelm Wundt. There, Wundt primarily studied feelings and sensations in a structured manner using objective, systematized measures and controls. This mathematical and experimental approach set the precedent for the scientific methods that experimental psychologists and research centers use today. 

For example, Wundt used his background in physiology to design an experiment on sensory processes in which each participant was exposed to the same stimulus, such as the sound of a metronome. Each participant was then asked to provide a report of the sensations they experienced, a process called introspection.

The goal of this experiment was to understand the underlying structure of sensory processes. That is, Wundt was interested in analyzing each of the elements of the human experience involved in sensing the stimulus – the thoughts, senses, feelings, and so forth.

Wundt believed that breaking down the process of a behavior – in this case, sensing a sound – could be done much like a chemist analyzes a chemical compound. If you examine the individual components, you can learn more about the structure underneath the individual components.

While Wundt’s process of introspection didn’t remain a long-lasting approach to studying psychological processes, his insistence on controlling the experimental environment has had a long-term impact on how psychological research is conducted.

The experiments he devised used the same conditions for the experimental subjects – the same stimuli, the same setting, the same lighting, and so forth. By controlling the environment in which research is taking place, Wundt was able to minimize potential confounding variables. Doing so is critically important for any research.

Because Wundt’s laboratory was the first in the world, he is often considered the father of modern psychology. That is, his contributions to the field shifted psychology from a philosophical pursuit to a scientific one.

In the United States, experimental psychology grew out of the efforts of G. Stanley Hall and George Trumbull Ladd .

• Stanley Hall is credited with developing the first experimental psychology lab in the United States, which was located at Johns Hopkins University. Though his career mostly focused on child development and evolutionary psychology, his most significant contribution to psychology was overseeing the early development of the field in the United States. He was the first American to get a doctorate in psychology and he oversaw 11 of the first 14 doctorates to be awarded in the U.S., including those to John Dewey, Lewis Terman, and James McKeen Cattell, each of whom went on to become influential figures in the field in their own rite.

Meanwhile, George Trumbull Ladd, who was a professor of psychology at Yale University, established one of the earliest experimental labs for psychology in the U.S. He is also credited with publishing the first experimental psychology textbook, entitled Elements of Physiological Psychology.

Another important figure in the history of experimental psychology was Wundt’s American contemporary, William James.

His textbook, The Principles of Psychology , is perhaps the most seminal work in the history of psychology. Published in 1890, the book offers insights into the experiments James performed over the course of his career teaching at Harvard. However, the book is not a manual on experimental psychology. In fact, James wasn’t particularly interested in experimental research.

Despite this, James was the first American to teach a psychology course in the United States. He also helped found functionalism, which was one of the earliest schools of thought in psychology. As a result of this, James is often referred to as the father of American psychology.

What is The Methodology of Experimental Psychology?

Almost everyone is familiar with certain experiments, such as a mouse trying to navigate a maze or a primate trying to figure out a puzzle. However, human experiments are much more complex. For example, the experimental psychologist must take into account extraneous variables, environmental conditions, and experimenter bias as potentially skewing the data that’s collected.

Additionally, experimental psychologists must choose an appropriate sample size, correctly define the operations of the experiment, and use sound statistical analyses. Experimental methods must be completely controlled and perfectly executed in order to stand up to peer review, which is one of the foundations of all scientific endeavors.

An experimental method in psychology can take several forms:

• Laboratory experiments , in which researchers carefully control every aspect of the experiment. This includes where, when, and how the experiment will take place, the number and type of participants, standardized procedures, and assignment of participants to the control or experimental group. Lab experiments are easy to replicate and do a good job of controlling for confounding variables. However, lab experiments can produce unnatural behaviors due to the artificial setting and experimenter bias can be an issue.
• Natural experiments , in which researchers conduct their experiments in a real-life setting. This type of research offers no control over the independent variable (and no control over potential confounding variables, either). However, because the research is conducted in a natural environment, it has better ecological validity than lab experiments and it can be used to study behaviors that would be unethical to study in a lab setting, again because the independent variable is naturally occurring. 
• Field experiments , in which research is conducted in a real-life setting, but with the ability to manipulate the independent variable. While this type of research doesn’t allow for control over confounding variables, it offers the advantage of most closely reflecting real behaviors with a lesser likelihood of demand characteristics influencing the final results.

What is The Science of Experimental Psychology?

Experimental psychologists and scientists all believe in the same basic four principles. First, determinism means that all phenomena have some sort of systematic cause. Second, empiricism means that objective observation is the key to interpreting the world around us. Third, parsimony means that scientists prefer a minimalist approach to developing and researching theories. That is, science embraces the principle of Occam’s razor, which means that the theory with the fewest assumptions should be the logical conclusion. Finally, the fourth principle is testability. All theories must be empirically tested with applied falsifiability.

In other words, experimental psychology follows the same maxims of the physical sciences. The purpose is to use the principles of the scientific method to empirically study human behavior to arrive at testable and repeatable conclusions.

To do so, experimental research methods must be reliable and valid. Reliability refers to the consistency of observations, or the repeatability of an observation. Examples of tests of reliability include the split-half method (in which the results from one half of participants is compared to the other half) and the test-retest method (in which measurements are taken of the same group multiple times to see if the results are consistent.

Validity refers to how well a test measures what it’s intended to measure. So, an intelligence test is a valid measure of intelligence. It is not, however, a valid measure of honesty.

Additionally, the science of experimental psychology is rooted in research design. There are many types of designs that experimental psychologists can use, including:

• Within-subjects designs , in which participants in the study are exposed to more than one condition, thereby allowing researchers to compare different data points on the same subject.
• Between-subjects designs , in which participants are exposed to only one condition, which enables researchers to compare data between different subjects.
• One-way designs , in which there is a single independent variable and often just two groups, one of which serves as the control group (which is not exposed to the treatment) and the experimental group (which is exposed to the treatment).
• Factorial designs , which feature two or more independent variables which occur at all levels and in combination with every other independent variable. These experiments are quantified based on their factorial design, such as a 2×3 design. This design has two independent variables, one of which has two levels and the other of which has three.

What Are Some Experiment Examples?

Since experimental psychologists are involved in every branch of psychology, there is an impressive variety of experimental categories.

Social psychology uses field experiments and objective observation to understand collective behavior. For example, researchers might construct a simulated scenario that tests how participants engage in altruistic behavior, such as helping an injured stranger.

On the other hand, cognitive psychologists can use complex equipment and software to analyze the neurological reactions of participants as they watch scary or violent images.

Finally, psychologists studying abnormal behavior, such as phobias or personality disorders, could test participants with these conditions against groups of people that have not been diagnosed with these disorders.

Over the years, there have been many highly influential psychological experiments using various scientific methods. And while their influence has had far-reaching ramifications on our understanding of human behavior, some of these experiments are now viewed as having been unethical. While an experimental psychologist can make a big difference in the world after a successful experimental, their experimental methods do still have to be rational and fair. Psychology research is pointless if people are getting hurt. 

Philip Zimbardo’s Stanford Prison Experiment:

In short, the goal of the experiment was to determine how people conform to social roles. To study this, Zimbardo constructed a makeshift jail in the basement of the psychology building on the Stanford campus. He then recruited 24 male students, each of which were randomly assigned to be a guard or a prisoner.

Prisoners were rounded up and brought to the “prison,” where they were booked and supervised by the group assigned to be guards. Both groups quickly adapted to the roles to which they had ben assigned, with some of the guards engaging in psychological torture of their charges. The experiment ended after just six days because the situation had become so intense and so dangerous.

Despite the questionable ethics of the experiment, it did shed light on conformity and social roles, and how people can very quickly and easily adopt roles they are expected to play, especially when those roles are highly stereotyped

What Are The Careers Options for Experimental Psychology?

According to the American Psychological Association , experimental psychologists seek to answer basic questions about human behavior and mental processes through applied research. These professional perform research to bring light to many topics.

For example, the most popular research topics include memory, emotion, perception and sensation. Typically, experimental psychologists work work in university research centers, but also work for private companies or even the government. Other experimental psychologists may also work in subfields. This may include education (to teach psychology courses), human resources and health care.

Whatever the work setting, you will need a doctorate in psychology to be an experimental psychologist. What’s more, you’ll need to specialize in a particular area of research and pursue post-doctoral studies in that area.

The job outlook for psychology as a whole is about average for the next few years. The Bureau of Labor Statistics (BLS) estimates that all psychology jobs will grow at a rate of three percent through 2029 . The BLS doesn’t provide data specific to experimental psychology, but it’s reasonable to assume that job growth in this field will be on par with the field of psychology as a whole.

In other words, with average job growth for the coming years, competition for experimental psychology jobs will likely be fierce. This is all the more reason to learn about the field, carefully plan your education, and seek out ways in which you can get real-world experience in experimental psychology. The better your combination of education and experience, the more likely you are to stand out in a crowd of other experimental psychology graduates.

Sean Jackson

B.A. Social Studies Education | University of Wyoming

M.S. Counseling | University of Wyoming

B.S. Information Technology | University of Massachusetts

Updated August 2021

Related Resources:

What Careers are in Experimental Psychology?

• The 25 Most Influential Psychological Experiments in History

What Is Human Factors Psychology?

• 5 Best Online Ph.D. Marriage and Family Counseling Programs
• Top 5 Online Doctorate in Educational Psychology
• 5 Best Online Ph.D. in Industrial and Organizational Psychology Programs
• Top 10 Online Master’s in Forensic Psychology
• 10 Most Affordable Counseling Psychology Online Programs
• 10 Most Affordable Online Industrial Organizational Psychology Programs
• 10 Most Affordable Online Developmental Psychology Online Programs
• 15 Most Affordable Online Sport Psychology Programs
• 10 Most Affordable School Psychology Online Degree Programs
• Top 50 Online Psychology Master’s Degree Programs
• Top 25 Online Master’s in Educational Psychology
• Top 25 Online Master’s in Industrial/Organizational Psychology
• Top 10 Most Affordable Online Master’s in Clinical Psychology Degree Programs
• Top 6 Most Affordable Online PhD/PsyD Programs in Clinical Psychology
• 50 Great Small Colleges for a Bachelor’s in Psychology
• 50 Most Innovative University Psychology Departments
• The 30 Most Influential Cognitive Psychologists Alive Today
• Top 30 Affordable Online Psychology Degree Programs
• 30 Most Influential Neuroscientists
• Top 40 Websites for Psychology Students and Professionals
• Top 30 Psychology Blogs
• 25 Celebrities With Animal Phobias
• Your Phobias Illustrated (Infographic)
• 15 Inspiring TED Talks on Overcoming Challenges
• 20 Most Unethical Experiments in Psychology
• 10 Fascinating Facts About the Psychology of Color
• 15 Scariest Mental Disorders of All Time
• 15 Things to Know About Mental Disorders in Animals
• 13 Most Deranged Serial Killers of All Time

Site Information

• About Online Psychology Degree Guide

What is Experimental Psychology?

The four canons of science, dependent and independent variables, operational definitions.

The operational definition is a way to define abstract ideas to make it observable and measurable. Going back to the study between emotions and memory, which are two very abstract ideas, researchers will need to provide operational definitions to measure the happiness of a person, as well as the strength of his or her memory. As an example, researchers can define happiness through a survey that is filled out by participants to gauge their current state; memory, on the other hand, can be tested by asking participants to recall the order of photos that will be shown to them later on.

Validity and Reliability

What does an experimental psychologist do, leave a reply cancel reply.

EXPERIMENTAL PSYCHOLOGY

Studying behaviour, motive, cognition in a laboratory or experimental setting. See empirical pstchology.

Leave a Reply

Your email address will not be published. Required fields are marked *

Latest Posts

A Quick Look at the History Behind Hypnosis

A Brief History of Brainwashing: The Science of Thought Control

A Deep Dive into the Social Psychology of Leadership

Counseling Approaches to Client Care: Theories to Apply in Practice

The Future Of Education: Can You Earn A Psychology Degree Online?

Insomnia & Mental Illness: What is the Correlation?

Stop Guessing: Here Are 3 Steps to Data-Driven Psychological Decisions

Getting Help with Grief: Understanding Therapy & How It Can Help

Exploring the Psychology of Risk and Reward

Understanding ADHD in Women: Symptoms, Treatment & Support

Meeting the Milestones: A Guide to Piaget's Child Developmental Stages

Counseling, Therapy, and Psychology: What Is The Difference?

Popular psychology terms, medical model, hypermnesia, affirmation, brainwashing, backup reinforcer, message-learning approach, affiliative behavior, acculturation, assertiveness, behavioral congruence, social instinct.

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Sage Choice

The psychology of experimental psychologists: Overcoming cognitive constraints to improve research: The 47th Sir Frederic Bartlett Lecture

Like many other areas of science, experimental psychology is affected by a “replication crisis” that is causing concern in many fields of research. Approaches to tackling this crisis include better training in statistical methods, greater transparency and openness, and changes to the incentives created by funding agencies, journals, and institutions. Here, I argue that if proposed solutions are to be effective, we also need to take into account human cognitive constraints that can distort all stages of the research process, including design and execution of experiments, analysis of data, and writing up findings for publication. I focus specifically on cognitive schemata in perception and memory, confirmation bias, systematic misunderstanding of statistics, and asymmetry in moral judgements of errors of commission and omission. Finally, I consider methods that may help mitigate the effect of cognitive constraints: better training, including use of simulations to overcome statistical misunderstanding; specific programmes directed at inoculating against cognitive biases; adoption of Registered Reports to encourage more critical reflection in planning studies; and using methods such as triangulation and “pre mortem” evaluation of study design to foster a culture of dialogue and criticism.

Introduction

The past decade has been a bruising one for experimental psychology. The publication of a paper by Simmons, Nelson, and Simonsohn (2011) entitled “False-positive psychology” drew attention to problems with the way in which research was often conducted in our field, which meant that many results could not be trusted. Simmons et al. focused on “undisclosed flexibility in data collection and analysis,” which is now variously referred to as p -hacking, data dredging, noise mining, or asterisk hunting: exploring datasets with different selections of variables and different analyses to attain a p -value lower than .05 and, subsequently, reporting only the significant findings. Hard on the heels of their demonstration came a wealth of empirical evidence from the Open Science Collaboration (2015) . This showed that less than half the results reported in reputable psychological journals could be replicated in a new experiment.

The points made by Simmons et al. (2011) were not new: indeed, they were anticipated in 1830 by Charles Babbage, who described “cooking” of data:

This is an art of various forms, the object of which is to give ordinary observations the appearance and character of those of the highest degree of accuracy. One of its numerous processes is to make multitudes of observations, and out of these to select only those which agree, or very nearly agree. If a hundred observations are made, the cook must be very unhappy if he cannot pick out fifteen or twenty which will do for serving up. (p. 178–179)

P -hacking refers to biased selection of data or analyses from within an experiment. Bias also affects which studies get published in the form of publication bias—the tendency for positive results to be overrepresented in the published literature. This is problematic because it gives an impression that findings are more consistent than is the case, which means that false theories can attain a state of “canonisation,” where they are widely accepted as true ( Nissen, Magidson, Gross, & Bergstrom, 2016 ). Figure 1 illustrates this with a toy simulation of a set of studies testing a difference between means from two conditions. If we have results from a series of experiments, three of which found a statistically significant difference and three of which did not, this provides fairly strong evidence that the difference is real (panel a). However, if we add a further four experiments that were not reported because results were null, the evidence cumulates in the opposite direction. Thus, omission of null studies can drastically alter our impression of the overall support for a hypothesis.

The impact of publication bias demonstrated with plots of cumulative log odds in favour of true versus null effect over a series of experiments. The log odds for each experiment can be computed with knowledge of alpha (.05) and power (.8); 1 denotes an experiment with significant difference between means, and 0, a null result. The starting point is zero, indicating that we assume a 50:50 chance of a true effect. For each significant result, the log odds of it coming from a true effect versus a null effect is log(.8/.05) = 2.77. For a null result, the log odds is log (.2/.95) = −1.55. The selected set of studies in panel (a) concludes with a log odds greater than 3, indicating that the likelihood of a true effect is 20 times greater than a null effect. However, panel (b), which includes additional null results (labelled in grey), leads to the opposite conclusion.

Since the paper by Simmons et al. (2011) , there has been a dramatic increase in replication studies. As a result, a number of well-established phenomena in psychology have come into question. Often it is difficult to be certain whether the original reports were false positives, whether the replication was flawed, or whether the effect of interest is only evident under specific conditions—see, for example, Hobson and Bishop (2016) on mu suppression in response to observed actions; Sripada, Kesller, and Jonides (2016) on ego depletion; Lehtonen et al. (2018) on an advantage in cognitive control for bilinguals; O’Donnell et al. (2018) on the professor-priming effect; and Oostenbroek et al. (2016) on neonatal imitation. What is clear is that the size, robustness, and generalisability of many classic effects are lower than previously thought.

Selective reporting, through p -hacking and publication bias, is not the only blight on our science. A related problem is many editors place emphasis on reporting results in a way that “tells a good story,” even if that means retrofitting our hypothesis to the data, i.e., HARKing or “hypothesising after the results are known” ( Kerr, 1998 ). Oberauer and Lewandowsky (2019) drew parallels between HARKing and p -hacking: in HARKing, there is post hoc selection of hypotheses, rather than selection of results or an analytic method. They proposed that HARKing is most widely used in fields where theories are so underspecified that they can accommodate many hypotheses and where there is a lack of “disconfirmatory diagnosticity,” i.e., failure to support a prediction is uninformative.

A lack of statistical power is a further problem for psychology—one that has been recognised since 1969 , when Jacob Cohen exhorted psychologists not to waste time and effort doing experiments that had too few observations to show an effect of interest. In other fields, notably clinical trials and genetics, after a period where non-replicable results proliferated, underpowered studies died out quite rapidly when journals adopted stringent criteria for publication (e.g., Johnston, Lahey, & Matthys, 2013 ), and funders began to require power analysis in grant proposals. Psychology, however, has been slow to catch up.

It is not just experimental psychology that has these problems—studies attempting to link psychological traits and disorders to genetic and/or neurobiological variables are, if anything, subject to greater challenges. A striking example comes from a meta-analysis of links between the serotonin transporter gene, 5-HTTPLR, and depression. This postulated association has attracted huge research interest over the past 20 years, and the meta-analysis included 450 studies. Contrary to expectation, it concluded that there was no evidence of association. In a blog post summarising findings, Alexander (2019) wrote,

. . . what bothers me isn’t just that people said 5-HTTLPR mattered and it didn’t. It’s that we built whole imaginary edifices, whole castles in the air on top of this idea of 5-HTTLPR mattering. We “figured out” how 5-HTTLPR exerted its effects, what parts of the brain it was active in, what sorts of things it interacted with, how its effects were enhanced or suppressed by the effects of other imaginary depression genes. This isn’t just an explorer coming back from the Orient and claiming there are unicorns there. It’s the explorer describing the life cycle of unicorns, what unicorns eat, all the different subspecies of unicorn, which cuts of unicorn meat are tastiest, and a blow-by-blow account of a wrestling match between unicorns and Bigfoot.

It is no exaggeration to say that our field is at a crossroads ( Pashler & Wagenmakers, 2012 ), and the 5-HTTLPR story is just a warning sign that practices that lead to bad science are widespread. If we continue to take the well-trodden path, using traditional methods for cooking data and asterisk hunting, we are in danger of losing attention, respect, and funding.

Much has been written about how we might tackle the so-called “replication crisis.” There have been four lines of attack. First, there have been calls for greater openness and transparency ( Nosek et al., 2015 ). Second, a case has been made for better training in methods (e.g., Rousselet, Pernet, & Wilcox, 2017 ). Third, it has been argued we need to change the way research has been conducted to incorporate pre-registration of research protocols, preferably in the format of Registered Reports, which are peer-reviewed prior to data collection ( Chambers, 2019 ). Fourth, it is recognised that for too long, the incentive structure of research has prioritised innovative, groundbreaking results over methodological quality. Indeed, Smaldino and McElreath (2016) suggested that one can model the success of scientists in a field as an evolutionary process, where prestigious publications lead to survival, leaving those whose work is less exciting to wither away and leave science. The common thread to these efforts is that they locate the mechanisms of bad science at the systemic level, in ways in which cultures and institutions reinforce norms and distribute resources. The solutions are, therefore, aimed at correcting these shortcomings by creating systems that make good behaviour easier and more rewarding and make poor behaviour more costly.

My view, however, is that institutional shortcomings are only part of the story: to improve scientific research, we also need to understand the mechanisms that maintain bad practices in individual humans. Bad science is usually done because somebody mistook it for good science. Understanding why individual scientists mistake bad science for good, and helping them to resist these errors, is a necessary component of the movement to improve psychology. I will argue that we need to understand how cognitive constraints lead to faulty reasoning if we are to get science back on course and persuade those who set the incentives to reform. Fortunately, as psychologists, we are uniquely well positioned to tackle this issue.

Experimental psychology has a rich tradition of studying human reasoning and decision-making, documenting the flaws and foibles that lead us to selectively process some types of information, make judgements on the basis of incomplete evidence, and sometimes behave in ways that seem frankly irrational. This line of work has had significant application to economics, politics, business studies, and law, but, with some notable exceptions (e.g., Hossenfelder, 2018 ; Mahoney, 1976 ), it has seldom been considered when studying the behaviour of research scientists. In what follows, I consider how our knowledge of human cognition can make sense of problematic scientific practices, and I propose ways we might use this information to find solutions.

Cognitive constraints that affect how psychological science is done

Table 1 lists four characteristics of human cognition that I focus on: I refer to these as “constraints” because they limit how we process, understand, or remember information, but it is important to note that they include some biases that can be beneficial in many contexts. The first constraint is confirmation bias. As Hahn and Harris (2014) noted, a range of definitions of “confirmation bias” exist—here, I will define it as the tendency to seek out evidence that supports our position. A further set of constraints has to do with understanding of probability. A lack of an intuitive grasp of probability contributes to both neglect of statistical power in study design and p -hacking in data analysis. Third, there is an asymmetry in moral reasoning that can lead us to treat errors of omission as less culpable than errors of commission, even when their consequences are equally serious ( Haidt & Baron, 1996 ). The final constraint featured in Bartlett’s (1932) work: reliance on cognitive schemata to fill in unstated information, leading to “reconstructive remembering,” which imbues memories with meaning while filtering out details that do not fit preconceptions.

Different types of cognitive constraints.

In what follows, I illustrate how these constraints assume particular importance at different stages of the research process, as shown in Table 2 .

Cognitive constraints that operate at different stages of the research process.

HARKing: hypothesising after the results are known.

Bias in experimental design

Confirmation bias and the failure to consider alternative explanations.

Scientific discovery involves several phases: the researcher needs to (a) assemble evidence, (b) look for meaningful patterns and regularities in the data, (c) formulate a hypothesis, and (d) test it empirically by gathering informative new data. Steps (a)–(c) may be designated as exploratory and step (d) as hypothesis testing or confirmatory ( Wagenmakers, Wetzels, Borsboom, van der Mass, & Kievit, 2012 ). Importantly, the same experiment cannot be used to both formulate and confirm a hypothesis. In practice, however, the distinction between the two types of experiment is often blurred.

Our ability to see patterns in data is vital at the exploratory stage of research: indeed, seeing something that nobody else has observed is a pinnacle of scientific achievement. Nevertheless, new ideas are often slow to be accepted, precisely because they do not fit the views of the time. One such example is described by Zilles and Amunts (2010) : Brodmann’s cytoarchitectonic map of the brain, described in 1909. This has stood the test of time and is still used over 100 years later, but for several decades, it was questioned by those who could not see the fine distinctions made by Brodmann. Indeed, criticisms of poor reproducibility and lack of objectivity were levelled against him.

Brodmann’s case illustrates that we need to be cautious about dismissing findings that depend on special expertise or unique insight of the observer. However, there are plenty of other instances in the history of science where invalid ideas persisted, especially if proposed by an influential or charismatic figure. Entire edifices of pseudoscience have endured because we are very bad at discarding theories that do not work; as Bartlett (1932) would predict, new information that is consistent with the theory will strengthen its representation in our minds, but inconsistent information will be ignored. Examples from the history of science include the rete mirabile , a mass of intertwined arteries that is found in sheep but wrongly included in anatomical drawings of humans for over 1,000 years because of the significance attributed to this structure by Galen ( Bataille et al., 2007 ); the planet Vulcan, predicted by Newton’s laws and seen by many astronomers until its existence was disproved by Einstein’s discoveries ( Levenson, 2015 ); and N-rays, non-existent rays seen by at least 40 people and analysed in 3,090 papers by 100 scientists between 1903 and 1906 ( Nye, 1980 ).

Popper’s (1934/ 1959 ) goal was to find ways to distinguish science from pseudoscience, and his contribution to philosophy of science was to emphasise that we should be bold in developing ideas but ruthless in attempts to falsify them. In an early attempt to test scientists’ grasp of Popperian logic, Mahoney (1976) administered a classic task developed by Wason (1960) to 84 scientists (physicists, biologists, psychologists, and sociologists). In this deceptively simple task, people are shown four cards and told that each card has a number on one side and a patch of colour on the other side. The cards are placed to show number 3, number 8, red, and blue, respectively (see Figure 2 ). The task is to identify which cards need to be turned over to test the hypothesis that if an even number appears on one side, then the opposite side is red. The subject can pick any number of cards. The correct response is to name the two cards that could disconfirm the hypothesis—the number 8 and the blue card. Fewer than 10% of the scientists tested by Mahoney identified both critical cards, more often selecting the number 8 and the red card.

Wason’s (1960) task: The subject is told, “Each card has a number on one side and a patch of colour on the other. You are asked to test the hypothesis that—for these 4 cards—if an even number appears on one side, then the opposite side is red. Which card(s) would you turn over to test the hypothesis?”

Although this study was taken as evidence of unscientific reasoning by scientists, that conclusion has since been challenged by those who have criticised both Popperian logic, in general, and the Wason selection task, in particular, as providing an unrealistic test of human rationality. For a start, the Wason task uses a deterministic hypothesis that can be disproved by a single piece of evidence. This is not a realistic model of biological or behavioural sciences, where we seldom deal with deterministic phenomena. Consider the claim that smoking causes lung cancer. Most of us accept that this is so, even though we know there are people who smoke and who do not get lung cancer and people who get lung cancer but never smoked. When dealing with probabilistic phenomena, a Bayesian approach makes more sense, whereby we consider the accumulated evidence to determine the relative likelihood of one hypothesis over another (as illustrated in Figure 1 ). Theories are judged as more or less probable, rather than true or false. Oaksford and Chater (1994) showed that, from a Bayesian perspective, typical selections made on the Wason task would be rational in contexts where the antecedent and consequent of the hypothesis (an even number and red colour) were both rare. Subsequently, Perfors and Navarro (2009) concluded that in situations where rules are relevant only for a minority of entities, then confirmation bias is an efficient strategy.

This kind of analysis has shifted the focus to discussions about how far, and under what circumstances, people are rational decision-makers. However, it misses a key point about scientific reasoning, which is that it involves an active process of deciding which evidence to gather, rather than merely a passive evaluation of existing evidence. It seems reasonable to conclude that, when presented with a particular set of evidence, people generally make decisions that are rational when evaluated against Bayesian standards. However, history suggests that we are less good at identifying which new evidence needs to be gathered to evaluate a theory. In particular, people appear to have a tendency to accept a hypothesis on the basis of “good enough” evidence, rather than actively seeking evidence for alternative explanations. Indeed, an early study by Doherty, Mynatt, Tweney, and Schiavo (1979) found that, when given an opportunity to select evidence to help decide which of two hypotheses was true (in a task where a fictitious pot had to be assigned as originating from one of the two islands that differed in characteristic features), people seemed unable to identify which information would be diagnostic and tended, instead, to select information that could neither confirm nor disconfirm their hypothesis.

Perhaps the strongest evidence for our poor ability to consider alternative explanations comes from the history of the development of clinical trials. Although James Lind is credited with doing the first trials for treatment of scurvy in 1747, it was only in 1948 that the randomised controlled trial became the gold standard for evaluating medical interventions ( Vallier & Timmerman, 2008 ). The need for controls is not obvious, and people who are not trained in this methodology will often judge whether a treatment is effective on the basis of a comparison on an outcome measure between a pre-treatment baseline and a post-treatment evaluation. The logic is that if a group of patients given the treatment does not improve, the treatment did not work. If they do show meaningful gains, then it did work. And we can even embellish this comparison with a test of statistical significance. This reasoning can be seen as entirely rational, and this can explain why so many people are willing to accept that alternative medicine is effective.

The problem with this approach is that the pre–post intervention comparison allows important confounds to creep in. For instance, early years practitioners argue that we should identify language problems in toddlers so that we can intervene early. They find that if 18-month-old late talkers are given intervention, only a minority still have language problems at 2 years and, therefore, conclude the intervention was effective. However, if an untreated control group is studied over the same period, we find very similar rates of improvement ( Wake et al., 2011 )—presumably due to factors such a spontaneous resolution of problems or regression to the mean, which will lead to systematic bias in outcomes. Researchers need training to recognise causes of bias and to take steps to overcome them: thinking about possible alternative explanations of an observed phenomenon does not come naturally, especially when the preliminary evidence looks strong.

Intervention studies provide the clearest evidence of what I term “premature entrenchment” of a theory: some other examples are summarised in Table 3 . Note that these examples do not involve poor replicability, quite the opposite. They are all cases where an effect, typically an association between variables, is reliably observed, and researchers then converge on accepting the most obvious causal explanation, without considering lines of evidence that might point to alternative possibilities.

Premature entrenchment: examples where the most obvious explanation for an observed association is accepted for many years, without considering alternative explanations that could be tested with different evidence.

fMRI: functional magnetic resonance imaging.

Premature entrenchment may be regarded as evidence that humans adopt Bayesian reasoning: we form a prior belief about what is the case and then require considerably more evidence to overturn that belief than to support it. This would explain why, when presented with virtually identical studies that either provided support for or evidence against astrology, psychologists were more critical of the latter ( Goodstein & Brazis, 1970 ). The authors of that study expressed concern about the “double standard” shown by biased psychologists who made unusually harsh demands of research in borderline areas, but from a Bayesian perspective, it is reasonable to use prior knowledge so that extraordinary claims require extraordinary evidence. Bayesian reasoning is useful in many situations: it allows us to act decisively on the basis of our long-term experience, rather than being swayed by each new incoming piece of data. However, it can be disastrous if we converge on a solution too readily on the basis of incomplete or inaccurate information. This will be exacerbated by publication bias, which distorts the evidential landscape.

For many years, the only methods available to counteract the tendency for premature entrenchment were exhortations to be self-critical (e.g., Feynman, 1974 ) and peer review. The problem with peer review is that it typically comes too late to be useful, after research is completed. In the final section of this article, I will consider some alternative approaches that bring in external appraisal of experimental designs at an earlier stage in the research process.

Misunderstanding of probability leading to underpowered studies

Some 17 years after Cohen’s seminal work on statistical power, Newcombe (1987) wrote,

Small studies continue to be carried out with little more than a blind hope of showing the desired effect. Nevertheless, papers based on such work are submitted for publication, especially if the results turn out to be statistically significant. (p. 657)

In clinical medicine, things have changed, and the importance of adequate statistical power is widely recognised among those conducting clinical trials. But in psychology, the “blind hope” has persisted, and we have to ask ourselves why this is.

My evidence here is anecdotal, but the impression is that many psychologists simply do not believe advice about statistical power, perhaps because there are so many underpowered studies published in the literature. When a statistician is consulted about sample size for a study, he or she will ask the researcher to estimate the anticipated effect size. This usually leads to a sample size estimate that is far higher than the researcher anticipated or finds feasible, leading to a series of responses not unlike the first four of the five stages of grief: denial, anger, bargaining, and depression. The final stage, acceptance, may, however, not be reached.

Of course, there are situations where small sample sizes are perfectly adequate: the key issue is how large the effect of interest is in relation to the variance. In some fields, such as psychophysics, you may not even need statistics—the famous “interocular trauma” test (referring to a result so obvious and clear-cut that it hits you between the eyes) may suffice. Indeed, in such cases, recruitment of a large sample would just be wasteful.

There are, however, numerous instances in psychology where people have habitually used sample sizes that are too small to reliably detect an effect of interest: see, for instance, the analysis by Poldrack et al. (2017) of well-known effects in functional magnetic resonance imaging (fMRI) or Oakes (2017) on looking-time experiments in infants. Quite often, a line of research is started when a large effect is seen in a small sample, but over time, it becomes clear that this is a case of “winner’s curse,” a false positive that is published precisely because it looks impressive but then fails to replicate when much larger sample sizes are used. There are some recent examples from studies looking at neurobiological or genetic correlates of individual differences, where large-scale studies have failed to support previously published associations that had appeared to be solid (e.g., De Kovel & Francks, 2019 , on genetics of handedness; Traut et al., 2018 , on cerebellar volume in autism; Uddén et al., 2019 , on genetic correlates of fMRI language-based activation).

A clue to the persistence of underpowered psychology studies comes from early work by Tversky and Kahneman (1971 , 1974 ). They studied a phenomenon that they termed “belief in the law of small numbers,” an exaggerated confidence in the validity of conclusions based on small samples, and showed that even those with science training tended to have strong intuitions about random sampling that were simply wrong. They illustrated this with the following problem:

A certain town is served by two hospitals. In the larger hospital about 45 babies are born each day, and in the smaller hospital about 15 babies are born each day. As you know, about 50% of all babies are boys. However, the exact percentage varies from day to day. Sometimes it may be higher than 50%, sometimes lower. For a period of 1 year, each hospital recorded the days on which more than 60% of the babies born were boys. Which hospital do you think recorded more such days? 1. The large hospital 2. The small hospital 3. About the same (that is, within 5% of each other)

Most people selected Option 3, whereas, as illustrated in Figure 3 , Option 2 is the correct answer—with only 15 births per day, the day-to-day variation in the proportion of boys will be much higher than with 45 births per day, and hence, more days will have more than 60% boys. One reason why our intuitions deceive us is because the sample size does not affect the average percentage of male births in the long run: this will be 50%, regardless of the hospital size. But sample size has a dramatic impact on the variability in the proportion of male births from day to day. More formally, if you have a big and small sample drawn from the same population, the expected estimate of the mean will be the same, but the standard error of that estimate will be greater for the small sample.

Simulated data showing proportions of males born in a small hospital with 15 births per day versus a large hospital with 45 births per day. The small hospital has more days where more than 60% of births are boys (points above red line).

Statistical power depends on the effect size, which, for a simple comparison of two means, can be computed as the difference in means divided by the pooled standard deviation. It follows that power is crucially dependent on the proportion of variance in observations that is associated with an effect of interest, relative to background noise. Where variance is high, it is much harder to detect the effect, and hence, small samples are often underpowered. Increasing the sample size is not the only way to improve power: other options include improving the precision of measurement, using more effective manipulations, or adopting statistical approaches to control noise ( Lazic, 2018 ). But in many situations, increasing the sample size is the preferred approach to enhance statistical power to detect an effect.

Bias in data analysis: p -hacking

P -hacking can take various forms, but they all involve a process of selective analysis. Suppose some researchers hypothesise that there is an association between executive function and implicit learning in a serial reaction time task, and they test this in a study using four measures of executive function. Even if there is only one established way of scoring each task, they have four correlations; this means that the probability that none of the correlations is significant at the .05 level is .95 4 —i.e., .815—and conversely, the probability that at least one is significant is .185. This probability can be massaged to even higher levels if the experimenters look at the data and then select an analytic approach that maximises the association: maybe by dropping outliers, by creating a new scoring method, combining measures in composites, and so on. Alternatively, the experimenters may notice that the strength of the correlation varies with the age or sex of participants and so subdivide the sample to coax at least a subset of data into significance. The key thing about p -hacking is that at the end of the process, the researchers selectively report the result that “worked,” with the implication that the p -value can be interpreted at face value. But it cannot: probability is meaningless if not defined in terms of a particular analytic context. P -hacking appears to be common in psychology ( John, Loewenstein, & Prelec, 2012 ). I argue here that this is because it arises from a conjunction of two cognitive constraints: failure to understand probability, coupled with a view that omission of information when reporting results is not a serious misdemeanour.

Failure to understand probability

In an influential career guide, published by the American Psychological Association, Bem (2004) explicitly recommended going against the “conventional view” of the research process, as this might lead us to miss exciting new findings. Instead readers were encouraged to

become intimately familiar with . . . the data. Examine them from every angle. Analyze the sexes separately. Make up new composite indexes. If a datum suggests a new hypothesis, try to find additional evidence for it elsewhere in the data. If you see dim traces of interesting patterns, try to reorganize the data to bring them into bolder relief. If there are participants you don’t like, or trials, observers, or interviewers who gave you anomalous results, drop them (temporarily). Go on a fishing expedition for something—anything—interesting. (p. 2)

For those who were concerned this might be inappropriate, Bem offered reassurance. Everything is fine because what you are doing is exploring your data. Indeed, he implied that anyone who follows the “conventional view” would be destined to do boring research that nobody will want to publish.

Of course, Bem (2004) was correct to say that we need exploratory research. The problem comes when exploratory research is repackaged as if it were hypothesis testing, with the hypothesis invented after observing the data (HARKing), and the paper embellished with p -values that are bound to be misleading because they were p -hacked from numerous possible values, rather than derived from testing an a priori hypothesis. If results from exploratory studies were routinely replicated, prior to publication, we would not have a problem, but they are not. So why did the American Psychological Association think it appropriate to publish Bem’s views as advice to young researchers? We can find some clues in the book overview, which explains that there is a distinction between the “formal” rules that students are taught and the “implicit” rules that are applied in everyday life, concluding that “This book provides invaluable guidance that will help new academics plan, play, and ultimately win the academic career game.” Note that the stated goal is not to do excellent research: it is to have “a lasting and vibrant career.” It seems, then, that there is recognition here that if you do things in the “conventional” way, your career will suffer. It is clear from Bem’s framing of his argument that he was aware that his advice was not “conventional,” but he did not think it was unethical—indeed, he implied it would be unfair on young researchers to do things conventionally as that will prevent them making exciting discoveries that will enable them to get published and rise up the academic hierarchy. While it is tempting to lament the corruption of a system that treats an academic career as a game, it is more important to consider why so many people genuinely believe that p -hacking is a valid, and indeed creative, approach to doing research.

The use of null-hypothesis significance testing has attracted a lot of criticism, with repeated suggestions over the years that p -values be banned. I favour the more nuanced view expressed by Lakens (2019) , who suggests that p -values have a place in science, provided they are correctly understood and used to address specific questions. There is no doubt, however, that many people do misunderstand the p -value. There are many varieties of misunderstanding, but perhaps the most common is to interpret the p -value as a measure of strength of evidence that can be attached to a given result, regardless of the context. It is easy to see how this misunderstanding arises: if we hold the sample size constant, then for a single comparison, there will be a linear relationship between the p -value and the effect size. However, whereas an effect size remains the same, regardless of the analytic context, a p -value is crucially context-dependent.

Suppose in the fictitious study of executive function described above, the researchers have 20 participants and four measures of executive function (A–D) that correlate with implicit learning with r values of .21, .47, .07, and −.01. The statistics package tells us that the corresponding two-tailed p -values are .374, .037, .769, and .966. A naive researcher may rejoice at having achieved significance with the second correlation. However, as noted above, the probability that at least one correlation of the four will have an associated p -value of less than .05 is 18%, not 5%. If we want to identify correlations that are unlikely under the null hypothesis, then we need to correct the alpha level (e.g., by doing a Bonferroni correction to adjust by the number of tests, i.e., .05/4 = .0125). At this point, the researchers see their significant result snatched from their grasp. This creates a strong temptation to just drop the three non-significant tests and not report them. Alternatively, one sometimes sees papers that report the original p -value but then state that it “did not survive” Bonferroni correction, but they, nevertheless, exhume it and interpret the uncorrected value. Researchers acting this way may not think that they are doing anything inappropriate, other than going against advice of pedantic statisticians, especially given Bem’s (2004) advice to follow the “implicit” rather than “formal” rules of research. However, this is simply wrong: as illustrated above, a p -value can only be interpreted in relation to the context in which it is computed.

One way of explaining the notion of p -hacking is to use the old-fashioned method of games of chance. I find this scenario helpful: we have a magician who claims he can use supernatural powers to deal a poker hand of “three of a kind” from an unbiased deck of cards. This type of hand will occur in around 1 of 50 draws from an unbiased deck. He points you to a man who, to his amazement, finds that his hand contains three of a kind. However, you then discover he actually tried his stunt with 50 people, and this man was the only one who got three of a kind. You are rightly disgruntled. This is analogous to p -hacking. The three-of-a-kind hand is real enough, but its unusualness, and hence its value as evidence of the supernatural, depends on the context of how many tests were done. The probability that needs to be computed here is not the probability of one specific result but rather the probability that specific result would come up at least once in 50 trials.

Asymmetry of sins of omission and commission

According to Greenwald (1975) “[I]t is a truly gross ethical violation for a researcher to suppress reporting of difficult-to-explain or embarrassing data to present a neat and attractive package to a journal editor” (p. 19).

However, this view is not universal.

Greenwald’s focus was on publication bias, i.e., failure to report an entire study, but the point he made about “prejudice” against null results also applies to cases of p -hacking where only “significant” results are reported, whereas others go unmentioned. It is easy to see why scientists might play down the inappropriateness of p -hacking, when it is so important to generate “significant” findings in a world with a strong prejudice against null results. But I suspect another reason why people tend to underrate the seriousness of p -hacking is because it involves an error of omission (failing to report the full context of a p -value), rather than an error of commission (making up data).

In studies of morality judgement, errors of omission are generally regarded as less culpable than errors of commission (see, e.g., Haidt & Baron, 1996 ). Furthermore, p -hacking may be seen to involve a particularly subtle kind of dishonesty because the statistics and their associated p -values are provided by the output of statistics software. They are mathematically correct when testing a specific, prespecified hypothesis: the problem is that, without the appropriate context, they imply stronger evidence than is justified. This is akin to what Rogers, Zeckhauser, Gino, Norton, and Schweitzer (2017) have termed “paltering,” i.e., the use of truthful statements to mislead, a topic they studied in the context of negotiations. An example was given of a person trying to sell a car that had twice needed a mechanic to fix it. Suppose the potential purchaser directly asks “Has the car ever had problems?” An error of commission is to deny the problems, but a paltering answer would be “This car drives very smoothly and is very responsive. Just last week it started up with no problems when the temperature was −5 degrees Fahrenheit.” Rogers et al. showed that negotiators were more willing to palter than to lie, although potential purchasers regarded paltering as only marginally less immoral than lying.

Regardless of the habitual behaviour of researchers, the general public does not find p -hacking acceptable. Pickett and Roche (2018) did an M-Turk experiment in which a community sample was asked to judge the morality of various scenarios, including this one:

A medical researcher is writing an article testing a new drug for high blood pressure. When she analyzes the data with either method A or B, the drug has zero effect on blood pressure, but when she uses method C, the drug seems to reduce blood pressure. She only reports the results of method C, which are the results that she wants to see.

Seventy-one percent of respondents thought this behaviour was immoral, 73% thought the researcher should receive a funding ban, and 63% thought the researcher should be fired.

Nevertheless, although selective reporting was generally deemed immoral, data fabrication was judged more harshly, confirming that in this context, as in those studied by Haidt and Baron (1996) , sins of commission are taken more seriously than errors of omission.

If we look at the consequences of a specific act of p -hacking, it can potentially be more serious than an act of data fabrication: this is most obvious in medical contexts, where suppression of trial results, either by omitting findings from within a study or by failing to publish studies with null results, can provide a badly distorted basis for clinical decision-making. In their simulations of evidence cumulation, Nissen et al. (2016) showed how p -hacking could compound the impact of publication bias and accelerate the premature “canonization” of theories; the alpha level that researchers assume applies to experimental results is distorted by p -hacking, and the expected rate of false positives is actually much higher. Furthermore, p -hacking is virtually undetectable because the data that are presented are real, but the necessary context for their interpretation is missing. This makes it harder to correct the scientific record.

Bias in writing up a study

Most writing on the “replication crisis” focuses on aspects of experimental design and observations, data analysis, and scientific reporting. The resumé of literature that is found in the introduction to empirical papers, as well as in literature review articles, is given less scrutiny. I will make the case that biased literature reviews are universal and have a major role in sustaining poor reproducibility because they lead to entrenchment of false theories, which are then used as the basis for further research.

It is common to see biased literature reviews that put a disproportionate focus on findings that are consistent with the author’s position. Researchers who know an area well may be aware of this, especially if their own work is omitted, but in general, cherry-picking of evidence is hard to detect. I will use a specific paper published in 2013 to illustrate my point, but I will not name the authors, as it would be invidious to single them out when the kinds of bias in their literature review are ubiquitous. In their paper, my attention was drawn to the following statement in the introduction:

Regardless of etiology, cerebellar neuropathology commonly occurs in autistic individuals. Cerebellar hypoplasia and reduced cerebellar Purkinje cell numbers are the most consistent neuropathologies linked to autism. … MRI studies report that autistic children have smaller cerebellar vermal volume in comparison to typically developing children.

I was surprised to read this because a few years ago, I had attended a meeting on neuroanatomical studies of autism and had come away with the impression that there were few consistent findings. I did a quick search for an up-to-date review, which turned up a meta-analysis ( Traut et al., 2018 ), that included 16 MRI studies published between 1997 and 2010, five of which reported larger cerebellar size in autism and one of which found smaller cerebellar size. In the article I was reading, one paper had been cited to support the MRI statement, but it referred to a study where the absolute size of the vermis did not differ from typically developing children but was relatively small in the autistic participants, after the overall (larger) size of the cerebellum had been controlled for.

Other papers cited to support the claims of cerebellar neuropathology included a couple of early post mortem neuroanatomical studies, as well as two reviews. The first of these ( DiCicco-Bloom et al., 2006 ) summarised presentations from a conference and supported the claims made by the authors. The other one, however ( Palmen, van Engeland, Hof, & Schmitz, 2004 ), expressed more uncertainty and noted a lack of correspondence between early neuroanatomical studies and subsequent MRI findings, concluding,

Although some consistent results emerge, the majority of the neuropathological data remain equivocal. This may be due to lack of statistical power, resulting from small sample sizes and from the heterogeneity of the disorder itself, to the inability to control for potential confounding variables such as gender, mental retardation, epilepsy and medication status, and, importantly, to the lack of consistent design in histopathological quantitative studies of autism published to date.

In sum, a confident statement “cerebellar neuropathology commonly occurs in autistic individuals,” accompanied by a set of references, converged to give the impression that there is consensus that the cerebellum is involved in autism. However, when we drill down, we find that the evidence is uncertain, with discrepancies between neuropathological studies and MRI and methodological concerns about the former. Meanwhile, this study forms part of a large body of research in which genetically modified mice with cerebellar dysfunction are used as an animal model of autism. My impression is that few of the researchers using these mouse models appreciate that the claim of cerebellar abnormality in autism is controversial among those working with humans because each paper builds on the prior literature. There is entrenchment of error, for two reasons. First, many researchers will take at face value the summary of previous work in a peer-reviewed paper, without going back to original cited sources. Second, even if a researcher is careful and scholarly and does read the cited work, they are unlikely to find relevant studies that were not included in the literature review.

It is easy to take an example like this and bemoan the lack of rigour in scientific writing, but this is to discount cognitive biases that make it inevitable that, unless we adopt specific safeguards against this, cherry-picking of evidence will be the norm. Three biases lead us in this direction: confirmation bias, moral asymmetry, and reliance on schemata.

Confirmation bias: cherry-picking prior literature

A personal example may serve to illustrate the way confirmation bias can operate subconsciously. I am interested in genetic effects on children’s language problems, and I was in the habit of citing three relevant twin studies when I gave talks on this topic. All these obtained similar results, namely that there was a strong genetic component to developmental language disorders, as evidenced by much higher concordance for disorder in pairs of monozygotic versus dizygotic twins. In 2005 , however, Hayiou-Thomas, Oliver, and Plomin published a twin study with very different findings, with low twin/co-twin concordance, regardless of zygosity. It was only when I came to write a review of this area and I checked the literature that I realised I had failed to mention the 2005 study in talks for a year or two, even though I had collaborated with the authors and was well aware of the findings. I had formed a clear view on heritability of language disorders, and so I had difficulty remembering results that did not agree. Subsequently, I realised we should try to understand why this study obtained different results and found a plausible explanation ( Bishop & Hayiou-Thomas, 2008 ). But I only went back for a further critical look at the study because I needed to make sense of the conflicting results. It is inevitable that we behave this way as we try to find generalisable results from a body of work, but it creates an asymmetry of attention and focus between work that we readily accept, because it fits, and work that is either forgotten or looked at more critically, because it does not.

A particularly rich analysis of citation bias comes from a case study by Greenberg (2009) , who took as his starting point papers concerned with claims that a protein, β amyloid, was involved in causing a specific form of muscle disease. Greenberg classified papers according to whether they were positive, negative, or neutral about this claim and carried out a network analysis to identify influential papers (those with many citations). He found that papers that were critical of the claim received far fewer citations than those that supported it, and this was not explained by lower quality. Animal model studies were almost exclusively justified by selective citation of positive studies. Consistent with the idea of “reconstructive remembering,” he also found instances where cited content was distorted, as well as cases where influential review papers amplified citation bias by focusing attention only on positive work. The net result was an information (perhaps better termed a disinformation) cascade that would lead to a lack of awareness of critical data, which never gets recognised. In effect, when we have agents that adopt Bayesian reasoning, if they are presented with distorted information, this creates a positive feedback loop that leads to increasing bias in the prior. Viewed this way, we can start to see how omission of relevant citations is not a minor peccadillo but a serious contributor to entrenchment of error. Further evidence of the cumulative impact of citation bias is shown in Figure 4 , which uses studies of intervention for depression. Because studies in this area are registered, it is possible to track the fate of unpublished as well as published studies. The researchers showed that studies with null results are far less likely to be published than those with positive findings, but even if the former are published, there is a bias against citing them.

The cumulative impact of reporting and citation biases on the evidence base for antidepressants. (a) Displays the initial, complete cohort of trials that were recorded in a registry, while (b) through (e) show the cumulative effect of biases. Each circle indicates a trial, while the colour indicates whether results were positive or negative or were reported to give a misleadingly positive impression(spin). Circles connected by a grey line indicate trials from the same publication. The progression from (a) to (b) shows that nearly all the positive trials but only half of those with null results were published, and reporting of null studies showed (c) bias or (d) spin in what was reported. In (e), the size of the circle indicates the (relative) number of citations received by that category of studies.

Source. Reprinted with permission from De Vries et al. (2018) .

While describing such cases of citation bias, it is worth pausing to consider one of the best-known examples of distorted thinking: experimenter bias. This is similar to confirmation bias, but rather than involving selective attention to specific aspects of a situation that fits with our preconceptions, it has a more active character, whereby the experimenter can unwittingly influence the outcome of a study. The best-known research on this topic was the original Rosenthal and Fode (1963) study, where students were informed that the rats they were studying were “maze-bright” or “maze-dull,” when in fact they did not differ. Nevertheless, the “maze-bright” group learned better, suggesting that the experimenter would try harder to train an animal thought to have potential. A related study by Rosenthal and Jacobson (1963) claimed that if teachers were told that a test had revealed that specific pupils were “ready to bloom,” they would do better on an IQ test administered at the end of the year, even though the children so designated were selected at random.

Both these studies are widely cited. It is less well known that work on experimenter bias was subjected to a scathing critique by Barber and Silver (1968) , entitled “Fact, fiction and the experimenter bias effect,” in which it was noted that work in this area suffered from poor methodological quality, in particular p -hacking. Barber and Silver did not deny that experimenter bias could affect results, but they concluded that these effects were far less common and smaller in magnitude than those implied by Rosenthal’s early work. Subsequently, Barber (1976) developed this critique further in his book Pitfalls in Human Research. Yet Rosenthal’s work is more highly cited and better remembered than that of Barber.

Rosenthal’s work provides a cautionary tale: although confirmation bias helps explain distorted patterns of citation, the evidence for maladaptive cognitive biases has been exaggerated. Furthermore, studies on confirmation bias often use artificial experiments, divorced from real life, and the criteria for deciding that reasoning is erroneous are often poorly justified ( Hahn & Harris, 2014 ). In future, it would be worthwhile doing more naturalistic explorations of people’s memory for studies that do and do not support a position when summarising scientific literature.

On a related point, in using confirmation bias as an explanation for persistence of weak theories, there is a danger that I am falling into exactly the trap that I am describing. For instance, I was delighted to find Greenberg’s (2009) paper, as it chimed very well with my experiences when reading papers about cerebellar deficits in autism. But would I have described and cited it here if it had shown no difference between citations for papers that did and did not support the β amyloid claim? Almost certainly not. Am I going to read all literature on citation bias to find out how common it is? That strategy would soon become impossible if I tried to do it for every idea I touch upon in this article.

Moral asymmetry between errors of omission and commission

The second bias that fortifies the distortions in a literature review is the asymmetry of moral judgement that I referred to when discussing p -hacking. To my knowledge, paltering has not been studied in the context of literature reviews, but my impression is that selective presentation of results that fit, while failing to mention important contextual factors (e.g., the vermis in those with autism is smaller but only when you have covaried for the total cerebellar size), is common. How far this is deliberate or due to reconstructive remembering, however, is impossible to establish.

It would also be of interest to conduct studies on people’s attitudes to the acceptability of cherry-picking of literature versus paltering (misleadingly selective reporting) or invention of a study. I would anticipate that most would regard cherry-picking as fairly innocuous, for several reasons: first, it could be an unintended omission; second, the consequences of omitting material from a review may be seen as less severe than introducing misinformation; and third, selective citation of papers that fit a narrative can have a positive benefit in terms of readability. There are also pragmatic concerns: some journals limit the word count for an introduction or reference list so that full citation of all relevant work is not possible and, finally, sanctioning people for harmful omissions would create apparently unlimited obligations ( Haidt & Baron, 1996 ). Quite simply, there is far too much literature for even the most diligent scholar to read.

Nevertheless, consequences of omission can be severe. The above examples of research on the serotonin transporter gene in depression, or cerebellar abnormality in autism, emphasise how failure to cite earlier null results can lead to a misplaced sense of confidence in a phenomenon, which is wasteful in time and money when others attempt to build on it. And the more we encounter a claim, the more likely it is to be judged as true, regardless of actual accuracy (see Pennycook, Cannon, & Rand, 2018 , for a topical example). As Ingelfinger (1976) put it, “faulty or inappropriate references . . . like weeds, tend to reproduce themselves and so permit even the weakest of allegations to acquire, with repeated citation, the guise of factuality” (p. 1076).

Reliance on schemata

Our brains cannot conceivably process all the information around us: we have to find a way to select what is important to function and survive. This involves a search for meaningful patterns, which once established, allow us to focus on what is relevant and ignore the rest. Scientific discovery may be seen as an elevated version of pattern discovery: we see the height of scientific achievement as discovering regularities in nature that allow us to make better predictions about how the world behaves and to create new technologies and interventions from the basic principles we have discovered.

Scientific progress is not a simple process of weighing up competing pieces of evidence in relation to a theory. Rather than simply choosing between one hypothesis and another, we try to understand a problem in terms of a schema. Bartlett (1932) was one of the first psychologists to study how our preconceptions, or schemata, create distortions in perception and memory. He introduced the idea of “reconstructive remembering,” demonstrating how people’s memory of a narrative changed over time in specific ways, to become more coherent and aligned with pre-existing schemata.

Bartlett’s (1932) work on reconstructive remembering can explain why we not only tend to ignore inconsistent evidence ( Duyx, Urlings, Swaen, Bouter, & Zeegers, 2017 ) but also are prone to distort the evidence that we do include ( Vicente & Brewer, 1993 ). If we put together the combined influence of confirmation bias and reconstructive remembering, it suggests that narrative literature reviews have a high probability of being inaccurate: both types of bias will lead to a picture of research converging on a compelling story, when the reality may be far less tidy ( Katz, 2013 ).

I have focused so far on bias in citing prior literature, but schemata also influence how researchers go about writing up results. If we just were to present a set of facts that did not cohere, our work would be difficult to understand and remember. As Chalmers, Hedges, and Cooper (2002) noted, this point was made in 1885 by Lord Raleigh in a presidential address to the British Association for the Advancement of Science:

If, as is sometimes supposed, science consisted in nothing but the laborious accumulation of facts, it would soon come to a standstill, crushed, as it were, under its own weight. The suggestion of a new idea, or the detection of a law, supersedes much that has previously been a burden on the memory, and by introducing order and coherence facilitates the retention of the remainder in an available form. ( Rayleigh, 1885 , p. 20)

Indeed, when we write up our research, we are exhorted to “tell a story,” which achieves the “order and coherence” that Rayleigh described. Since his time, ample literature on narrative comprehension has confirmed that people fill in gaps in unstated information and find texts easier to comprehend and memorise when they fit a familiar narrative structure ( Bower & Morrow, 1990 ; Van den Broek, 1994 ).

This resonates with Dawkins’ ( 1976 ) criteria for a meme, i.e., an idea that persists by being transmitted from person to person. Memes need to be easy to remember, understand, and communicate, and so narrative accounts make far better memes than dry lists of facts. From this perspective, narrative serves a useful function in providing a scaffolding that facilitates communication. However, while this is generally a useful, and indeed essential, aspect of human cognition, in scientific communication, it can lead to propagation of false information. Bartlett (1932) noted that remembering is hardly ever really exact, “and it is not at all important that it should be so.” He was thinking of the beneficial aspects of schemata, in allowing us to avoid information overload and to focus on what is meaningful. However, as Dawkins emphasised, survival of a meme does not depend on it being useful or true. An idea such as the claim that vaccination causes autism is a very effective meme, but it has led to resurgence of diseases that were close to being eradicated.

In communicating scientific results, we need to strike a fine balance between presenting a precis of findings that is easily communicated and moving towards an increase in knowledge. I would argue the pendulum may have swung too far in the direction of encouraging researchers to tell good narratives. Not just media outlets, but also many journal editors and reviewers, encourage authors to tell simple stories that are easy to understand, and those who can produce these may be rewarded with funding and promotion.

The clearest illustration of narrative supplanting accurate reporting comes from the widespread use of HARKing, which was encouraged by Bem (2004) when he wrote,

There are two possible articles you can write: (a) the article you planned to write when you designed your study or (b) the article that makes the most sense now that you have seen the results. They are rarely the same, and the correct answer is (b).

Of course, formulating a hypothesis on the basis of observed data is a key part of the scientific process. However, as noted above, it is not acceptable to use the same data to both formulate and test the hypothesis—replication in a new sample is needed to avoid being misled by the play of chance and littering literature with false positives ( Lazic, 2016 ; Wagenmakers et al., 2012 ).

Kerr (1998) considered why HARKing is a successful strategy and pointed out that it allowed the researcher to construct an account of an experiment that fits a good story script:

Positing a theory serves as an effective “initiating event.” It gives certain events significance and justifies the investigators’ subsequent purposeful activities directed at the goal of testing the hypotheses. And, when one HARKs, a “happy ending” (i.e., confirmation) is guaranteed. (p. 203)

In this regard, Bem’s advice makes perfect sense: “A journal article tells a straightforward tale of a circumscribed problem in search of a solution. It is not a novel with subplots, flashbacks, and literary allusions, but a short story with a single linear narrative line.”

We have, then, a serious tension in scientific writing. We are expected to be scholarly and honest, to report all our data and analyses and not to hide inconvenient truths. At the same time, if we want people to understand and remember our work, we should tell a coherent story from which unnecessary details have been expunged and where we cut out any part of the narrative that distracts from the main conclusions.

Kerr (1998) was clear that HARKing has serious costs. As well as translating type I errors into hard-to-eradicate theory, he noted that it presents a distorted view of science as a process which is far less difficult and unpredictable than is really the case. We never learn what did not work because inconvenient results are suppressed. For early career researchers, it can lead to cynicism when they learn that the rosy picture portrayed in the literature was achieved only by misrepresentation.

Overcoming cognitive constraints to do better science

One thing that is clear from this overview is that we have known about cognitive constraints for decades, yet they continue to affect scientific research. Finding ways to mitigate the impact of these constraints should be a high priority for experimental psychologists. Here, I draw together some general approaches that might be used to devise an agenda for research improvement. Many of these ideas have been suggested before but without much consideration of cognitive constraints that may affect their implementation. Some methods, such as training, attempt to overcome the constraints directly in individuals: others involve making structural changes to how science is done to counteract our human tendency towards unscientific thinking. None of these provides a total solution: rather, the goal is to tweak the dials that dictate how people think and behave, to move us closer to better scientific practices.

It is often suggested that better training is needed to improve replicability of scientific results, yet the focus tends to be on formal instruction in experimental design and statistics. Less attention has been given to engendering a more intuitive understanding of probability, or counteracting cognitive biases, though there are exceptions, such as the course by Steel, Liermann, and Guttorp (2018) , which starts with a consideration of “How the wiring of the human brain leads to incorrect conclusions from data.” One way of inducing a more intuitive sense of statistics and p -values is by using data simulations. Simulation is not routinely incorporated in statistics training, but free statistical software now makes this within the grasp of all ( Tintle et al., 2015 ). This is a powerful way to experience how easy it is to get a “significant” p -value when running multiple tests. Students are often surprised when they generate repeated runs of a correlation matrix of random numbers with, say, five variables and find at least one “significant” correlation in about one in four runs. Once you understand that there is a difference between the probability associated with getting a specific result on a single test, predicted in advance, versus the probability of that result coming up at least once in a multitude of tests, then the dangers of p -hacking become easier to grasp.

Data simulation could also help overcome the misplaced “belief in the law of small numbers” ( Tversky & Kahneman, 1974 ). By generating datasets with a known effect size, and then taking samples from these and subjecting them to statistical test, the student can learn to appreciate just how easy it is to miss a true effect (type II error) if the study is underpowered.

There is small literature evaluating attempts to specifically inoculate people against certain types of cognitive bias. For instance, Morewedge et al. (2015) developed instructional videos and computer games designed to reduce a series of cognitive biases, including confirmation bias, and found these to be effective over the longer term. Typically, however, such interventions focus on hypothetical scenarios outside the scope of experimental psychology. They might improve scientific quality of research projects if adjusted to make them relevant to conducting and appraising experiments.

Triangulation of methods in study design

I noted above that for science to progress, we need to overcome a tendency to settle on the first theory that seems “good enough” to account for observations. Any method that forces the researcher to actively search for alternative explanations is, therefore, likely to stimulate better research.

The notion of triangulation ( Munafò & Davey Smith, 2018 ) was developed in the field of epidemiology, where reliance is primarily on observational data, and experimental manipulation is not feasible. Inferring causality from correlational data is hazardous, but it is possible to adopt a strategic approach of combining complementary approaches to analysis, each of which has different assumptions, strengths, and weaknesses. Epidemiology progresses when different explanations for correlational data are explicitly identified and evaluated, and converging evidence is obtained ( Lawlor, Tilling, & Davey Smith, 2016 ). This approach could be extended to other disciplines, by explicitly requiring researchers to use at least two different methods with different potential biases when evaluating a specific hypothesis.

A “culture of criticism”

Smith (2006) described peer review as “a flawed process, full of easily identified defects with little evidence that it works” (p. 182). Yet peer review provides one way of forcing researchers to recognise when they are so focused on a favoured theory that they are unable to break away. Hossenfelder (2018) has argued that the field of particle physics has stagnated because of a reluctance to abandon theories that are deemed “beautiful.” We are accustomed to regarding physicists as superior to psychologists in terms of theoretical and methodological sophistication. In general, they place far less emphasis than we do on statistical criteria for evidence, and where they do use statistics, they understand probability theory and adopt very stringent levels of significance. Nevertheless, according to Hossenfelder, they are subject to cognitive and social biases that make them reluctant to discard theories. She concludes her book with an Appendix on “What you can do to help,” and as well as advocating better understanding of cognitive biases, she recommends some cultural changes to address these. These include building “a culture of criticism.” In principle, we already have this—talks and seminars should provide a forum for research to be challenged—but in practice, critiquing another’s work is often seen as clashing with social conventions of being supportive to others, especially when it is conducted in public.

Recently, two other approaches have been developed, with the potential to make a “culture of criticism” more useful and more socially acceptable. Registered Reports ( Chambers, 2019 ) is an approach that was devised to prevent publication bias, p -hacking, and HARKing. This format moves the peer review process to a point before data collection so that results cannot influence editorial decisions. An unexpected positive consequence is that peer review comes at a point when it can be acted upon to improve the experimental design. Where reviewers of Registered Reports ask “how could we disprove the hypothesis?” and “what other explanations should we consider?” this can generate more informative experiments.

A related idea is borrowed from business practices and is known as the “pre mortem” approach ( Klein, 2007 ). Project developers gather together and are asked to imagine that a proposed project has gone ahead and failed. They are then encouraged to write down reasons why this has happened, allowing people to voice misgivings that they may have been reluctant to state openly, so they can be addressed before the project has begun. It would be worth evaluating the effectiveness of pre-mortems for scientific projects. We could strengthen this approach by incorporating ideas from Bang and Frith (2017) , who noted that group decision-making is most likely to be effective when the group is diverse and people can express their views anonymously. With both Registered Reports and the study pre-mortem, reviewers can have a role as critical friends who can encourage researchers to identify ways to improve a project before it is conducted. This can be a more positive experience for the reviewer, who may otherwise have no option but to recommend rejection of a study with flawed methodology.

Counteracting cherry-picking of literature

Turning to cherry-picking of prior literature, the established solution is the systematic review, where clear criteria are laid out in advance so that a comprehensive search can be made of all relevant studies ( Siddaway, Wood, & Hedges, 2019 ). The systematic review is only as good as the data that go into it, however, and if a field suffers from substantial publication bias and/or p -hacking, then, rather than tackling error entrenchment, it may add to it. With the most scrupulous search strategy, relevant papers with null results can be missed because positive results are mentioned in titles and abstracts of papers, whereas null results are not ( Lazic, 2016 , p. 15). This can mean that, if a study is looking at many possible associations (e.g., with brain regions or with genes), studies that considered a specific association but failed to find support for it will be systematically disregarded. This may explain why it seems to take 30 or 40 years for some erroneous entrenched theories to be abandoned. The situation may improve with increasing availability of open data. Provided data are adequately documented and accessible, the problem of missing relevant studies may be reduced.

Ultimately, the problem of biased reviews may not be soluble just by changing people’s citation habits. Journal editors and reviewers could insist that abstracts follow a structured format and report all variables that were tested, not just those that gave significant results. A more radical approach by funders may be needed to disrupt this wasteful cycle. When a research team applies to test a new idea, they could first be required to (a) conduct a systematic review (unless one has been recently done) and (b) replicate the original findings on which the work is based: this is the opposite to what happens currently, where novelty and originality are major criteria for funding. In addition, it could be made mandatory for any newly funded research idea to be investigated by at least two independent laboratories and using at least two different approaches (triangulation). All these measures would drastically slow down science and may be unfeasible where research needs highly specialised equipment, facilities, or skills that are specific to one laboratory. Nevertheless, slower science may be preferable to the current system where there are so many examples of false leads being pursued for decades, with consequent waste of resources.

Reconciling storytelling with honesty

Perhaps the hardest problem is how to reconcile our need for narrative with a “warts and all” account of research. Consider this advice from Bem (2004) —which I suspect many journal editors would endorse:

Contrary to the conventional wisdom, science does not care how clever or clairvoyant you were at guessing your results ahead of time. Scientific integrity does not require you to lead your readers through all your wrongheaded hunches only to show—voila!—they were wrongheaded. A journal article should not be a personal history of your stillborn thoughts . . . Your overriding purpose is to tell the world what you have learned from your study. If your results suggest a compelling framework for their presentation, adopt it and make the most instructive findings your centerpiece . . . Think of your dataset as a jewel. Your task is to cut and polish it, to select the facets to highlight, and to craft the best setting for it.

As Kerr (1998) pointed out, HARKing gives a misleading impression of what was found, which can be particularly damaging for students, who on reading literature may form the impression that it is normal for scientists to have their predictions confirmed and think of themselves as incompetent when their own experiments do not work out that way. One of the goals of pre-registration is to ensure that researchers do not omit inconvenient facts when writing up a study—or if they do, at least make it possible to see that this has been done. In the field of clinical medicine, impressive progress has been made in methodology, with registration now a requirement for clinical trials ( International Committee of Medical Journal Editors, 2019 ). Yet, Goldacre et al. (2019) found that even when a trial was registered, it was common for researchers to change the primary outcome measure without explanation, and it has been similarly noted that pre-registrations in psychology are often too ambiguous to preclude p -hacking ( Veldkamp et al., 2018 ). Registered Reports ( Chambers, 2019 ) adopt stricter standards that should prevent HARKing, but the author may struggle to maintain a strong narrative because messy reality makes a less compelling story than a set of results subjected to Bem’s (2004) cutting and polishing process.

Rewarding credible research practices

A final set of recommendations has to do with changing the culture so that incentives are aligned with efforts to counteract unhelpful cognitive constraints, and researchers are rewarded for doing reproducible, replicable research, rather than for grant income or publications in high-impact journals ( Forstmeier, Wagenmakers, & Parker, 2016 ; Pulverer, 2015 ). There is already evidence that funders are concerned to address problems with credibility of biomedical research ( Academy of Medical Sciences, 2015 ), and rigour and reproducibility are increasingly mentioned in grant guidelines (e.g., https://grants.nih.gov/policy/reproducibility/index.htm ). One funder, Cancer Research UK, is innovating by incorporating Registered Reports in a two-stage funding model ( Munafò, 2017 ). We now need publishers and institutions to follow suit and ensure that researchers are not disadvantaged by adopting a self-critical mind-set and engaging in practices of open and reproducible science ( Poldrack, 2019 ).

Acknowledgments

My thanks to Kate Nation, Matt Jaquiery, Joe Chislett, Laura Fortunato, Uta Frith, Stefan Lewandowsky, and Karalyn Patterson for invaluable comments on an early draft of this manuscript.

Declaration of conflicting interests: The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding: The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The author is supported by a Principal Research Fellowship from the Wellcome Trust (programme grant no. 082498) and European Research Council advanced grant no. 694189.