how to conduct literature review in nursing

Searching the ‘grey literature’ is also an important factor in reducing publication bias. It is often the case that only studies with positive results and statistical significance are published. This creates a certain bias inherent in the published literature. This bias can, to some degree, be mitigated by the inclusion of results from the so-called grey literature, including unpublished work, abstracts, conference proceedings and PhD theses ( Higgins and Green, 2011 ; Bettany-Saltikov, 2012 ; Cognetti et al, 2015 ). Biases in a systematic review can lead to overestimating or underestimating the results ( Jahan et al, 2016 ).

An example search strategy from a published review looking at web use for the appraisal of physical health conditions can be seen in Box 2 . High-quality reviews usually detail which databases were searched and the number of items retrieved from each.

((web OR Internet OR “search engine” OR google OR online OR on line’’) AND (“help seeking” OR “help-seeking” OR “information seeking” OR “information-seeking”) AND (symptom OR symptoms OR diagnoses OR diagnosis))

A balance between high recall and high precision is often required in order to produce the best results. An oversensitive search, or one prone to including too much noise, can mean missing important studies or producing too many search results ( Cognetti et al, 2015 ). Following a search, the exported citations can be added to citation management software (such as Mendeley or Endnote) and duplicates removed.

Title and abstract screening

Initial screening begins with the title and abstracts of articles being read and included or excluded from the review based on their relevance. This is usually carried out by at least two researchers to reduce bias ( Bashir and Conlon, 2018 ). After screening any discrepancies in agreement should be resolved by discussion, or by an additional researcher casting the deciding vote ( Bashir and Conlon, 2018 ). Statistics for inter-rater reliability exist and can be reported, such as percentage of agreement or Cohen's kappa ( Box 3 ) for two reviewers and Fleiss' kappa for more than two reviewers. Agreement can depend on the background and knowledge of the researchers and the clarity of the inclusion and exclusion criteria. This highlights the importance of providing clear, well-defined criteria for inclusion that are easy for other researchers to follow.

Full-text review

Following title and abstract screening, the remaining articles/sources are screened in the same way, but this time the full texts are read in their entirety and included or excluded based on their relevance. Reasons for exclusion are usually recorded and reported. Extraction of the specific details of the studies can begin once the final set of papers is determined.

Data extraction

At this stage, the full-text papers are read and compared against the inclusion criteria of the review. Data extraction sheets are forms that are created to extract specific data about a study (12 Jahan et al, 2016 ) and ensure that data are extracted in a uniform and structured manner. Extraction sheets can differ between quantitative and qualitative reviews. For quantitative reviews they normally include details of the study's population, design, sample size, intervention, comparisons and outcomes ( Bettany-Saltikov, 2012 ; Mueller et al, 2017 ).

Quality appraisal

The quality of the studies used in the review should also be appraised. Caldwell et al (2005) discussed the need for a health research evaluation framework that could be used to evaluate both qualitative and quantitative work. The framework produced uses features common to both research methodologies, as well as those that differ ( Caldwell et al, 2005 ; Dixon-Woods et al, 2006 ). Figure 3 details the research critique framework. Other quality appraisal methods do exist, such as those presented in Box 4 . Quality appraisal can also be used to weight the evidence from studies. For example, more emphasis can be placed on the results of large randomised controlled trials (RCT) than one with a small sample size. The quality of a review can also be used as a factor for exclusion and can be specified in inclusion/exclusion criteria. Quality appraisal is an important step that needs to be undertaken before conclusions about the body of evidence can be made ( Sambunjak and Franic, 2012 ). It is also important to note that there is a difference between the quality of the research carried out in the studies and the quality of how those studies were reported ( Sambunjak and Franic, 2012 ).

The quality appraisal is different for qualitative and quantitative studies. With quantitative studies this usually focuses on their internal and external validity, such as how well the study has been designed and analysed, and the generalisability of its findings. Qualitative work, on the other hand, is often evaluated in terms of trustworthiness and authenticity, as well as how transferable the findings may be ( Bettany-Saltikov, 2012 ; Bashir and Conlon, 2018 ; Siddaway et al, 2019 ).

Reporting a review (the PRISMA statement)

The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) provides a reporting structure for systematic reviews/meta-analysis, and consists of a checklist and diagram ( Figure 4 ). The stages of identifying potential papers/sources, screening by title and abstract, determining eligibility and final inclusion are detailed with the number of articles included/excluded at each stage. PRISMA diagrams are often included in systematic reviews to detail the number of papers included at each of the four main stages (identification, screening, eligibility and inclusion) of the review.

Data synthesis

The combined results of the screened studies can be analysed qualitatively by grouping them together under themes and subthemes, often referred to as meta-synthesis or meta-ethnography ( Siddaway et al, 2019 ). Sometimes this is not done and a summary of the literature found is presented instead. When the findings are synthesised, they are usually grouped into themes that were derived by noting commonality among the studies included. Inductive (bottom-up) thematic analysis is frequently used for such purposes and works by identifying themes (essentially repeating patterns) in the data, and can include a set of higher-level and related subthemes (Braun and Clarke, 2012). Thomas and Harden (2008) provide examples of the use of thematic synthesis in systematic reviews, and there is an excellent introduction to thematic analysis by Braun and Clarke (2012).

The results of the review should contain details on the search strategy used (including search terms), the databases searched (and the number of items retrieved), summaries of the studies included and an overall synthesis of the results ( Bettany-Saltikov, 2012 ). Finally, conclusions should be made about the results and the limitations of the studies included ( Jahan et al, 2016 ). Another method for synthesising data in a systematic review is a meta-analysis.

Limitations of systematic reviews

Apart from the many advantages and benefits to carrying out systematic reviews highlighted throughout this article, there remain a number of disadvantages. These include the fact that not all stages of the review process are followed rigorously or even at all in some cases. This can lead to poor quality reviews that are difficult or impossible to replicate. There also exist some barriers to the use of evidence produced by reviews, including ( Wallace et al, 2012 ):

• Lack of awareness and familiarity with reviews
• Lack of access
• Lack of direct usefulness/applicability.

Meta-analysis

When the methods used and the analysis are similar or the same, such as in some RCTs, the results can be synthesised using a statistical approach called meta-analysis and presented using summary visualisations such as forest plots (or blobbograms) ( Figure 5 ). This can be done only if the results can be combined in a meaningful way.

Meta-analysis can be carried out using common statistical and data science software, such as the cross-platform ‘R’ ( https://www.r-project.org), or by using standalone software, such as Review Manager (RevMan) produced by the Cochrane community ( https://tinyurl.com/revman-5), which is currently developing a cross-platform version RevMan Web.

Carrying out a systematic review is a time-consuming process, that on average takes between 6 and 18 months and requires skill from those involved. Ideally, several reviewers will work on a review to reduce bias. Experts such as librarians should be consulted and included where possible in review teams to leverage their expertise.

Systematic reviews should present the state of the art (most recent/up-to-date developments) concerning a specific topic and aim to be systematic and reproducible. Reproducibility is aided by transparent reporting of the various stages of a review using reporting frameworks such as PRISMA for standardisation. A high-quality review should present a summary of a specific topic to a high standard upon which other professionals can base subsequent care decisions that increase the quality of evidence-based clinical practice.

• Systematic reviews remain one of the most trusted sources of high-quality information from which to make clinical decisions
• Understanding the components of a review will help practitioners to better assess their quality
• Many formal frameworks exist to help structure and report reviews, the use of which is recommended for reproducibility
• Experts such as librarians can be included in the review team to help with the review process and improve its quality

CPD reflective questions

• Where should high-quality qualitative research sit regarding the hierarchies of evidence?
• What background and expertise should those conducting a systematic review have, and who should ideally be included in the team?
• Consider to what extent inter-rater agreement is important in the screening process

NUR 288: Nursing Concepts IV

• Library Research Day

Literature Review Research

What is a literature review, how to complete a literature review, how to write a literature review.

• Peer Reviewed
• Research Databases
• Web Searching
• Evaluating Health Information
• APA Citing Sources
• DOLS Virtual Poster 2021

Researching Instruction will cover: 

Discussion about what is a literature review and how to complete a literature review. 

Use of Smart Search and Google Scholar

A literature review is an account of what has been published on a topic by accredited scholars and researchers. Occasionally you will be asked to write one as a separate assignment (sometimes in the form of an annotated bibliography —see the bottom of the next page), but more often it is part of the introduction to an essay, research report, or thesis. In writing the literature review, your purpose is to convey to your reader what knowledge and ideas have been established on a topic, and what their strengths and weaknesses are.

From  Taylor, D. (n/a). The Literature Review: A Few Tips On Conducting It. University of Toronto, Health Sciences Writing Centre. 

• The Literature Review: A Few Tips On Conducting It Writing Advice from the University of Toronto, Canada.
• Record Title: Writing & Research. Writing a Literature Review. Neill, C. (2017). Writing & Research. Writing a Literature Review. Radiation Therapist, 26(1), 89–91.

There are several steps in developing a literature review.  These include:

• Define your paper’s goal
• Literature review will match paper’s goal
• Review articles related to your paper’s topic
• Articles are written by scholars
• Identify top scholars in the field about your topic
• Include most pertinent publications by those scholars
• Summarize articles you identified
• Provide the importance of the article as it relates to your thesis/project statement
• Establish its relevance to the discussion
• What where the earliest ideas on the?
• How did grow and evolve in the academic conversation?
• As you write you will include author and date
• Create comprehensive citation for each article
• Follow APA format
• Literature Reviews From Purdue OWL
• << Previous: Library Research Day
• Next: Peer Reviewed >>
• Last Updated: May 14, 2024 2:11 PM
• URL: https://libguides.wccnet.edu/NursingConceptsIV

Health (Nursing, Medicine, Allied Health)

• Find Articles/Databases
• Reference Resources
• Evidence Summaries & Clinical Guidelines
• Drug Information
• Health Data & Statistics
• Patient/Consumer Facing Materials
• Images and Streaming Video
• Grey Literature
• Mobile Apps & "Point of Care" Tools
• Tests & Measures This link opens in a new window
• Citing Sources
• Selecting Databases
• Framing Research Questions
• Crafting a Search
• Narrowing / Filtering a Search
• Expanding a Search
• Cited Reference Searching
• Saving Searches
• Term Glossary
• Critical Appraisal Resources
• What are Literature Reviews?
• Conducting & Reporting Systematic Reviews
• Finding Systematic Reviews
• Tutorials & Tools for Literature Reviews
• Finding Full Text

What are Systematic Reviews? (3 minutes, 24 second YouTube Video)

Systematic Literature Reviews: Steps & Resources

These steps for conducting a systematic literature review are listed below . 

Also see subpages for more information about:

• The different types of literature reviews, including systematic reviews and other evidence synthesis methods
• Tools & Tutorials

Literature Review & Systematic Review Steps

• Develop a Focused Question
• Scope the Literature  (Initial Search)
• Refine & Expand the Search
• Limit the Results
• Download Citations
• Abstract & Analyze
• Create Flow Diagram
• Synthesize & Report Results

1. Develop a Focused   Question 

Consider the PICO Format: Population/Problem, Intervention, Comparison, Outcome

Focus on defining the Population or Problem and Intervention (don't narrow by Comparison or Outcome just yet!)

"What are the effects of the Pilates method for patients with low back pain?"

Tools & Additional Resources:

• PICO Question Help
• Stillwell, Susan B., DNP, RN, CNE; Fineout-Overholt, Ellen, PhD, RN, FNAP, FAAN; Melnyk, Bernadette Mazurek, PhD, RN, CPNP/PMHNP, FNAP, FAAN; Williamson, Kathleen M., PhD, RN Evidence-Based Practice, Step by Step: Asking the Clinical Question, AJN The American Journal of Nursing : March 2010 - Volume 110 - Issue 3 - p 58-61 doi: 10.1097/01.NAJ.0000368959.11129.79

2. Scope the Literature

A "scoping search" investigates the breadth and/or depth of the initial question or may identify a gap in the literature. 

Eligible studies may be located by searching in:

• Background sources (books, point-of-care tools)
• Article databases
• Trial registries
• Grey literature
• Cited references
• Reference lists

When searching, if possible, translate terms to controlled vocabulary of the database. Use text word searching when necessary.

Use Boolean operators to connect search terms:

• Combine separate concepts with AND  (resulting in a narrower search)
• Connecting synonyms with OR  (resulting in an expanded search)

Search:  pilates AND ("low back pain"  OR  backache )

Video Tutorials - Translating PICO Questions into Search Queries

• Translate Your PICO Into a Search in PubMed (YouTube, Carrie Price, 5:11) 
• Translate Your PICO Into a Search in CINAHL (YouTube, Carrie Price, 4:56)

3. Refine & Expand Your Search

Expand your search strategy with synonymous search terms harvested from:

• database thesauri
• reference lists
• relevant studies

Example: 

(pilates OR exercise movement techniques) AND ("low back pain" OR backache* OR sciatica OR lumbago OR spondylosis)

As you develop a final, reproducible strategy for each database, save your strategies in a:

• a personal database account (e.g., MyNCBI for PubMed)
• Log in with your NYU credentials
• Open and "Make a Copy" to create your own tracker for your literature search strategies

4. Limit Your Results

Use database filters to limit your results based on your defined inclusion/exclusion criteria.  In addition to relying on the databases' categorical filters, you may also need to manually screen results.  

• Limit to Article type, e.g.,:  "randomized controlled trial" OR multicenter study
• Limit by publication years, age groups, language, etc.

NOTE: Many databases allow you to filter to "Full Text Only".  This filter is  not recommended . It excludes articles if their full text is not available in that particular database (CINAHL, PubMed, etc), but if the article is relevant, it is important that you are able to read its title and abstract, regardless of 'full text' status. The full text is likely to be accessible through another source (a different database, or Interlibrary Loan).  

• Filters in PubMed
• CINAHL Advanced Searching Tutorial

5. Download Citations

Selected citations and/or entire sets of search results can be downloaded from the database into a citation management tool. If you are conducting a systematic review that will require reporting according to PRISMA standards, a citation manager can help you keep track of the number of articles that came from each database, as well as the number of duplicate records.

In Zotero, you can create a Collection for the combined results set, and sub-collections for the results from each database you search.  You can then use Zotero's 'Duplicate Items" function to find and merge duplicate records.

• Citation Managers - General Guide

6. Abstract and Analyze

• Migrate citations to data collection/extraction tool
• Screen Title/Abstracts for inclusion/exclusion
• Screen and appraise full text for relevance, methods, 
• Resolve disagreements by consensus

Covidence is a web-based tool that enables you to work with a team to screen titles/abstracts and full text for inclusion in your review, as well as extract data from the included studies.

• Covidence Support
• Critical Appraisal Tools
• Data Extraction Tools

7. Create Flow Diagram

The PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses) flow diagram is a visual representation of the flow of records through different phases of a systematic review.  It depicts the number of records identified, included and excluded.  It is best used in conjunction with the PRISMA checklist .

Example from: Stotz, S. A., McNealy, K., Begay, R. L., DeSanto, K., Manson, S. M., & Moore, K. R. (2021). Multi-level diabetes prevention and treatment interventions for Native people in the USA and Canada: A scoping review. Current Diabetes Reports, 2 (11), 46. https://doi.org/10.1007/s11892-021-01414-3

• PRISMA Flow Diagram Generator (ShinyApp.io, Haddaway et al. )
• PRISMA Diagram Templates  (Word and PDF)
• Make a copy of the file to fill out the template
• Image can be downloaded as PDF, PNG, JPG, or SVG
• Covidence generates a PRISMA diagram that is automatically updated as records move through the review phases

8. Synthesize & Report Results

There are a number of reporting guideline available to guide the synthesis and reporting of results in systematic literature reviews.

It is common to organize findings in a matrix, also known as a Table of Evidence (ToE).

• Reporting Guidelines for Systematic Reviews
• Download a sample template of a health sciences review matrix  (GoogleSheets)

Steps modified from: 

Cook, D. A., & West, C. P. (2012). Conducting systematic reviews in medical education: a stepwise approach.   Medical Education , 46 (10), 943–952.

• << Previous: Critical Appraisal Resources
• Next: What are Literature Reviews? >>
• Last Updated: Sep 5, 2024 9:32 AM
• URL: https://guides.nyu.edu/health

MSU Libraries

• Need help? Ask Us

Nursing Literature Reviews

• Literature and Other Types of Reviews
• Starting Your Search
• Developing a Research Question and the Literature Search Process
• Conducting a Literature Search
• Levels of Evidence
• Creating a PRISMA Table
• Literature Table and Synthesis
• Other Resources

Introduction

There are many types of literature reviews, but all should follow a similar search process.  Below are a few types of literature reviews, as well as definitions and examples. Much of this information can be found in the article A Typology of Reviews: An Analysis of 14 Review Types and Associated Methodologies.

Additional information about types of reviews, including an updated list of 48 types of reviews can be found in the article Meeting the Review Family: Exploring Review Types and Associated Information Retrieval Techniques

Literature Review : This is a generic term that can cover a wide range of subjects, and varies in completeness and comprehensiveness. They are typically narrative, and analysis may be chronological, conceptual, thematic, or however the author decides to organize the material. Anesthesia Personnel's Experiences With Digital Anesthesia Information Management Systems: A Literature Review.

Scoping Review : A preliminary assessment of the size and scope of available published literature. A scoping review is intended to identify current research and the extent of such research, and determine if a more comprehensive review is viable. Can include research in progress, and the completeness of searching is determined by time/scope. Traumatic Subarachnoid Hemorrhage: A Scoping Review

Mapping Review: Looks at existing literature and maps out future directions and current gaps in the research literature. Search may be determined by time/scope. Classification of Mild Stroke: A Mapping Review

Rapid Review : Assessment of what is already known about a policy or practice issue. Uses systematic review methods to search and critically evaluate existing research, but search is limited by time constraints. The Current State Of Telehealth Evidence: A Rapid Review

State-of-the-art/state-of-the-literature review: Addresses current matters as opposed to other types of reviews that address retrospective and current approaches. Comprehensive searching of the literature, and looks for current state of knowledge and sets priorities for future investigation and research. Artificial Intelligence for the Otolaryngologist: A State of the Art Review

Integrative Review: Combines empirical and theoretical research to examine research on a given area. Includes non-experimental research, and can include case studies, observational studies, theories, guidelines, etc., and is generally used to inform healthcare policy and practice. An Integrative Review of Yoga and Mindfulness-Based Approaches for Children and Adolescents with Asthma

Systematic Review: Seeks to systematically search, appraise, and synthesize research evidence. Requires exhaustive, comprehensive searching, including searching of grey literature. The efficacy of rehabilitation in people with Guillain-Barrè syndrome: a systematic review of randomized controlled trials

Systematic Review and Meta-Analysis: Includes all of systematic review, but requires quantitative analysis for the meta-analysis piece. The Efficacy and Safety of Disease-Modifying Osteoarthritis Drugs for Knee and Hip Osteoarthritis—a Systematic Review and Network Meta-Analysis

Umbrella Review: Specifically refers to searching for reviews only-usually systematic reviews only. Should discuss what is known, unknown, and recommendations for future research. Coffee consumption and health: umbrella review of meta-analyses of multiple health outcomes

Which Type of Review Should I Do?

• Which Review Is Right For You? A great diagram and flow chart for picking the type of review that is right for you, based on scope, time, and research team.
• Which Review Is Right For You? A survey tool designed to help determine the most appropriate method for your review. It's not a prescriptive tool, but is intended to help identify options for your review.
• Next: Starting Your Search >>
• Last Updated: May 10, 2024 9:36 AM
• URL: https://libguides.lib.msu.edu/nursinglitreview

University Library

• Research Guides
• Literature Reviews
• Evidence-Based Practice
• Books & Media

What is a Literature Review?

Key questions for a literature review, examples of literature reviews, useful links, evidence matrix for literature reviews.

• Annotated Bibliographies

The Scholarly Conversation

A literature review provides an overview of previous research on a topic that critically evaluates, classifies, and compares what has already been published on a particular topic. It allows the author to synthesize and place into context the research and scholarly literature relevant to the topic. It helps map the different approaches to a given question and reveals patterns. It forms the foundation for the author’s subsequent research and justifies the significance of the new investigation.

A literature review can be a short introductory section of a research article or a report or policy paper that focuses on recent research. Or, in the case of dissertations, theses, and review articles, it can be an extensive review of all relevant research.

• The format is usually a bibliographic essay; sources are briefly cited within the body of the essay, with full bibliographic citations at the end.
• The introduction should define the topic and set the context for the literature review. It will include the author's perspective or point of view on the topic, how they have defined the scope of the topic (including what's not included), and how the review will be organized. It can point out overall trends, conflicts in methodology or conclusions, and gaps in the research.
• In the body of the review, the author should organize the research into major topics and subtopics. These groupings may be by subject, (e.g., globalization of clothing manufacturing), type of research (e.g., case studies), methodology (e.g., qualitative), genre, chronology, or other common characteristics. Within these groups, the author can then discuss the merits of each article and analyze and compare the importance of each article to similar ones.
• The conclusion will summarize the main findings, make clear how this review of the literature supports (or not) the research to follow, and may point the direction for further research.
• The list of references will include full citations for all of the items mentioned in the literature review.

A literature review should try to answer questions such as

• Who are the key researchers on this topic?
• What has been the focus of the research efforts so far and what is the current status?
• How have certain studies built on prior studies? Where are the connections? Are there new interpretations of the research?
• Have there been any controversies or debate about the research? Is there consensus? Are there any contradictions?
• Which areas have been identified as needing further research? Have any pathways been suggested?
• How will your topic uniquely contribute to this body of knowledge?
• Which methodologies have researchers used and which appear to be the most productive?
• What sources of information or data were identified that might be useful to you?
• How does your particular topic fit into the larger context of what has already been done?
• How has the research that has already been done help frame your current investigation ?

Example of a literature review at the beginning of an article: Forbes, C. C., Blanchard, C. M., Mummery, W. K., & Courneya, K. S. (2015, March). Prevalence and correlates of strength exercise among breast, prostate, and colorectal cancer survivors . Oncology Nursing Forum, 42(2), 118+. Retrieved from http://go.galegroup.com.sonoma.idm.oclc.org/ps/i.do?p=HRCA&sw=w&u=sonomacsu&v=2.1&it=r&id=GALE%7CA422059606&asid=27e45873fddc413ac1bebbc129f7649c Example of a comprehensive review of the literature: Wilson, J. L. (2016). An exploration of bullying behaviours in nursing: a review of the literature.   British Journal Of Nursing ,  25 (6), 303-306. For additional examples, see:

Galvan, J., Galvan, M., & ProQuest. (2017). Writing literature reviews: A guide for students of the social and behavioral sciences (Seventh ed.). [Electronic book]

Pan, M., & Lopez, M. (2008). Preparing literature reviews: Qualitative and quantitative approaches (3rd ed.). Glendale, CA: Pyrczak Pub. [ Q180.55.E9 P36 2008]

• Write a Literature Review (UCSC)
• Literature Reviews (Purdue)
• Literature Reviews: overview (UNC)
• Review of Literature (UW-Madison)

The  Evidence Matrix  can help you  organize your research  before writing your lit review.  Use it to  identify patterns  and commonalities in the articles you have found--similar methodologies ?  common  theoretical frameworks ? It helps you make sure that all your major concepts covered. It also helps you see how your research fits into the context  of the overall topic.

• Evidence Matrix Special thanks to Dr. Cindy Stearns, SSU Sociology Dept, for permission to use this Matrix as an example.
• << Previous: Misc
• Next: Annotated Bibliographies >>
• Last Updated: Sep 5, 2024 10:10 AM
• URL: https://libguides.sonoma.edu/nursing

Doctorate of Nursing Practice (DNP) Project Resources

What type of review, how to conduct a review, find articles, evaluating articles, full text access tool.

• How to Get Full Articles
• How to Read a Scientific Paper
• Matrix Template & Example
• Research Instruments
• Organize Your Research
• Statistics + Data Analysis
• Writing & Publishing

Researchers, academics and librarians all use various terms to describe different types of literature reviews. Indeed there is often inconsistency between the ways the types are discussed. Here are a couple of simple explanations.

The image below describes common review types in terms of speed, detail, risk of bias and comprehensiveness:

"Schematic of the main differences between the types of literature review" by Brennan, M. L., Arlt, S. P., Belshaw, Z., Buckley, L., Corah, L., Doit, H., Fajt, V. R., Grindlay, D., Moberly, H. K., Morrow, L. D., Stavisky, J., & White, C. (2020). Critically Appraised Topics (CATs) in veterinary medicine: Applying evidence in clinical practice. Frontiers in Veterinary Science, 7 , 314. https://doi.org/10.3389/fvets.2020.00314 is licensed under CC BY 3.0

The table below has been adapted from a widely used typology of fourteen types of reviews, (Grant & Booth, 2009).  Here are four of the most common types:

For a more detailed list of review types, see:

Grant, M.J. & Booth, A. (2009).  A typology of reviews: An analysis of 14 review types and associated methodologies. Health Information & Libraries Journal, 26 (2), 91-108.  DOI: 10.1111/j.1471-1842.2009.00848.x

The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) outlines standards of practice completing a systematic review to ensure consistency and high-quality results.

• PRISMA 2020 Explanations
• PRISMA 2020 Checklist
• PRISMA 2020 Flow Diagram
• PRISMA Flow Diagram Generator
• PRISMA-S: an extension to the PRISMA Statement for Reporting Literature Searches in Systematic Reviews

Techniques from the PRISMA process can also be used with other types of reviews to have a systematic process for searching and evaluating results.

• Article Evaluation Handout Key questions to evaluate each section of original research studies (RCTs, cohort studies, etc)

LibKey Nomad is a browser extension that connects you with articles that are library-licensed or open access. LibKey Nomad is available for Google Chrome, Firefox, Safari, Microsoft Edge, Brave, and Vivaldi.

Here is a short demonstration of how it works:

When using LibKey Nomad, you will automatically be directed to the best available version of an article so you don't need to search in multiple places. When the full text of an article is not available, you will be provided with options to request the article through  Interlibrary Loan (ILL).

LibKey Nomad does not require you to create a personal account. It does not track users or hold credentials, and it is only active when you are on the web page of a scholarly publisher or database.  

Getting Started

• Install the LibKey Nomad extension for your browser of choice. (Note: A user account is  not  required to use the extension.)
• Search for and select "James Madison University" from the list of institutions. See images below.
• When you are on publisher websites, Wikipedia, PubMed, and more, you can l ook for the LibKey Nomad button to download PDFs.

Publisher Websites

When you're browsing publisher sites and other web pages, LibKey Nomad provides a link to download the PDF of an article available through the JMU Libraries. It is not necessary to first connect through the JMU Libraries website. If the full text of the article is not available through the JMU Libraries subscriptions, an "Access Options" button will appear, and you will see options for requesting the article through ILL. See example in the image below.

LibKey Nomad also works with Wikipedia. When you're viewing the list of references for an entry, a "Download PDF" button will appear after the citations, if the full text of the article is available from JMU Libraries. An "Article Link" button may appear for citations that are from sources that do not support linking directly to the PDF. For articles not available from JMU Libraries, an "Access Options..." button will appear that provides options for requesting the articles through ILL. See example in the image below.

When you're searching PubMed, LibKey Nomad shows options for downloading a PDF, linking to an article, or additional access options. You can also view the cover images of the journals that articles appear in. LibKey Nomad also provides a link to the complete issue of a journal. Click "View Complete Issue" to browse the table of contents for the issue in which an article appears, helping you find similar articles from the same publication. See example in the image below.

• << Previous: Home
• Next: Research Instruments >>
• Last Updated: Jun 12, 2024 9:46 AM
• URL: https://guides.lib.jmu.edu/nursing/dnp

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• v.112(1); Jan-Feb 2015

Systematically Reviewing the Literature: Building the Evidence for Health Care Quality

There are important research and non-research reasons to systematically review the literature. This article describes a step-by-step process to systematically review the literature along with links to key resources. An example of a graduate program using systematic literature reviews to link research and quality improvement practices is also provided.

Introduction

Systematic reviews that summarize the available information on a topic are an important part of evidence-based health care. There are both research and non-research reasons for undertaking a literature review. It is important to systematically review the literature when one would like to justify the need for a study, to update personal knowledge and practice, to evaluate current practices, to develop and update guidelines for practice, and to develop work related policies. 1 A systematic review draws upon the best health services research principles and methods to address: What is the state of the evidence on the selected topic? The systematic process enables others to reproduce the methods and to make a rational determination of whether to accept the results of the review. An abundance of articles on systematic reviews exist focusing on different aspects of systematic reviews. 2 – 9 The purpose of this article is to describe a step by step process of systematically reviewing the health care literature and provide links to key resources.

Systematic Review Process: Six Key Steps

Six key steps to systematically review the literature are outlined in Table 1 and discussed here.

Systematic Review Steps

1. Formulate the Question and Refine the Topic

When preparing a topic to conduct a systematic review, it is important to ask at the outset, “What exactly am I looking for?” Hopefully it seems like an obvious step, but explicitly writing a one or two sentence statement of the topic before you begin to search is often overlooked. It is important for several reasons; in particular because, although we usually think we know what we are searching for, in truth our mental image of a topic is often quite fuzzy. The act of writing something concise and intelligible to a reader, even if you are the only one who will read it, clarifies your thoughts and can inspire you to ask key questions. In addition, in subsequent steps of the review process, when you begin to develop a strategy for searching the literature, your topic statement is the ready raw material from which you can extract the key concepts and terminology for your strategies. The medical and related health literature is massive, so the more precise and specific your understanding of your information need, the better your results will be when you search.

2. Search, Retrieve, and Select Relevant Articles

The retrieval tools chosen to search the literature should be determined by the purpose of the search. Questions to ask include: For what and by whom will the information be used? A topical expert or a novice? Am I looking for a simple fact? A comprehensive overview on the topic? Exploration of a new topic? A systematic review? For the purpose of a systematic review of journal research in the area of health care, PubMed or Medline is the most appropriate retrieval tool to start with, however other databases may be useful ( Table 2 ). In particular, Google Scholar allows one to search the same set of articles as PubMed/MEDLINE, in addition to some from other disciplines, but it lacks a number of key advanced search features that a skilled searcher can exploit in PubMed/MEDLINE.

Examples of Electronic Bibliographic Databases Specific to Health Care

Note: These databases may be available through university or hospital library systems.

An effective way to search the literature is to break the topic into different “building blocks.” The building blocks approach is the most systematic and works the best in periodical databases such as PubMed/MEDLINE. The “blocks” in a “building blocks” strategy consist of the key concepts in the search topic. For example, let’s say we are interested in researching about mobile phone-based interventions for monitoring of patient status or disease management. We could break the topic into the following concepts or blocks: 1. Mobile phones, 2. patient monitoring, and 3. Disease management. Gather synonyms and related terms to represent each concept and match to available subject headings in databases that offer them. Organize the resulting concepts into individual queries. Run the queries and examine your results to find relevant items and suggest query modifications to improve your results. Revise and re-run your strategy based on your observations. Repeat this process until you are satisfied or further modifications produce no improvements. For example in Medline, these terms would be used in this search and combined as follows: cellular phone AND (ambulatory monitoring OR disease management), where each of the key word phrases is an official subject heading in the MEDLINE vocabulary. Keep detailed notes on the literature search, as it will need to be reported in the methods section of the systematic review paper. Careful noting of search strategies also allows you to revisit a topic in the future and confidently replicate the same results, with the addition of those subsequently published on your topic.

3. Assess Quality

There is no consensus on the best way to assess study quality. Many quality assessment tools include issues such as: appropriateness of study design to the research objective, risk of bias, generalizability, statistical issues, quality of the intervention, and quality of reporting. Reporting guidelines for most literature types are available at the EQUATOR Network website ( http://www.equator-network.org/ ). These guidelines are a useful starting point; however they should not be used for assessing study quality.

4. Extract Data and Information

Extract information from each eligible article into a standardized format to permit the findings to be summarized. This will involve building one or more tables. When making tables each row should represent an article and each column a variable. Not all of the information that is extracted into the tables will end up in the paper. All of the information that is extracted from the eligible articles will help you obtain an overview of the topic, however you will want to reserve the use of tables in the literature review paper for the more complex information. All tables should be introduced and discussed in the narrative of the literature review. An example of an evidence summary table is presented in Table 3 .

Example of an evidence summary table

Notes: BP = blood pressure, HbA1c = Hemoglobin A1c, Hypo = hypoglycemic, I = Internet, NS = not significant, PDA = personal digital assistant, QOL = quality of life, SMBG = self-monitored blood glucose, SMS = short message service, V = voice

5. Analyze and Synthesize Data and information

The findings from individual studies are analyzed and synthesized so that the overall effectiveness of the intervention can be determined. It should also be observed at this time if the effect of an intervention is comparable in different studies, participants, and settings.

6. Write the Systematic Review

The PRISMA 12 and ENTREQ 13 checklists can be useful resources when writing a systematic review. These uniform reporting tools focus on how to write coherent and comprehensive reviews that facilitate readers and reviewers in evaluating the relative strengths and weaknesses. A systematic literature review has the same structure as an original research article:

TITLE : The systematic review title should indicate the content. The title should reflect the research question, however it should be a statement and not a question. The research question and the title should have similar key words.

STRUCTURED ABSTRACT: The structured abstract recaps the background, methods, results and conclusion in usually 250 words or less.

INTRODUCTION: The introduction summarizes the topic or problem and specifies the practical significance for the systematic review. The first paragraph or two of the paper should capture the attention of the reader. It might be dramatic, statistical, or descriptive, but above all, it should be interesting and very relevant to the research question. The topic or problem is linked with earlier research through previous attempts to solve the problem. Gaps in the literature regarding research and practice should also be noted. The final sentence of the introduction should clearly state the purpose of the systematic review.

METHODS: The methods provide a specification of the study protocol with enough information so that others can reproduce the results. It is important to include information on the:

• Eligibility criteria for studies: Who are the patients or subjects? What are the study characteristics, interventions, and outcomes? Were there language restrictions?
• Literature search: What databases were searched? Which key search terms were used? Which years were searched?
• Study selection: What was the study selection method? Was the title screened first, followed by the abstract, and finally the full text of the article?
• Data extraction: What data and information will be extracted from the articles?
• Data analysis: What are the statistical methods for handling any quantitative data?

RESULTS: The results should also be well-organized. One way to approach the results is to include information on the:

• Search results: What are the numbers of articles identified, excluded, and ultimately eligible?
• Study characteristics: What are the type and number of subjects? What are the methodological features of the studies?
• Study quality score: What is the overall quality of included studies? Does the quality of the included studies affect the outcome of the results?
• Results of the study: What are the overall results and outcomes? Could the literature be divided into themes or categories?

DISCUSSION: The discussion begins with a nonnumeric summary of the results. Next, gaps in the literature as well as limitations of the included articles are discussed with respect to the impact that they have on the reliability of the results. The final paragraph provides conclusions as well as implications for future research and current practice. For example, questions for future research on this topic are revealed, as well as whether or not practice should change as a result of the review.

REFERENCES: A complete bibliographical list of all journal articles, reports, books, and other media referred to in the systematic review should be included at the end of the paper. Referencing software can facilitate the compilation of citations and is useful in terms of ensuring the reference list is accurate and complete.

The following resources may be helpful when writing a systematic review:

CEBM: Centre for Evidence-based Medicine. Dedicated to the practice, teaching and dissemination of high quality evidence based medicine to improve health care Available at: http://www.cebm.net/ .

CITING MEDICINE: The National Library of Medicine Style Guide for Authors, Editors, and Publishers. This resource provides guidance in compiling, revising, formatting, and setting reference standards. Available at http://www.ncbi.nlm.nih.gov/books/NBK7265/ .

EQUATOR NETWORK: Enhancing the QUAlity and Transparency Of health Research. The EQUATOR Network promotes the transparent and accurate reporting of research studies. Available at: http://www.equator-network.org/ .

ICMJE RECOMMENDATIONS: International Committee of Medical Journal Editors Recommendations for the Conduct, Reporting, Editing and Publication of Scholarly Work in Medical Journals. The ICJME recommendations are followed by a large number of journals. Available at: http://www.icmje.org/about-icmje/faqs/icmje-recommendations/ .

PRISMA STATEMENT: Preferred Reporting Items for Systematic Reviews and Meta-Analyses. Authors can utilize the PRISMA Statement checklist to improve the reporting of systematic reviews and meta-analyses. Available at: http://prisma-statement.org .

THE COCHRANE COLLABORATION: A reliable source for making evidence generated through research useful for informing decisions about health. Available at: http://www.cochrane.org/ .

Examples of Systematic Reviews To Link Research and Quality Improvement

Over the past 17 years more than 300 learners, including physicians, nurses, and health administrators have completed a course as part of a Master of Health Administration or a Master of Science in Health Informatics degree at the University of Missouri. An objective of the course is to educate health informatics and health administration professionals about how to utilize a systematic, scientific, and evidence-based approach to literature searching, appraisal, and synthesis. Learners in the course conduct a systematic review of the literature on a health care topic of their choosing that could suggest quality improvement in their organization. Students select topics that make sense in terms of their core educational competencies and are related to their work. The categories of topics include public health, leadership, information management, health information technology, electronic medical records, telehealth, patient/clinician safety, treatment/screening evaluation cost/finance, human resources, planning and marketing, supply chain, education/training, policies and regulations, access, and satisfaction. Some learners have published their systematic literature reviews 14 – 15 . Qualitative comments from the students indicate that the course is well received and the skills learned in the course are applicable to a variety of health care settings.

Undertaking a literature review includes identification of a topic of interest, searching and retrieving the appropriate literature, assessing quality, extracting data and information, analyzing and synthesizing the findings, and writing a report. A structured step-by-step approach facilitates the development of a complete and informed literature review.

Suzanne Austin Boren, PhD, MHA, (above) is Associate Professor and Director of Academic Programs, and David Moxley, MLIS, is Clinical Instructor and Associate Director of Executive Programs. Both are in the Department of Health Management and Informatics at the University of Missouri School of Medicine.

Contact: ude.iruossim.htlaeh@snerob

None reported.

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• My Bibliography
• Collections
• Citation manager

Save citation to file

Email citation, add to collections.

• Create a new collection
• Add to an existing collection

Add to My Bibliography

Your saved search, create a file for external citation management software, your rss feed.

• Search in PubMed
• Search in NLM Catalog
• Add to Search

A practical overview of how to conduct a systematic review

Affiliation.

• 1 Central Manchester University Hospitals NHS Foundation Trust, Manchester, England.
• PMID: 27897766
• DOI: 10.7748/ns.2016.e10316

With an increasing focus on evidence-based practice in health care, it is important that nurses understand the principles underlying systematic reviews. Systematic reviews are used in healthcare to present a comprehensive, policy-neutral, transparent and reproducible synthesis of evidence. This article provides a practical overview of the process of undertaking systematic reviews, explaining the rationale for each stage. It provides guidance on the standard methods applicable to every systematic review: writing and registering a protocol; planning a review; searching and selecting studies; data collection; assessing the risk of bias; and interpreting results.

Keywords: evidence-based practice; healthcare research; literature review; research; review; systematic literature review; systematic review.

PubMed Disclaimer

Similar articles

• Conducting systematic reviews of association (etiology): The Joanna Briggs Institute's approach. Moola S, Munn Z, Sears K, Sfetcu R, Currie M, Lisy K, Tufanaru C, Qureshi R, Mattis P, Mu P. Moola S, et al. Int J Evid Based Healthc. 2015 Sep;13(3):163-9. doi: 10.1097/XEB.0000000000000064. Int J Evid Based Healthc. 2015. PMID: 26262566
• Summarizing systematic reviews: methodological development, conduct and reporting of an umbrella review approach. Aromataris E, Fernandez R, Godfrey CM, Holly C, Khalil H, Tungpunkom P. Aromataris E, et al. Int J Evid Based Healthc. 2015 Sep;13(3):132-40. doi: 10.1097/XEB.0000000000000055. Int J Evid Based Healthc. 2015. PMID: 26360830
• The future of Cochrane Neonatal. Soll RF, Ovelman C, McGuire W. Soll RF, et al. Early Hum Dev. 2020 Nov;150:105191. doi: 10.1016/j.earlhumdev.2020.105191. Epub 2020 Sep 12. Early Hum Dev. 2020. PMID: 33036834
• Effects of e-learning in a continuing education context on nursing care: a review of systematic qualitative, quantitative and mixed studies reviews (protocol). Rouleau G, Gagnon MP, Côté J, Payne-Gagnon J, Hudson E, Bouix-Picasso J, Dubois CA. Rouleau G, et al. BMJ Open. 2017 Oct 16;7(10):e018441. doi: 10.1136/bmjopen-2017-018441. BMJ Open. 2017. PMID: 29042394 Free PMC article. Review.
• What guidance is available for researchers conducting overviews of reviews of healthcare interventions? A scoping review and qualitative metasummary. Pollock M, Fernandes RM, Becker LA, Featherstone R, Hartling L. Pollock M, et al. Syst Rev. 2016 Nov 14;5(1):190. doi: 10.1186/s13643-016-0367-5. Syst Rev. 2016. PMID: 27842604 Free PMC article. Review.
• An analysis of current practices in undertaking literature reviews in nursing: findings from a focused mapping review and synthesis. Aveyard H, Bradbury-Jones C. Aveyard H, et al. BMC Med Res Methodol. 2019 May 16;19(1):105. doi: 10.1186/s12874-019-0751-7. BMC Med Res Methodol. 2019. PMID: 31096917 Free PMC article.
• Search in MeSH

LinkOut - more resources

Other literature sources.

• scite Smart Citations
• Citation Manager

NCBI Literature Resources

MeSH PMC Bookshelf Disclaimer

The PubMed wordmark and PubMed logo are registered trademarks of the U.S. Department of Health and Human Services (HHS). Unauthorized use of these marks is strictly prohibited.

Doctor of Nursing Practice (DNP): Conducting a Literature Review

• Find Articles
• Reference Resources
• Writing & Citing
• Conducting a Literature Review
• Video/Tutorial Resources
• Web Resources
• Finding Full Text Articles
• Searching for Quantitative and Qualitative Articles
• Presentation Tips

Selected Literature Review Books

Tips for Literature Reviews

• Search the nursing/healthcare literature to find studies relevant to your topic or PICOT question.
• Appraise your findings
• Summarize research studies
• Compare and contrast studies
• Synthesize the key concepts of your readings

Selected Online Resources

• DOI Help: Digital Object Identifiers (DOIs) How to find an article's Digital Object Identifier, or DOI.
• Evaluating Research Literature A guide from Kennesaw State University on how to assess and evaluate research studies.
• Literature Review: A Self-Guided Tutorial A step-by-step guide to conducting a literature review.
• Literature Reviews This guide provides detailed information about conducting a literature review
• Literature Reviews This guide from NYU gives details about various types of literature reviews and a roadmap for evidence synthesis.
• Literature Reviews & Research Methods This JSU subject guide offers advice on doing literature reviews and types of research methods
• RefWorks: Analysis & Synthesis How to use RefWorks to help with the synthesis of articles for a literature review. Part of a comprehensive subject guide from Kennesaw State University.
• What is a Literature Review? This page offers information on literature reviews

Welcome to the Houston Cole Library Guide pages! The Conducting a Literature Review Guide gives you links to key resources to help you get started finding and organizing your resources. Information is also available at Houston Cole Library's How to Conduct a Literature Review page.

Finding Information and Keeping Track of References

Selected databases to search for literature are listed below. RefWorks is a reference manager designed to keep track of the references cited in your papers, and to create bibliographies.

Don't Fear the Literature Review

• Literature Review Tutorial This short tutorial breaks down the aspects of a literature review and how to perform one with emphasis on tips to take the anxiety out of the process.

Literature Reviews: An Overview for Graduate Students

Health and Sciences Librarian

Other Useful Subject Guides

• Basic Academic Research by John Upchurch Last Updated Aug 9, 2024 12 views this year
• Evidence Based Practice in Nursing by Paula Barnett-Ellis Last Updated Sep 4, 2024 509 views this year
• How to Conduct a Literature Review by Karlie Johnson Last Updated Aug 9, 2024 242 views this year
• Medicine, Health, and Nursing by Paula Barnett-Ellis Last Updated Sep 4, 2024 578 views this year
• RefWorks by Karlie Johnson Last Updated Aug 9, 2024 14 views this year
• Scholarly Writing and Publishing by Paula Barnett-Ellis Last Updated Aug 25, 2024 55 views this year
• << Previous: Writing & Citing
• Next: Video/Tutorial Resources >>
• Last Updated: Sep 4, 2024 2:45 PM
• URL: https://libguides.jsu.edu/dnp

• International
• Alumni As a member of our Alumni Network, you are always part of the University of Central Lancashire.
• Business and Enterprise We have a strong reputation for applied business research, entrepreneurship and innovation, offering a wide range of specialist business support.
• News Read the the latest news and updates from across the University.
• Work here Discover more about working at one of England’s largest universities in the heart of the North West. Find out more about working at UCLan and browse our latest job opportunities.
• Events at UCLan and beyond We host a wide variety of events open to the public. These include performances and exhibitions, public lectures, festivals and more...
• Visit us Find out more about great ways to visit our campus from Open Days to Academic conferences. There's something for everyone.
• Online Store
• Academic staff
• Research We have a growing and vibrant community of academics producing high quality research.
• Research Services We provide a professional, effective and holistic offering for researchers, from ideas and training to funding, dissemination and governance.
• Research Themes Our research fits under six broad themes, ranging from arts and culture to transformative science and technology.
• Research Centres Our interdisciplinary Research Centres are the home of world-class research that brings together expertise from across the University enabling us to provide real-world solutions and opportunities that have an impact on society.
• Postgraduate research degrees We offer a vibrant and world-class research environment, matched by excellent facilities, comprehensive research support and award-winning training opportunities.
• Researcher Development
• Research Strategy and Policy
• Training and development We work in partnership with organisations from all sectors to develop and deliver tailored training programmes.
• Recruit our Students Are you looking for someone to bring new ideas and a fresh approach to your business? A UCLan student or graduate could be just what you need.
• Support for SMEs Support and funding for small and medium sized businesses
• Apprenticeships Looking to sponsor or employ a degree apprentice? We offer a wide range of higher and degree apprenticeships.
• Develop your business
• Professional Services
• International students If you’re looking to study at a great university in the North West, the University of Central Lancashire is a great choice to start or continue your studies in the UK.
• International We are an international, multi-campus university leading the way in modern learning.
• Explore our subjects
• A-Z of courses
• EU and International tuition fees
• Find your country or region
• How to apply to university in the UK
• Welcome and arrival information Everything you need to know about starting university
• Student experience For everything about University life, from sports clubs to join, to student support.
• Campuses and partners We have a large and diverse campus network, with campus locations in Preston, Burnley, Westlakes and Cyprus.
• Student support We're here for you, no matter what. We offer expert advice and support in key areas such as health and inclusion to financial matters, visas and more.
• Accommodation Our student accommodation options offer great value for money in a safe, vibrant and inclusive halls community.
• Offer holders
• Undergraduate study As an undergraduate student, you’ll benefit from the perfect mix of academic study and a range of opportunities that will set you up for a lifetime of success.
• Postgraduate study Whether you’re looking to develop your knowledge, enhance your career or change direction completely, our postgraduate courses can support your goals.
• Degree Apprenticeships Earn your qualification in a paid position within a company or organisation.
• Professional Development & Short Courses We offer professional development for you or your team to help plug a skills gap, offer continuing professional development or support career progression.
• How to apply to university
• Fees and funding
• International students
• Work or Study Abroad
• Return to Study
• University Open Days
• University exhibitions
• University Clearing and Adjustment
• Student Policies and Regulations

Search CLok

• Accessibility
• Guest Login

You are here: Home » Research » How to conduct a literature review: a process that should be familiar to nurses

How to conduct a literature review: a process that should be familiar to nurses

Official URL: http://www.hivnursing.net/past_articles/volume_sea...

Writing and research can be challenging for nurses at undergraduate and postgraduate level; however, understanding the process and developing the skills to conduct a literature review with a staged strategy will positively affect care delivery. Nurses have a responsibility to deliver care based on the best evidence available. Therefore, developing the necessary skills to conduct a literature review is beneficial to both nurses and those in their care.

Repository Staff Only: item control page

• Campuses and partners
• Report and Support
• Accessibility Statement
• Cookie Notice
• Modern Slavery Statement
• Legal and Regulatory Information
• Charitable Status

Libraries & Cultural Resources

Research guides, research in education.

• Where to Start

Education Literature Review Guide

General literature review guides.

• Citation Resources
• Contact Us!

• << Previous: Where to Start
• Next: Citation Resources >>
• Last Updated: Sep 3, 2024 5:13 PM
• URL: https://libguides.ucalgary.ca/edresearch

Libraries & Cultural Resources

• 403.220.8895

• Open access
• Published: 02 September 2024

Clinician perspectives and recommendations regarding design of clinical prediction models for deteriorating patients in acute care

• Robin Blythe   ORCID: orcid.org/0000-0002-3643-4332 1 ,
• Sundresan Naicker   ORCID: orcid.org/0000-0002-2392-4981 1 ,
• Nicole White   ORCID: orcid.org/0000-0002-9292-0773 1 ,
• Raelene Donovan   ORCID: orcid.org/0000-0003-0737-7719 2 ,
• Ian A. Scott   ORCID: orcid.org/0000-0002-7596-0837 3 , 4 ,
• Andrew McKelliget 2 &
• Steven M McPhail   ORCID: orcid.org/0000-0002-1463-662X 1 , 4  

BMC Medical Informatics and Decision Making volume  24 , Article number:  241 ( 2024 ) Cite this article

Metrics details

Successful deployment of clinical prediction models for clinical deterioration relates not only to predictive performance but to integration into the decision making process. Models may demonstrate good discrimination and calibration, but fail to match the needs of practising acute care clinicians who receive, interpret, and act upon model outputs or alerts. We sought to understand how prediction models for clinical deterioration, also known as early warning scores (EWS), influence the decision-making of clinicians who regularly use them and elicit their perspectives on model design to guide future deterioration model development and implementation.

Nurses and doctors who regularly receive or respond to EWS alerts in two digital metropolitan hospitals were interviewed for up to one hour between February 2022 and March 2023 using semi-structured formats. We grouped interview data into sub-themes and then into general themes using reflexive thematic analysis. Themes were then mapped to a model of clinical decision making using deductive framework mapping to develop a set of practical recommendations for future deterioration model development and deployment.

Fifteen nurses ( n  = 8) and doctors ( n  = 7) were interviewed for a mean duration of 42 min. Participants emphasised the importance of using predictive tools for supporting rather than supplanting critical thinking, avoiding over-protocolising care, incorporating important contextual information and focusing on how clinicians generate, test, and select diagnostic hypotheses when managing deteriorating patients. These themes were incorporated into a conceptual model which informed recommendations that clinical deterioration prediction models demonstrate transparency and interactivity, generate outputs tailored to the tasks and responsibilities of end-users, avoid priming clinicians with potential diagnoses before patients were physically assessed, and support the process of deciding upon subsequent management.

Conclusions

Prediction models for deteriorating inpatients may be more impactful if they are designed in accordance with the decision-making processes of acute care clinicians. Models should produce actionable outputs that assist with, rather than supplant, critical thinking.

• This article explored decision-making processes of clinicians using a clinical prediction model for deteriorating patients, also known as an early warning score.

• Our study identified that the clinical utility of deterioration models may lie in their assistance in generating, evaluating, and selecting diagnostic hypotheses, an important part of clinical decision making that is underrepresented in the prediction modelling literature.

• Nurses in particular stressed the need for models that encourage critical thinking and further investigation rather than prescribe strict care protocols.

Peer Review reports

The number of ‘clinical prediction model’ articles published on PubMed has grown rapidly over the past two decades, from 1,918 articles identified with these search terms published in 2002 to 26,326 published in 2022. A clinical prediction model is defined as any multivariable model that provides patient-level estimates of the probability or risk of a disease, condition or future event [ 1 , 2 , 3 ].

Recent systematic and scoping reviews report a lack of evidence that clinical decision support systems based on prediction models are associated with improved patient outcomes once implemented in acute care [ 4 , 5 , 6 , 7 ]. One potential reason may be that some models are not superior to clinical judgment in reducing missed diagnoses or correctly classifying non-diseased patients [ 8 ]. While improving predictive accuracy is important, this appears insufficient for improving patient outcomes, suggesting that more attention should be paid to the process and justification of how prediction models are designed and deployed [ 9 , 10 ].

If model predictions are to influence clinical decision-making, they must not only demonstrate acceptable accuracy, but also be implemented and adopted at scale in clinical settings. This requires consideration of how they are integrated into clinical workflows, how they generate value for users, and how clinicians perceive and respond to their outputs of predicted risks [ 11 , 12 ]. These concepts are tenets of user-centred design, which focuses on building systems based on the needs and responsibilities of those who will use them. User-centred decision support tools can be designed in a variety of ways, but may benefit from understanding the characteristics of the users and the local environment in which tools are implemented, [ 13 ] the nature of the tasks end-users are expected to perform, [ 14 ] and the interface between the user and the tools [ 15 ].

Prediction models for clinical deterioration

A common task for prediction models integrated into clinical decision support systems is in predicting or recognising clinical deterioration, also known as early warning scores. Clinical deterioration is defined as the transition of a patient from their current health state to a worse one that puts them at greater risk of adverse events and death [ 16 ]. Early warning scores were initially designed to get the attention of skilled clinicians when patients began to deteriorate, but have since morphed into complex multivariable prediction models [ 17 ]. As with many other clinical prediction models, early warning scores often fail to demonstrate better patient outcomes once deployed [ 4 , 18 ]. The clinical utility of early warning scores likely rests on two key contextual elements: the presence of uncertainty, both in terms of diagnosis and prognosis, and the potential for undesirable patient outcomes if an appropriate care pathway is delayed or an inappropriate one is chosen [ 19 ].

The overarching goal of this qualitative study was to determine how prediction models for clinical deterioration, or early warning scores, could be better tailored to the needs of end-users to improve inpatient care. This study had three aims. First, to understand the experiences and perspectives of nurses and doctors who use early warning scores. Second, to identify the tasks these clinicians performed when managing deteriorating patients, the decision-making processes that guided these tasks, and how these could be conceptualised schematically. Finally, to address these tasks and needs with actionable, practical recommendations for enhancing future deterioration prediction model development and deployment.

To achieve our study aims, we conducted semi-structured interviews of nurses and doctors at two large, digitally mature hospitals. We first asked clinicians to describe their backgrounds, perspectives, and experience with early warning scores to give context to our analysis. We then examined the tasks and responsibilities of participants and the decision-making processes that guided these tasks using reflexive thematic analysis, an inductive method that facilitated the identification of general themes. We then identified a conceptual decision-making framework from the literature to which we mapped these themes to understand how they may lead to better decision support tools. Finally, we used this framework to formulate recommendations for deterioration prediction model design and deployment. These steps are presented graphically in a flow diagram (Fig.  1 ).

Schema of study goal, aims and methods

The study was conducted at one large tertiary and one medium-sized metropolitan hospital in Brisbane, Australia. The large hospital contained over 1,000 beds, handling over 116,000 admissions and approximately 150,000 deterioration alerts per year in 2019. Over the same period, the medium hospital contained 175 beds, handling over 31,000 admissions and approximately 42,000 deterioration alerts per year. These facilities had a high level of digital maturity, including fully integrated electronic medical records.

Clinical prediction model for deteriorating patients

The deterioration monitoring system used at both hospitals was the Queensland Adult Deterioration Detection System (Q-ADDS) [ 20 , 21 ]. Q-ADDS uses an underlying prediction model to convert patient-level vital signs from a single time of observation into an ordinal risk score describing an adult patient’s risk of acute deterioration. Vital signs collected are respiratory rate (breaths/minute), oxygen flow rate (L/minute), arterial oxygen saturation (percent), blood pressure (mmHg), heart rate (beats/minute), temperature (degrees Celsius), level of consciousness (Alert-Voice-Pain-Unresponsive) and increased or new onset agitation. Increased pain and urine output are collected but not used for score calculation [ 21 ]. The Q-ADDS tool is included in the supplementary material.

Vital signs are entered into the patient’s electronic medical record, either imported from the vital signs monitoring device at the patient’s bedside or from manual entry by nurses. Calculations are made automatically within Q-ADDS to generate an ordinal risk score per patient observation. Scores can be elevated to levels requiring a tiered escalation response if a single vital sign is greatly deranged, or if several observations are deranged by varying degrees. Scores range from 0 to 8+, with automated alerts and escalation protocols ranging from more frequent observations for lower scores to immediate activation of the medical emergency team (MET) at higher scores.

The escalation process for Q-ADDS is highly structured, mandated and well documented [ 21 ]. Briefly, when a patient’s vital signs meet a required alert threshold, the patient’s nurse is required to physically assess the patient and, depending on the level of severity predicted by Q-ADDS, notify the patient’s doctor (escalation). The doctor is then required to be notified of the patient’s Q-ADDS score, potentially review the patient, and discuss any potential changes to care with the nurse. Both nurses and doctors can escalate straight to MET calls or an emergency ‘code blue’ call (requiring cardiopulmonary resuscitation or assisted ventilation) at any time if necessary.

Participant recruitment

Participant recruitment began in February 2022 and concluded in March 2023, disrupted by the COVID-19 pandemic. Eligibility criteria were nurses or doctors at each hospital with direct patient contact who either receive or respond, respectively, to Q-ADDS alerts. An anticipated target sample size of 15 participants was established prior to recruitment, based on expected constraints in recruitment due to clinician workloads and the expected length of interviews relative to their scope, as guided by prior research [ 22 ]. As the analysis plan involved coding interviews iteratively as they were conducted, the main justification for ceasing recruitment was when no new themes relating to the study objectives were generated during successive interviews as the target sample size was approached [ 23 ].

Study information was broadly distributed via email to nurses and doctors in patient-facing roles across hospitals. Nurse unit managers were followed up during regular nursing committee meetings to participate or assist with recruitment within their assigned wards. Doctors were followed up by face-to-face rounding. Snowball sampling, in which participants were encouraged to refer their colleagues for study participation, was employed whenever possible. In all cases, study authors explained study goals and distributed participant consent forms prior to interview scheduling with the explicit proviso that participation was completely voluntary and anonymous to all but two study authors (RB and SN).

Interview process

We used a reflexive framework method to develop an open-ended interview template [ 24 ] that aligned with our study aims. Interview questions were informed by the non-adoption, abandonment, scale-up, spread and sustainability (NASSS) framework [ 25 ]. The NASSS framework relates the end-user perceptions of the technology being evaluated to its value proposition for the clinical situation to which it is being applied. We selected a reflexive method based on the NASSS for our study as we wanted to allow end-users to speak freely about the barriers they faced when using prediction models for clinical deterioration, but did not limit participants to discussing only topics that could fit within the NASSS framework.

Participants were first asked about their background and clinical expertise. They were then invited to share their experiences and perspectives with using early warning scores to manage deteriorating patients. This was used as a segue for participants to describe the primary tasks required of them when evaluating and treating a deteriorating patient. Participants were encouraged to talk through their decision-making process when fulfilling these tasks, and to identify any barriers or obstacles to achieving those tasks that were related to prediction models for deteriorating patients. Participants were specifically encouraged to identify any sources of information that were useful for managing deteriorating patients, including prediction models for other, related disease groups like sepsis, and to think of any barriers or facilitators for making that information more accessible. Finally, participants were invited to suggest ways to improve early warning scores, and how those changes may lead to benefits for patients and clinicians.

As we employed a reflexive methodology to allow clinicians to speak freely about their perspectives and opinions, answers to interview questions were optional and open-ended, allowing participants to discuss relevant tangents. Separate interview guides were developed for nurses and doctors as the responsibilities and information needs of these two disciplines in managing deteriorating patients often differ. Nurses are generally charged with receiving and passing on deterioration alerts, while doctors are generally charged with responding to alerts and making any required changes to patient care plans [ 4 ]. Interview guides are contained in the supplement.

Due to clinician workloads, member checking, a form of post-interview validation in which participants retrospectively confirm their interview answers, was not used. To ensure participants perceived the interviewers as being impartial, two study authors not employed by the hospital network and not involved in direct patient care (RB and SN) were solely responsible for conducting interviews and interrogating interview transcripts. Interviews were recorded and transcribed verbatim, then re-checked for accuracy.

Inductive thematic analysis

Transcripts were analysed using a reflexive thematic methodology informed by Braun and Clarke [ 26 ]. This method was selected because it facilitated exploring the research objectives rather than being restricted to the domains of a specific technology adoption framework, which may limit generalisability [ 27 ]. Interviews were analysed over five steps to identify emergent themes.

Each interview was broken down into segments by RB and SN, where segments corresponded to a distinct opinion.

Whenever appropriate, representative quotes for each distinct concept were extracted.

Segments were grouped into sub-themes.

Sub-themes were grouped into higher-order themes, or general concepts.

Steps 1 through 4 were iteratively repeated by RB and supervised by SN.

As reflexive methods incorporate the experiences and expertise of the analysts, our goal was to extract any sub-themes relevant to the study aims and able to be analysed in the context of early warning scores, prediction models, or decision support tools for clinical deterioration. The concepts explored during this process were not exhaustive, but repeated analysis and re-analysis of participant transcripts helped to ensure all themes could be interpreted in the context of our three study aims: background and perspectives, tasks and decision-making, and recommendations for future practice.

Deductive mapping to a clinical decision-making framework

Once the emergent themes from the inductive analysis were defined, we conducted a brief scan of PubMed for English-language studies that investigated how the design of clinical decision support systems relate to clinical decision-making frameworks. The purpose of this exercise was to identify a framework against which we could map the previously elicited contexts, tasks, and decision-making of end-users in developing a decision-making model that could then be used to support the third aim of formulating recommendations to enhance prediction model development and deployment.

RB and SN then mapped higher-order themes from the inductive analysis to the decision-making model based on whether there was a clear relationship between each theme and a node in the model (see Results).

Recommendations for improving prediction model design were derived by reformatting the inductive themes based on the stated preferences of the participants. These recommendations were then assessed by the remaining authors and the process repeated iteratively until authors were confident that all recommendations were concordant with the decision-making model.

Participant characteristics

Our sample included 8 nurses and 7 doctors of varying levels of expertise and clinical specialties; further information is contained in the supplement. Compared to doctors, nurse participants were generally more experienced, often participating in training or mentoring less experienced staff. Clinical specialities of nurses were diverse, including orthopaedics, cancer services, medical assessment and planning unit, general medicine, and pain management services. Doctor participants ranged from interns with less than a year of clinical experience up to consultant level, including three doctors doing training rotations and two surgical registrars. Clinical specialties of doctors included geriatric medicine, colorectal surgery, and medical education.

Interviews and thematic analysis

Eleven interviews were conducted jointly by RB and SN, one conducted by RB, and three by SN. Interviews were scheduled for up to one hour, with a mean duration of 42 min. Six higher-order themes were identified. These were: added value of more information; communication of model outputs; validation of clinical intuition; capability for objective measurement; over-protocolisation of care; and model transparency and interactivity (Table  1 ). Some aspects of care, including the need for critical thinking and the informational value of discerning trends in patient observations, were discussed in several contexts, making them relevant to more than one higher-order theme.

Added value of other information

Clinicians identified that additional data or variables important for decision making were often omitted from the Q-ADDS digital interface. Such variables included current medical conditions, prescribed medications and prior observations, which were important for interpreting current patient data in the context of their baseline observations under normal circumstances (e.g., habitually low arterial oxygen saturation due to chronic obstructive pulmonary disease) or in response to an acute stimulus (e.g., expected hypotension for next 4 to 8 h while treatment for septic shock is underway).

“The trend is the biggest thing [when] looking at the data , because sometimes people’s observations are deranged forever and it’s not abnormal for them to be tachycardic , whereas for someone else , if it’s new and acute , then that’s a worry.” – Registrar.

Participants frequently emphasised the critical importance of looking at patients holistically, or that patients were more than the sum of the variables used to predict risk. Senior nurses stressed that prediction models were only one part of patient evaluation, and clinicians should be encouraged to incorporate both model outputs and their own knowledge and experiences in decision making rather than trust models implicitly. Doctors also emphasised this holistic approach, adding that they placed more importance on hearing a nurse was concerned for the patient than seeing the model output. Critical thinking about future management was frequently raised in this context, with both nurses and doctors insisting that model predictions and the information required for contextualising risk scores should be communicated together when escalating the patient’s care to more senior clinicians.

Model outputs

Model outputs were discussed in two contexts. First, doctors perceived that ordinal risk scores generated by Q-ADDS felt arbitrary compared to receiving probabilities of a future event, for example cardiorespiratory decompensation, that required a response such as resuscitation or high-level treatment. However, nurses did not wholly embrace probabilities as outputs, instead suggesting that recommendations for how they should respond to different Q-ADDS scores were more important. This difference may reflect the different roles of alert receivers (nurses) and alert responders (doctors).

“[It’s helpful] if you use probabilities… If your patient has a sedation score of 2 and a respiratory rate of 10 , [giving them] a probability of respiratory depression would be helpful. However , I don’t find many clinicians , and certainly beginning practitioners , think in terms of probabilities.” – Clinical nurse consultant.

Second, there was frequent mention of alert fatigue in the context of model outputs. One doctor and two nurses felt there was insufficient leeway for nurses to exercise discretion in responding to risk scores, leading to many unnecessary alert-initiated actions. More nuance in the way Q-ADDS outputs were delivered to clinicians with different roles was deemed important to avoid model alerts being perceived as repetitive and unwarranted. However, three other doctors warned against altering MET call criteria in response to repetitive and seemingly unchanging risk scores and that at-risk patients should, as a standard of care, remain under frequent observation. Frustrations centred more often around rigidly tying repetitive Q-ADDS outputs to certain mandated actions, leading to multiple clinical reviews in a row for a patient whose trajectory was predictable, for example a patient with stable heart failure having a constantly low blood pressure. This led to duplication of nursing effort (e.g., repeatedly checking the blood pressure) and the perception that prediction models were overly sensitive.

“It takes away a lot of nurses’ critical judgement. If someone’s baseline systolic [blood pressure] is 95 [mmHg] , they’re asymptomatic and I would never hear about it previously. We’re all aware that this is where they sit and that’s fine. Now they are required to notify me in the middle of the night , “Just so you know , they’ve dropped to 89 [below an alert threshold of 90mmHg].“” – Junior doctor.

Validation of clinical intuition

Clinicians identified the ability of prediction models to validate their clinical intuition as both a benefit and a hindrance, depending on how outputs were interpreted and acted upon. Junior clinicians appreciated early warning scores giving them more support to escalate care to senior clinicians, as a conversation starter or framing a request for discussion. Clinicians described how assessing the patient holistically first, then obtaining model outputs to add context and validate their diagnostic hypotheses, was very useful in deciding what care should be initiated and when.

“You kind of rule [hypotheses] out… you go to the worst extreme: is it something you need to really be concerned about , especially if their [score] is quite high? You’re thinking of common complications like blood clots , so that presents as tachycardic… I’m thinking of a PE [pulmonary embolism] , then you do the nursing interventions.” – Clinical nurse manager.

While deterioration alerts were often seen as triggers to think about potential causes for deterioration, participants noted that decision making could be compromised if clinicians were primed by model outputs to think of different diagnoses before they had fully assessed the patient at the bedside. Clinicians described the dangers of tunnel vision or, before considering all available clinical information, investigating favoured diagnoses to the exclusion of more likely causes.

“[Diagnosis-specific warnings are] great , [but] that’s one of those things that can lead to a bit of confirmation bias… It’s a good trigger to articulate , “I need to look for sources of infection when I go to escalate"… but then , people can get a little bit sidetracked with that and ignore something more blatant in front of them. I’ve seen people go down this rabbit warren of being obsessed with the “fact” that it was sepsis , but it was something very , very unrelated.” – Nurse educator.

Objective measurement

Clinicians perceived that prediction models were useful as more objective measures of patients’ clinical status that could ameliorate clinical uncertainty or mitigate cognitive biases. In contrast to the risk of confirmation bias arising from front-loading model outputs suggesting specific diagnoses, prediction models could offer a second opinion that could help clinicians recognise opposing signals in noisy data that, in particular, assisted in considering serious diagnoses that shouldn’t be missed (e.g., sepsis), or more frequent and easily treated diagnoses (e.g., dehydration). Prediction models were also useful when they disclosed several small, early changes in patient status that provided an opportunity for early intervention.

“Maybe [the patient has] a low grade fever , they’re a bit tachycardic. Maybe [sepsis] isn’t completely out of the blue for this person. If there was some sort of tool , that said there’s a reasonable chance that they could have sepsis here , I would use that to justify the option of going for blood cultures and maybe a full septic screen. If [I’m indecisive] , that sort of information could certainly push me in that direction.” – Junior doctor.

Clinicians frequently mentioned that prediction models would have been more useful when first starting clinical practice, but become less useful with experience. However, clinicians noted that at any experience level, risk scoring was considered most useful as a triage/prioritisation tool, helping decide which patients to see first, or which clinical concerns to address first.

“[Doctors] can easily triage a patient who’s scoring 4 to 5 versus 1 to 3. If they’re swamped , they can change the escalation process , or triage appropriately with better communication.” – Clinical nurse manager.

Clinicians also stressed that predictions were not necessarily accurate because measurement error or random variation, especially one-off outlier values for certain variables, was a significant contributor to false alerts and inappropriate responses. For example, a single unusually high respiratory rate generated an unusually high risk score, prompting an unnecessary alert.

Over-protocolisation of care

The sentiment most commonly expressed by all experienced nursing participants and some doctors was that nurses were increasingly being trained to solely react to model outputs with fixed response protocols, rather than think critically about what is happening to patients and why. It was perceived that prediction models may actually reduce the capacity for clinicians to process and internalise important information. For example, several nurses observed their staff failing to act on their own clinical suspicions that patients were deteriorating because the risk score had not exceeded a response threshold.

“We’ve had patients on the ward that have had quite a high tachycardia , but it’s not triggering because it’s below the threshold to trigger… [I often need to make my staff] make the clinical decision that they can call the MET anyway , because they have clinical concern with the patient.” – Clinical nurse consultant.

A source of great frustration for many nurses was the lack of critical thinking by their colleagues of possible causes when assessing deteriorating patients. They wanted their staff to investigate whether early warning score outputs or other changes in patient status were caused by simple, easily fixable issues such as fitting the oxygen mask properly and helping the patient sit up to breathe more easily, or whether they indicated more serious underlying pathophysiology. Nurses repeatedly referenced the need for clinicians to always be asking why something was happening, not simply reacting to what was happening.

“[Models should also be] trying to get back to critical thinking. What I’m seeing doesn’t add up with the monitor , so I should investigate further than just simply calling the code.” – Clinical nurse educator.

Model transparency and interactivity

Clinicians frequently requested more transparent and interactive prediction models. These included a desire to receive more training in how prediction models worked and how risk estimates were generated mathematically, and being able to visualise important predictors of deterioration and the absolute magnitude of their effects (effect sizes) in intuitive ways. For example, despite receiving training in Q-ADDS, nurses expressed frustrations that nobody at the hospital seemed to understand how it worked in generating risk scores. Doctors were interested in being able to visualise the relative size and direction of effect of different model variables, potentially using colour-coding, combined with other contextual patient data like current vital sign trends and medications, and presented on one single screen.

The ability to modify threshold values for model variables and see how this impacted risk scores, and what this may then mean for altering MET calling criteria, was also discussed. For example, in an older patient with an acute ischaemic stroke, a persistently high, asymptomatic blood pressure value is an expected bodily response to this acute insult over the first 24–48 h. In the absence of any change to alert criteria, recurrent alerts would be triggered which may encourage overtreatment and precipitous lowering of the blood pressure with potential to cause harm. Altering the criteria to an acceptable or “normal” value for this clinical scenario (i.e. a higher than normal blood pressure) may generate a lower, more patient-centred risk estimate and less propensity to overtreat. This ability to tinker with the model may also enhance understanding of how it works.

“I wish I could alter criteria and see what the score is after that , with another set of observations. A lot of the time… I wonder what they’re sitting at , now that I’ve [altered] the bit that I’m not concerned about… It would be quite helpful to refresh it and have their score refreshed as the new score.” – Junior doctor.

Derivation of the decision-making model

Guided by the responses of our participants regarding their decision-making processes, our literature search identified a narrative review by Banning (2008) that reported previous work by O’Neill et al. (2005) [ 28 , 29 ]. While these studies referred to models of nurse decision-making, we selected a model (Fig.  2 ) that also appropriately described the responses of doctors in our participant group and matched the context of using clinical decision support systems to support clinical judgement. As an example, when clinicians referenced needing to look for certain data points to give context to a patient assessment, this was mapped to nodes relating to “Current patient data,” “Changes to patient status/data,” and “Hypothesis-driven assessment.”

Decision-making model(Adapted from Neill’s clinical decision making framework [2005] and modified by Banning [2006]) with sequential decision nodes

Mapping of themes to decision-making model

The themes from Table  1 were mapped to the nodes in the decision-making model based on close alignment with participant responses (see Fig.  3 ). This mapping is further explained below, where the nodes in the model are described in parentheses.

Value of additional information for decision-making : participants stressed the importance of understanding not only the data going into the prediction model, but also how that data changed over time as trends, and the data that were not included in the model. (Current patient data, changes to patient status/data)

Format, frequency, and relevance of outputs : participants suggested a change in patient data should not always lead to an alert. Doctors, but not necessarily nurses, proposed outputs displayed as probabilities rather than scores, tying model predictions to potential diagnoses or prognoses. (Changes to patient status/data, hypothesis generation)

Using models to validate but not supersede clinical intuition : Depending on the exact timing of model outputs within the pathway of patient assessment, participants found predictions could either augment or hinder the hypothesis generation process. (Hypothesis generation)

Measuring risks objectively : Risk scores can assist with triaging or prioritising patients by urgency or prognostic risk, thereby potentially leading to early intervention to identify and/or prevent adverse events. (Clinician concerns, hypothesis generation)

Supporting critical thinking and reducing over-protocolised care : by acting as triggers for further assessment, participants suggested prediction models can support or discount diagnostic hypotheses, lead to root-cause identification, and facilitate interim cares, for example by ensuring good fit of nasal prongs. (Provision of interim care, hypothesis generation, hypothesis-driven assessment)

Model transparency and interactivity : understanding how prediction models worked, being able to modify or add necessary context to model predictions, and understanding the relative contribution of different predictors could better assist the generation and selection of different hypotheses that may explain a given risk score. (Hypothesis generation, recognition of clinical pattern and hypothesis selection)

Mapping of the perceived relationships between higher-order themes and nodes in the decision-making model shown in Fig.  2

Recommendations for improving the design of prediction models

Based on the mapping of themes to the decision-making model, we formulated four recommendations for enhancing the development and deployment of prediction models for clinical deterioration.

Improve accessibility and transparency of data included in the model. Provide an interface that allows end-users to see what predictor variables are included in the model, their relative contributions to model outputs, and facilitate easy access to data not included in the model but still relevant for model-informed decisions, e.g., trends of predictor variables over time.

Present model outputs that are relevant to the end-user receiving those outputs, their responsibilities, and the tasks they may be obliged to perform, while preserving the ability of clinicians to apply their own discretionary judgement.

In situations associated with diagnostic uncertainty, avoid tunnel vision from priming clinicians with possible diagnostic explanations based on model outputs, prior to more detailed clinical assessment of the patient.

Support critical thinking whereby clinicians can apply a more holistic view of the patient’s condition, take all relevant contextual factors into account, and be more thoughtful in generating and selecting causal hypotheses.

This qualitative study involving front-line acute care clinicians who respond to early warning score alerts has generated several insights into how clinicians perceive the use of prediction models for clinical deterioration. Clinicians preferred models that facilitated critical thinking, allowed an understanding of the impact of variables included and excluded from the model, provided model outputs specific to the tasks and responsibilities of different disciplines of clinicians, and supported decision-making processes in terms of hypotheses and choice of management, rather than simply responding to alerts in a pre-specified, mandated manner. In particular, preventing prediction models from supplanting critical thinking was repeatedly emphasised.

Reduced staffing ratios, less time spent with patients, greater reliance on more junior workforce, and increasing dependence on automated activation of protocolised management are all pressures that could lead to a decline in clinical reasoning skills. This problem could be exacerbated by adding yet more predictive algorithms and accompanying protocols for other clinical scenarios, which may intensify alert fatigue and disrupt essential clinical care. However, extrapolating our results to areas other than clinical deterioration should be done with caution. An opposing view may be that using prediction models to reduce the burden of routine surveillance may allow redirection of critical thinking skills towards more useful tasks, a question that has not been explored in depth in the clinical informatics literature.

Clinicians expressed interest in models capable of providing causal insights into clinical deterioration. This is neither a function nor capability of most risk prediction models, requiring different assumptions and theoretical frameworks [ 30 ]. Despite this limitation, risk nomograms, visualisations of changes in risk with changes in predictor variables, and other interactive tools for estimating risk may be useful adjuncts for clinical decision-making due to the ease with which input values can be manipulated.

Contributions to the literature

Our research supports and extends the literature on the acceptability of risk prediction models within clinical decision support systems. Common themes in the literature supporting good practices in clinical informatics and which are also reflected in our study include: alert fatigue; the delivery of more relevant contextual information; [ 31 ] the value of patient histories; [ 32 , 33 ] ranking relevant information by clinical importance, including colour-coding; [ 34 , 35 ] not using computerised tools to replace clinical judgement; [ 32 , 36 , 37 ] and understanding the analytic methods underpinning the tool [ 38 ]. One other study has investigated the perspectives of clinicians of relatively simple, rules-based prediction models similar to Q-ADDS. Kappen et al [ 12 ] conducted an impact study of a prediction model for postoperative nausea and vomiting and also found that clinicians frequently made decisions in an intuitive manner that incorporated information both included and absent from prediction models. However, the authors recommended a more directive than assistive approach to model-based recommendations, possibly due to a greater focus on timely prescribing of effective prophylaxis or treatment.

The unique contribution of our study is a better understanding of how clinicians may use prediction models to generate and validate diagnostic hypotheses. The central role of critical thinking and back-and-forth interactions between clinician and model in our results provide a basis for future research using more direct investigative approaches like cognitive task analysis [ 39 ]. Our study has yielded a set of cognitive insights into decision making that can be applied in tandem with statistical best practice in designing, validating and implementing prediction models. [ 19 , 40 , 41 ].

Relevance to machine learning and artificial intelligence prediction models for deterioration

Our results may generalise to prediction models based on machine learning (ML) and artificial intelligence (AI), according to results of several recent studies. Tonekaboni et al [ 42 ] investigated clinician preferences for ML models in the intensive care unit and emergency department using hypothetical scenarios. Several themes appear both in our results and theirs: a need to understand the impact of both included and excluded predictors on model performance; the role of uncertain or noisy data in prediction accuracy; and the influence of trends or patient trajectories in decision making. Their recommendations for more transparent models and the delivery of model outputs designed for the task at hand align closely with ours. The authors’ focus on clinicians’ trust in the model was not echoed by our participants.

Eini-Porat et al [ 43 ] conducted a comprehensive case study of ML models in both adult and paediatric critical care. Their results present several findings supported by our participants despite differences in clinical environments: the value of trends and smaller changes in several vital signs that could cumulatively signal future deterioration; the utility of triage and prioritisation in time-poor settings; and the use of models as triggers for investigating the cause of deterioration.

As ML/AI models proliferate in the clinical deterioration prediction space, [ 44 ] it is important to deeply understand the factors that may influence clinician acceptance of more complex approaches. As a general principle, these methods often strive to input as many variables or transformations of those variables as possible into the model development process to improve predictive accuracy, incorporating dynamic updating to refine model performance. While this functionality may be powerful, highly complex models are not easily explainable, require careful consideration of generalisability, and can prevent clinicians from knowing when a model is producing inaccurate predictions, with potential for patient harm when critical healthcare decisions are being made [ 45 , 46 , 47 ]. Given that our clinicians emphasised the need to understand the model, know which variables are included and excluded, and correctly interpret the format of the output, ML/AI models in the future will need to be transparent in their development and their outputs easily interpretable.

Limitations

The primary limitations of our study were that our sample was drawn from two hospitals with high levels of digital maturity in a metropolitan region of a developed country, with a context specific to clinical deterioration. Our sample of 15 participants may be considered small but is similar to that of other studies with a narrow focus on clinical perspectives [ 42 , 43 ]. All these factors can limit generalisability to other settings or to other prediction models. As described in the methods, we used open-ended interview templates and generated our inductive themes reflexively, which is vulnerable to different types of biases compared to more structured preference elicitation methods with rigidly defined analysis plans. Member checking may have mitigated this bias, but was not possible due to the time required from busy clinical staff.

Our study does not directly deal with methodological issues in prediction model development, [ 41 , 48 ] nor does it provide explicit guidance on how model predictions should be used in clinical practice. Our findings should also not be considered an exhaustive list of concerns clinicians have with prediction models for clinical deterioration, nor may they necessarily apply to highly specialised clinical areas, such as critical care. Our choice of decision making framework was selected because it demonstrated a clear, intuitive causal pathway for model developers to support the clinical decision-making process. However, other, equally valid frameworks may have led to different conclusions, and we encourage more research in this area.

This study elicited clinician perspectives of models designed to predict and manage impending clinical deterioration. Applying these perspectives to a decision-making model, we formulated four recommendations for the design of future prediction models for deteriorating patients: improved transparency and interactivity, tailoring models to the tasks and responsibilities of different end-users, avoiding priming clinicians with diagnostic predictions prior to in-depth clinical review, and finally, facilitating the diagnostic hypothesis generation and assessment process.

Availability of data and materials

Due to privacy concerns and the potential identifiability of participants, interview transcripts are not available. However, interview guides are available in the supplement.

Jenkins DA, Martin GP, Sperrin M, Riley RD, Debray TPA, Collins GS, Peek N. Continual updating and monitoring of clinical prediction models: time for dynamic prediction systems? Diagn Prognostic Res. 2021;5(1):1.

Article   Google Scholar  

Collins GS, Reitsma JB, Altman DG, Moons KG. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. BMJ. 2015;350:g7594.

Article   PubMed   Google Scholar  

Moons KG, Altman DG, Reitsma JB, Ioannidis JP, Macaskill P, Steyerberg EW, et al. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med. 2015;162(1):W1–73.

Blythe R, Parsons R, White NM, Cook D, McPhail SM. A scoping review of real-time automated clinical deterioration alerts and evidence of impacts on hospitalised patient outcomes. BMJ Qual Saf. 2022;31(10):725–34.

Fahey M, Crayton E, Wolfe C, Douiri A. Clinical prediction models for mortality and functional outcome following ischemic stroke: a systematic review and meta-analysis. PLoS ONE. 2018;13(1):e0185402.

Article   PubMed   PubMed Central   Google Scholar  

Fleuren LM, Klausch TLT, Zwager CL, Schoonmade LJ, Guo T, Roggeveen LF, et al. Machine learning for the prediction of sepsis: a systematic review and meta-analysis of diagnostic test accuracy. Intensive Care Med. 2020;46(3):383–400.

White NM, Carter HE, Kularatna S, Borg DN, Brain DC, Tariq A, et al. Evaluating the costs and consequences of computerized clinical decision support systems in hospitals: a scoping review and recommendations for future practice. J Am Med Inform Assoc. 2023;30(6):1205–18.

Sanders S, Doust J, Glasziou P. A systematic review of studies comparing diagnostic clinical prediction rules with clinical judgment. PLoS ONE. 2015;10(6):e0128233.

Abell B, Naicker S, Rodwell D, Donovan T, Tariq A, Baysari M, et al. Identifying barriers and facilitators to successful implementation of computerized clinical decision support systems in hospitals: a NASSS framework-informed scoping review. Implement Sci. 2023;18(1):32.

van der Vegt AH, Campbell V, Mitchell I, Malycha J, Simpson J, Flenady T, et al. Systematic review and longitudinal analysis of implementing Artificial Intelligence to predict clinical deterioration in adult hospitals: what is known and what remains uncertain. J Am Med Inf Assoc. 2024;31(2):509–24.

Davenport T, Kalakota R. The potential for artificial intelligence in healthcare. Future Healthc J. 2019;6(2):94–8.

Kappen TH, van Loon K, Kappen MA, van Wolfswinkel L, Vergouwe Y, van Klei WA, et al. Barriers and facilitators perceived by physicians when using prediction models in practice. J Clin Epidemiol. 2016;70:136–45.

Witteman HO, Dansokho SC, Colquhoun H, Coulter A, Dugas M, Fagerlin A, Giguere AM, Glouberman S, Haslett L, Hoffman A, Ivers N. User-centered design and the development of patient decision aids: protocol for a systematic review. Systematic reviews. 2015;4:1−8.

Zhang J, Norman DA. Representations in distributed cognitive tasks. Cogn Sci. 1994;18(1):87–122.

Johnson CM, Johnson TR, Zhang J. A user-centered framework for redesigning health care interfaces. J Biomed Inf. 2005;38(1):75–87.

Jones D, Mitchell I, Hillman K, Story D. Defining clinical deterioration. Resuscitation. 2013;84(8):1029–34.

Morgan RJ, Wright MM. In defence of early warning scores. Br J Anaesth. 2007;99(5):747–8.

Article   CAS   PubMed   Google Scholar  

Smith ME, Chiovaro JC, O’Neil M, Kansagara D, Quinones AR, Freeman M, et al. Early warning system scores for clinical deterioration in hospitalized patients: a systematic review. Annals Am Thorac Soc. 2014;11(9):1454–65.

Baker T, Gerdin M. The clinical usefulness of prognostic prediction models in critical illness. Eur J Intern Med. 2017;45:37–40.

Campbell V, Conway R, Carey K, Tran K, Visser A, Gifford S, et al. Predicting clinical deterioration with Q-ADDS compared to NEWS, between the flags, and eCART track and trigger tools. Resuscitation. 2020;153:28–34.

The Australian Commission on Safety and Quality in Health is the publisher, and the publisher location is Sydney, Australia. https://www.safetyandquality.gov.au/sites/default/files/migrated/35981-ChartDevelopment.pdf .

Vasileiou K, Barnett J, Thorpe S, Young T. Characterising and justifying sample size sufficiency in interview-based studies: systematic analysis of qualitative health research over a 15-year period. BMC Med Res Methodol. 2018;18(1):148.

Hennink MM, Kaiser BN, Marconi VC. Code saturation versus meaning saturation: how many interviews are Enough? Qual Health Res. 2017;27(4):591–608.

Gale NK, Heath G, Cameron E, Rashid S, Redwood S. Using the framework method for the analysis of qualitative data in multi-disciplinary health research. BMC Med Res Methodol. 2013;13(1):1–8.

Greenhalgh T, Wherton J, Papoutsi C, Lynch J, Hughes G, A’Court C, et al. Beyond adoption: a New Framework for Theorizing and evaluating nonadoption, abandonment, and challenges to the Scale-Up, Spread, and sustainability of Health and Care technologies. J Med Internet Res. 2017;19(11):e367.

Braun V, Clarke V. One size fits all? What counts as quality practice in (reflexive) thematic analysis? Qualitative Res Psychol. 2021;18(3):328–52.

Campbell KA, Orr E, Durepos P, Nguyen L, Li L, Whitmore C, et al. Reflexive thematic analysis for applied qualitative health research. Qualitative Rep. 2021;26(6):2011–28.

Google Scholar  

Banning M. A review of clinical decision making: models and current research. J Clin Nurs. 2008;17(2):187–95.

O’Neill ES, Dluhy NM, Chin E. Modelling novice clinical reasoning for a computerized decision support system. J Adv Nurs. 2005;49(1):68–77.

Arnold KF, Davies V, de Kamps M, Tennant PWG, Mbotwa J, Gilthorpe MS. Reflection on modern methods: generalized linear models for prognosis and intervention—theory, practice and implications for machine learning. Int J Epidemiol. 2020;49(6):2074–82.

Article   PubMed Central   Google Scholar  

Westerbeek L, Ploegmakers KJ, de Bruijn GJ, Linn AJ, van Weert JCM, Daams JG, et al. Barriers and facilitators influencing medication-related CDSS acceptance according to clinicians: a systematic review. Int J Med Informatics. 2021;152:104506.

Henshall C, Marzano L, Smith K, Attenburrow MJ, Puntis S, Zlodre J, et al. A web-based clinical decision tool to support treatment decision-making in psychiatry: a pilot focus group study with clinicians, patients and carers. BMC Psychiatry. 2017;17(1):265.

Weingart SN, Simchowitz B, Shiman L, Brouillard D, Cyrulik A, Davis RB, et al. Clinicians’ assessments of electronic medication safety alerts in ambulatory care. Arch Intern Med. 2009;169(17):1627–32.

Baysari MT, Zheng WY, Van Dort B, Reid-Anderson H, Gronski M, Kenny E. A late attempt to involve end users in the design of medication-related alerts: Survey Study. J Med Internet Res. 2020;22(3):e14855.

Trafton J, Martins S, Michel M, Lewis E, Wang D, Combs A, et al. Evaluation of the acceptability and usability of a decision support system to encourage safe and effective use of opioid therapy for chronic, noncancer pain by primary care providers. Pain Med. 2010;11(4):575–85.

Wipfli R, Betrancourt M, Guardia A, Lovis C. A qualitative analysis of prescription activity and alert usage in a computerized physician order entry system. Stud Health Technol Inform. 2011;169:940–4.

PubMed   Google Scholar  

Cornu P, Steurbaut S, De Beukeleer M, Putman K, van de Velde R, Dupont AG. Physician’s expectations regarding prescribing clinical decision support systems in a Belgian hospital. Acta Clin Belg. 2014;69(3):157–64.

Ahearn MD, Kerr SJ. General practitioners’ perceptions of the pharmaceutical decision-support tools in their prescribing software. Med J Australia. 2003;179(1):34–7.

Swaby L, Shu P, Hind D, Sutherland K. The use of cognitive task analysis in clinical and health services research - a systematic review. Pilot Feasibility Stud. 2022;8(1):57.

Steyerberg EW. Applications of prediction models. In: Steyerberg EW, editor. Clinical prediction models. New York: Springer-; 2009. pp. 11–31.

Chapter   Google Scholar  

Steyerberg EW, Vergouwe Y. Towards better clinical prediction models: seven steps for development and an ABCD for validation. Eur Heart J. 2014;35(29):1925–31.

Tonekaboni S, Joshi S, McCradden MD, Goldenberg A. What Clinicians Want: Contextualizing Explainable Machine Learning for Clinical End Use. In: Doshi-Velez F, Fackler J, Jung K, Kale D, Ranganath R, Wallace B, Wiens J, editors. Proceedings of the 4th Machine Learning for Healthcare Conference; Proceedings of Machine Learning Research: PMLR; 2019;106:359–80.

Eini-Porat B, Amir O, Eytan D, Shalit U. Tell me something interesting: clinical utility of machine learning prediction models in the ICU. J Biomed Inform. 2022;132:104107.

Muralitharan S, Nelson W, Di S, McGillion M, Devereaux PJ, Barr NG, Petch J. Machine learning-based early warning systems for clinical deterioration: systematic scoping review. J Med Internet Res. 2021;23(2):e25187.

Rudin C. Stop Explaining Black Box Machine Learning Models for high stakes decisions and use interpretable models instead. Nat Mach Intell. 2019;1(5):206–15.

Blythe R, Parsons R, Barnett AG, McPhail SM, White NM. Vital signs-based deterioration prediction model assumptions can lead to losses in prediction performance. J Clin Epidemiol. 2023;159:106–15.

Futoma J, Simons M, Panch T, Doshi-Velez F, Celi LA. The myth of generalisability in clinical research and machine learning in health care. Lancet Digit Health. 2020;2(9):e489–92.

Steyerberg EW, Uno H, Ioannidis JPA, van Calster B, Collaborators. Poor performance of clinical prediction models: the harm of commonly applied methods. J Clin Epidemiol. 2018;98:133–43.

Download references

Acknowledgements

We would like to thank the participants who made time in their busy clinical schedules to speak to us and offer their support in recruitment.

This work was supported by the Digital Health Cooperative Research Centre (“DHCRC”). DHCRC is funded under the Commonwealth’s Cooperative Research Centres (CRC) Program. SMM was supported by an NHMRC-administered fellowships (#1181138).

Author information

Authors and affiliations.

Australian Centre for Health Services Innovation and Centre for Healthcare Transformation, School of Public Health and Social Work, Faculty of Health, Queensland University of Technology, 60 Musk Ave, Kelvin Grove, Brisbane, QLD, 4059, Australia

Robin Blythe, Sundresan Naicker, Nicole White & Steven M McPhail

Princess Alexandra Hospital, Metro South Health, Woolloongabba, QLD, Australia

Raelene Donovan & Andrew McKelliget

Queensland Digital Health Centre, Faculty of Medicine, University of Queensland, Brisbane, QLD, Australia

Ian A. Scott

Digital Health and Informatics Directorate, Metro South Health, Woolloongabba, QLD, Australia

Ian A. Scott & Steven M McPhail

You can also search for this author in PubMed   Google Scholar

Contributions

RB: conceptualisation, data acquisition, analysis, interpretation, writing. SN: data acquisition, analysis, interpretation, writing. NW: interpretation, writing. RD: data acquisition, interpretation, writing. IS: data acquisition, analysis, interpretation, writing. AM: data acquisition, interpretation, writing. SM: conceptualisation, data acquisition, analysis, interpretation, writing. All authors have approved the submitted version and agree to be accountable for the integrity and accuracy of the work.

Corresponding author

Correspondence to Robin Blythe .

Ethics declarations

Ethics approval and consent to participate.

This study was approved by the Metro South Human Research Ethics Committee (HREC/2022/QMS/84205). Informed consent was obtained prior to interview scheduling, with all participants filling out a participant information and consent form. Consent forms were approved by the ethics committee. Participation was entirely voluntary, and could be withdrawn at any time. All responses were explicitly deemed confidential, with only the first two study authors and the participant privy to the research data. Interviews were then conducted in accordance with Metro South Health and Queensland University of Technology qualitative research regulations. For further information, please contact the corresponding author.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary material 1., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Blythe, R., Naicker, S., White, N. et al. Clinician perspectives and recommendations regarding design of clinical prediction models for deteriorating patients in acute care. BMC Med Inform Decis Mak 24 , 241 (2024). https://doi.org/10.1186/s12911-024-02647-4

Download citation

Received : 06 September 2023

Accepted : 23 August 2024

Published : 02 September 2024

DOI : https://doi.org/10.1186/s12911-024-02647-4

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Clinical prediction models
• Clinical decision support systems
• Early warning score
• Clinical deterioration
• Clinical decision-making

BMC Medical Informatics and Decision Making

ISSN: 1472-6947

• General enquiries: [email protected]

Loading metrics

Open Access

Peer-reviewed

Research Article

Perpetrators of gender-based workplace violence amongst nurses and physicians–A scoping review of the literature

Roles Conceptualization, Data curation, Formal analysis, Methodology, Software, Validation, Writing – original draft, Writing – review & editing

* E-mail: [email protected]

Affiliation Lawrence S. Bloomberg Faculty of Nursing, University of Toronto, Toronto, Ontario, Canada

Roles Data curation, Formal analysis, Investigation, Validation, Writing – review & editing

Affiliation Princess Margaret Cancer Centre, Toronto, Canada

Roles Conceptualization, Investigation, Methodology, Validation, Writing – review & editing

Roles Formal analysis, Investigation, Writing – original draft

Roles Conceptualization, Formal analysis, Investigation, Methodology, Supervision, Validation, Writing – review & editing

• Basnama Ayaz, 
• Graham Dozois, 
• Andrea L. Baumann, 
• Adam Fuseini, 
• Sioban Nelson

• Published: September 6, 2024
• https://doi.org/10.1371/journal.pgph.0003646
• Peer Review
• Reader Comments

In healthcare settings worldwide, workplace violence (WPV) has been extensively studied. However, significantly less is known about gender-based WPV and the characteristics of perpetrators. We conducted a comprehensive scoping review on Type II (directed by consumers) and Type III (perpetuated by healthcare workers) gender based-WPV among nurses and physicians globally. For the review, we followed the Preferred Reporting Items for Systematic and Meta Analyses extension for Scoping Review (PRISMA-ScR). The protocol for the comprehensive review was registered on the Open Science Framework on January 14, 2022, at https://osf.io/t4pfb/ . A systematic search in five health and social science databases yielded 178 relevant studies that indicated types of perpetrators, with only 34 providing descriptive data for perpetrators’ gender. Across both types of WPV, men (65.1%) were more frequently responsible for perpetuating WPV compared to women (28.2%) and both genders (6.7%). Type II WPV, demonstrated a higher incidence of violence against women; linked to the gendered roles, stereotypes, and societal expectations that allocate specific responsibilities based on gender. Type III WPV was further categorized into Type III-A (horizontal) and Type III-B (vertical). With Type III WPV, gendered power structures and stereotypes contributed to a permissive environment for violence by men and women that victimized more women. These revelations emphasize the pressing need for gender-sensitive strategies for addressing WPV within the healthcare sector. Policymakers must prioritize the security of healthcare workers, especially women, through reforms and zero-tolerance policies. Promoting gender equality and empowerment within the workforce and leadership is pivotal. Additionally, creating a culture of inclusivity, support, and respect, led by senior leadership, acknowledging WPV as a structural issue and enabling an open dialogue across all levels are essential for combating this pervasive problem.

Citation: Ayaz B, Dozois G, Baumann AL, Fuseini A, Nelson S (2024) Perpetrators of gender-based workplace violence amongst nurses and physicians–A scoping review of the literature. PLOS Glob Public Health 4(9): e0003646. https://doi.org/10.1371/journal.pgph.0003646

Editor: Tanmay Bagade, The University of Newcastle Australia: University of Newcastle, AUSTRALIA

Received: May 22, 2024; Accepted: August 2, 2024; Published: September 6, 2024

Copyright: © 2024 Ayaz et al. This is an open access article distributed under the terms of the Creative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: All relevant data are included in the manuscript and its supporting information files.

Funding: The authors received no specific funding for this work.

Competing interests: The authors have declared that no competing interests exist.

Introduction

The International Labour Office (ILO), the International Council of Nurses (ICN), the World Health Organization (WHO), and Public Services International (PSI) defined WPV as "incidents where staff are abused, threatened or assaulted in circumstances related to their work, including commuting to and from work, involving an explicit or implicit challenge to their safety, well-being or health" [ 1 ]. The ILO [ 2 ] further defined "gender-based violence and harassment means violence and harassment directed at persons because of their sex or gender, or affecting persons of a particular sex or gender disproportionately".

Irrespective of industry, workplace violence (WPV) can cause lasting trauma and injuries and is a serious threat to human and health resources. WPV includes physical and psychological violence, including physical assault, verbal abuse, sexual harassment, and bullying. Gender-based workplace violence (GB-WPV), which is experienced across operational layers of an organization (horizontal) and organizational hierarchy (vertical), reinforces the differential risk for exposure and outcomes of violence for men and women [ 3 ]. Despite extensive research on workplace violence in healthcare, GB-WPV, its perpetrators, and its impact on healthcare professionals remains understudied. We presented the sex-segregated prevalence and risk factors for WPV somewhere else [ 4 ]. The earlier paper focused on the scope and scale of workplace violence (WPV), risk factors and its impact on men and women. As part of the same scoping review protocol, this paper reports on GB-WPV perpetrators. It specifically focuses on explaining the root causes of violent acts by individuals and the triggers and circumstances to provide gender-sensitive recommendations.

A systematic review [ 5 ] of the consequences of exposure to WPV in the healthcare setting based on 68 studies reported psychological and emotional effects such as post-traumatic stress, depression, anger, and fear. These effects impact work productivity, leading to increased sick leaves, poor job satisfaction, burnout, and higher attrition rates, particularly for women [ 5 ]. Studies have also shown that men are more likely to commit physical violence [ 6 ] and sexual harassment [ 7 , 8 ], while women are more often engaged in verbal abuse [ 9 ]. Moreover, gender stereotypes and inequalities in the distribution of roles and responsibilities can worsen power imbalances [ 3 ]. By recognizing and understanding these issues, employers and organizations can more effectively prevent and deal with gender-based workplace violence, ensuring a safer and more equitable work environment for everyone.

The classification of workplace violence has evolved, delineating distinct categories shaped by the nature of its perpetrators. The current working taxonomy categorizes WPV into four types based on the perpetrators of violence. This typology, as shown in Table 1 , emerged from a collaborative effort of a workshop on workplace violence intervention research held in Washington, DC, in 2000. The findings of this endeavour were subsequently published by the U.S. Department of Justice in 2001 [ 13 ]. Since then, this framework has been adopted by multiple organizations [ 10 , 11 , 14 ], and by researchers [ 12 , 15 ].

• PPT PowerPoint slide
• PNG larger image
• TIFF original image

https://doi.org/10.1371/journal.pgph.0003646.t001

This paper explores the dynamics of workplace violence by categorizing and summarizing both Type II WPV, from patients and significant others, and Type III WPV (horizontal and vertical), which pertain to violence perpetuated by colleagues, supervisors, and administrators within the organization. Additionally, we explore the nuances of GB-WPV, considering both the perpetrators and nurses and physicians as victims of WPV. We summarized perpetrators based on their gender and synthesized the factors attributed to Type II and III, which are prevalent forms of violence reported in the literature. Type I and Type IV violence are beyond the scope of this paper as they focus on a security-based rather than workplace culture interventions. Understanding the factors contributing to these types of WPV is crucial to developing effective preventive interventions and strategies. Currently, there is a dearth of information identifying the characteristics of individuals who are more likely to commit GB-WPV and the characteristics of those targeted by such offences. This review addresses this gap by synthesizing data from studies that reported on the gender/sex data for various forms (please see S1 Text : Definitions of the Forms of Violence) of WPV and perpetrators of WPV among nurses and physicians.

While WPV affects individuals across the gender spectrum and in different professional groups, women are disproportionately affected. Some studies attributed it to their preponderance in the health workforce, their overrepresentation in lower positions in organizational and professional hierarchies, and societal gender norms in most cultures that subjugate women [ 9 , 16 ]. Recognizing that workplace violence is fundamentally intertwined with broader societal structures rooted in socioeconomic, cultural, and institutional factors, we underscore the necessity of a systematic approach to address this issue—one that is integrated, participatory, culturally and gender-sensitive, and non-discriminatory [ 1 ]. While current interventions aimed at addressing WPV primarily focus on assessing the effectiveness of training interventions to prevent and manage WPV in healthcare settings [ 17 – 19 ], they often lack gender-segregated findings for their effectiveness. Clarifying the existing situation on the gender of victims and perpetrators for specific Type/s of violence would help develop gender-sensitive interventions and policies to more effectively address WPV. This scoping review focuses on understanding GB-WPV and its perpetrators in the global health workforce, including nurses and physicians. Our preliminary search for a scoping review revealed that GB-WPV affects men, women, and non-binary persons. However, most studies included in this review reported gender as binary (men and women); only a few studies included non-binary personnel (for sample-see Table 2 , in results section). Therefore, we defined gender as a binary for this review and deliberated on it in the discussion section.

https://doi.org/10.1371/journal.pgph.0003646.t002

Our specific objectives set out for this paper were:

• Describe the proportions of WPV and related perpetrator/s contributing to Type II (from patients/clients/families) and Type III (worker-to-worker) violence among nurses and physicians in different contexts.
• Summarize the gendered perpetration of Type II and Type III WPV against men and women in the health workforce.
• Identify gaps in the state of knowledge to recommend direction for future empirical research studies.

Protocol registration and study design

Following the Joanna Briggs Institute (JBI) revised guidelines, we conducted a scoping review. The protocol for this review was registered on the Open Science Framework on January 14, 2022, and is accessible at https://osf.io/t4pfb/ and S2 Text : Registered Protocol. The scoping literature review design addressed the research questions and accommodated the heterogeneous and complex nature of the literature. This method is appropriate for exploring the extent of the literature, mapping and summarizing the evidence, and identifying and analyzing knowledge gaps to inform future research. The framework used for this review consists of eight steps; they are built upon the seminal framework of Arksey and O’Malley’s scoping review, which was further developed by Levac and colleagues. The revised guidelines of JBI align these eight steps with the Preferred Reporting Items for Systematic and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR), ensuring rigour, transparency, and trustworthiness in Reporting the conduct of the scoping review. The first step of the scoping review framework is to align with research objectives, the title, and the inclusion criteria, as well as the exclusion criteria (see Box 1 ). Please see S1 Checklist : PRISMA-ScR Checklist.

Box 1. Study selection criteria.

Inclusion Criteria for Studies

1. The study participants included nurses and/ or physicians who experienced WPV during their careers.

2. Provided sex-segregated data for any form of violence and any type of perpetrators among nurses and physicians, including students, globally.

3. Published in English and after 2010.

Exclusion criteria

4. Studies that did not provide sex-segregated data and information for perpetrators.

5. Exclude systematic/ scoping reviews, concept or theoretical papers, and theses.

Search strategy

The research team collaborated with a health sciences librarian to develop a comprehensive search strategy. The systematic search focused on published literature in various databases, including Ovid MEDLINE: Epub Ahead of Print, In-Process and Other Non-Indexed Citations, which were translated in CINAHL Plus, APA PsycINFO, Web of Sciences, and Gender Studies Databases, Applied Social Sciences Index & Abstracts (ASSIA), and Sociological Abstracts ( S3 Text : Ovid MEDLINE search strategy, which was translated in all other databases). The search terms related to the population (midwifery, nursing, and physicians), concepts (violence and gender-based violence), and context (healthcare) were combined appropriately based on the scoping review objectives. These terms were identified through a preliminary literature search on various aspects of workplace violence in Google Scholar. The final search results were exported to EndNote, a citation manager, to de-duplicate sources from multiple databases. After de-duplication, the sources were imported into the Covidence online software program that streamlined the screening process by two independent reviewers. The final search of the literature review for this study was conducted on 11 th February 2024.

Evidence screening and selection

The identified sources were screened based on the inclusion criteria ( S4 Text : Excluded Sources). Two independent reviewers screened the titles and abstracts to shortlist the sources. Discrepancies were resolved through discussion and consensus, with a complete source review conducted, if necessary, followed by a full-text review against the inclusion criteria by two reviewers. The selection process is presented in the PRISMA diagram ( Fig 1 ). Given the overall objective of the review to map the most frequent forms and prevalence of GB-WPV for midwives, nurses, and physicians in different contexts and clinical settings, a quality assessment of the identified sources was not conducted.

https://doi.org/10.1371/journal.pgph.0003646.g001

Data analysis and synthesis of results

This paper is a component of a multi-part scoping review; it reports on the perpetrators of WPV from gender-segregated prevalence data reported from a global context among the health workforce, including nurses and physicians. The prevalence and risk factors have been reported elsewhere [ 4 ]. This paper reports on Type II and Type III (vertical and horizontal) WPV perpetrators. Data from all sources ( S1 Data ) that reported sex/gender segregated findings and provided information for the types of perpetrators were included in mapping the prevalence of GB-WPV (See Table 2 ) for several types/forms of WPV and the clinical setting across countries/special regions. We could not calculate a mean score for various forms of violence based on gender for all the studies that provided information on perpetrators because of the wide variability in the operational definitions of the terms and the concepts in these studies. These studies also did not consistently provide quantifiable data for the Types of perpetrators. Only 34 studies (19%) provided the gender of perpetrators. We summarized the proportion of male and female perpetrators in those studies for Type II, Type III-A (horizontal) and Type III-B (Vertical) violence (see Table 3 ).

https://doi.org/10.1371/journal.pgph.0003646.t003

After de-duplication, 8435 possible references were imported for screening in the Covidence. These studies were screened against the title by one person, 1551 were shortlisted to be screened (for title and abstract) by two independent reviewers, and 402 were assessed for full-text eligibility. After applying the inclusion and exclusion criteria, 178 [ 6 – 9 , 15 , 20 – 125 ] studies were retained (PRISMA diagram, Fig 1 ) and analyzed to report on perpetrators that provided gender-segregated findings for WPV and information on various types of perpetrators ( Table 2 ). We included studies published between 2010–2024. The most common study design was quantitative, cross-sectional (n = 168), mixed methods (n = 4), and qualitative methods (n = 6).

Perpetrators for the three types of violence

A total of 178 studies provided information on the perpetrators of either Type II (consumers/patients, including patients’ companions), Type III-A (from colleagues), and Type III-B (from administrators and superior authorities within and between professions) violence. Studies included in this review did not consistently provide data for all types of violence and perpetrators; instead, they provided data for any Type/s. Of 178 studies, 141 (79%) reported perpetrators for Type II violence, followed by 93 (52.2%) for Type III-B (vertical) and 92 (51.6%) for Type III-A (horizontal) violence. Only 40 (22.5%) studies [ 9 , 21 – 53 , 164 , 168 , 172 , 177 , 178 , 191 ] reported information about all three types of violence.

While the search terms yielded many studies, there was significantly less information on the gender of perpetrators of WPV. Of the 178 studies reported on perpetrators, only 34 studies provided data for perpetrators’ gender (detailed in Table 3 ). Across the three types of violence, more men (65%) were responsible for perpetrating WPV compared to women (28%). Both men and women perpetuated violence in the remaining 7% of cases. Of the 34 studies, 25 studies reported on Type II violence, predominantly perpetrated by men, encompassing general violence [ 37 , 40 , 53 – 60 ], physical violence [ 6 , 35 , 44 , 61 – 63 , 178 ], verbal violence [ 6 , 35 , 44 , 53 , 63 , 172 , 178 ], and sexual harassment [ 8 , 41 , 44 , 63 , 64 ]. In most of these studies, women experienced a higher prevalence of violence than men. Gender-based workplace violence against nurses emerged as a pressing issue for Type II (56.2%) violence in ten studies [ 6 , 8 , 35 , 40 , 53 – 56 , 168 , 178 ]; men perpetrated 80% of the violence while women were responsible for only 19% violence, and almost all studies reported a higher prevalence of WPV against female nurses. A recent study [ 168 ] from 79 countries, though reported gender was not significant for WPV, being a nurse had higher odds of experiencing WPV (OR = 1.95; 95% CI 1.46 to 2.59, p<0.001) than a physician (OR = 1.70; 95% CI 1.33 to 2.18, p<0.001). In this study, most perpetrators were consumers (56%), followed by supervisors (16%) and colleagues (9%), or a combination of all (19%).

Violence perpetrated by colleagues (Type III-A) was reported by 15 studies, including seven for physicians [ 41 , 43 , 65 – 68 , 172 ], three studies for nurses [ 40 , 53 , 178 ], and five that included both professionals [ 29 , 37 , 44 , 60 , 63 ]. Approximately 24% of violence was perpetrated by colleagues (Type III-A) among nurses and physicians. More perpetrators were men (63.5%) than women (23%), and some violence by colleagues was reported as perpetrated by both men and women (13.5%). Only one study [ 29 ] reported higher rates of bullying by women (37.9%) than men (10.5%) and by both genders (51.6%). Two other studies reported higher mobbing behaviours (20% Vs. 69%) (192) and (8%vs.93%) (72) by women. In these studies, most perpetrators (40.7%) were supervisors and senior colleagues (Type III-B). Victims were both physicians (53.1%) and nurses (53.6%) with similar intensity, but a higher number of women (n = 195, 56.4%) were exposed to bullying than men (n = 18, 36%). Additionally, those who experienced bullying had lower levels of psychological health status. Bullying from colleagues (26.4%) and patients/consumers (7.7%) was perceived as less harmful than bullying from supervisors (Type III-B), which was also less reported because of the fear of consequences.

Of the 34 studies reporting on the gendered perpetuation of WPV ( Table 3 ), 24 reported on Type III-B (vertical) violence, which was more prevalent among physicians (51.5%) than nurses (16%). When it did occur among nurses, more men (77%) perpetuated Type III-B violence than women (18%) and both men and women (5%). Several studies highlighted physicians as perpetrators of WPV against nurses regardless of gender [ 8 , 51 , 53 ]. Similarly, more men (67.5%) than women (24.2%) and both genders (8.2%) perpetuated Type III-B violence among physicians. In seven of ten studies (70%) for Type III-B violence among physicians, male supervisors and administrators perpetuated sexual harassment [ 41 , 64 – 69 ]. Four studies reported bullying [ 43 , 70 , 172 ] and emotional abuse [ 71 ], which was also perpetrated by men.

Medical residents appear to be particularly vulnerable to Type III-B violence, with more than 60% of studies [ 41 , 43 , 64 , 66 , 67 , 71 , 172 , 175 ] reporting this type of violence in medical residency programs. Furthermore, several studies highlighted that the perpetrator of sexual harassment was most often of the opposite sex [ 63 , 64 , 66 ]. For instance, Freedman-Weiss et al. [ 66 ] reported that male residents experienced 65.9% of harassment from men compared to 81.8% from women. On the other hand, female residents reported experiencing more harassment from men (97.7%) compared to women (42.4%). In the same study, the main perpetrators for female resident victims were attending physicians (72.9%), followed by nurses (68.5%), senior colleagues (44.7%) and same-level residents (23.5%). Among male residents, nurses were the most common perpetrator of WPV (69%), followed by attending physicians (62%), senior colleagues (41.9%) and same-level residents (25.6%).

Healthcare professionals in lower hierarchical positions, such as nurses and residents, often contend with stressful conditions and managerial or administrative abuse and harassment, posing challenges to patient care, institutional integrity, and the healthcare system. These experiences also detrimentally impact the victims’ health and career progression. For instance, Tekin and Bulut [ 51 ] found that Turkish nurses who experienced Type III-B violence reported feelings of anger, humiliation, confusion and sadness. Moreover, these experiences also led to strained relationships with others, decreased performance, and caused them to consider leaving the profession. Although this study did not specify the gender of the offender, women experienced significantly higher verbal abuse. The highest perpetuation for all forms of abuse, including verbal (85.7%), physical (46.4%) and sexual (94.4%), was from physicians. In these cases, gender and status within the organizational hierarchy played a critical role in perpetuating Type III-A and III-B WPV, which requires serious attention from employers and health organizations to address GB-WPV through a gender-sensitive approach.

Our examination explores the complexities of gender dynamics concerning both the perpetrator and the victims of workplace violence within the global healthcare community, mainly focusing on nurses and physicians. While 178 studies provided information about perpetrators and sex-segregated findings for workplace violence, only 34 studies (19%) reported the gender of the perpetrator for Type II and Type III violence. These findings provided insights into how gender and an individual’s position within the organization create unique vulnerabilities to WPV. The consequences of such violence against health workers not only affect patient care but also have broader implications for healthcare organizations and workforce landscapes. In our review, men were found to be the primary instigators, accounting for 65% of incidences of WPV, while women were responsible for 28% of instances. Both men and women perpetrated the remaining 7% of incidents. Additionally, our analysis identified distinctive behaviour patterns among male and female offenders. Recognizing that each type of violence requires a different approach for its management and prevention, we will discuss the divergent behavioural patterns of men and women perpetrators of Type II and Type III violence. We examine the underlying factors contributing to these differences and discuss the implications of adopting gender-sensitive approaches to prevent and manage GB-WPV.

Type II WPV- Client/patient

Of the 34 studies that provided the gender of perpetrators for any type/s of violence, the majority (74%, n = 25) reported on Type II WPV perpetrated by patients, their families, or visitors. In this context, male perpetrators were more prevalent, targeting both nurses (77.9%) and physicians (70%). The majority of studies that reported on Type II violence indicated a higher prevalence of various forms of violence against female nurses and physicians. The higher perpetration of WPV by men can be linked to societal norms associating aggression and dominance with masculinity [ 193 ]. At the same time, violence against a feminized nursing workforce is normalized as part of the job [ 24 , 75 , 98 , 193 ]. This link between societal norms and assigned roles was evident in several studies [ 76 , 125 ], which is deliberated in the following section.

Type II violence typically targets healthcare providers in the performance of their professional duties and is characterized by acts of physical violence [ 6 , 35 , 44 , 61 – 63 , 166 ]; verbal violence [ 6 , 35 , 44 , 53 , 63 ], and sexual harassment/ violence [ 8 , 41 , 44 , 63 , 64 , 167 ]. Most of these studies reported a higher prevalence of WPV for women for all forms of violence [ 8 , 9 , 44 , 53 , 62 , 64 , 169 , 174 , 176 ]. The social norms, which stem from social relations dictate gender roles and responsibilities, and healthcare institutions are no exception to these forces. For example, a study conducted in Italy that included all areas of practice and the entire health workforce, investigating determinants of aggression against the health workforce reported women were 1.37 times more likely to experience aggression from consumers and colleagues. In this study, nurses experienced the highest number of episodes of violence (64%). Most of these aggressive acts occurred during assistance and supportive care to patients (38%) [ 125 ]. On the other hand, men were not immune to WPV, particularly physical [ 44 , 61 , 166 , 185 ] and both physical and verbal violence in the emergency department in Saudi Arabia, Turkey and China [ 37 , 59 , 189 ]. In Turkey, male physicians experienced higher violence (62.4%) in contrast to their female counterparts (37.6%) [ 59 ]. A similar pattern emerged in Saudi Arabia, with male physicians and nurses reporting a higher prevalence (57.8%), than their female counterparts (42.8%) [ 37 ]. These three studies identified several factors for the high occurrence of WPV from patients and their relatives, including dissatisfaction with the treatment, long wait times and lack of staff [ 37 , 59 ], overcrowding and lack of security [ 37 ]. Though these highlighted factors are important to explain the occurrence of workplace violence for both men and women in the workforce, in the Saudi context, culture seems to have a protective factor for women, where public abuse from men is socially unacceptable [ 88 ]. Similarly, three other studies in Jordan attributed the higher prevalence among male physicians to culture and the existence of laws that intensify legal penalties against women abusers [ 87 ], the cultural norm of altruism and tolerance towards females, particularly physical violence [ 42 ], and a lack of encouragement for reporting WPV by females as part of the male-dominant culture [ 150 ]. Additionally, the higher occurrence of physical violence for men can also be explained by the cultural expectation of masculinity.

In contrast, women’s experience of severe sexual harassment was associated with pregnancy, family responsibilities, and occupational segregation [ 63 ]. Newman et al. [ 63 ] explained that occupational segregation also creates a vertical hierarchy where women are assigned to lower-level tasks (typically front-line care providers). The WHO report analyzed gender and equity in the health and social workforce ‘delivered by women, led by men’ (2019) and acknowledged occupational segregation as universal, which is reinforced by the broader societal norms and creates discriminatory practices with regard to gender and occupational roles [ 194 ]. In these lower positions, women experience sexual harassment from male colleagues, male patients and community members [ 16 , 194 ]. Considering the prevalence of Type II violence for both men and women linked to sex-segregated responsibilities and societal structures. Jafree [ 195 ] calls on policymakers to ensure security and protection for the health workforce, particularly women; legislative reforms for healthcare governance and zero-tolerance policies for violence were also recommended. Several other sources, too, advocate for zero-tolerance policies and emphasize the need for a managerial approach that takes all complaints seriously, reports investigation outcomes, and enforces sanctions to eliminate impunity [ 9 , 92 , 131 ]. Collaborative community efforts are required to acknowledge and alter the patriarchal culture and reduce violence against women by creating awareness about the public role through various forums, including the media [ 28 , 79 , 94 , 195 ].

Several contributing factors have been identified in the context of Type II WPV, such as noise levels, inadequate communication skills [ 74 ], perceived/actual staff incompetence or unsympathetic attitudes, dissatisfaction with service provision, prolonged wait times, and poor communication [ 53 , 196 ]. These circumstances can escalate emotions and increase the likelihood of violent encounters. Furthermore, specific treatment specialties, such as emergency departments [ 35 , 75 , 191 ], psychiatric units [ 76 , 77 ], and geriatric care [ 26 , 76 ], have demonstrated a higher risk of Type II workplace violence. Factors specific to these settings include a lack of privacy and personal space, unrealistic expectations of clients, insufficient staffing and resources, poor staff skills mix, healthcare systems and processes not understood by clients, perceived favouritism, overcrowding in emergency departments, delays in providing analgesia, and inflexible visiting hours [ 196 ]. These challenges, compounded by a shortage of skilled professionals, unclear expectations and communication, scheduling issues, and environmental stressors can generate increased stress and, thus, uncertainty. Addressing these factors constitutes the initial step in decreasing or eliminating the risk of violence for both men and women [ 197 ].

Both primary research and systematic reviews have acknowledged the difficulty associated with addressing multifactorial violence, given the diversity in population and setting and the types/classifications of violence [ 94 , 95 , 102 , 198 ]. However, these sources did not provide information about perpetuators, particularly gendered nature. For instance, a recent umbrella review examined 32 systematic reviews for WPV prevalence and characteristics. This comprehensive assessment reported that the overall prevalence from the meta-analysis of 11 reviews was 57.9%, ranging from 34.1% to 78.9% among healthcare providers and most affected were nurses working in psychiatric wards [ 198 ]. This prevalence aligns with the findings of this review. Of note, the umbrella review too did not provide information on perpetrators and prevalence based on gender and stated that the included reviews had reported variable results for men and women; however, it did underscore how gender imbalances in emergency departments could increase the risk of violence among women. Several studies in our review recommended ensuring gender equality in the health workforce and leadership positions to reduce the prevalence of WPV among women [ 9 , 30 , 63 , 80 ].

Type III WPV-Worker-on-Worker

Type III-A (Horizontal or lateral) workplace violence perpetrated by one healthcare worker against another may stem from interpersonal conflicts, workplace stress [ 12 ], or other factors contributing to a hostile work environment. Among studies that provided data on Type III-A violence, most perpetrators were men (63.5%) compared to women (23%). Horizontal WPV was reported more frequently by physicians [ 41 , 43 , 65 – 69 ] than among nurses [ 40 , 53 ]. The studies that sampled both nurses and physicians [ 29 , 37 , 44 , 60 , 63 ] also reported that men perpetuated all forms of violence in most cases for both male and female victims [ 37 , 44 , 63 ]. In some instances, women experienced violence from both men and women [ 63 ].

Type III violence is also rooted in cultural norms and societal expectations that allocate roles and responsibilities based on gender [ 63 ]; in most cultures, women are responsible for childbearing and rearing and men hold decision-making positions. This phenomenon transcends the household and is also seen in the workplace and healthcare institutions [ 9 , 68 , 78 ]. These gendered roles and responsibilities often position men in leadership positions while women are assigned to caring roles with less authority and responsibility, perpetuating discriminatory practices that negatively impact women [ 9 , 63 , 70 ]. This dynamic prevails in both wealthy and lower- and middle-income countries in varied behaviors. For instance, in Australia and New Zealand, women experienced significantly higher discrimination (31% vs. 8%) and sexual harassment (23% vs. 0.5%) than men, primarily due to family responsibilities, lack of mentorship and rigid promotion criteria [ 70 ]. In Rwanda, women’s experiences of childbearing and care, including managing pregnancy, motherhood and work, and the widespread negative stereotyping of women at work led to discrimination that co-occurred with sexual harassment within health workplaces [ 63 ]. Jacobson et al. [ 12 ] report on Type III-A violence in medical residency programs, and women experienced a significantly higher frequency of work-related incidents from colleagues and support staff, explaining the higher workload for women due to the coexistence of family responsibilities. Additionally, relational and managerial issues, including organizational affairs within large, complex health organizations, shifting duties and cohabitation of various teams on the same unit, were identified as factors contributing to Type III-A violence in Italy [ 53 ].

This type of interpersonal violence, including violence against women, is prevalent in science, technology, engineering and math (STEM), which are considered male-dominant disciplines [ 199 , 200 ], unlike healthcare, where 70% of the workforce globally are women and higher rates of violence are associated with their roles and responsibilities and the gendered workplace hierarchy [ 194 ]. In STEM, violence against women can be explained by the backlash effect, in which gender equality is associated with higher prevalence [ 200 ].

Given the social reality of women’s lives and career development in healthcare, flexible human resource development and management policies could empower women to balance their work and family responsibilities. Zampieroni et al. [ 53 ] recommend adopting realistic workloads and skill-mixed staffing, promoting gender equality in staff allocation, and participatory leadership to overcome relational conflict and managerial actions that enhance working conditions. Nurse managers must play the role of cultural gatekeepers, hold individuals accountable and foster staff empowerment; utilizing research-informed methods such as ‘cognitive rehearsal and crucial conversations’ [ 20 ] and conducting team-building workshops will assist in mitigating the impact of horizontal violence [ 21 ].

Type III-B (vertical) violence is primarily perpetrated by senior colleagues, supervisors, and administrative personnel occupying higher positions in the organizational hierarchy than the victim. Among the 34 studies, 66% reported perpetrators’ gender for Type III-B violence; men perpetuated 77% among nurses and 67.5% among physicians. The causative factors for vertical violence included organizational structure, leadership and administrative authorities, and power struggles in the health workplaces. These factors not only perpetuated WPV but also prohibited reporting of the instances due to fear of reprisal [ 29 , 63 , 66 ]. Two prevalent forms of violence linked to hierarchical/ organizational structure were sexual harassment and bullying/mobbing. The majority of studies reporting Type III-B violence reported sexual violence from male supervisors and administrators [ 8 , 41 , 44 , 63 – 69 ], particularly in medical residency programs—placing these trainee residents in a vulnerable position [ 41 , 66 , 67 ]. Additionally, vertical violence was the only type reported to be perpetuated by women at higher levels in the organizational hierarchy, particularly bullying (women: 37.9% vs. men: 10.5%) among nurses and physicians [ 29 ]. Additionally, two studies reported higher rates of mobbing behaviours by women than men among healthcare professionals, including nurses and physicians [ 72 , 192 ].

Type III-B violence is emblematic of the hierarchical and inflexible organizational culture historically dominated by male medical professionals. This stemmed from beliefs and negative stereotypes, such as women being weak, unwilling to speak up, indecisive and incompetent [ 63 ]. Additionally, perceived competence was expressed as a predictor for bullying among women [ 42 , 153 ]. Such perceptions reinforce the structural power held by men, particularly with male managers and physicians. The patriarchal institutional structures provide power domination among women as well, who could use their power to oppress individuals under their control. A qualitative study [ 201 ] from Estonia exposed this dynamic of domination and sexual harassment among nurses; it highlighted the association between power and the use of sexualized language. A female nurse stated, “I am more disturbed by their patronizing behaviour"; the nurse characterizes physicians’ attitude as: “I am a man, I am a physician, I can do and say whatever comes to my mind” (Nurse 18, p.30). Lamesoo [ 201 ] further explained that nurses placed themselves in the hospital hierarchy between physicians and patients and acknowledged that they could not challenge a physician’s incivility. However, these nurses can easily ask patients to refrain from such behaviour without hesitation because patients have less power than nurses, and patients are expected to follow hospital rules [ 201 ] dictated by nurses. These instances explain organizational power as a protective factor for offenders. However, women’s underrepresentation in positions of power places them in a vulnerable position.

Another qualitative study in Uganda by Newman et al. [ 9 ] reported from key informant interviews in the Uganda health system that "we have women over-represented in the bottom of any organization and for the men, it is an upward or inverted pyramid whereby as you go up the power ladder…. There is a tendency to abuse that power and they don’t even think that they are abusing it because they have grown up thinking they may be flattering the women…". The authors further stated that "Sexual coercion started during recruitment of health workers and continued after hiring, perpetrated by men in hierarchically superior decision-making positions supervisors, senior managers (including human resources) or medical superintendents” [ 9 ]. These severe human rights violations necessitate a transformation in the mindset of individuals in the workforce and a cultural shift at organizational levels to rectify the dominant, hierarchical and permissive environment [ 65 ]. Ensuring gender equality at the upper echelons of healthcare organizations and in decision-making positions is crucial to establishing a secure and equitable environment for all, regardless of gender. A scoping review of three evidence-based guidelines and 33 systematic reviews on strategies to prevent and manage WPV in healthcare settings reported a correlation between strong leadership to cultivate and enforce a culture of inclusivity, support and respect as a prerequisite for successful prevention of WPV [ 202 ]. Therefore, healthcare organizations’ leadership must proactively seek organizational solutions to end gender-based WPV and prioritize gender equality and protecting employees’ rights as part of their human resources for health (HRH) policy [ 9 ].

Sexual harassment in academia was found to be an issue across various contexts, particularly among women in medical residency programs. A study [ 78 ] in a U.S. medical college reported that one-third of respondents experienced sexual harassment, including medical students (51.7%), residents/fellows (31%) and faculty members (25%), which was inversely proportional to their position in the program. Similarly, sexual harassment was more prevalent among women in vascular surgery in the U.S. [ 67 ], ophthalmology in Australia and New Zealand [ 79 ] and cardiothoracic surgery, reported by a global survey [ 28 ], and rates of sexual harassment in almost all contexts were higher among female trainees. In one instance, in the U.S., male (70%) and female (69%) residents [ 41 ] in obstetrics and gynecology residents experienced sexual harassment at similar levels [ 41 ]. Additionally, one study in the U.S. with a large, representative sample (n = 6000) from a national survey reported that higher women’s representation within a specialty was associated with lower sexual harassment for both men and women from coworkers and patients [ 80 ]. This observation held true in the Canadian context where reporting of sexual harassment incidents was low (2.9%) in a study with female participants constituting 53% of the sample [ 33 ]. These women did report slightly higher rates of intimidation, harassment, and discrimination (IHD) based on gender (males 40.4%; females 48.0%). Hence, findings underscore the recurring recommendation of gender equality in the health workforce and leadership positions and the role of leadership in preventing Type II and Type III violence, including harassment.

Acknowledging sexual harassment as a prevalent problem is the crucial initial step in formulating a successful strategy to prevent its occurrence [ 65 , 203 ]; a comprehensive strategy should encompass a zero-tolerance statement across the specialty with a transparent and fair mechanism for reporting sexual harassment [ 65 , 78 ]. Moreover, it is essential to provide trainees with both direct face-to-face and electronic routes for anonymous and confidential reporting to alleviate concerns related to personal reattribution and academic detriment [ 64 , 78 ]. Standardized, transparent reporting mechanisms with well-delineated consequences for the offender must be established. Additionally, the institutions should ensure the availability of links to all the required resources is the first hit on online searches, displaying posters/presentations/ads [ 78 ].

Recognizing harassment as an institutional structural issue, senior leadership can have a protective role by serving as role models. A qualitative study conducted in Germany [ 197 ] representing women nurses (50%) and physicians (50%) explored preventive options for sexual harassment in academia. The findings revealed that leadership commitment and clear statements can significantly influence multiple levels by demonstrating openness to address taboo topics, raise awareness, and place the issue at the decision table. A participant stated, "A culture of political correctness is communicated from the top down, with the management committee and senior management acting as role models” (p.12). Another participant stated, "It is the senior staff that creates a team culture that should be supportive and transparent, with clear boundaries… .. I have an open door and open eyes policy and try to initiate rituals that allow us to work together in the correct way” (p.12). While commitment and stated actions are essential, meaningful cultural change necessitates the consistent, active, structured, and continued engagement of all health workforce members, including students and trainees, staff and especially from senior leadership. Senior leadership must be actively engaged in this process, particularly male leaders. Therefore, engaging individuals at various levels in open, nonjudgmental conversations is paramount to breaking the silence [ 30 ] and ingraining these principles into the organizational culture.

Limitations

First, in our comprehensive review of workplace violence (WPV), not all studies reported on perpetrators of WPV. Therefore, we included all studies that indicated perpetrators/ sources of violence. We categorized these sources into distinctive categories of Type II and Type III WPV. Limitations to this approach include the heterogeneity of the forms of violence reported by the included studies according to gender. While studies reported victims’ exposure to Types II, and/or III, the gender of perpetrators in each case was not specified. As a result, we presented the prevalence of the various forms and categorized the perpetrators’ type for all the studies (178) in Table 2 . The final set of studies (19%) that reported on the gender of the perpetrators was analyzed. Since fewer studies provided information about the gender of perpetrators across the types/forms of violence, future research must focus on conducting and reporting gender-segregated findings for perpetrators that will strengthen recognition of the gender-based WPV and could lead to gender-sensitive strategies at the local and international levels. Another limitation of our review was that most of the included studies operationalized gender as a binary. A few studies included either non-binary (less than 4%) [ 80 , 98 ] individuals or mentioned as others (less than 4%) [ 28 , 30 ] or unknown (less than 9%) [ 85 , 128 ], in the analysis of the total population, reported in Table 2 . Even these studies did not report findings for those minority populations or address it as a limitation. Therefore, we reported findings based on gender binary. All these studies, which represented non-binary individuals, were conducted in the USA [ 30 , 80 , 98 , 128 ] and Canada [ 85 ]; in these contexts, gender diversity and inclusion are acknowledged as compared to most Low-and -middle-income countries where sex is equated with gender. These studies did not recognize it as a limitation; only one study, which reported on survey data from the Association of American Medical Colleges (AAMC) National Sample Survey of Physicians (NSSP), expressed excluding the non-binary data because of the lower sample [ 80 ]. Considering this limitation, we recommend that future research include gender-diverse populations.

The review revealed a higher prevalence of Type II and Type III WPV among women compared to men. In parallel, it was observed that men predominantly perpetrated all forms of violence against both men and women healthcare providers. Only Type III-B violence, including bullying/ mobbing, was occasionally perpetuated by women. Both Types II and Type III violence have roots in societal structures, and women were more frequently victimized. This increased victimization of women can be attributed to their lower status in society and in the healthcare settings that assign roles and responsibilities based on this status. Additionally, women’s reproductive realities, including managing pregnancy, motherhood and work, and widespread negative stereotyping contributed to their vulnerability to gender-based WPV.

Conversely, men’s domination in leadership, decision-making and supervisory positions in most contexts creates a hierarchical and permissive environment that perpetuates violence against women. Therefore, understanding gender implications concerning both the victim and perpetrator among the critical health workforce of nurses and physicians across the globe is essential. Healthcare organizations and professional stakeholders must seriously consider zero-tolerance policies, transparent mechanisms for handling violent incidents, and the provision of appropriate support to victims. These measures will empower individual professionals, enhance patient care, and positively impact healthcare institutions and society as a whole.

Supporting information

S1 checklist. prisma-scr checklist..

https://doi.org/10.1371/journal.pgph.0003646.s001

S1 Text. Definitions of forms of workplace violence.

https://doi.org/10.1371/journal.pgph.0003646.s002

S2 Text. Registered protocol.

https://doi.org/10.1371/journal.pgph.0003646.s003

S3 Text. Ovid MEDLINE search strategy.

https://doi.org/10.1371/journal.pgph.0003646.s004

S4 Text. Sources excluded.

https://doi.org/10.1371/journal.pgph.0003646.s005

S1 Data. Data for full text review.

https://doi.org/10.1371/journal.pgph.0003646.s006

• 1. International Labour Organization, International Council of Nurses, World Health Organization, Public Services International. Framework guidelines for addressing workplace violence in the health sector. [Internet]. International Labour Office; 2002 [cited 2021 Apr 19]. Available from: http://www.icn.ch/images/stories/documents/pillars/sew/sew_framework_guidelines_for_addressing_workplace_violence.pdf
• 2. International Labour Organization. Convention C190—Violence and Harassment Convention, 2019 (No. 190) [Internet]. 2019 [cited 2024 Jul 10]. Available from: https://normlex.ilo.org/dyn/normlex/en/f?p=NORMLEXPUB:12100:0::NO::P12100_ILO_CODE:C190
• View Article
• Google Scholar
• PubMed/NCBI
• 10. Public Services Health and Safety Association. Public Services Health and Safety Association | Workplace Violence—Complying with the OHS Act [Internet]. Public Services Health and Safety Association. [cited 2023 Nov 14]. Available from: https://www.pshsa.ca/resources/workplace-violence-complying-with-the-ohs-act
• 11. Registered Nurses’ Association of Ontario. Best Practice Guideline:Preventing Violence, Harassment and Bullying Against Health Workers [Internet]. RNAO; 2019 [cited 2023 Nov 14]. Available from: https://rnao.ca/bpg/guidelines/preventing-violence-harassment-and-bullying-against-health-workers
• 13. Workplace Violence: A Report to the Nation | Office of Justice Programs [Internet]. 2001 [cited 2023 Sep 27]. Available from: https://www.ojp.gov/ncjrs/virtual-library/abstracts/workplace-violence-report-nation
• 14. International Labour Organization. Statistics of work, employment and labour underutilization: 19th International Conference of Labour Statisticians [Internet]. Geneva; 2013. (ICLS/19/2013/2). Available from: https://www.ilo.org/wcmsp5/groups/public/—dgreports/—stat/documents/publication/wcms_220535.pdf
• 194. World Health Organization. Delivered by women, led by men: a gender and equity analysis of the global health and social workforce [Internet]. Geneva: World Health Organization; 2019 [cited 2023 Mar 29]. 60 p. (Human Resources for Health Observer Series;24). Available from: https://apps.who.int/iris/handle/10665/311322

Information

• Author Services

Initiatives

You are accessing a machine-readable page. In order to be human-readable, please install an RSS reader.

All articles published by MDPI are made immediately available worldwide under an open access license. No special permission is required to reuse all or part of the article published by MDPI, including figures and tables. For articles published under an open access Creative Common CC BY license, any part of the article may be reused without permission provided that the original article is clearly cited. For more information, please refer to https://www.mdpi.com/openaccess .

Feature papers represent the most advanced research with significant potential for high impact in the field. A Feature Paper should be a substantial original Article that involves several techniques or approaches, provides an outlook for future research directions and describes possible research applications.

Feature papers are submitted upon individual invitation or recommendation by the scientific editors and must receive positive feedback from the reviewers.

Editor’s Choice articles are based on recommendations by the scientific editors of MDPI journals from around the world. Editors select a small number of articles recently published in the journal that they believe will be particularly interesting to readers, or important in the respective research area. The aim is to provide a snapshot of some of the most exciting work published in the various research areas of the journal.

Original Submission Date Received: .

• Active Journals
• Find a Journal
• Proceedings Series
• For Authors
• For Reviewers
• For Editors
• For Librarians
• For Publishers
• For Societies
• For Conference Organizers
• Open Access Policy
• Institutional Open Access Program
• Special Issues Guidelines
• Editorial Process
• Research and Publication Ethics
• Article Processing Charges
• Testimonials
• Preprints.org
• SciProfiles
• Encyclopedia

Article Menu

• Subscribe SciFeed
• Recommended Articles
• Google Scholar
• on Google Scholar
• Table of Contents

Find support for a specific problem in the support section of our website.

Please let us know what you think of our products and services.

Visit our dedicated information section to learn more about MDPI.

JSmol Viewer

Literature review on collaborative project delivery for sustainable construction: bibliometric analysis.

1. Introduction

2. literature review, 2.1. collaborative project delivery, 2.2. design build (db), 2.3. construction manager at risk (cmar), 2.4. integrated project delivery method (ipd), 2.5. sustainability, 2.6. sustainable construction, 2.7. benefits of eci comparing case studies, 2.8. collaborative delivery models, 3. methodology, 3.1. research methods, 3.2. database research, 4.1. ipd, design-build, and cmar overview, 4.1.1. yearly publication distribution of db cmar and ipd, 4.1.2. major country analysis, 4.1.3. most relevant and influential journals, 4.1.4. corresponding author countries, 4.2. keyword analysis, 4.2.1. high-frequency keyword analysis, 4.2.2. co-occurrence network analysis, 4.2.3. analysis of keywords’ frequency over time, 5. discussion, 5.1. findings of advantages and disadvantages of ipd, db, and cmar for sustainable construction, 5.1.1. advantages of ipd, 5.1.2. advantages of design-build, 5.1.3. advantages of construction manager at risk, 5.1.4. disadvantages of ipd, 5.1.5. disadvantages of design-build, 5.1.6. disadvantages of construction manager at risk, 5.2. most suitable cpd technique for sustainable construction based on literature review, 5.2.1. limitations, 5.2.2. recommendations for future research, 6. future trend, 6.1. enhancing innovation through collaborative project delivery, 6.2. open communication and block chain technology, 6.3. multi-party agreement, 6.4. utilizing artificial intelligence in decision support systems, 7. conclusions, author contributions, institutional review board statement, informed consent statement, data availability statement, conflicts of interest.

• Giachino, J.; Cecil, M.; Husselbee, B.; Matthews, C. Alternative Project Delivery: Construction Management at Risk, Design-Build and Public-Private Partnerships. In Proceedings of the Utility Management Conference 2016, San Diego, CA, USA, 24–26 February 2016. [ Google Scholar ]
• Shrestha, P.P.; Maharjan, R.; Batista, J.R. Performance of Design-Build and Construction Manager-at-Risk Methods in Water and Wastewater Projects. Pract. Period. Struct. Des. Constr. 2019 , 24 , 04018029. [ Google Scholar ] [ CrossRef ]
• Shrestha, P.P.; Batista, J. Lessons Learned in Design-Build and Construction-Manager-at-Risk Water and Wastewater Project. J. Leg. Aff. Dispute Resolut. Eng. Constr. 2020 , 12 , 04520002. [ Google Scholar ] [ CrossRef ]
• Xia, B.; Chan, A.P.C. Identification of Selection Criteria for Operational Variations of The Design-Build System: A Delphi Study in China. J. Civ. Eng. Manag. 2012 , 18 , 173–183. [ Google Scholar ] [ CrossRef ]
• Shane, J.S.; Bogus, S.M.; Molenaar, K.R. Municipal Water/Wastewater Project Delivery Performance Comparison. J. Manag. Eng. 2013 , 29 , 251–258. [ Google Scholar ] [ CrossRef ]
• Sullivan, J.; El Asmar, M.; Chalhoub, J.; Obeid, H. Two Decades of Performance Comparisons for Design-Build, Construction Manager at Risk, and Design-Bid-Build: Quantitative Analysis of the State of Knowledge on Project Cost, Schedule, and Quality. J. Constr. Eng. Manag. 2017 , 143 , 04017009. [ Google Scholar ] [ CrossRef ]
• Raouf, A.M.; Al-Ghamdi, S. Effectiveness of Project Delivery Systems in Executing Green Buildings. J. Constr. Eng. Manag. 2019 , 145 , 03119005. [ Google Scholar ] [ CrossRef ]
• Francom, T.; El Asmar, M.; Ariaratnam, S.T. Performance Analysis of Construction Manager at Risk on Pipeline Engineering and Construction Projects. J. Manag. Eng. 2016 , 32 , 04016016. [ Google Scholar ] [ CrossRef ]
• Gransberg, D.D.; Shane, J.S.; Transportation Research Board. Construction Manager-at-Risk Project Delivery for Highway Programs ; The National Academies Press: Washington, DC, USA, 2010. [ Google Scholar ]
• Rahman, M.M.; Kumaraswamy, M.M. Potential for Implementing Relational Contracting and Joint Risk Management. J. Manag. Eng. 2004 , 20 , 178–189. [ Google Scholar ] [ CrossRef ]
• Feghaly, J.; El Asmar, M.; Ariaratnam, S.; Bearup, W. Selecting project delivery methods for water treatment plants. Eng. Constr. Archit. Manag. 2019 , 27 , 936–951. [ Google Scholar ] [ CrossRef ]
• Park, H.-S.; Lee, D.; Kim, S.; Kim, J.-L. Comparing Project Performance of Design-Build and Design-Bid-Build Methods for Large-sized Public Apartment Housing Projects in Korea. J. Asian Archit. Build. Eng. 2015 , 14 , 323–330. [ Google Scholar ] [ CrossRef ]
• Shrestha, P.P.; Batista, J.; Maharajan, R. Risks involved in using alternative project delivery (APD) methods in water and wastewater projects. Procedia Eng. 2016 , 145 , 219–223. [ Google Scholar ] [ CrossRef ]
• Hettiaarachchige, N.; Rathnasinghe, A.; Ranadewa, K.; Thurairajah, N. Thurairajah, Lean Integrated Project Delivery for Construction Procurement: The Case of Sri Lanka. Buildings 2022 , 12 , 524. [ Google Scholar ] [ CrossRef ]
• Kent, D.C.; Becerik-Gerber, B. Understanding Construction Industry Experience and Attitudes toward Integrated Project Delivery. J. Constr. Eng. Manag. 2010 , 136 , 815–825. [ Google Scholar ] [ CrossRef ]
• Franz, B.; Leicht, R.; Molenaar, K.; Messner, J. Impact of Team Integration and Group Cohesion on Project Delivery Performance. J. Constr. Eng. Manag. 2017 , 143 , 04016088. [ Google Scholar ] [ CrossRef ]
• Engebø, A.; Klakegg, O.J.; Lohne, J.; Lædre, O. A collaborative project delivery method for design of a high-performance building. Int. J. Manag. Proj. Bus. 2020 , 13 , 1141–1165. [ Google Scholar ] [ CrossRef ]
• Ahmed, S.; El-Sayegh, S. Critical Review of the Evolution of Project Delivery Methods in the Construction Industry. Buildings 2020 , 11 , 11. [ Google Scholar ] [ CrossRef ]
• Bond-Barnard, T.J.; Fletcher, L.; Steyn, H. Linking trust and collaboration in project teams to project management success. Int. J. Manag. Proj. Bus. 2018 , 11 , 432–457. [ Google Scholar ] [ CrossRef ]
• Rodrigues, M.R.; Lindhard, S.M. Lindhard, Benefits and challenges to applying IPD: Experiences from a Norwegian mega-project. Constr. Innov. 2021 , 23 , 287–305. [ Google Scholar ] [ CrossRef ]
• Kaminsky, J. The fourth pillar of infrastructure sustainability: Tailoring civil infrastructure to social context. Constr. Manag. Econ. 2015 , 33 , 299–309. [ Google Scholar ] [ CrossRef ]
• Al Khalil, M.I. Selecting the appropriate project delivery method using AHP. Int. J. Proj. Manag. 2002 , 20 , 469–474. [ Google Scholar ] [ CrossRef ]
• Ibbs, C.W.; Kwak, Y.H.; Ng, T.; Odabasi, A.M. Project Delivery Systems and Project Change: Quantitative Analysis. J. Constr. Eng. Manag. 2003 , 129 , 382–387. [ Google Scholar ] [ CrossRef ]
• Jansen, J.; Beck, A. Overcoming the Challenges of Large Diameter Water Project in North Texas via CMAR Delivery Method. In Proceedings of the Pipelines 2020, San Antonio, TX, USA, 9–12 August 2020; Conference Held Virtually. pp. 264–271. [ Google Scholar ] [ CrossRef ]
• Bingham, E.; Gibson, G.E.; Asmar, M.E. Measuring User Perceptions of Popular Transportation Project Delivery Methods Using Least Significant Difference Intervals and Multiple Range Tests. J. Constr. Eng. Manag. 2018 , 144 , 04018033. [ Google Scholar ] [ CrossRef ]
• Cho, Y.J. A review of construction delivery systems: Focus on the construction management at risk system in the Korean public construction market. KSCE J. Civ. Eng. 2016 , 20 , 530–537. [ Google Scholar ] [ CrossRef ]
• Rosayuru, H.D.R.R.; Waidyasekara, K.G.A.S.; Wijewickrama, M.K.C.S. Sustainable BIM based integrated project delivery system for construction industry in Sri Lanka. Int. J. Constr. Manag. 2022 , 22 , 769–783. [ Google Scholar ] [ CrossRef ]
• Pishdad-Bozorgi, P.; Beliveau, Y.J. Symbiotic Relationships between Integrated Project Delivery (IPD) and Trust. Int. J. Constr. Educ. Res. 2016 , 12 , 179–192. [ Google Scholar ] [ CrossRef ]
• Sherif, M.; Abotaleb, I.; Alqahtani, F.K. Alqahtani, Application of Integrated Project Delivery (IPD) in the Middle East: Implementation and Challenges. Buildings 2022 , 12 , 467. [ Google Scholar ] [ CrossRef ]
• Manata, B.; Garcia, A.J.; Mollaoglu, S.; Miller, V.D. The effect of commitment differentiation on integrated project delivery team dynamics: The critical roles of goal alignment, communication behaviors, and decision quality. Int. J. Proj. Manag. 2021 , 39 , 259–269. [ Google Scholar ] [ CrossRef ]
• Kraatz, J.A.; Sanchez, A.X.; Hampson, K.D. Hampson, Digital Modeling, Integrated Project Delivery and Industry Transformation: An Australian Case Study. Buildings 2014 , 4 , 453–466. [ Google Scholar ] [ CrossRef ]
• Zhang, L.; He, J.; Zhou, S. Sharing Tacit Knowledge for Integrated Project Team Flexibility: Case Study of Integrated Project Delivery. J. Constr. Eng. Manag. 2013 , 139 , 795–804. [ Google Scholar ] [ CrossRef ]
• El Asmar, M.; Hanna, A.S.; Loh, W.-Y. Quantifying Performance for the Integrated Project Delivery System as Compared to Established Delivery Systems. J. Constr. Eng. Manag. 2013 , 139 , 04013012. [ Google Scholar ] [ CrossRef ]
• Ghassemi, R.; Becerik-Gerber, B. Transitioning to integrated project delivery: Potential barriers and lessons learned. Lean Constr. J. 2011 , 32–52. Available online: https://leanconstruction.org/resources/lean-construction-journal/lcj-back-issues/2011-issue/ (accessed on 11 August 2024).
• Mei, T.; Guo, Z.; Li, P.; Fang, K.; Zhong, S. Influence of Integrated Project Delivery Principles on Project Performance in China: An SEM-Based Approach. Sustainability 2022 , 14 , 4381. [ Google Scholar ] [ CrossRef ]
• Ilozor, B.D.; Kelly, D.J. Building information modeling and integrated project delivery in the commercial construction industry: A conceptual study. J. Eng. Proj. Prod. Manag. 2012 , 2 , 23–36. [ Google Scholar ] [ CrossRef ]
• Zabihi, H.; Habib, F.; Mirsaeedie, L. Sustainability in Building and Construction: Revising Definitions and Concepts. Int. J. Emerg. Sci. 2012 , 2 , 570–578. [ Google Scholar ]
• Young, J.W.S. A Framework for the Ultimate Environmental Index—Putting Atmospheric Change Into Context With Sustainability. Environ. Monit. Assess. 1997 , 46 , 135–149. [ Google Scholar ] [ CrossRef ]
• Ding, G.K.C. Sustainable construction—The role of environmental assessment tools. J. Environ. Manag. 2008 , 86 , 451–464. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Conte, E. The Era of Sustainability: Promises, Pitfalls and Prospects for Sustainable Buildings and the Built Environment. Sustainability 2018 , 10 , 2092. [ Google Scholar ] [ CrossRef ]
• Standardized Method of Life Cycle Costing for Construction Procurement. A Supplement to BS ISO 15686-5. Buildings and Constructed Assets. Service Life Planning. Life Cycle Costing ; BSI British Standards: London, UK, 2008. [ CrossRef ]
• Sustainability|Free Full-Text|A Hybrid Multi-Criteria Decision Support System for Selecting the Most Sustainable Structural Material for a Multistory Building Construction. Available online: https://www.mdpi.com/2071-1050/15/4/3128 (accessed on 2 April 2024).
• Korkmaz, S.; Riley, D.; Horman, M. Piloting Evaluation Metrics for Sustainable High-Performance Building Project Delivery. J. Constr. Eng. Manag. 2010 , 136 , 877–885. [ Google Scholar ] [ CrossRef ]
• Ng, M.S.; Graser, K.; Hall, D.M. Digital fabrication, BIM and early contractor involvement in design in construction projects: A comparative case study. Archit. Eng. Des. Manag. 2021 , 19 , 39–55. [ Google Scholar ] [ CrossRef ]
• Moradi, S.; Kähkönen, K.; Sormunen, P. Analytical and Conceptual Perspectives toward Behavioral Elements of Collaborative Delivery Models in Construction Projects. Buildings 2022 , 12 , 316. [ Google Scholar ] [ CrossRef ]
• Zupic, I.; Čater, T. Bibliometric Methods in Management and Organization. 2015. Available online: https://journals.sagepub.com/doi/abs/10.1177/1094428114562629 (accessed on 3 April 2024).
• Rozas, L.W.; Klein, W.C. The Value and Purpose of the Traditional Qualitative Literature Review. J. Evid.-Based Soc. Work 2010 , 7 , 387–399. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Cobo, M.J.; López-Herrera, A.G.; Herrera-Viedma, E.; Herrera, F. Science mapping software tools: Review, analysis, and cooperative study among tools. J. Am. Soc. Inf. Sci. Technol. 2011 , 62 , 1382–1402. [ Google Scholar ] [ CrossRef ]
• Cancino, C.A.; Merigó, J.M.; Coronado, F.C. A bibliometric analysis of leading universities in innovation research. J. Innov. Knowl. 2017 , 2 , 106–124. [ Google Scholar ] [ CrossRef ]
• Pedro, L.F.M.G.; Barbosa, C.M.M.d.O.; Santos, C.M.d.N. A critical review of mobile learning integration in formal educational contexts. Int. J. Educ. Technol. High. Educ. 2018 , 15 , 10. [ Google Scholar ] [ CrossRef ]
• Wen, S.; Tang, H.; Ying, F.; Wu, G. Exploring the Global Research Trends of Supply Chain Management of Construction Projects Based on a Bibliometric Analysis: Current Status and Future Prospects. Buildings 2023 , 13 , 373. [ Google Scholar ] [ CrossRef ]
• Hosseini, M.R.; Martek, I.; Zavadskas, E.K.; Aibinu, A.A.; Arashpour, M.; Chileshe, N. Critical evaluation of off-site construction research: A Scientometric analysis. Autom. Constr. 2018 , 87 , 235–247. [ Google Scholar ] [ CrossRef ]
• Toyin, J.O.; Mewomo, M.C. Mewomo, Overview of BIM contributions in the construction phase: Review and bibliometric analysis. J. Inf. Technol. Constr. 2023 , 28 , 500–514. [ Google Scholar ] [ CrossRef ]
• Kahvandi, Z.; Saghatforoush, E.; Alinezhad, M.; Noghli, F. Integrated Project Delivery (IPD) Research Trends. J. Eng. 2017 , 7 , 99–114. [ Google Scholar ] [ CrossRef ]
• Hale, D.R.; Shrestha, P.P.; Gibson, G.E.; Migliaccio, G.C. Empirical Comparison of Design/Build and Design/Bid/Build Project Delivery Methods. J. Constr. Eng. Manag. 2009 , 135 , 579–587. [ Google Scholar ] [ CrossRef ]
• Mollaoglu-Korkmaz, S.; Swarup, L.; Riley, D. Delivering Sustainable, High-Performance Buildings: Influence of Project Delivery Methods on Integration and Project Outcomes. J. Manag. Eng. 2013 , 29 , 71–78. [ Google Scholar ] [ CrossRef ]
• Ugwu, O.O.; Haupt, T.C. Key performance indicators and assessment methods for infrastructure sustainability—a South African construction industry perspective. Build. Environ. 2007 , 42 , 665–680. [ Google Scholar ] [ CrossRef ]
• Kines, P.; Andersen, L.P.S.; Spangenberg, S.; Mikkelsen, K.L.; Dyreborg, J.; Zohar, D. Improving construction site safety through leader-based verbal safety communication. J. Safety Res. 2010 , 41 , 399–406. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Ballard, G. The Lean Project Delivery System: An Update. 2008. [ Google Scholar ]
• Bynum, P.; Issa, R.R.A.; Olbina, S. Building information modeling in support of sustainable design and construction. J. Constr. Eng. Manag. 2013 , 139 , 24–34. [ Google Scholar ] [ CrossRef ]
• Choudhry, R.M.; Fang, D.; Lingard, H. Measuring Safety Climate of a Construction Company. J. Constr. Eng. Manag. 2009 , 135 , 890–899. [ Google Scholar ] [ CrossRef ]
• Wardani, M.A.E.; Messner, J.I.; Horman, M.J. Comparing procurement methods for Design-Build projects. J. Constr. Eng. Manag. 2006 , 132 , 230–238. [ Google Scholar ] [ CrossRef ]
• Liu, J.; Zhao, X.; Yan, P. Risk Paths in International Construction Projects: Case Study from Chinese Contractors. J. Constr. Eng. Manag. 2016 , 142 . [ Google Scholar ] [ CrossRef ]
• El-Sayegh, S. Evaluating the effectiveness of project delivery methods. J. Constr. Manag. Econ. 2008 , 23 , 457–465. [ Google Scholar ]
• Fang, C.; Marle, F.; Zio, E.; Bocquet, J.-C. Network theory-based analysis of risk interactions in large engineering projects. Reliability Eng. Syst. Safety 2012 , 106 , 1–10. [ Google Scholar ] [ CrossRef ]
• Franz, B.; Leicht, R.M. Initiating IPD Concepts on Campus Facilities with a ‘Collaboration Addendum’. In Proceedings of the Construction Research Congress 2012, West Lafayette, IN, USA, 21–23 May 2012; pp. 61–70. [ Google Scholar ] [ CrossRef ]
• Kim, H.; Kim, K.; Kim, H. Vision-Based Object-Centric Safety Assessment Using Fuzzy Inference: Monitoring Struck-By Accidents with Moving Objects. J. Comput. Civil Eng. 2016 , 30 . [ Google Scholar ] [ CrossRef ]
• Zhou, Y.; Ding, L.Y.; Chen, L.J. Application of 4D visualization technology for safety management in metro construction. Automation Constr. 2013 , 34 , 25–36. [ Google Scholar ] [ CrossRef ]
• Wanberg, J.; Harper, C.; Hallowell, M.R.; Rajendran, S. Relationship between Construction Safety and Quality Performance. J. Constr. Eng. Manag. 2013 , 139 . [ Google Scholar ] [ CrossRef ]
• Shrestha, P.P.; O’Connor, J.T.; Gibson, G.E. Performance comparison of large Design-Build and Design-Bid-Build highway projects. J. Constr. Eng. Manag. 2012 , 138 , 1–13. [ Google Scholar ] [ CrossRef ]
• Torabi, S.A.; Hassini, E. Multi-site production planning integrating procurement and distribution plans in multi-echelon supply chains: An interactive fuzzy goal programming approach. Int. J. Prod. Res. 2009 , 47 , 5475–5499. [ Google Scholar ] [ CrossRef ]
• Baradan, S.; Usmen, M. Comparative Injury and Fatality Risk Analysis of Building Trades. J. Constr. Eng. Manag.-ASCE 2006 , 132 . [ Google Scholar ] [ CrossRef ]
• Levitt, R.E. CEM Research for the Next 50 Years: Maximizing Economic, Environmental, and Societal Value of the Built Environment1. J. Constr. Eng. Manag. 2007 , 133 , 619–628. [ Google Scholar ] [ CrossRef ]
• Araya, F. Modeling the spread of COVID-19 on construction workers: An agent-based approach. Saf. Sci. 2021 , 133 , 105022. [ Google Scholar ] [ CrossRef ]
• Zheng, X.; Le, Y.; Chan, A.P.; Hu, Y.; Li, Y. Review of the application of social network analysis (SNA) in construction project management research. Int. J. Proj. Manag. 2016 , 34 , 1214–1225. [ Google Scholar ] [ CrossRef ]
• Elghaish, F.; Abrishami, S. A centralised cost management system: Exploiting EVM and ABC within IPD. Eng. Constr. Archit. Manag. 2021 , 28 , 549–569. [ Google Scholar ] [ CrossRef ]
• Smith, R.E.; Mossman, A.; Emmitt, S. Lean and integrated project delivery. Lean Constr. J. 2011 , 1–16. [ Google Scholar ]
• Bröchner, J.; Badenfelt, U. Changes and change management in construction and IT projects. Autom. Constr. 2011 , 20 , 767–775. [ Google Scholar ] [ CrossRef ]
• Monteiro, A.; Mêda, P.; Martins, J.P. Framework for the coordinated application of two different integrated project delivery platforms. Autom. Constr. 2014 , 38 , 87–99. [ Google Scholar ] [ CrossRef ]
• Azhar, N.; Kang, Y.; Ahmad, I.U. Factors influencing integrated project delivery in publicly owned construction projects: An information modelling perspective. Procedia Eng. 2014 , 77 , 213–221. [ Google Scholar ] [ CrossRef ]
• Mihic, M.; Sertic, J.; Zavrski, I. Integrated Project Delivery as Integration between Solution Development and Solution Implementation. Procedia Soc. Behav. Sci. 2014 , 119 , 557–565. [ Google Scholar ] [ CrossRef ]
• Nawi, M.N.M.; Haron, A.T.; Hamid, Z.A.; Kamar, K.A.M.; Baharuddin, Y. Improving integrated practice through building information modeling-integrated project delivery (BIM-IPD) for Malaysian industrialised building system (IBS) Construction Projects. Malays. Constr. Res. J. 2014 , 15 , 29–38. Available online: https://dsgate.uum.edu.my/jspui/handle/123456789/1651 (accessed on 24 April 2024).
• Ma, Z.; Zhang, D.; Li, J. A dedicated collaboration platform for Integrated Project Delivery. Autom. Constr. 2018 , 86 , 199–209. [ Google Scholar ] [ CrossRef ]
• Yadav, S.; Kanade, G. Application of Revit as Building Information Modeling (BIM) for Integrated Project Delivery (IPD) to Building Construction Project—A Review. Int. Res. J. Eng. Technol. 2018 , 5 , 11–14. [ Google Scholar ]
• Salim, M.S.; Mahjoob, A.M.R. Integrated project delivery (IPD) method with BIM to improve the project performance: A case study in the Republic of Iraq. Asian J. Civ. Eng. 2020 , 21 , 947–957. [ Google Scholar ] [ CrossRef ]
• Ling, Y.Y.; Lau, B.S.Y. A case study on the management of the development of a large-scale power plant project in East Asia based on design-build arrangement. Int. J. Proj. Manag. 2002 , 20 , 413–423. [ Google Scholar ] [ CrossRef ]
• Dalui, P.; Elghaish, F.; Brooks, T.; McIlwaine, S. Integrated Project Delivery with BIM: A Methodical Approach Within the UK Consulting Sector. J. Inf. Technol. Constr. 2021 , 26 , 922–935. [ Google Scholar ] [ CrossRef ]
• Pishdad-Bozorgi, P. Case Studies on the Role of Integrated Project Delivery (IPD) Approach on the Establishment and Promotion of Trust. Int. J. Constr. Educ. Res. 2017 , 13 , 102–124. [ Google Scholar ] [ CrossRef ]
• Singleton, M.S.; Hamzeh, F.R. Implementing integrated project delivery on department of the navy construction projects: Lean Construction Journal. Lean Constr. J. 2011 , 17–31. [ Google Scholar ]
• Tran, D.Q.; Nguyen, L.D.; Faught, A. Examination of communication processes in design-build project delivery in building construction. Eng. Constr. Archit. Manag. 2017 , 24 , 1319–1336. [ Google Scholar ] [ CrossRef ]
• Park, J.; Kwak, Y.H. Design-Bid-Build (DBB) vs. Design-Build (DB) in the U.S. public transportation projects: The choice and consequences. Int. J. Proj. Manag. 2017 , 35 , 280–295. [ Google Scholar ] [ CrossRef ]
• Wiss, R.A.; Roberts, R.T.; Phraner, S.D. Beyond Design-Build-Operate-Maintain: New Partnership Approach Toward Fixed Guideway Transit Projects. Transp. Res. Rec. J. Transp. Res. Board 2000 , 1704 , 13–18. [ Google Scholar ] [ CrossRef ]
• Xia, B.; Chan, A.P. Key competences of design-build clients in China. J. Facil. Manag. 2010 , 8 , 114–129. [ Google Scholar ] [ CrossRef ]
• DeBernard, D.M. Beyond Collaboration—The Benefits of Integrated Project Delivery ; AIA Soloso Website: Washington, DC, USA, 2008. [ Google Scholar ]
• Chen, Q.; Jin, Z.; Xia, B.; Wu, P.; Skitmore, M. Time and Cost Performance of Design–Build Projects. J. Constr. Eng. Manag. 2016 , 142 , 04015074. [ Google Scholar ] [ CrossRef ]
• Xia, B.; Chan, P. Review of the design-build market in the People’s Republic of China. J. Constr. Procure. 2008 , 14 , 108–117. [ Google Scholar ]
• Mcwhirt, D.; Ahn, J.; Shane, J.S.; Strong, K.C. Military construction projects: Comparison of project delivery methods. J. Facil. Manag. 2011 , 9 , 157–169. [ Google Scholar ] [ CrossRef ]
• Minchin, R.E.; Li, X.; Issa, R.R.; Vargas, G.G. Comparison of Cost and Time Performance of Design-Build and Design-Bid-Build Delivery Systems in Florida. J. Constr. Eng. Manag. 2013 , 139 , 04013007. [ Google Scholar ] [ CrossRef ]
• Adamtey, S.; Onsarigo, L. Effective tools for projects delivered by progressive design-build method. In Proceedings of the CSCE Annual Conference 2019, Laval, QC, Canada, 12–15 June 2019; pp. 1–10. [ Google Scholar ]
• Adamtey, S.A. A Case Study Performance Analysis of Design-Build and Integrated Project Delivery Methods. Int. J. Constr. Educ. Res. 2021 , 17 , 68–84. [ Google Scholar ] [ CrossRef ]
• Gad, G.M.; Adamtey, S.A.; Gransberg, D.D. Gransberg, Trends in Quality Management Approaches to Design–Build Transportation Projects. Transp. Res. Rec. J. Transp. Res. Board. 2015 , 2504 , 87–92. [ Google Scholar ] [ CrossRef ]
• Sari, E.M.; Irawan, A.P.; Wibowo, M.A.; Siregar, J.P.; Praja, A.K.A. Project delivery systems: The partnering concept in integrated and non-integrated construction projects. Sustainability 2022 , 15 , 86. [ Google Scholar ] [ CrossRef ]
• Chakra, H.A.; Ashi, A. Comparative analysis of design/build and design/bid/build project delivery systems in Lebanon. J. Ind. Eng. Int. 2019 , 15 , 147–152. [ Google Scholar ] [ CrossRef ]
• Perkins, R.A. Sources of Changes in Design–Build Contracts for a Governmental Owner. J. Constr. Eng. Manag. 2009 , 135 , 588–593. [ Google Scholar ] [ CrossRef ]
• Palaneeswaran, E.; Kumaraswamy, M.M. Contractor Selection for Design/Build Projects. J. Constr. Eng. Manag. 2000 , 126 , 331–339. [ Google Scholar ] [ CrossRef ]
• Chan, A.P.C. Evaluation of enhanced design and build system a case study of a hospital project. Constr. Manag. Econ. 2000 , 18 , 863–871. [ Google Scholar ] [ CrossRef ]
• Shrestha, P.P.; Davis, B.; Gad, G.M. Investigation of Legal Issues in Construction-Manager-at-Risk Projects: Case Study of Airport Projects. J. Leg. Aff. Dispute Resolut. Eng. Constr. 2020 , 12 , 04520022. [ Google Scholar ] [ CrossRef ]
• Marston, S. CMAR Project Delivery Method Generates Team Orientated Project Management with Win/Win Mentality. In Proceedings of the Pipelines 2020, San Antonio, TX, USA, 9–12 August 2020; pp. 167–170. [ Google Scholar ] [ CrossRef ]
• Francom, T.; El Asmar, M.; Ariaratnam, S.T. Ariaratnam, Longitudinal Study of Construction Manager at Risk for Pipeline Rehabilitation. J. Pipeline Syst. Eng. Pract. 2017 , 8 , 04017001. [ Google Scholar ] [ CrossRef ]
• Peña-Mora, F.; Tamaki, T. Effect of Delivery Systems on Collaborative Negotiations for Large-Scale Infrastructure Projects. J. Manag. Eng. 2001 , 17 , 105–121. [ Google Scholar ] [ CrossRef ]
• Mahdi, I.M.; Alreshaid, K. Decision support system for selecting the proper project delivery method using analytical hierarchy process (AHP). Int. J. Proj. Manag. 2005 , 23 , 564–572. [ Google Scholar ] [ CrossRef ]
• Randall, T.; Pool, S.; Limke, J.; Bradney, A. CMaR Delivery of Critical Water and Wastewater Pipelines. In Proceedings of the Pipelines 2020, San Antonio, TX, USA, 9–12 August 2020; Conference Held Virtually. pp. 280–289. [ Google Scholar ] [ CrossRef ]
• Perrenoud, A.; Reyes, M.; Ghosh, S.; Coetzee, M. Collaborative Risk Management of the Approval Process of Building Envelope Materials. In Proceedings of the AEI 2017, Oklahoma City, OK, USA, 11–13 April 2017; pp. 806–816. [ Google Scholar ] [ CrossRef ]
• Parrott, B.C.; Bomba, M.B. Integrated Project Delivery and Building Information Modeling: A New Breed of Contract. 2010. Available online: https://content.aia.org/sites/default/files/2017-03/Integrated%20project%20delivery%20and%20BIM-%20A%20new%20breed%20of%20contract.pdf (accessed on 18 November 2023).
• Cheng, R. IPD Case Studies. Report. March 2012. Available online: http://conservancy.umn.edu/handle/11299/201408 (accessed on 1 May 2024).
• Lee, H.W.; Anderson, S.M.; Kim, Y.-W.; Ballard, G. Ballard, Advancing Impact of Education, Training, and Professional Experience on Integrated Project Delivery. Pract. Period. Struct. Des. Constr. 2014 , 19 , 8–14. [ Google Scholar ] [ CrossRef ]
• Hoseingholi, M.; Jalal, M.P. Jalal, Identification and Analysis of Owner-Induced Problems in Design–Build Project Lifecycle. J. Leg. Aff. Dispute Resolut. Eng. Constr. 2017 , 9 , 04516013. [ Google Scholar ] [ CrossRef ]
• Öztaş, A.; Ökmen, Ö. Risk analysis in fixed-price design–build construction projects. Build. Environ. 2004 , 39 , 229–237. [ Google Scholar ] [ CrossRef ]
• Lee, D.-E.; Arditi, D. Total Quality Performance of Design/Build Firms Using Quality Function Deployment. J. Constr. Eng. Manag. 2006 , 132 , 49–57. [ Google Scholar ] [ CrossRef ]
• Garner, B.; Richardson, K.; Castro-Lacouture, D. Design-Build Project Delivery in Military Construction: Approach to Best Value Procurement. J. Adv. Perform. Inf. Value 2008 , 1 , 35–50. [ Google Scholar ] [ CrossRef ]
• Graham, P. Evaluation of Design-Build Practice in Colorado Project IR IM(CX) 025-3(113) ; Colorado Department of Transportation: Denver, CO, USA, 2001. [ Google Scholar ]
• Parami Dewi, A.; Too, E.; Trigunarsyah, B. Implementing design build project delivery system in Indonesian road infrastructure projects. In Innovation and Sustainable Construction in Developing Countries (CIB W107 Conference 2011) ; Uwakweh, B.O., Ed.; Construction Publishing House/International Council for Research and Innovation in Building and C: Hanoi, Vietnam, 2011; pp. 108–117. [ Google Scholar ]
• Arditi, D.; Lee, D.-E. Assessing the corporate service quality performance of design-build contractors using quality function deployment. Constr. Manag. Econ. 2003 , 21 , 175–185. [ Google Scholar ] [ CrossRef ]
• Rao, T. . Is Design-Build Right for Your Next WWW Project? presented at the WEFTEC 2009, Water Environment Federation. January 2009, pp. 6444–6458. Available online: https://www.accesswater.org/publications/proceedings/-297075/is-design-build-right-for-your-next-www-project- (accessed on 3 April 2024).
• Touran, A.; Molenaar, K.R.; Gransberg, D.D.; Ghavamifar, K. Decision Support System for Selection of Project Delivery Method in Transit. Transp. Res. Rec. 2009 , 2111 , 148–157. [ Google Scholar ] [ CrossRef ]
• Culp, G. Alternative Project Delivery Methods for Water and Wastewater Projects: Do They Save Time and Money? Leadersh. Manag. Eng. 2011 , 11 , 231–240. [ Google Scholar ] [ CrossRef ]
• Ling, F.Y.Y.; Poh, B.H.M. Problems encountered by owners of design–build projects in Singapore. Int. J. Proj. Manag. 2008 , 26 , 164–173. [ Google Scholar ] [ CrossRef ]
• Pishdad-Bozorgi, P.; de la Garza, J.M. Comparative Analysis of Design-Bid-Build and Design-Build from the Standpoint of Claims. In Proceedings of the Construction Research Congress 2012, West Lafayette, IN, USA, 21–23 May 2012. [ Google Scholar ] [ CrossRef ]
• Walewski, J.; Gibson, G.E., Jr.; Jasper, J. Project Delivery Methods and Contracting Approaches Available for Implementation by the Texas Department of Transportation. University of Texas at Austin. Center for Transportation Research. 2001. Available online: https://rosap.ntl.bts.gov/view/dot/14863 (accessed on 3 April 2024).
• Alleman, D.; Antoine, A.; Gransberg, D.D.; Molenaar, K.R. Comparison of Qualifications-Based Selection and Best-Value Procurement for Construction Manager–General Contractor Highway Construction. 2017. Available online: https://journals.sagepub.com/doi/abs/10.3141/2630-08 (accessed on 2 April 2024).
• Gransberg, N.J.; Gransberg, D.D. Public Project Construction Manager-at-Risk Contracts: Lessons Learned from a Comparison of Commercial and Infrastructure Projects. J. Leg. Aff. Dispute Resolut. Eng. Constr. 2020 , 12 , 04519039. [ Google Scholar ] [ CrossRef ]
• Anderson, S.D.; Damnjanovic, I. Selection and Evaluation of Alternative Contracting Methods to Accelerate Project Completion ; The National Academies Press: Washington, DC, USA, 2008; Available online: http://elibrary.pcu.edu.ph:9000/digi/NA02/2008/23075.pdf (accessed on 26 April 2024).
• Shrestha, P.P.; Batista, J.; Maharjan, R. Impediments in Using Design-Build or Construction Management-at-Risk Delivery Methods for Water and Wastewater Projects. In Proceedings of the Construction Research Congress 2016, San Juan, PR, USA, 31 May–2 June 2016; pp. 380–387. [ Google Scholar ] [ CrossRef ]
• Chateau, L. Environmental acceptability of beneficial use of waste as construction material—State of knowledge, current practices and future developments in Europe and in France. J. Hazard. Mater. 2007 , 139 , 556–562. [ Google Scholar ] [ CrossRef ]
• Lam, T.I.; Chan, H.W.E.; Chau, C.K.; Poon, C.S. An Overview of the Development of Green Specifications in the Construction Industry. In Proceedings of the International Conference on Urban Sustainability [ICONUS], 1 January 2008; pp. 295–301. Available online: https://research.polyu.edu.hk/en/publications/an-overview-of-the-development-of-green-specifications-in-the-con (accessed on 2 May 2024).
• Tabish, S.Z.S.; Jha, K.N. Success Traits for a Construction Project. J. Constr. Eng. Manag. 2012 , 138 , 1131–1138. [ Google Scholar ] [ CrossRef ]
• Niroumand, H.; Zain, M.; Jamil, M. A guideline for assessing of critical parameters on Earth architecture and Earth buildings as a sustainable architecture in various countries. Renew. Sustain. Energy Rev. 2013 , 28 , 130–165. [ Google Scholar ] [ CrossRef ]
• Rogulj, K.; Jajac, N. Achieving a Construction Barrier–Free Environment: Decision Support to Policy Selection. J. Manag. Eng. 2018 , 34 , 04018020. [ Google Scholar ] [ CrossRef ]
• Sackey, S.; Kim, B.-S. Environmental and Economic Performance of Asphalt Shingle and Clay Tile Roofing Sheets Using Life Cycle Assessment Approach and TOPSIS. J. Constr. Eng. Manag. 2018 , 144 , 04018104. [ Google Scholar ] [ CrossRef ]
• Carretero-Ayuso, M.J.; García-Sanz-Calcedo, J.; Rodríguez-Jiménez, C.E. Rodríguez-Jiménez, Characterization and Appraisal of Technical Specifications in Brick Façade Projects in Spain. J. Perform. Constr. Facil. 2018 , 32 , 04018012. [ Google Scholar ] [ CrossRef ]
• Golabchi, A.; Guo, X.; Liu, M.; Han, S.; Lee, S.; AbouRizk, S. An integrated ergonomics framework for evaluation and design of construction operations. Autom. Constr. 2018 , 95 , 72–85. [ Google Scholar ] [ CrossRef ]
• Jha, K.; Iyer, K. Commitment, coordination, competence and the iron triangle. Int. J. Proj. Manag. 2007 , 25 , 527–540. [ Google Scholar ] [ CrossRef ]
• Tabassi, A.A.; Ramli, M.; Roufechaei, K.M.; Tabasi, A.A. Team development and performance in construction design teams: An assessment of a hierarchical model with mediating effect of compensation. Constr. Manag. Econ. 2014 , 32 , 932–949. [ Google Scholar ] [ CrossRef ]
• Chen, Y.; Okudan, G.E.; Riley, D.R. Sustainable performance criteria for construction method selection in concrete buildings. Autom. Constr. 2010 , 19 , 235–244. [ Google Scholar ] [ CrossRef ]
• Doloi, H.; Sawhney, A.; Iyer, K.; Rentala, S. Analysing factors affecting delays in Indian construction projects. Int. J. Proj. Manag. 2012 , 30 , 479–489. [ Google Scholar ] [ CrossRef ]
• Kog, Y.C.; Loh, P.K. Critical Success Factors for Different Components of Construction Projects. J. Constr. Eng. Manag. 2012 , 138 , 520–528. [ Google Scholar ] [ CrossRef ]
• Gunduz, M.; Almuajebh, M. Critical success factors for sustainable construction project management. Sustainability 2020 , 12 , 1990. [ Google Scholar ] [ CrossRef ]
• Cao, D.; Li, H.; Wang, G.; Luo, X.; Tan, D. Relationship Network Structure and Organizational Competitiveness: Evidence from BIM Implementation Practices in the Construction Industry. J. Manag. Eng. 2018 , 34 , 04018005. [ Google Scholar ] [ CrossRef ]
• Clevenger, C.M. Development of a Project Management Certification Plan for a DOT. J. Manag. Eng. 2018 , 34 , 06018002. [ Google Scholar ] [ CrossRef ]
• Bygballe, L.E.; Swärd, A. Collaborative Project Delivery Models and the Role of Routines in Institutionalizing Partnering. Proj. Manag. J. 2019 , 50 , 161–176. [ Google Scholar ] [ CrossRef ]
• Collins, W.; Parrish, K. The Need for Integrated Project Delivery in the Public Sector. In Proceedings of the Construction Research Congress 2014, Atlanta, GA, USA, 19–21 May 2014; pp. 719–728. [ Google Scholar ] [ CrossRef ]
• Turk, Ž.; Klinc, R. Potentials of Blockchain Technology for Construction Management. Procedia Eng. 2017 , 196 , 638–645. [ Google Scholar ] [ CrossRef ]
• Elghaish, F.; Abrishami, S.; Hosseini, M.R. Integrated project delivery with blockchain: An automated financial system. Autom. Constr. 2020 , 114 , 103182. [ Google Scholar ] [ CrossRef ]
• Fish, A. Integrated Project Delivery: The Obstacles of Implementation. May 2011. Available online: http://hdl.handle.net/2097/8554 (accessed on 3 April 2024).
• Pan, Y.; Zhang, L. Roles of artificial intelligence in construction engineering and management: A critical review and future trends. Autom. Constr. 2020 , 122 , 103517. [ Google Scholar ] [ CrossRef ]
• Mellit, A.; Kalogirou, S.A. Artificial intelligence techniques for photovoltaic applications: A review. Prog. Energy Combust. Sci. 2008 , 34 , 574–632. [ Google Scholar ] [ CrossRef ]
• Smith, C.J.; Wong, A.T.C. Advancements in Artificial Intelligence-Based Decision Support Systems for Improving Construction Project Sustainability: A Systematic Literature Review. Informatics 2022 , 9 , 43. [ Google Scholar ] [ CrossRef ]
• Villa, F. Semantically driven meta-modelling: Automating model construction in an environmental decision support system for the assessment of ecosystem services flows. In Information Technologies in Environmental Engineering ; Athanasiadis, I.N., Rizzoli, A.E., Mitkas, P.A., Gómez, J.M., Eds.; Springer: Berlin, Heidelberg, 2009; pp. 23–36. [ Google Scholar ]
• Minhas, M.R.; Potdar, V. Decision Support Systems in Construction: A Bibliometric Analysis. Buildings 2020 , 10 , 108. [ Google Scholar ] [ CrossRef ]

Click here to enlarge figure

Share and Cite

Babalola, O.G.; Alam Bhuiyan, M.M.; Hammad, A. Literature Review on Collaborative Project Delivery for Sustainable Construction: Bibliometric Analysis. Sustainability 2024 , 16 , 7707. https://doi.org/10.3390/su16177707

Babalola OG, Alam Bhuiyan MM, Hammad A. Literature Review on Collaborative Project Delivery for Sustainable Construction: Bibliometric Analysis. Sustainability . 2024; 16(17):7707. https://doi.org/10.3390/su16177707

Babalola, Olabode Gafar, Mohammad Masfiqul Alam Bhuiyan, and Ahmed Hammad. 2024. "Literature Review on Collaborative Project Delivery for Sustainable Construction: Bibliometric Analysis" Sustainability 16, no. 17: 7707. https://doi.org/10.3390/su16177707

Article Metrics

Article access statistics, further information, mdpi initiatives, follow mdpi.

Subscribe to receive issue release notifications and newsletters from MDPI journals

Identifying and selecting the next generation of nursing leaders through effective succession planning: a policy analysis

• Perspective
• Open access
• Published: 04 September 2024
• Volume 3 , article number  73 , ( 2024 )

Cite this article

You have full access to this open access article

• Mutaz I. Othman 1 ,
• Islam Oweidat 2 ,
• Abdulqadir J. Nashwan   ORCID: orcid.org/0000-0003-4845-4119 1 ,
• Hothaifah Hijazi 1 &
• Ahmed Abu jaber 1  

1 Altmetric

Succession planning is a vital organizational process that facilitates the smooth transition of leadership and safeguarding organizational knowledge. This research paper seeks to determine the most efficient and effective method for identifying and selecting potential leaders within an organization by examining succession planning policies and their effectiveness in identifying and selecting potential leaders in diverse organizational contexts. This paper utilizes policy analysis methodology to analyze succession planning policies' components, objectives, and outcomes across various industries and sectors. The analysis is based on an extensive review of literature, policy documents, and case studies. This study addresses the methods and tools used to identify and select potential leaders using a scorecard structure. In conclusion, the nine-box grid is a valuable tool that assesses employees' performance and potential for leadership positions.

Similar content being viewed by others

Strategies for Healthcare Organizations in Succession Planning

Succession planning for large and small organizations: a practical review of professional business corporations, succession planning models, conceptual maps: ethical considerations and best practices, explore related subjects.

• Artificial Intelligence

Avoid common mistakes on your manuscript.

1 Introduction

The healthcare environment is known as dynamic and unstable, and healthcare is intensely affected by the change. So, healthcare leadership is becoming more complex. The shortage of competent workers affects service recipients and long-term growth [ 1 , 2 ]. Several countries lack trained healthcare professionals, particularly nurses. The World Health Organization (2016) reports a global shortage of 14.5 million doctors, nurses, and midwives. This shortage could harm vital programs in the health sector by 2030.

A needs-based study by the World Health Organization (WHO) (2016) anticipates a 7.6 million nurse shortage by 2030 [ 1 ]. That requires healthcare leadership to understand and build strategies to improve staff retention and ensure a smooth transition when staff retires [ 3 ]. Establish workforce planning based on current and emerging health service and community needs. Those needs require a crucial and effective strategy or policy to maintain the best organizational performance, as recommendation seven calls for nurses to prepare and enable nurses to lead change to advance health [ 4 ].

Nurse leaders prepare and enable nurses to lead change to advance health. This recommendation insists that "nurses, nursing education programs, and nursing associations should prepare the nursing workforce to assume leadership positions across all levels, while public, private, and governmental health care decision-makers should ensure that leadership positions are available to and filled by nurses" [ 4 ]. So today, nursing leadership should adopt new strategies to improve the retention of the nursing workforce and prepare and enable nurses to lead change to advance health [ 5 ]. Succession planning can be utilized in every area of healthcare succession planning [ 6 ].

Developing a succession plan policy usually starts with developing a strategic plan, identifying and assessing the key positions, and then identifying the potential leaders [ 7 ]. These employees are assessed as having leadership ability, organizational commitment, and motivation to grow and succeed in more senior positions. A systematic method for candidate selection is one of the most crucial components in developing a solid succession plan [ 8 ]. Even if an employee is working well in their current position, that does not guarantee that they will be successful in a more senior role [ 9 ].

There are distinct between leadership potential and performance; the assessment of leadership potential includes an assessment of an individual's potential for assuming future leadership positions, typically focusing on characteristics such as adaptability and strategic thinking. In contrast, a performance assessment is a process that assesses an individual's present job-related accomplishments, with a particular focus on past and current achievements. Both variables are crucial when assessing an individual's preparedness for leadership roles, but with different objectives [ 10 ]. It's critical to have a strategy in place for accurately identifying high-performing employees. Therefore, we must consider this when selecting potential succession candidates. So, determine measurable criteria to evaluate each candidate's potential [ 11 ].

The identifying and selecting criteria on which a potential leader will be nominated were not addressed in the policy for succession planning. Identifying a high-potential leader is an ideal strategic objective for succession planning [ 12 ]. Additionally, it is an essential step in succession planning [ 7 , 12 ]. So, in our policy analysis Here, the question is, "What is the most effective and efficient method of identifying and selecting those potential leaders in the organization?".

2 Background

Organizations usually struggle to identify potential leaders for further development. As healthcare is a dynamic work environment, it is crucial to provide a solid leadership pool for the future [ 13 ]. Identifying and selecting the most successful potential leaders are crucial strategic objectives for maintaining an organization's viability and competitiveness [ 14 ].

The breakdown of leadership is crucial for any organization's survival and long-term sustainability [ 15 ]. The breakdown of leaders and managers can be a terrible occurrence with far-reaching effects within and beyond the organization. In other words, succession planning, strategy, leadership, and culture are interconnected and comprise many aspects of an organization's vision [ 7 ]. A significant aspect of an organization's long-term viability is how succession planning is integrated with organizational strategy and culture, future leader training and development, and change management to ensure business continuity [ 16 , 17 ]. Succession planning is used to identify, select, and develop future leaders; it must be planned, implemented, and measured to ensure a positive conclusion [ 18 ].

The literature shows that most organizations do not have formal or written succession plans [ 19 ]. According to the literature, succession planning, and an organization's future leader's selection, development, and growth should be a focused, clearly articulated approach to benefit the firm, including all stakeholders [ 20 ]. This kind of planning might be important for developing leaders in different stakeholders.

Creating an effective succession plan for healthcare leadership allows for the development and support of future leaders through training programs. On the other hand, the employee must demonstrate leadership interest and potential [ 16 ]. Incorporating methods established in other healthcare departments, the nursing industry may be able to serve as a model for other healthcare leaders' succession planning strategies [ 21 ]. Suppose organizations do not effectively select, develop, and retain future leaders. In that case, they will encounter an unpredictable future characterized by the breakdown of leadership, which is crucial for any organization's survival and long-term sustainability. By leadership vacancies, the loss of key individuals, and possibly unstable operations [ 22 ]. Even though there is a huge and varied body of knowledge on succession planning, there is still much to learn. The literature also reveals that nursing has a well-established succession process and that healthcare leaders, in general, can benefit from modeling how nursing prepares for leadership transitions. Nursing leaders are crucial in healthcare organizations as they link leadership and nurses [ 23 ]. A lack of succession planning can be expensive and raise hospital expenses when healthcare institutions seek senior positions [ 24 ].

The research literature argues that the process of identifying future leaders includes the utilization of a variety of criteria and methodologies. In accordance with the Knoll study from 2021, it is possible to assess global leadership potential (GLP) by considering a variety of traits, attitudes, and competencies. Identifying GLP entails three stages: nomination, assessment, and confirmation [ 25 ]. According to Norman (2020), the assessment of leadership potential can be achieved by utilizing a comprehensive approach that incorporates behavioral simulations, psychometrics, and processing tests [ 26 ].

In a study conducted by Panait in 2017, a self-assessment questionnaire was Created to identify and evaluate leadership abilities and characteristics [ 27 ]. According to Knaub (2018), the exclusive reliance on social network analysis may not be adequate for identifying leaders[ 28 ]. Therefore, overall, the reviews indicate that identifying and selecting future leaders requires a comprehensive strategy considering different criteria and methods. Still, the articles didn't identify which methods are recommended. The author shows that using different methods can result in inconsistencies and unsuccessful utilization. Therefore, it is important to identify and select the best approach to ensure its effectiveness in implementation.

This paper aims to identify the most effective method for identifying and selecting potential organizational leaders. The main objective is to assess different methods and measure their effectiveness and efficiency in this context. We aim to offer practical recommendations to improve the organization's leadership development and succession planning.

The creativity and motivation of a future leader will contribute to the success of the organization; recognizing these employees' abilities enables department heads to focus on strategic development and evaluation, which could happen based on pre-defined qualifying criteria and appraise the right candidate to determine their strengths and development requirements.

Identifying and selecting willing employees based on measurable abilities using a procedure that ensures every person with leadership potential is examined fairly and fully for potential leadership roles.

In our policy analysis plan, we selected the rational model approach due to its well-organized framework, supported by research, emphasizing evidence-based analysis and transparent decision-making [ 29 ]. We assumed that the various alternatives would be of interest to achieving the objectives and responding to our policy analysis question. This analysis revised the most encountered alternatives for identifying and selecting potential organizational leaders.

The evaluation criteria for each alternative must account for its capacity to achieve policy objectives and outcomes, reduce costs, be politically and administratively feasible, and be suitable for long-term goals. The criteria will determine how effective and fair the different options are to determine how well the goals and objectives are met.

3.1 Alternative 1: Nomination by supervisor

Some organizations approach internal recruitment more closely, asking managers to nominate high-performing employees for internal positions. When individuals are familiar with employees' work in different departments in smaller organizations, this informal approach can be quite effective. However, this technique may appear to reflect discriminatory practices [ 30 ]. Organizations often have a well-defined process for performance evaluation; however, many managers are confused about evaluating potential, resulting in incorrect potential assessments [ 10 ]. Unconscious biases could also affect how employees are judged if there aren't clear ways to pick team members with high potential.

3.2 Alternative 2: 360 Assessment Tool

A 360-degree assessment tool allows you to get input from others on an applicant's potential leadership skills. When used correctly, it will most likely assist you in identifying and selecting your potential leader. A 360-degree assessment tool is a game-changing approach to providing employees with consistent feedback, support, and possibilities for advancement. It is a significant improvement over traditional approaches to learning and development, as well as annual performance reviews [ 31 ].

3.3 Alternative 3: Interview

An interview is a sort of interaction sometimes used to identify the best applicant. This is essentially a conversation between two or more people seeking to know and understand more about one another. The interviewee will usually talk about their opinions, views, and background, while the interviewer will ask about their knowledge, skills, and abilities [ 32 ]. One of the most significant benefits of conducting interviews is gaining knowledge about your organization's potential leaders' skills and abilities. However, there are some disadvantages to this method [ 33 ]. The method also takes a very long time and requires a significant amount of stress response. It's important to remember that interviewing is a two-way street. Suppose you go out of your way to ensure the interviewee has a positive experience that benefits them. This might also assist you in selecting the best people for the position [ 34 ].

3.4 Alternative 4: 9-Box Grid

The 9-box grid is a typical method of identifying and classifying potential intended to assist organizations in understanding the kinds of potential they have and determining where to focus their future development and budget [ 35 ]. Anyone can utilize the 9-box Grid, but it is mostly used by HR professionals, managers, and other development professionals. The 9-box grid is frequently used in succession planning [ 36 ]. A complete succession plan, on the other hand, extends beyond identifying and selecting talent in an organization.

The results present a comparison between different alternatives. Six criteria make up the evaluation tool. Here, we used a Likert scale of 1 to 5 to rate each criterion. (A score of 1 indicates a very low likelihood; a score of 2 indicates a low likelihood; a score of 3 indicates a moderate likelihood; a score of 4 indicates a high likelihood, and a score of 5 indicates a very high likelihood.) The evaluation carried out by the group in this analysis paper uses a few different ways to determine the best alternative in selecting potential managers, including supervisor nomination, a 360 assessment tool, an interview, and a 9-Box Grid. Each group member was required to rate every alternative separately and submit their results to their team leader. The results show that the 9-box-grid tool receives the highest score when using the scorecard method, as illustrated in Table  1 , which happened after the group's team leader calculated the submitted comparisons by utilizing evaluation criteria shown in Table  2 and came up with the mean scores.

The 9-box grid is a useful performance management tool that managers use for several reasons. It is simple [ 37 ], and the 9-box grid has a straightforward structure, as shown in Table  3 . Also, it can be used in various organizations because it needs little background research or data collection, and it is possible to perform this task based on first-hand observation [ 38 ]. The 9-box grid produces a visual representation of a firm's talent pool. It's a tool that can be used to compare potential leaders and facilitate debate and decision-making [ 39 ]. The benefit of implementing a 9-box grid is that it saves both time and money [ 39 ].

Most other alternatives may initially seem attractive, but they have a high long-term cost [ 40 ]. Here are a few limitations when implementing the 9-box Grid. The distinction between performance and potential can be challenging, especially if neither idea has a clear definition or comprehension [ 41 ]. If you choose to be honest and transparent with your staff and reveal performance measurements, you risk negatively affecting them and decreasing their satisfaction. Having a "poor performance" or "low potential" can have a detrimental impact on lower-level employees, and with good reason [ 42 ].

There are three phases to constructing a 9-box grid for your organization's succession planning [ 37 ]:

4.1 Assess your employees' performance

The first phase is evaluating each employee's level of performance. The actual criteria for evaluating performance vary depending on an organization's needs. However, each employee must be assigned to one of three groups: Low performance: The individual does not meet the work criteria and does not meet the company's targets and objectives. They lack motivation and alignment with the company's vision [ 37 ]. Moderate performance: the employee's job requirements, personal targets, and ambitions are only partially met [ 37 ]. High performance: The employee meets all their job description and personal goals and performs consistently throughout all tasks [ 37 ].

4.2 Evaluate an employee's potential

The next step is to evaluate each employee's potential. Employees' potential is based on how much they are expected to grow in the future, how willing they are to learn new skills, and how well they can use what they know in everyday situations. One of the problems is that organizations aren't as good at judging employees' potential as they are at judging their performance. Performance refers to previous conduct, while potential refers to expected future behavior to simplify the distinction between the two concepts.

4.3 Create a 3 × 3 grid by integrating performance and potential.

The next stage is plotting all employees on a 3 × 3 grid, resulting in your 9-box grid, after assessing them as low, moderate, or high on both performance and potential. Managers can also see where each employee ranks according to the matrix.

5 Discussion

Those in leadership positions are responsible for challenging the status quo, innovating, and rallying their people to propel enterprises to new heights of success [ 43 ]; in contrast, in leadership positions, selection failures can cost an organization a large amount of money and disrupt its potential growth for several years. Therefore, organizations must carefully consider their leadership choices [ 44 ]. It is necessary to have a leadership selection procedure that identifies the abilities, qualities, and competencies of potential candidates and decides whether they are appropriate for these positions [ 45 ].

One of the greatest practices for filling leadership positions is to create an internal pool of potential leaders who can be accessed at any time. Our goals and objectives could include analyzing productivity, turnover, days to fill, cost per hire, and quality of hire, as well as measuring productivity, turnover, days to fill, recruitment costs, and quality of talent management. Various indicators in which management is polled regarding the recruitment and selection process, their staff, or patient experience. Collaboration and organizational success within their departments and throughout the organization.

The 9 Box Grid is a tool that provides a comprehensive view of an organization's intellectual equity. The research reveals that utilizing the 9-box grid is a practicable tool for succession planning and talent management. This tool helps organizations assess and categorize their workforce according to their performance and potential, enabling the identification of potential leaders and the development of strategies for developing an effective leadership succession plan [ 37 , 42 , 46 , 47 , 48 ]. However, despite having higher job performance assessments than males, one study indicated that women receive significantly lower "potential" ratings after reviewing 9-box grid data for close to 30,000 employees. The implementation of the 9-box tool has the potential to jeopardize an organization's efforts toward fostering diversity, equity, and inclusion by unintentionally impeding the progression of underrepresented individuals into leadership positions. As a result of these individuals' potential demotivation and departure from the organization in search of better possibilities, diversity levels may be reduced overall. Therefore, the subjectivity of the 9-box Grid can affect individuals from historically marginalized backgrounds [ 48 ].

5.1 Implications

This article can inspire further studies in succession planning since it relates to healthcare and nursing contexts. Nurses and healthcare organizations can utilize the findings of this study to optimize leadership development programs, therefore adapting strategies to address the needs of the healthcare professions. Nurses have the potential to enhance their comprehension of the avenues leading to leadership positions within healthcare organizations, influencing their professional paths and goals. Implementing a well-designed leadership succession plan can significantly impact the entire quality of patient care, offering advantages for both nurses and their patients. Leadership may acknowledge the significance of fostering a talent pipeline inside their organizations, ensuring smooth succession when leadership roles become vacant.

Healthcare education programs have the potential to incorporate succession planning aspects into their curriculum to adequately equip aspiring healthcare employees for positions of leadership in the future. Policymakers may consider the implications of the findings on healthcare policy, which could potentially impact policies related to succession planning within healthcare organizations. Also, policymakers may see a potential connection between implementing effective leadership succession planning and enhancing healthcare quality, which encourages them to allocate resources towards efforts to develop competence in leadership. A research study on succession planning in healthcare organizations can have significant consequences in various areas, including research, nursing practice, leadership development, education, and policymaking.

Incorporating that knowledge can effectively inform appropriate approaches, facilitate improvements in the development of leadership qualities, and ultimately result in enhanced patient care and improved healthcare outcomes.

5.2 Strength and limitations

The present policy analysis study has the potential to provide policymakers and stakeholders with carefully analyzed information and evidence, enabling them to make sensible and informed decisions. In addition to providing realistic policy recommendations or alternatives based on research findings, these conclusions can serve as valuable guidance for policymakers in formulating effective policies. As the availability of precise and Comprehensive data can be a limitation, especially when studying the implications of policy, there are many limitations in this policy analysis paper. Another limitation is our comprehensive evaluation to identify and select potential organizational leaders. We made an effort to consider a variety of alternatives, but we may have missed certain alternatives. Furthermore, it is worth mentioning that our study did not proceed to the implementation phase. Therefore, the practical implications and effectiveness of the selected approach have yet to be tested.

6 Conclusion

Identifying potential by comparing performance to potential. The outcomes of this activity can successfully reveal the most appropriate succession, thereby identifying employees with high potential and performance. This term is used to describe employees who are most prepared to lead.

During the talent identification and selection phase, the 9-Box Grid is the primary instrument for assessing the potential and establishing organizational workforce strategic plans. Each department employee is carefully examined and assigned to the appropriate quadrant (box) in this task. By completing the 9 Box Grid tasks, accurate plans can be made for the development and retention of each staff member. To evaluate the successful operation for selecting potential leaders to achieve the aims of implementing it, we must determine which indications best represent the method defined and established within the organization.

Data availability

Not applicable.

World Health Organization. Global strategy on human resources for health: workforce 2030.

Lockhart L. Strategies to reduce nursing turnover. Nurs Made Incred Easy. 2020;18(2):56.

Article   Google Scholar  

DeCenzo DA, Robbins SP, Verhulst SL. Fundamentals of human resource management. John Wiley & Sons; 2016.

National Academies of Sciences, Engineering, and Medicine. Assessing progress on the Institute of Medicine report. The Future of Nursing; 2016.

Wei H, et al. Nurse leaders’ strategies to foster nurse resilience. J Nurs Manag. 2019;27(4):681–7.

Article   PubMed   Google Scholar  

Smith A. Eur J Public Health. 2019; 29 (Supplement_4):p. ckz185.

Al Suwaidi M, et al. Determinants linked to executive succession planning in public sector organizations. Vision. 2020;24(3):284–99.

Griffin RW, Phillips JM, Gully SM. Organizational behavior: Managing people and organizations. CENGAGE learning; 2016.

Stone RJ, Cox A, Gavin M, Carpini J. Human resource management. John Wiley & Sons; 2024.

Church AH, et al. Is there potential in assessing for high-potential? Evaluating the relationships between performance ratings, leadership assessment data, designated high-potential status and promotion outcomes in a global organization. Leadersh Q. 2021;32(5): 101516.

Hile K, Agnor M, Roesgen L, Lamb C.  A Comprehensive Tool for Principal Succession Planning  (Doctoral dissertation, Saint Louis University).

Smith AD. Corporate succession planning: accident or part of the strategic benchmarking process? Int J Process Manag Benchmark. 2022;12(6):695–729.

Behie SW, Pasman HJ, Khan FI, Shell K, Alarfaj A, El-Kady AH, Hernandez M. Leadership 4.0: The changing landscape of industry management in the smart digital era. Process Saf Environ Protect. 2023;172:317-28.

Obianuju AA, Ibrahim UA, Zubairu UM. Succession planning as a critical management imperative: A systematic review. Modern Manag Rev. 2021;26(4):69–92.

Pavlov OV, Katsamakas E. Will colleges survive the storm of declining enrollments? A computational model. PLoS ONE. 2020;15(8): e0236872.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Masenya TM. Integrating talent and knowledge management practices in the new normal business environment: developing future leaders in public sector organizations. In Navigating the new normal of business with enhanced human resource management strategies 2022 (p. 113–144). IGI global.

Acheampong J, Baidoo P, Omari C. Archives of the social sciences: a journal of collaborative memory. Arch Soc Sci A J Collab Mem. 2023;1(1):67–77.

Google Scholar  

Okwakpam JA. Effective succession planning: A roadmap to employee retention. Kuwait Chapter Arab J Bus Manag Rev. 2019;8(2):1–10.

Yawson RM. Human resource development and executive leadership succession planning in nonprofits. In Proceedings of the 56th Annual Eastern Academy of Management Conference. 2019 (p. 1–16).

Chang CY, Besel K. Cultivating next generation of healthcare leaders in Havana: Barriers and recommendation for succession planning. Int J Healthc Manag. 2021;14(4):1062–70.

Rimita KN.  Leader readiness in a volatile, uncertain, complex, and ambiguous (VUCA) business environment  (Doctoral dissertation, Walden University).

Tucker CA. Succession planning for academic nursing. J Prof Nurs. 2020;36(5):334–42.

Beitz JM. The perioperative succession crisis: A cross-sectional study of clinical realities and strategies for academic nursing. Nurs Econ. 2019;37(4):179–97.

Knoll C, Sternad D. Identifying global leadership potential. J Manag Dev. 2021;40(4):253–72.

Norman J. Leadership potential: Measurement beyond psychometrics . The British Psychological Society. 2020;12(2).

Panait C. Developing leadership skills. Identifying leadership qualities and attributes. Rev Air Force Acad. 2017(1):167.

Knaub AV, Henderson C, Fisher KQ. Finding the leaders: an examination of social network analysis and leadership identification in STEM education change. Int J STEM Educ. 2018;5(1):26.

Article   PubMed   PubMed Central   Google Scholar  

Gray D, De Haan E, Bonneywell S. Coaching the ‘ideal worker’: female leaders and the gendered self in a global corporation. Eur J Train Dev. 2019;43(7/8):661–81.

Jann W, Wegrich K. Theories of the policy cycle. In: Handbook of public policy analysis. Routledge; 2017. p. 69–88.

González-Gil M, et al. 360-Degree evaluation: Towards a comprehensive, integrated assessment of performance on clinical placement in nursing degrees: A descriptive observational study. Nurse Educ Today. 2020;95: 104594.

Solarino AM, Aguinis H. Challenges and best-practice recommendations for designing and conducting interviews with elite informants. J Manag Stud. 2021;58(3):649–72.

Team IE. Advantages and disadvantages of interviews (and useful tips). 2023; Available from: https://in.indeed.com/career-advice/interviewing/advantages-and-disadvantages-of-interviews .

Maurer R. The pros and cons of virtual and in-person interviews. 2016; Available from: https://www.shrm.org/resourcesandtools/hr-topics/talent-acquisition/pages/pros-and-cons-virtual-in-person-interviews.aspx .

Park HM. The context matters to the exclusive talent management: how to measure and pay in South Korea. Glob Bus Rev. 2023; 09721509231157014.

Boštjančič E, Slana Z. The role of talent management comparing medium-sized and large companies–major challenges in attracting and retaining talented employees. Front Psychol. 2018;9:1750.

Traynor S, Wellens MA, Krishnamoorthy V. Succession management intro, 9-box, and talent review forms. In: SAP success factors talent: volume 2: a complete guide to configuration, administration, and best practices: succession and development. Springer; 2021. p. 289–328.

Chapter   Google Scholar  

Garbellano S, Ughetto E, Esfahani HD. Talent acquisition and development in global companies (based on a study on nexans Group). 2020.

Gaonkar S, Khan D, Manisha AS. Impact of gamification on learning and development. J Adv Educ Philos. 2022;6:63–70.

Sparrow P, Hird M, Cooper CL, Sparrow P, Hird M, Cooper CL. Strategic talent management. Palgrave Macmillan UK; 2015.

Mistry R. Dilemmas in the realm of leadership talent development. NHRD Netw J. 2017;10(2):45–52.

Munira S. Importance of skills inventory and training in succession planning-how does it help in employee performance: The case of Heidelberg Cement Bangladesh Limited (HCBL), Chittagong Plant. 2022.

Rivera MJ. Leveraging innovation and intrapreneurship as a source for organizational growth. Int J Innov Sci. 2017;9(2):137–52.

O’Reilly III CA, Tushman ML. Lead and disrupt: How to solve the innovator's dilemma. Stanford University Press. 2021.

Lynn DB. Succession management strategies in public sector organizations. In: Administrative Leadership in the Public Sector. Routledge; 2016. p. 375–89.

Newhall S. Aligning the talent development and succession planning processes: Don’t allow critical leadership talent to fall by the wayside. Dev Learn Organ Int J. 2015;29(5):3–6.

Jones P, Rivers R. Development of a staff nurse to chief nursing officer succession planning program. Nurse Lead. 2021;19(6):646–54.

Martin A. Talent management: Preparing a “Ready” agile workforce. Int J Pediatr Adolesc Med. 2015;2(3–4):112–6.

Benson A, Li D, Shue K. Potential and the gender promotion gap. Unpublished Working Paper. 2021.

Download references

Acknowledgements

The Qatar National Library funded the publication of this article.

Author information

Authors and affiliations.

Nursing Department, Hamad Medical Corporation, Doha, Qatar

Mutaz I. Othman, Abdulqadir J. Nashwan, Hothaifah Hijazi & Ahmed Abu jaber

Faculty of Nursing, Zarqa University, P.O. Box 3050, Doha, Qatar

Islam Oweidat

You can also search for this author in PubMed   Google Scholar

Contributions

MIO: Conceptualization. MIO, IO, AJN, HH, AA: Methods, Literature search, Manuscript preparation. All authors have accepted responsibility for the entire content of this manuscript and approved its submission.

Corresponding author

Correspondence to Abdulqadir J. Nashwan .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interest.

The authors declare no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Othman, M.I., Oweidat, I., Nashwan, A.J. et al. Identifying and selecting the next generation of nursing leaders through effective succession planning: a policy analysis. Discov Health Systems 3 , 73 (2024). https://doi.org/10.1007/s44250-024-00130-5

Download citation

Received : 25 September 2023

Accepted : 05 August 2024

Published : 04 September 2024

DOI : https://doi.org/10.1007/s44250-024-00130-5

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Succession planning
• Potential leaders
• Policy analysis
• The 9-box grid
• Find a journal
• Publish with us
• Track your research