the features of a research article

• Search Menu
• Sign in through your institution
• Advance articles
• Editor's Choice
• Supplements
• French Abstracts
• Portuguese Abstracts
• Spanish Abstracts
• Author Guidelines
• Submission Site
• Open Access
• About International Journal for Quality in Health Care
• About the International Society for Quality in Health Care
• Editorial Board
• Advertising and Corporate Services
• Journals Career Network
• Self-Archiving Policy
• Dispatch Dates
• Contact ISQua
• Journals on Oxford Academic
• Books on Oxford Academic

Article Contents

Primacy of the research question, structure of the paper, writing a research article: advice to beginners.

• Article contents
• Figures & tables
• Supplementary Data

Thomas V. Perneger, Patricia M. Hudelson, Writing a research article: advice to beginners, International Journal for Quality in Health Care , Volume 16, Issue 3, June 2004, Pages 191–192, https://doi.org/10.1093/intqhc/mzh053

• Permissions Icon Permissions

Writing research papers does not come naturally to most of us. The typical research paper is a highly codified rhetorical form [ 1 , 2 ]. Knowledge of the rules—some explicit, others implied—goes a long way toward writing a paper that will get accepted in a peer-reviewed journal.

A good research paper addresses a specific research question. The research question—or study objective or main research hypothesis—is the central organizing principle of the paper. Whatever relates to the research question belongs in the paper; the rest doesn’t. This is perhaps obvious when the paper reports on a well planned research project. However, in applied domains such as quality improvement, some papers are written based on projects that were undertaken for operational reasons, and not with the primary aim of producing new knowledge. In such cases, authors should define the main research question a posteriori and design the paper around it.

Generally, only one main research question should be addressed in a paper (secondary but related questions are allowed). If a project allows you to explore several distinct research questions, write several papers. For instance, if you measured the impact of obtaining written consent on patient satisfaction at a specialized clinic using a newly developed questionnaire, you may want to write one paper on the questionnaire development and validation, and another on the impact of the intervention. The idea is not to split results into ‘least publishable units’, a practice that is rightly decried, but rather into ‘optimally publishable units’.

What is a good research question? The key attributes are: (i) specificity; (ii) originality or novelty; and (iii) general relevance to a broad scientific community. The research question should be precise and not merely identify a general area of inquiry. It can often (but not always) be expressed in terms of a possible association between X and Y in a population Z, for example ‘we examined whether providing patients about to be discharged from the hospital with written information about their medications would improve their compliance with the treatment 1 month later’. A study does not necessarily have to break completely new ground, but it should extend previous knowledge in a useful way, or alternatively refute existing knowledge. Finally, the question should be of interest to others who work in the same scientific area. The latter requirement is more challenging for those who work in applied science than for basic scientists. While it may safely be assumed that the human genome is the same worldwide, whether the results of a local quality improvement project have wider relevance requires careful consideration and argument.

Once the research question is clearly defined, writing the paper becomes considerably easier. The paper will ask the question, then answer it. The key to successful scientific writing is getting the structure of the paper right. The basic structure of a typical research paper is the sequence of Introduction, Methods, Results, and Discussion (sometimes abbreviated as IMRAD). Each section addresses a different objective. The authors state: (i) the problem they intend to address—in other terms, the research question—in the Introduction; (ii) what they did to answer the question in the Methods section; (iii) what they observed in the Results section; and (iv) what they think the results mean in the Discussion.

In turn, each basic section addresses several topics, and may be divided into subsections (Table 1 ). In the Introduction, the authors should explain the rationale and background to the study. What is the research question, and why is it important to ask it? While it is neither necessary nor desirable to provide a full-blown review of the literature as a prelude to the study, it is helpful to situate the study within some larger field of enquiry. The research question should always be spelled out, and not merely left for the reader to guess.

Typical structure of a research paper

The Methods section should provide the readers with sufficient detail about the study methods to be able to reproduce the study if so desired. Thus, this section should be specific, concrete, technical, and fairly detailed. The study setting, the sampling strategy used, instruments, data collection methods, and analysis strategies should be described. In the case of qualitative research studies, it is also useful to tell the reader which research tradition the study utilizes and to link the choice of methodological strategies with the research goals [ 3 ].

The Results section is typically fairly straightforward and factual. All results that relate to the research question should be given in detail, including simple counts and percentages. Resist the temptation to demonstrate analytic ability and the richness of the dataset by providing numerous tables of non-essential results.

The Discussion section allows the most freedom. This is why the Discussion is the most difficult to write, and is often the weakest part of a paper. Structured Discussion sections have been proposed by some journal editors [ 4 ]. While strict adherence to such rules may not be necessary, following a plan such as that proposed in Table 1 may help the novice writer stay on track.

References should be used wisely. Key assertions should be referenced, as well as the methods and instruments used. However, unless the paper is a comprehensive review of a topic, there is no need to be exhaustive. Also, references to unpublished work, to documents in the grey literature (technical reports), or to any source that the reader will have difficulty finding or understanding should be avoided.

Having the structure of the paper in place is a good start. However, there are many details that have to be attended to while writing. An obvious recommendation is to read, and follow, the instructions to authors published by the journal (typically found on the journal’s website). Another concerns non-native writers of English: do have a native speaker edit the manuscript. A paper usually goes through several drafts before it is submitted. When revising a paper, it is useful to keep an eye out for the most common mistakes (Table 2 ). If you avoid all those, your paper should be in good shape.

Common mistakes seen in manuscripts submitted to this journal

Huth EJ . How to Write and Publish Papers in the Medical Sciences , 2nd edition. Baltimore, MD: Williams & Wilkins, 1990 .

Browner WS . Publishing and Presenting Clinical Research . Baltimore, MD: Lippincott, Williams & Wilkins, 1999 .

Devers KJ , Frankel RM. Getting qualitative research published. Educ Health 2001 ; 14 : 109 –117.

Docherty M , Smith R. The case for structuring the discussion of scientific papers. Br Med J 1999 ; 318 : 1224 –1225.

Email alerts

Citing articles via.

• Recommend to your Library

Affiliations

• Online ISSN 1464-3677
• Print ISSN 1353-4505
• Copyright © 2024 International Society for Quality in Health Care and Oxford University Press
• About Oxford Academic
• Publish journals with us
• University press partners
• What we publish
• New features  
• Open access
• Institutional account management
• Rights and permissions
• Get help with access
• Accessibility
• Advertising
• Media enquiries
• Oxford University Press
• Oxford Languages
• University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

• Copyright © 2024 Oxford University Press
• Cookie settings
• Cookie policy
• Privacy policy
• Legal notice

This Feature Is Available To Subscribers Only

Sign In or Create an Account

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

• Research Guides

Reading for Research: Social Sciences

Structure of a research article.

• Structural Read

Guide Acknowledgements

How to Read a Scholarly Article from the Howard Tilton Memorial Library at Tulane University

Strategic Reading for Research   from the Howard Tilton Memorial Library at Tulane University

Bridging the Gap between Faculty Expectation and the Student Experience: Teaching Students toAnnotate and Synthesize Sources

Librarian for Sociology, Environmental Sociology, MHS and Public Policy Studies

Academic writing has features that vary only slightly across the different disciplines. Knowing these elements and the purpose of each serves help you to read and understand academic texts efficiently and effectively, and then apply what you read to your paper or project.

Social Science (and Science) original research articles generally follow IMRD: Introduction- Methods-Results-Discussion

Introduction

• Introduces topic of article
• Presents the "Research Gap"/Statement of Problem article will address
• How research presented in the article will solve the problem presented in research gap.
• Literature Review. presenting and evaluating previous scholarship on a topic.  Sometimes, this is separate section of the article. 

​Method & Results

• How research was done, including analysis and measurements.  
• Sometimes labeled as "Research Design"
• What answers were found
• Interpretation of Results (What Does It Mean? Why is it important?)
• Implications for the Field, how the study contributes to the existing field of knowledge
• Suggestions for further research
• Sometimes called Conclusion

You might also see IBC: Introduction - Body - Conclusion

• Identify the subject
• State the thesis 
• Describe why thesis is important to the field (this may be in the form of a literature review or general prose)

Body  

• Presents Evidence/Counter Evidence
• Integrate other writings (i.e. evidence) to support argument 
• Discuss why others may disagree (counter-evidence) and why argument is still valid
• Summary of argument
• Evaluation of argument by pointing out its implications and/or limitations 
• Anticipate and address possible counter-claims
• Suggest future directions of research
• Next: Structural Read >>
• Last Updated: Jan 19, 2024 10:44 AM
• URL: https://researchguides.library.vanderbilt.edu/readingforresearch

What is Research?: Parts of a Research Article

• The Truth about Research
• Research Steps
• Evaluating Sources
• Parts of a Research Article

While each article is different, here are some common pieces you'll see in many of them...

• The title of the article should give you some clues as to the topic it addresses.
• The abstract allows readers to quickly review the overall content of the article. It should give you an idea of the topic of the article, while also providing any key details--such as the questions address in the article and the general results of the studies conducted.
• The introduction introduces the general topic and provides some background information, eventually narrowing it down to the specific issues addressed in the article.
• The literature review describes past research on the topic and relates it to the specific topic covered by the article.  Not all articles will have a literature review.
• The methods section addresses the research design and methodology used by the author to come to the conclusions they have in this article.  This gives others the ability to replicate the study.  Not all articles will have this, since there will be many articles that don't involve an actual study.
• The results section presents the results of any studies or analysis that has been conducted.  Not all articles will have this, either.
• The discussion/conclusion addresses the implications or future of the field.  It may also address where future research is needed.
• The list references or bibliography is the alphabetized list of resources used for the article.  The format of the citations is often determined by what that field's preferred format is.  Common citations formats include APA, Chicago, and MLA.  This is a necessity in an article--and it helps you identify more possible resources for your own paper.
• Components of a Research Paper Useful site that goes more in depth on these sections.
• Parts of a Citation A really wonderful site by the Nash Community College Library.
• << Previous: Evaluating Sources
• Last Updated: Jul 20, 2017 9:23 PM
• URL: https://libguides.uno.edu/whatisresearch

• Interlibrary Loan and Scan & Deliver
• Course Reserves
• Purchase Request
• Collection Development & Maintenance
• Current Negotiations
• Ask a Librarian
• Instructor Support
• Library How-To
• Research Guides
• Research Support
• Study Rooms
• Research Rooms
• Partner Spaces
• Loanable Equipment
• Print, Scan, Copy
• 3D Printers
• Poster Printing
• OSULP Leadership
• Strategic Plan

AEC 406: Projects (Ecampus)

• Citations & References
• Characteristics of Scholarly Articles

Journal Information

Author and affiliation, specialized vocabulary, research sponsors, introduction, literature review, methodology.

• Getting Journal Articles
• Borrowing Books
• Annotated Bibliograpahy

One of the first things you want to do is look at the journal citation . The information about the article, such as the journal title, volume, issue, and pages is usually on the first page of the article. You will need this information to cite the article correctly in your paper. Sometimes you can tell that the article is scholarly just by the name of the journal, but not always. Scholarly articles are generally long so looking at the number of pages is one indicator that it is scholarly.

You may also want to find out more about the journal in which the article was published. For example, you may want to know if the article is in a peer-reviewed journal. 

One of the first things to look for is the author or authors. In a research article, the authors will list their affiliation, usually with a university or research institution. In this example, the author's affiliation is clearly shown on the first page of the article. In a research article, you will never have an anonymous author or need to look for the author's name or affiliation.

An abstract is a summary of the main article. An abstract will include information about why the research study was done, what the methodology was and something about the findings of the author(s). The abstract is always at the beginning of the article and will either be labeled "abstract" or will be set apart from the rest of the article by a different font or margins.

The abstract should tell you what the research study is about, how the research was done (methodology), who the research sample was, what the authors found and why this is important to the field.

In many articles there are key words or terms that describe the content of the article. This is another good place to look for additional search terms.

Some research is sponsored by an institution, such as the U.S. government or a non-profit or private company. If the research project is sponsored by someone, you will want to investigate who that is and if that could impact how the research was done. For example, a pharmaceutical company that sponsors research on a particular medication may mean that the research is not entirely bias-free.

Most articles will start with an introductory section, which may be labeled introduction. This section introduces the research study, the thesis statement and why the research being conducted is important.

Questions to ask while you read:

• What is the thesis? What are the authors trying to prove or disprove?
• What is the contribution that the authors are making to the field?

The literature review section of an article is a summary or analysis of all the research the author read before doing his/her own research. This section may be part of the introduction or in a section called Background. It provides the background on who has done related research, what that research has or has not uncovered and how the current research contributes to the conversation on the topic. When you read the lit review ask:

• Does the review of the literature logically lead up to the research questions?
• Do the authors review articles relevant to their research study?
• Do the authors show where there are gaps in the literature?

The lit review is also a good place to find other sources you may want to read on this topic to help you get the bigger picture.

The methodology section or methods section tells you how the author(s) went about doing their research. It should let you know a) what method they used to gather data (survey, interviews, experiments, etc.), why they chose this method, and what the limitations are to this method.

The methodology section should be detailed enough that another researcher could replicate the study described. When you read the methodology or methods section:

• What kind of research method did the authors use? Is it an appropriate method for the type of study they are conducting?
• How did the authors get their tests subjects? What criteria did they use?
• What are the contexts of the study that may have affected the results (e.g. environmental conditions, lab conditions, timing questions, etc.)
• Is the sample size representative of the larger population (i.e., was it big enough?)
• Are the data collection instruments and procedures likely to have measured all the important characteristics with reasonable accuracy?
• Does the data analysis appear to have been done with care, and were appropriate analytical techniques used? 

A good researcher will always let you know about the limitations of his or her research.

Research articles are full of data . The data should be complete and directly support the conclusions the authors' draw about their research question.

Tables, graphs, and charts are good indicators that this is a research article. The tables should represent the data in a clear and readable manner.

The results section in a scholarly article is where the author(s) talk about what they found in their research study. Most scholarly articles will have a section labeled results or findings.

The discussion section is where the author(s) write about what they found and what they think it means. The authors may also draw some conclusions about the research and what significance it has in this section. This section will also tell you what some of the issues were with the research or using a specific population for a research study.

The final section is usually called the conclusion or recommendations. Here is where the authors summarize what they found, why they think their research is significant and, if appropriate, make recommendations about future actions or future research that needs to be conducted. In some cases, the conclusion is part of the discussion section.

• << Previous: Step 2
• Next: Getting Journal Articles >>
• Last Updated: Jul 19, 2024 2:43 PM
• URL: https://guides.library.oregonstate.edu/AEC406

Contact Info

121 The Valley Library Corvallis OR 97331–4501

Phone: 541-737-3331

Services for Persons with Disabilities

In the Valley Library

• Oregon State University Press
• Special Collections and Archives Research Center
• Undergrad Research & Writing Studio
• Graduate Student Commons
• Tutoring Services
• Northwest Art Collection

Digital Projects

• Oregon Explorer
• Oregon Digital
• ScholarsArchive@OSU
• Digital Publishing Initiatives
• Atlas of the Pacific Northwest
• Marilyn Potts Guin Library  
• Cascades Campus Library
• McDowell Library of Vet Medicine

• Hirsh Health Sciences
• Webster Veterinary

Guide to Scholarly Articles

• What is a Scholarly Article?
• Scholarly vs. Popular vs. Trade Articles

Types of Scholarly Articles

Qualitative, quantitative, and mixed-methods articles, why does this matter.

• Anatomy of Scholarly Articles
• Tips for Reading Scholarly Articles

Scholarly articles come in many different formats each with their own function in the scholarly conversation. The following are a few of the major types of scholarly articles you are likely to encounter as you become a part of the conversation. Identifying the different types of scholarly articles and knowing their function will help you become a better researcher.

Original/Empirical Studies

• Note: Empirical studies can be subdivided into qualitative studies, quantitative studies, or mixed methods studies. See below for more information  
• Usefulness for research:  Empirical studies are useful because they provide current original research on a topic which may contain a hypothesis or interpretation to advance or to disprove. 

Literature Reviews

• Distinguishing characteristic:  Literature reviews survey and analyze a clearly delaminated body of scholarly literature.  
• Usefulness for research: Literature reviews are useful as a way to quickly get up to date on a particular topic of research.

Theoretical Articles

• Distinguishing characteristic:  Theoretical articles draw on existing scholarship to improve upon or offer a new theoretical perspective on a given topic.
• Usefulness for research:  Theoretical articles are useful because they provide a theoretical framework you can apply to your own research.

Methodological Articles

• Distinguishing characteristic:  Methodological articles draw on existing scholarship to improve or offer new methodologies for exploring a given topic.
• Usefulness for research:  Methodological articles are useful because they provide a methodologies you can apply to your own research.

Case Studies

• Distinguishing characteristic:  Case studies focus on individual examples or instances of a phenomenon to illustrate a research problem or a a solution to a research problem.
• Usefulness for research:  Case studies are useful because they provide information about a research problem or data for analysis.

Book Reviews

• Distinguishing characteristic:  Book reviews provide summaries and evaluations of individual books.
• Usefulness for research:  Book reviews are useful because they provide summaries and evaluations of individual books relevant to your research.

Adapted from the Publication manual of the American Psychological Association : the official guide to APA style. (Sixth edition.). (2013). American Psychological Association.

Qualitative articles  ask "why" questions where as  quantitative  articles  ask "how many/how much?" questions. These approaches are are not mutually exclusive. In fact, many articles combine the two in a  mixed-methods  approach. 

We can think of these different kinds of scholarly articles as different tools designed for different tasks. What research task do you need to accomplish? Do you need to get up to date on a give topic? Find a literature review. Do you need to find a hypothesis to test or to extend? Find an empirical study. Do you need to explore methodologies? Find a methodological article.

• << Previous: Scholarly vs. Popular vs. Trade Articles
• Next: Anatomy of Scholarly Articles >>
• Last Updated: Aug 23, 2023 8:53 AM
• URL: https://researchguides.library.tufts.edu/scholarly-articles

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• J Grad Med Educ
• v.3(4); 2011 Dec

Qualities of Qualitative Research: Part I

Many important medical education research questions cry out for a qualitative research approach: How do teacher characteristics affect learning? Why do learners choose particular specialties? How is professionalism influenced by experiences, mentors, or the curriculum? The medical paradigm, the “hard” science most often taught in medical schools, usually employs quantitative approaches. 1 As a result, clinicians 5 be less familiar with qualitative research or its applicability to medical education questions. For these why types of questions, qualitative or mixed qualitative and quantitative approaches 5 be more appropriate and helpful. 2 Thus, we wish to encourage submissions to the Journal of Graduate Medical Education that are for qualitative purposes or use qualitative methods.

This editorial is the first in a series of two, and it will provide an introduction to qualitative approaches and compare features of quantitative and qualitative research. The second editorial will review in more detail the approaches for selecting participants, analyzing data, and ensuring rigor and study quality in qualitative research. The aims of the editorials are to enhance readers' understanding of articles using this approach and to encourage more researchers to explore qualitative approaches.

Theory and Methodology

Good research follows from a reasonable starting point, a theoretical concept or perspective. Quantitative research uses a positivist perspective in which evidence is objectively and systematically obtained to prove a causal model or hypothesis; what works is the focus. 3 Alternatively, qualitative approaches focus on how and why something works, to build understanding. 3 In the positivist model, study objects (eg, learners) are independent of the researchers, and knowledge or facts are determined through direct observations. Also, the context in which the observations occur is controlled or assumed to be stable. In contrast, in a qualitative paradigm researchers might interact with the study objects (learners) to collect observations, which are highly context specific. 3

Qualitative research has often been differentiated from quantitative as hypothesis generating rather than hypothesis testing . 4 Qualitative research methods “explore, describe, or generate theory, especially for uncertain and ‘immature’ concepts; sensitive and socially dependent concepts; and complex human intentions and motivations.” 4 In education, qualitative research strives to understand how learning occurs through close study of small numbers of learners and a focus on the individual. It attempts to explain a phenomenon or relationship. Typically, results from qualitative research have been assumed to apply only to the small groups studied, such that generalizability of the results to other populations is not expected. For this reason, qualitative research is considered to be hypothesis generating, although some experts dispute this limitation. 5 table 1 presents a comparison of qualitative and quantitative approaches.

Quantitative Versus Qualitative Research

When Qualitative Studies Make Sense

Qualitative studies are helpful to understand why and how; quantitative studies focus on cause and effect, how much, and numeric correlations. Qualitative approaches are used when the potential answer to a question requires an explanation, not a straightforward yes/no. Generally, qualitative research is concerned with cases rather than variables, and understanding differences rather than calculating the mean of responses. 4 In-depth interviews, focus groups, case studies, and open-ended questions are often employed to find these answers. A qualitative study is concerned with the point of view of the individual under study. 6

For example, the changes in duty hours for residents in 2003 have generated many quantitative research articles, which have counted and correlated the changes in numbers of procedures, patient safety parameters, resident test results, and resident sleep hours. However, to determine why residents still sleep about the same number of hours since 2003, one could start from a qualitative framework in order to understand residents' decisions regarding sleep. Similarly, to understand how residents perceive the influence of resident work hour restrictions on aspects of professionalism, a qualitative study would start with the learners rather than by measuring and correlating scores on professionalism assessments. Because learning takes place in social environments characterized by complex interactions, the quantitative “cause and effect” model is often too simplistic. 7

A variety of ways to collect information are available to researchers, such as observation, field notes, reflexive journals, interviews, focus groups, and analysis of documents and materials; table 2 provides examples of these methods. Interviews and focus groups are usually audiorecorded and transcribed for analysis, whereas observations are recorded in field notes by the observer.

Potential Data Sources for Qualitative Research 8

After data collection, accepted methods are employed to interpret the data. Researchers review the observations and report their impressions in a structured format, with subsequent analysis also standardized. table 3 provides one example of an analysis plan. Strategies to ensure rigor in data collection and trustworthiness of the data and data analysis will be discussed in the second editorial in the series.

Iterative Team Process to Interpret Data 8

In contrast to quantitative methods, subjective responses are critical findings, both in participant responses and observer reactions. The unique or outlier response has value in contributing to understanding the experience of others, and thus individual responses are not lost in the aggregation of findings or in the development of research group consensus. 2 , 4 Qualitative methods acknowledge the “myth of objectivity” between researcher and subjects of study. 7 In fact, the researcher is unlikely to be a purely detached observer.

Ethical Issues

As qualitative researchers usually attempt to study subjects and interactions in their “natural settings,” ethical issues frequently arise. Because of the sensitive nature of some discussions as well as the relationship between researchers and participants, informed consent is often required. The very reason for doing qualitative research—to discover why and how, particularly for thorny topics—can lead to potential exposure of sensitive opinions, feelings, and personal information. Thus, consideration of how to protect participants from harm is essential from the very onset of the study.

Quality Assessment

Qualitative researchers need to show that their findings are credible. As with quantitative approaches, a strong research project starts with a basic review of existing knowledge: a solid literature search. However, in contrast to quantitative approaches, most qualitative paradigms do not look to find a single “truth,” but rather multiple views of a context-specific “reality.” The concepts of validity and reliability originally evolved from the quantitative tradition, and therefore their accepted definitions are considered inadequate for qualitative research. Instead, concepts of precision, credibility, and transferability are key aspects of evaluating a qualitative study. 9

Although some experts find that reliability has little relevance to qualitative studies, others propose the term “dependability” as the analogous metric for this type of research. Dependability is gained though consistency of data, which is evaluated through transparent research steps and research findings. 9 , 10 Trustworthiness and rigor are terms used to establish credible findings. One technique often used to enhance trustworthiness and rigor is triangulation, in which multiple data sources (eg, observation, interviews, and recordings), multiple analytic methods, or multiple researchers are used to study the question. 9 The overall goal is to minimize and understand potential bias while ensuring the researcher's “truthfulness” of interpretation. 9

A potentially helpful appraisal checklist for qualitative studies, developed by Coté and Turgeon, 11 is found in table 4 . This appraisal checklist has not been examined systematically. table 5 includes a list of terms commonly used in qualitative research. Approaches to ensure rigor and trustworthiness in qualitative research will be addressed in greater detail in Part 2.

Sample Quality Appraisal Checklist for Qualitative Studies 11 , a

Commonly Used Terms in Qualitative Research 8

Both quantitative and qualitative approaches have strengths and weaknesses; medical education research will benefit from each type of inquiry. The best approach will depend on the kind of question asked, and the best methods will be those most appropriate to the question. 4 To learn more about this topic, the references below are a useful start, as is talking to colleagues engaged in qualitative research at your institution or in your specialty.

Gail M. Sullivan, MD, MPH, is Editor-in-Chief, Journal of Graduate Medical Education; and Joan Sargeant, PhD, is Professor in the Division of Medical Education, Dalhousie University, Halifax, Nova Scotia, Canada.

1. Researching, Writing and Presenting Information - A How To Guide: Writing a Feature Article

• Brainstorming and Planning
• Effective Research
• Note-taking
• Writing an Essay
• Writing a Discussion
• Writing an Exposition
• Writing a Rationale
• Years 11 & 12
• Writing a Blog Post
• Writing a Feature Article
• Preparing Oral Presentations
• Creating a Podcast
• Technologies
• Effective Proofreading Skills
• Glossary of Common Instruction Terms
• Glossary of Literary Terms
• How to use Appendices
• How to paraphrase

Writing an Article

Feature articles explore issues, experiences, opinions and ideas. They present research in an engaging and detailed piece of writing. Features articles are written using language and content tailored to their chosen audience. Always refer to your task guidelines for specific instructions from your teacher.

A Step by Step Guide To Planning Your Article

1. Topic - what is the idea, issue or experience that you intend to explore?

2. Audience - who is the target audience of the publication that will contain your feature article?

3. Purpose - why are you exploring this issue, idea or experience?

4. Research the publication. Remember that each publication has a specific target audience and a distinct style of writing. If you’re writing for a well-known magazine, journal or newspaper, find some examples of feature articles to get an idea of the layout, structure and style.

5. Research your topic. Research will ground your article in fact. Good details to include in your article are statistics, quotes, definitions, anecdotes, references to other media (print, film, television, radio) or references to local venues or events (if for a regional/local publication).

• Draws attention to the main idea of the article
• Encourages the reader to engage with the article

Introduction - the first paragraph

• Establishes tone
• Provides necessary background information
• Includes a hook or unusual statement
• Heightens drama or importance of topic to increase appeal
• May include subheadings
• Personal viewpoints
• Quotes, interviews, expert opinions
• Specific names, places and dates
• Photographs, diagrams, tables and graphs
• Suggests an appropriate course of action
• Encourages reader to change attitude or opinion
• Reinforces article's main idea

Language features of an Article

The language features of an article will depend upon the purpose and audience; usually, the vocabulary of the article will fit the topic content, and who it is targeted at.

• Direct quotes - personalises the topic
• Imagery and description - engage reader's imagination
• Facts & research - validate the viewpoints being presented
• Anecdotes - personalise & maintain interest
• Relevant jargon - increases authenticity
• Personal tone - created using informal, colloquial language and first person narrative where relevant to purpose and audience
• << Previous: Writing a Blog Post
• Next: Preparing Oral Presentations >>
• Last Updated: Sep 15, 2022 2:24 PM
• URL: https://stcc.libguides.com/c.php?g=926682

More results...

Complex science beautifully .

Research Features distils academic work into engaging content, truly connecting science with society.

Research Features is sponsored by the researchers we feature. Their work is expertly transformed into captivating content we publish.

Our skilled marketing team disseminate your content online for the world to see. You’ll then receive a report showing you the results of our global reach.

Subscribe to our Newsletter – we will never use your email address for anything other than to send you a digital publication link.

Innovations and insights in rail and energy projects

HLA genes in Type 2 diabetes: Protection and risk

Latest articles..

Decoding insights: A structured approach to applied research

Animal resistance to stress in the desert

Random articles..

Earthwatch: Working for the good of the planet

Mesoscale ocean currents and improved climate modelling

Promote your work.

Discover latest publications.

Thought leaders., e-mrs: rewarding collaborative materials research.

• Thought Leaders

A global challenge: On the lookout for research ideas for the prevention of breast cancer

Battling the stigma: the world’s first vagina museum, gacd: tackling chronic non-communicable diseases head on, literature and the laboratory: where art and science meet, gun violence: a biopsychosocial disease, gender equality as a precondition for excellence: the core mission of epws in european research policy, the institute for integrative health: igniting an epidemic of health, canadian blood services: the road to regaining public trust, dietary supplements: fact or fiction and how to tell the difference, mrs: the strange matter of materials science, asm: living in a material world, latest publications..

Research Features Magazine – 153

Our team at Research Features is passionate about making the

Research Features Magazine – 152

At Research Features, our aim is to spotlight cutting-edge research

Research Features Magazine – 151

Academic research is the foundation for much of our knowledge,

Research Features Magazine – 150

Research features magazine – 149, research features magazine – 148, research features magazine – 147, health & medicine., self-care 2030: towards a seismic shift in perceiving and practising mental healthcare.

• Health & Medicine

Niclosamide restricts entry protein CD147 – A novel therapeutic prospect in SARS-CoV-2 infections

Breaking barriers: recent advancements in pancreatic cancer detection with graphene sensors, the impact of national holidays on postoperative radiotherapy: a study from canada, blog latest., artificial intelligence: perhaps our most powerful weapon in the battle with cancer, the tenth edition of rosen research review: celebrating a successful collaboration, carcinisation – what’s so special about crabs, british science week 2024, social media reach.

• Systematic review
• Open access
• Published: 07 August 2024

Models and frameworks for assessing the implementation of clinical practice guidelines: a systematic review

• Nicole Freitas de Mello   ORCID: orcid.org/0000-0002-5228-6691 1 , 2 ,
• Sarah Nascimento Silva   ORCID: orcid.org/0000-0002-1087-9819 3 ,
• Dalila Fernandes Gomes   ORCID: orcid.org/0000-0002-2864-0806 1 , 2 ,
• Juliana da Motta Girardi   ORCID: orcid.org/0000-0002-7547-7722 4 &
• Jorge Otávio Maia Barreto   ORCID: orcid.org/0000-0002-7648-0472 2 , 4  

Implementation Science volume  19 , Article number:  59 ( 2024 ) Cite this article

179 Accesses

5 Altmetric

Metrics details

The implementation of clinical practice guidelines (CPGs) is a cyclical process in which the evaluation stage can facilitate continuous improvement. Implementation science has utilized theoretical approaches, such as models and frameworks, to understand and address this process. This article aims to provide a comprehensive overview of the models and frameworks used to assess the implementation of CPGs.

A systematic review was conducted following the Cochrane methodology, with adaptations to the "selection process" due to the unique nature of this review. The findings were reported following PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) reporting guidelines. Electronic databases were searched from their inception until May 15, 2023. A predetermined strategy and manual searches were conducted to identify relevant documents from health institutions worldwide. Eligible studies presented models and frameworks for assessing the implementation of CPGs. Information on the characteristics of the documents, the context in which the models were used (specific objectives, level of use, type of health service, target group), and the characteristics of each model or framework (name, domain evaluated, and model limitations) were extracted. The domains of the models were analyzed according to the key constructs: strategies, context, outcomes, fidelity, adaptation, sustainability, process, and intervention. A subgroup analysis was performed grouping models and frameworks according to their levels of use (clinical, organizational, and policy) and type of health service (community, ambulatorial, hospital, institutional). The JBI’s critical appraisal tools were utilized by two independent researchers to assess the trustworthiness, relevance, and results of the included studies.

Database searches yielded 14,395 studies, of which 80 full texts were reviewed. Eight studies were included in the data analysis and four methodological guidelines were additionally included from the manual search. The risk of bias in the studies was considered non-critical for the results of this systematic review. A total of ten models/frameworks for assessing the implementation of CPGs were found. The level of use was mainly policy, the most common type of health service was institutional, and the major target group was professionals directly involved in clinical practice. The evaluated domains differed between the models and there were also differences in their conceptualization. All the models addressed the domain "Context", especially at the micro level (8/12), followed by the multilevel (7/12). The domains "Outcome" (9/12), "Intervention" (8/12), "Strategies" (7/12), and "Process" (5/12) were frequently addressed, while "Sustainability" was found only in one study, and "Fidelity/Adaptation" was not observed.

Conclusions

The use of models and frameworks for assessing the implementation of CPGs is still incipient. This systematic review may help stakeholders choose or adapt the most appropriate model or framework to assess CPGs implementation based on their specific health context.

Trial registration

PROSPERO (International Prospective Register of Systematic Reviews) registration number: CRD42022335884. Registered on June 7, 2022.

Peer Review reports

Contributions to the literature

Although the number of theoretical approaches has grown in recent years, there are still important gaps to be explored in the use of models and frameworks to assess the implementation of clinical practice guidelines (CPGs). This systematic review aims to contribute knowledge to overcome these gaps.

Despite the great advances in implementation science, evaluating the implementation of CPGs remains a challenge, and models and frameworks could support improvements in this field.

This study demonstrates that the available models and frameworks do not cover all characteristics and domains necessary for a complete evaluation of CPGs implementation.

The presented findings contribute to the field of implementation science, encouraging debate on choices and adaptations of models and frameworks for implementation research and evaluation.

Substantial investments have been made in clinical research and development in recent decades, increasing the medical knowledge base and the availability of health technologies [ 1 ]. The use of clinical practice guidelines (CPGs) has increased worldwide to guide best health practices and to maximize healthcare investments. A CPG can be defined as "any formal statements systematically developed to assist practitioner and patient decisions about appropriate health care for specific clinical circumstances" [ 2 ] and has the potential to improve patient care by promoting interventions of proven benefit and discouraging ineffective interventions. Furthermore, they can promote efficiency in resource allocation and provide support for managers and health professionals in decision-making [ 3 , 4 ].

However, having a quality CPG does not guarantee that the expected health benefits will be obtained. In fact, putting these devices to use still presents a challenge for most health services across distinct levels of government. In addition to the development of guidelines with high methodological rigor, those recommendations need to be available to their users; these recommendations involve the diffusion and dissemination stages, and they need to be used in clinical practice (implemented), which usually requires behavioral changes and appropriate resources and infrastructure. All these stages involve an iterative and complex process called implementation, which is defined as the process of putting new practices within a setting into use [ 5 , 6 ].

Implementation is a cyclical process, and the evaluation is one of its key stages, which allows continuous improvement of CPGs development and implementation strategies. It consists of verifying whether clinical practice is being performed as recommended (process evaluation or formative evaluation) and whether the expected results and impact are being reached (summative evaluation) [ 7 , 8 , 9 ]. Although the importance of the implementation evaluation stage has been recognized, research on how these guidelines are implemented is scarce [ 10 ]. This paper focused on the process of assessing CPGs implementation.

To understand and improve this complex process, implementation science provides a systematic set of principles and methods to integrate research findings and other evidence-based practices into routine practice and improve the quality and effectiveness of health services and care [ 11 ]. The field of implementation science uses theoretical approaches that have varying degrees of specificity based on the current state of knowledge and are structured based on theories, models, and frameworks [ 5 , 12 , 13 ]. A "Model" is defined as "a simplified depiction of a more complex world with relatively precise assumptions about cause and effect", and a "framework" is defined as "a broad set of constructs that organize concepts and data descriptively without specifying causal relationships" [ 9 ]. Although these concepts are distinct, in this paper, their use will be interchangeable, as they are typically like checklists of factors relevant to various aspects of implementation.

There are a variety of theoretical approaches available in implementation science [ 5 , 14 ], which can make choosing the most appropriate challenging [ 5 ]. Some models and frameworks have been categorized as "evaluation models" by providing a structure for evaluating implementation endeavors [ 15 ], even though theoretical approaches from other categories can also be applied for evaluation purposes because they specify concepts and constructs that may be operationalized and measured [ 13 ]. Two frameworks that can specify implementation aspects that should be evaluated as part of intervention studies are RE-AIM (Reach, Effectiveness, Adoption, Implementation, Maintenance) [ 16 ] and PRECEDE-PROCEED (Predisposing, Reinforcing and Enabling Constructs in Educational Diagnosis and Evaluation-Policy, Regulatory, and Organizational Constructs in Educational and Environmental Development) [ 17 ]. Although the number of theoretical approaches has grown in recent years, the use of models and frameworks to evaluate the implementation of guidelines still seems to be a challenge.

This article aims to provide a complete map of the models and frameworks applied to assess the implementation of CPGs. The aim is also to subside debate and choices on models and frameworks for the research and evaluation of the implementation processes of CPGs and thus to facilitate the continued development of the field of implementation as well as to contribute to healthcare policy and practice.

A systematic review was conducted following the Cochrane methodology [ 18 ], with adaptations to the "selection process" due to the unique nature of this review (details can be found in the respective section). The review protocol was registered in PROSPERO (registration number: CRD42022335884) on June 7, 2022. This report adhered to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines [ 19 ] and a completed checklist is provided in Additional File 1.

Eligibility criteria

The SDMO approach (Types of Studies, Types of Data, Types of Methods, Outcomes) [ 20 ] was utilized in this systematic review, outlined as follows:

Types of studies

All types of studies were considered for inclusion, as the assessment of CPG implementation can benefit from a diverse range of study designs, including randomized clinical trials/experimental studies, scale/tool development, systematic reviews, opinion pieces, qualitative studies, peer-reviewed articles, books, reports, and unpublished theses.

Studies were categorized based on their methodological designs, which guided the synthesis, risk of bias assessment, and presentation of results.

Study protocols and conference abstracts were excluded due to insufficient information for this review.

Types of data

Studies that evaluated the implementation of CPGs either independently or as part of a multifaceted intervention.

Guidelines for evaluating CPG implementation.

Inclusion of CPGs related to any context, clinical area, intervention, and patient characteristics.

No restrictions were placed on publication date or language.

Exclusion criteria

General guidelines were excluded, as this review focused on 'models for evaluating clinical practice guidelines implementation' rather than the guidelines themselves.

Studies that focused solely on implementation determinants as barriers and enablers were excluded, as this review aimed to explore comprehensive models/frameworks.

Studies evaluating programs and policies were excluded.

Studies that only assessed implementation strategies (isolated actions) rather than the implementation process itself were excluded.

Studies that focused solely on the impact or results of implementation (summative evaluation) were excluded.

Types of methods

Not applicable.

All potential models or frameworks for assessing the implementation of CPG (evaluation models/frameworks), as well as their characteristics: name; specific objectives; levels of use (clinical, organizational, and policy); health system (public, private, or both); type of health service (community, ambulatorial, hospital, institutional, homecare); domains or outcomes evaluated; type of recommendation evaluated; context; limitations of the model.

Model was defined as "a deliberated simplification of a phenomenon on a specific aspect" [ 21 ].

Framework was defined as "structure, overview outline, system, or plan consisting of various descriptive categories" [ 21 ].

Models or frameworks used solely for the CPG development, dissemination, or implementation phase.

Models/frameworks used solely for assessment processes other than implementation, such as for the development or dissemination phase.

Data sources and literature search

The systematic search was conducted on July 31, 2022 (and updated on May 15, 2023) in the following electronic databases: MEDLINE/PubMed, Centre for Reviews and Dissemination (CRD), the Cochrane Library, Cumulative Index to Nursing and Allied Health Literature (CINAHL), EMBASE, Epistemonikos, Global Health, Health Systems Evidence, PDQ-Evidence, PsycINFO, Rx for Change (Canadian Agency for Drugs and Technologies in Health, CADTH), Scopus, Web of Science and Virtual Health Library (VHL). The Google Scholar database was used for the manual selection of studies (first 10 pages).

Additionally, hand searches were performed on the lists of references included in the systematic reviews and citations of the included studies, as well as on the websites of institutions working on CPGs development and implementation: Guidelines International Networks (GIN), National Institute for Health and Care Excellence (NICE; United Kingdom), World Health Organization (WHO), Centers for Disease Control and Prevention (CDC; USA), Institute of Medicine (IOM; USA), Australian Department of Health and Aged Care (ADH), Healthcare Improvement Scotland (SIGN), National Health and Medical Research Council (NHMRC; Australia), Queensland Health, The Joanna Briggs Institute (JBI), Ministry of Health and Social Policy of Spain, Ministry of Health of Brazil and Capes Theses and Dissertations Catalog.

The search strategy combined terms related to "clinical practice guidelines" (practice guidelines, practice guidelines as topic, clinical protocols), "implementation", "assessment" (assessment, evaluation), and "models, framework". The free term "monitoring" was not used because it was regularly related to clinical monitoring and not to implementation monitoring. The search strategies adapted for the electronic databases are presented in an additional file (see Additional file 2).

Study selection process

The results of the literature search from scientific databases, excluding the CRD database, were imported into Mendeley Reference Management software to remove duplicates. They were then transferred to the Rayyan platform ( https://rayyan.qcri.org ) [ 22 ] for the screening process. Initially, studies related to the "assessment of implementation of the CPG" were selected. The titles were first screened independently by two pairs of reviewers (first selection: four reviewers, NM, JB, SS, and JG; update: a pair of reviewers, NM and DG). The title screening was broad, including all potentially relevant studies on CPG and the implementation process. Following that, the abstracts were independently screened by the same group of reviewers. The abstract screening was more focused, specifically selecting studies that addressed CPG and the evaluation of the implementation process. In the next step, full-text articles were reviewed independently by a pair of reviewers (NM, DG) to identify those that explicitly presented "models" or "frameworks" for assessing the implementation of the CPG. Disagreements regarding the eligibility of studies were resolved through discussion and consensus, and by a third reviewer (JB) when necessary. One reviewer (NM) conducted manual searches, and the inclusion of documents was discussed with the other reviewers.

Risk of bias assessment of studies

The selected studies were independently classified and evaluated according to their methodological designs by two investigators (NM and JG). This review employed JBI’s critical appraisal tools to assess the trustworthiness, relevance and results of the included studies [ 23 ] and these tools are presented in additional files (see Additional file 3 and Additional file 4). Disagreements were resolved by consensus or consultation with the other reviewers. Methodological guidelines and noncomparative and before–after studies were not evaluated because JBI does not have specific tools for assessing these types of documents. Although the studies were assessed for quality, they were not excluded on this basis.

Data extraction

The data was independently extracted by two reviewers (NM, DG) using a Microsoft Excel spreadsheet. Discrepancies were discussed and resolved by consensus. The following information was extracted:

Document characteristics : author; year of publication; title; study design; instrument of evaluation; country; guideline context;

Usage context of the models : specific objectives; level of use (clinical, organizational, and policy); type of health service (community, ambulatorial, hospital, institutional); target group (guideline developers, clinicians; health professionals; health-policy decision-makers; health-care organizations; service managers);

Model and framework characteristics : name, domain evaluated, and model limitations.

The set of information to be extracted, shown in the systematic review protocol, was adjusted to improve the organization of the analysis.

The "level of use" refers to the scope of the model used. "Clinical" was considered when the evaluation focused on individual practices, "organizational" when practices were within a health service institution, and "policy" when the evaluation was more systemic and covered different health services or institutions.

The "type of health service" indicated the category of health service where the model/framework was used (or can be used) to assess the implementation of the CPG, related to the complexity of healthcare. "Community" is related to primary health care; "ambulatorial" is related to secondary health care; "hospital" is related to tertiary health care; and "institutional" represented models/frameworks not specific to a particular type of health service.

The "target group" included stakeholders related to the use of the model/framework for evaluating the implementation of the CPG, such as clinicians, health professionals, guideline developers, health policy-makers, health organizations, and service managers.

The category "health system" (public, private, or both) mentioned in the systematic review protocol was not found in the literature obtained and was removed as an extraction variable. Similarly, the variables "type of recommendation evaluated" and "context" were grouped because the same information was included in the "guideline context" section of the study.

Some selected documents presented models or frameworks recognized by the scientific field, including some that were validated. However, some studies adapted the model to this context. Therefore, the domain analysis covered all models or frameworks domains evaluated by (or suggested for evaluation by) the document analyzed.

Data analysis and synthesis

The results were tabulated using narrative synthesis with an aggregative approach, without meta-analysis, aiming to summarize the documents descriptively for the organization, description, interpretation and explanation of the study findings [ 24 , 25 ].

The model/framework domains evaluated in each document were studied according to Nilsen et al.’s constructs: "strategies", "context", "outcomes", "fidelity", "adaptation" and "sustainability". For this study, "strategies" were described as structured and planned initiatives used to enhance the implementation of clinical practice [ 26 ].

The definition of "context" varies in the literature. Despite that, this review considered it as the set of circumstances or factors surrounding a particular implementation effort, such as organizational support, financial resources, social relations and support, leadership, and organizational culture [ 26 , 27 ]. The domain "context" was subdivided according to the level of health care into "micro" (individual perspective), "meso" (organizational perspective), "macro" (systemic perspective), and "multiple" (when there is an issue involving more than one level of health care).

The "outcomes" domain was related to the results of the implementation process (unlike clinical outcomes) and was stratified according to the following constructs: acceptability, appropriateness, feasibility, adoption, cost, and penetration. All these concepts align with the definitions of Proctor et al. (2011), although we decided to separate "fidelity" and "sustainability" as independent domains similar to Nilsen [ 26 , 28 ].

"Fidelity" and "adaptation" were considered the same domain, as they are complementary pieces of the same issue. In this study, implementation fidelity refers to how closely guidelines are followed as intended by their developers or designers. On the other hand, adaptation involves making changes to the content or delivery of a guideline to better fit the needs of a specific context. The "sustainability" domain was defined as evaluations about the continuation or permanence over time of the CPG implementation.

Additionally, the domain "process" was utilized to address issues related to the implementation process itself, rather than focusing solely on the outcomes of the implementation process, as done by Wang et al. [ 14 ]. Furthermore, the "intervention" domain was introduced to distinguish aspects related to the CPG characteristics that can impact its implementation, such as the complexity of the recommendation.

A subgroup analysis was performed with models and frameworks categorized based on their levels of use (clinical, organizational, and policy) and the type of health service (community, ambulatorial, hospital, institutional) associated with the CPG. The goal is to assist stakeholders (politicians, clinicians, researchers, or others) in selecting the most suitable model for evaluating CPG implementation based on their specific health context.

Search results

Database searches yielded 26,011 studies, of which 107 full texts were reviewed. During the full-text review, 99 articles were excluded: 41 studies did not mention a model or framework for assessing the implementation of the CPG, 31 studies evaluated only implementation strategies (isolated actions) rather than the implementation process itself, and 27 articles were not related to the implementation assessment. Therefore, eight studies were included in the data analysis. The updated search did not reveal additional relevant studies. The main reason for study exclusion was that they did not use models or frameworks to assess CPG implementation. Additionally, four methodological guidelines were included from the manual search (Fig.  1 ).

PRISMA diagram. Acronyms: ADH—Australian Department of Health, CINAHL—Cumulative Index to Nursing and Allied Health Literature, CDC—Centers for Disease Control and Prevention, CRD—Centre for Reviews and Dissemination, GIN—Guidelines International Networks, HSE—Health Systems Evidence, IOM—Institute of Medicine, JBI—The Joanna Briggs Institute, MHB—Ministry of Health of Brazil, NICE—National Institute for Health and Care Excellence, NHMRC—National Health and Medical Research Council, MSPS – Ministerio de Sanidad Y Política Social (Spain), SIGN—Scottish Intercollegiate Guidelines Network, VHL – Virtual Health Library, WHO—World Health Organization. Legend: Reason A –The study evaluated only implementation strategies (isolated actions) rather than the implementation process itself. Reason B – The study did not mention a model or framework for assessing the implementation of the intervention. Reason C – The study was not related to the implementation assessment. Adapted from Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 2021;372:n71. https://doi.org/10.1136/bmj.n71 . For more information, visit:

According to the JBI’s critical appraisal tools, the overall assessment of the studies indicates their acceptance for the systematic review.

The cross-sectional studies lacked clear information regarding "confounding factors" or "strategies to address confounding factors". This was understandable given the nature of the study, where such details are not typically included. However, the reviewers did not find this lack of information to be critical, allowing the studies to be included in the review. The results of this methodological quality assessment can be found in an additional file (see Additional file 5).

In the qualitative studies, there was some ambiguity regarding the questions: "Is there a statement locating the researcher culturally or theoretically?" and "Is the influence of the researcher on the research, and vice versa, addressed?". However, the reviewers decided to include the studies and deemed the methodological quality sufficient for the analysis in this article, based on the other information analyzed. The results of this methodological quality assessment can be found in an additional file (see Additional file 6).

Documents characteristics (Table  1 )

The documents were directed to several continents: Australia/Oceania (4/12) [ 31 , 33 , 36 , 37 ], North America (4/12 [ 30 , 32 , 38 , 39 ], Europe (2/12 [ 29 , 35 ] and Asia (2/12) [ 34 , 40 ]. The types of documents were classified as cross-sectional studies (4/12) [ 29 , 32 , 34 , 38 ], methodological guidelines (4/12) [ 33 , 35 , 36 , 37 ], mixed methods studies (3/12) [ 30 , 31 , 39 ] or noncomparative studies (1/12) [ 40 ]. In terms of the instrument of evaluation, most of the documents used a survey/questionnaire (6/12) [ 29 , 30 , 31 , 32 , 34 , 38 ], while three (3/12) used qualitative instruments (interviews, group discussions) [ 30 , 31 , 39 ], one used a checklist [ 37 ], one used an audit [ 33 ] and three (3/12) did not define a specific instrument to measure [ 35 , 36 , 40 ].

Considering the clinical areas covered, most studies evaluated the implementation of nonspecific (general) clinical areas [ 29 , 33 , 35 , 36 , 37 , 40 ]. However, some studies focused on specific clinical contexts, such as mental health [ 32 , 38 ], oncology [ 39 ], fall prevention [ 31 ], spinal cord injury [ 30 ], and sexually transmitted infections [ 34 ].

Usage context of the models (Table  1 )

Specific objectives.

All the studies highlighted the purpose of guiding the process of evaluating the implementation of CPGs, even if they evaluated CPGs from generic or different clinical areas.

Levels of use

The most common level of use of the models/frameworks identified to assess the implementation of CPGs was policy (6/12) [ 33 , 35 , 36 , 37 , 39 , 40 ]. In this level, the model is used in a systematic way to evaluate all the processes involved in CPGs implementation and is primarily related to methodological guidelines. This was followed by the organizational level of use (5/12) [ 30 , 31 , 32 , 38 , 39 ], where the model is used to evaluate the implementation of CPGs in a specific institution, considering its specific environment. Finally, the clinical level of use (2/12) [ 29 , 34 ] focuses on individual practice and the factors that can influence the implementation of CPGs by professionals.

Type of health service

Institutional services were predominant (5/12) [ 33 , 35 , 36 , 37 , 40 ] and included methodological guidelines and a study of model development and validation. Hospitals were the second most common type of health service (4/12) [ 29 , 30 , 31 , 34 ], followed by ambulatorial (2/12) [ 32 , 34 ] and community health services (1/12) [ 32 ]. Two studies did not specify which type of health service the assessment addressed [ 38 , 39 ].

Target group

The focus of the target group was professionals directly involved in clinical practice (6/12) [ 29 , 31 , 32 , 34 , 38 , 40 ], namely, health professionals and clinicians. Other less related stakeholders included guideline developers (2/12) [ 39 , 40 ], health policy decision makers (1/12) [ 39 ], and healthcare organizations (1/12) [ 39 ]. The target group was not defined in the methodological guidelines, although all the mentioned stakeholders could be related to these documents.

Model and framework characteristics

Models and frameworks for assessing the implementation of cpgs.

The Consolidated Framework for Implementation Research (CFIR) [ 31 , 38 ] and the Promoting Action on Research Implementation in Health Systems (PARiHS) framework [ 29 , 30 ] were the most commonly employed frameworks within the selected documents. The other models mentioned were: Goal commitment and implementation of practice guidelines framework [ 32 ]; Guideline to identify key indicators [ 35 ]; Guideline implementation checklist [ 37 ]; Guideline implementation evaluation tool [ 40 ]; JBI Implementation Framework [ 33 ]; Reach, effectiveness, adoption, implementation and maintenance (RE-AIM) framework [ 34 ]; The Guideline Implementability Framework [ 39 ] and an unnamed model [ 36 ].

Domains evaluated

The number of domains evaluated (or suggested for evaluation) by the documents varied between three and five, with the majority focusing on three domains. All the models addressed the domain "context", with a particular emphasis on the micro level of the health care context (8/12) [ 29 , 31 , 34 , 35 , 36 , 37 , 38 , 39 ], followed by the multilevel (7/12) [ 29 , 31 , 32 , 33 , 38 , 39 , 40 ], meso level (4/12) [ 30 , 35 , 39 , 40 ] and macro level (2/12) [ 37 , 39 ]. The "Outcome" domain was evaluated in nine models. Within this domain, the most frequently evaluated subdomain was "adoption" (6/12) [ 29 , 32 , 34 , 35 , 36 , 37 ], followed by "acceptability" (4/12) [ 30 , 32 , 35 , 39 ], "appropriateness" (3/12) [ 32 , 34 , 36 ], "feasibility" (3/12) [ 29 , 32 , 36 ], "cost" (1/12) [ 35 ] and "penetration" (1/12) [ 34 ]. Regarding the other domains, "Intervention" (8/12) [ 29 , 31 , 34 , 35 , 36 , 38 , 39 , 40 ], "Strategies" (7/12) [ 29 , 30 , 33 , 35 , 36 , 37 , 40 ] and "Process" (5/12) [ 29 , 31 , 32 , 33 , 38 ] were frequently addressed in the models, while "Sustainability" (1/12) [ 34 ] was only found in one model, and "Fidelity/Adaptation" was not observed. The domains presented by the models and frameworks and evaluated in the documents are shown in Table  2 .

Limitations of the models

Only two documents mentioned limitations in the use of the model or frameworks. These two studies reported limitations in the use of CFIR: "is complex and cumbersome and requires tailoring of the key variables to the specific context", and "this framework should be supplemented with other important factors and local features to achieve a sound basis for the planning and realization of an ongoing project" [ 31 , 38 ]. Limitations in the use of other models or frameworks are not reported.

Subgroup analysis

Following the subgroup analysis (Table  3 ), five different models/frameworks were utilized at the policy level by institutional health services. These included the Guideline Implementation Evaluation Tool [ 40 ], the NHMRC tool (model name not defined) [ 36 ], the JBI Implementation Framework + GRiP [ 33 ], Guideline to identify key indicators [ 35 ], and the Guideline implementation checklist [ 37 ]. Additionally, the "Guideline Implementability Framework" [ 39 ] was implemented at the policy level without restrictions based on the type of health service. Regarding the organizational level, the models used varied depending on the type of service. The "Goal commitment and implementation of practice guidelines framework" [ 32 ] was applied in community and ambulatory health services, while "PARiHS" [ 29 , 30 ] and "CFIR" [ 31 , 38 ] were utilized in hospitals. In contexts where the type of health service was not defined, "CFIR" [ 31 , 38 ] and "The Guideline Implementability Framework" [ 39 ] were employed. Lastly, at the clinical level, "RE-AIM" [ 34 ] was utilized in ambulatory and hospital services, and PARiHS [ 29 , 30 ] was specifically used in hospital services.

Key findings

This systematic review identified 10 models/ frameworks used to assess the implementation of CPGs in various health system contexts. These documents shared similar objectives in utilizing models and frameworks for assessment. The primary level of use was policy, the most common type of health service was institutional, and the main target group of the documents was professionals directly involved in clinical practice. The models and frameworks presented varied analytical domains, with sometimes divergent concepts used in these domains. This study is innovative in its emphasis on the evaluation stage of CPG implementation and in summarizing aspects and domains aimed at the practical application of these models.

The small number of documents contrasts with studies that present an extensive range of models and frameworks available in implementation science. The findings suggest that the use of models and frameworks to evaluate the implementation of CPGs is still in its early stages. Among the selected documents, there was a predominance of cross-sectional studies and methodological guidelines, which strongly influenced how the implementation evaluation was conducted. This was primarily done through surveys/questionnaires, qualitative methods (interviews, group discussions), and non-specific measurement instruments. Regarding the subject areas evaluated, most studies focused on a general clinical area, while others explored different clinical areas. This suggests that the evaluation of CPG implementation has been carried out in various contexts.

The models were chosen independently of the categories proposed in the literature, with their usage categorized for purposes other than implementation evaluation, as is the case with CFIR and PARiHS. This practice was described by Nilsen et al. who suggested that models and frameworks from other categories can also be applied for evaluation purposes because they specify concepts and constructs that may be operationalized and measured [ 14 , 15 , 42 , 43 ].

The results highlight the increased use of models and frameworks in evaluation processes at the policy level and institutional environments, followed by the organizational level in hospital settings. This finding contradicts a review that reported the policy level as an area that was not as well studied [ 44 ]. The use of different models at the institutional level is also emphasized in the subgroup analysis. This may suggest that the greater the impact (social, financial/economic, and organizational) of implementing CPGs, the greater the interest and need to establish well-defined and robust processes. In this context, the evaluation stage stands out as crucial, and the investment of resources and efforts to structure this stage becomes even more advantageous [ 10 , 45 ]. Two studies (16,7%) evaluated the implementation of CPGs at the individual level (clinical level). These studies stand out for their potential to analyze variations in clinical practice in greater depth.

In contrast to the level of use and type of health service most strongly indicated in the documents, with systemic approaches, the target group most observed was professionals directly involved in clinical practice. This suggests an emphasis on evaluating individual behaviors. This same emphasis is observed in the analysis of the models, in which there is a predominance of evaluating the micro level of the health context and the "adoption" subdomain, in contrast with the sub-use of domains such as "cost" and "process". Cassetti et al. observed the same phenomenon in their review, in which studies evaluating the implementation of CPGs mainly adopted a behavioral change approach to tackle those issues, without considering the influence of wider social determinants of health [ 10 ]. However, the literature widely reiterates that multiple factors impact the implementation of CPGs, and different actions are required to make them effective [ 6 , 46 , 47 ]. As a result, there is enormous potential for the development and adaptation of models and frameworks aimed at more systemic evaluation processes that consider institutional and organizational aspects.

In analyzing the model domains, most models focused on evaluating only some aspects of implementation (three domains). All models evaluated the "context", highlighting its significant influence on implementation [ 9 , 26 ]. Context is an essential effect modifier for providing research evidence to guide decisions on implementation strategies [ 48 ]. Contextualizing a guideline involves integrating research or other evidence into a specific circumstance [ 49 ]. The analysis of this domain was adjusted to include all possible contextual aspects, even if they were initially allocated to other domains. Some contextual aspects presented by the models vary in comprehensiveness, such as the assessment of the "timing and nature of stakeholder engagement" [ 39 ], which includes individual engagement by healthcare professionals and organizational involvement in CPG implementation. While the importance of context is universally recognized, its conceptualization and interpretation differ across studies and models. This divergence is also evident in other domains, consistent with existing literature [ 14 ]. Efforts to address this conceptual divergence in implementation science are ongoing, but further research and development are needed in this field [ 26 ].

The main subdomain evaluated was "adoption" within the outcome domain. This may be attributed to the ease of accessing information on the adoption of the CPG, whether through computerized system records, patient records, or self-reports from healthcare professionals or patients themselves. The "acceptability" subdomain pertains to the perception among implementation stakeholders that a particular CPG is agreeable, palatable or satisfactory. On the other hand, "appropriateness" encompasses the perceived fit, relevance or compatibility of the CPG for a specific practice setting, provider, or consumer, or its perceived fit to address a particular issue or problem [ 26 ]. Both subdomains are subjective and rely on stakeholders' interpretations and perceptions of the issue being analyzed, making them susceptible to reporting biases. Moreover, obtaining this information requires direct consultation with stakeholders, which can be challenging for some evaluation processes, particularly in institutional contexts.

The evaluation of the subdomains "feasibility" (the extent to which a CPG can be successfully used or carried out within a given agency or setting), "cost" (the cost impact of an implementation effort), and "penetration" (the extent to which an intervention or treatment is integrated within a service setting and its subsystems) [ 26 ] was rarely observed in the documents. This may be related to the greater complexity of obtaining information on these aspects, as they involve cross-cutting and multifactorial issues. In other words, it would be difficult to gather this information during evaluations with health practitioners as the target group. This highlights the need for evaluation processes of CPGs implementation involving multiple stakeholders, even if the evaluation is adjusted for each of these groups.

Although the models do not establish the "intervention" domain, we thought it pertinent in this study to delimit the issues that are intrinsic to CPGs, such as methodological quality or clarity in establishing recommendations. These issues were quite common in the models evaluated but were considered in other domains (e.g., in "context"). Studies have reported the importance of evaluating these issues intrinsic to CPGs [ 47 , 50 ] and their influence on the implementation process [ 51 ].

The models explicitly present the "strategies" domain, and its evaluation was usually included in the assessments. This is likely due to the expansion of scientific and practical studies in implementation science that involve theoretical approaches to the development and application of interventions to improve the implementation of evidence-based practices. However, these interventions themselves are not guaranteed to be effective, as reported in a previous review that showed unclear results indicating that the strategies had affected successful implementation [ 52 ]. Furthermore, model domains end up not covering all the complexity surrounding the strategies and their development and implementation process. For example, the ‘Guideline implementation evaluation tool’ evaluates whether guideline developers have designed and provided auxiliary tools to promote the implementation of guidelines [ 40 ], but this does not mean that these tools would work as expected.

The "process" domain was identified in the CFIR [ 31 , 38 ], JBI/GRiP [ 33 ], and PARiHS [ 29 ] frameworks. While it may be included in other domains of analysis, its distinct separation is crucial for defining operational issues when assessing the implementation process, such as determining if and how the use of the mentioned CPG was evaluated [ 3 ]. Despite its presence in multiple models, there is still limited detail in the evaluation guidelines, which makes it difficult to operationalize the concept. Further research is needed to better define the "process" domain and its connections and boundaries with other domains.

The domain of "sustainability" was only observed in the RE-AIM framework, which is categorized as an evaluation framework [ 34 ]. In its acronym, the letter M stands for "maintenance" and corresponds to the assessment of whether the user maintains use, typically longer than 6 months. The presence of this domain highlights the need for continuous evaluation of CPGs implementation in the short, medium, and long term. Although the RE-AIM framework includes this domain, it was not used in the questionnaire developed in the study. One probable reason is that the evaluation of CPGs implementation is still conducted on a one-off basis and not as a continuous improvement process. Considering that changes in clinical practices are inherent over time, evaluating and monitoring changes throughout the duration of the CPG could be an important strategy for ensuring its implementation. This is an emerging field that requires additional investment and research.

The "Fidelity/Adaptation" domain was not observed in the models. These emerging concepts involve the extent to which a CPG is being conducted exactly as planned or whether it is undergoing adjustments and adaptations. Whether or not there is fidelity or adaptation in the implementation of CPGs does not presuppose greater or lesser effectiveness; after all, some adaptations may be necessary to implement general CPGs in specific contexts. The absence of this domain in all the models and frameworks may suggest that they are not relevant aspects for evaluating implementation or that there is a lack of knowledge of these complex concepts. This may suggest difficulty in expressing concepts in specific evaluative questions. However, further studies are warranted to determine the comprehensiveness of these concepts.

It is important to note the customization of the domains of analysis, with some domains presented in the models not being evaluated in the studies, while others were complementarily included. This can be seen in Jeong et al. [ 34 ], where the "intervention" domain in the evaluation with the RE-AIM framework reinforced the aim of theoretical approaches such as guiding the process and not determining norms. Despite this, few limitations were reported for the models, suggesting that the use of models in these studies reflects the application of these models to defined contexts without a deep critical analysis of their domains.

Limitations

This review has several limitations. First, only a few studies and methodological guidelines that explicitly present models and frameworks for assessing the implementation of CPGs have been found. This means that few alternative models could be analyzed and presented in this review. Second, this review adopted multiple analytical categories (e.g., level of use, health service, target group, and domains evaluated), whose terminology has varied enormously in the studies and documents selected, especially for the "domains evaluated" category. This difficulty in harmonizing the taxonomy used in the area has already been reported [ 26 ] and has significant potential to confuse. For this reason, studies and initiatives are needed to align understandings between concepts and, as far as possible, standardize them. Third, in some studies/documents, the information extracted was not clear about the analytical category. This required an in-depth interpretative process of the studies, which was conducted in pairs to avoid inappropriate interpretations.

Implications

This study contributes to the literature and clinical practice management by describing models and frameworks specifically used to assess the implementation of CPGs based on their level of use, type of health service, target group related to the CPG, and the evaluated domains. While there are existing reviews on the theories, frameworks, and models used in implementation science, this review addresses aspects not previously covered in the literature. This valuable information can assist stakeholders (such as politicians, clinicians, researchers, etc.) in selecting or adapting the most appropriate model to assess CPG implementation based on their health context. Furthermore, this study is expected to guide future research on developing or adapting models to assess the implementation of CPGs in various contexts.

The use of models and frameworks to evaluate the implementation remains a challenge. Studies should clearly state the level of model use, the type of health service evaluated, and the target group. The domains evaluated in these models may need adaptation to specific contexts. Nevertheless, utilizing models to assess CPGs implementation is crucial as they can guide a more thorough and systematic evaluation process, aiding in the continuous improvement of CPGs implementation. The findings of this systematic review offer valuable insights for stakeholders in selecting or adjusting models and frameworks for CPGs evaluation, supporting future theoretical advancements and research.

Availability of data and materials

Abbreviations.

Australian Department of Health and Aged Care

Canadian Agency for Drugs and Technologies in Health

Centers for Disease Control and

Consolidated Framework for Implementation Research

Cumulative Index to Nursing and Allied Health Literature

Clinical practice guideline

Centre for Reviews and Dissemination

Guidelines International Networks

Getting Research into Practice

Health Systems Evidence

Institute of Medicine

The Joanna Briggs Institute

Ministry of Health of Brazil

Ministerio de Sanidad y Política Social

National Health and Medical Research Council

National Institute for Health and Care Excellence

Promoting action on research implementation in health systems framework

Predisposing, Reinforcing and Enabling Constructs in Educational Diagnosis and Evaluation-Policy, Regulatory, and Organizational Constructs in Educational and Environmental Development

Preferred Reporting Items for Systematic Reviews and Meta-Analyses

International Prospective Register of Systematic Reviews

Reach, effectiveness, adoption, implementation, and maintenance framework

Healthcare Improvement Scotland

United States of America

Virtual Health Library

World Health Organization

Medicine I of. Crossing the Quality Chasm: A New Health System for the 21st Century. 2001. Available from: http://www.nap.edu/catalog/10027 . Cited 2022 Sep 29.

Field MJ, Lohr KN. Clinical Practice Guidelines: Directions for a New Program. Washington DC: National Academy Press. 1990. Available from: https://www.nap.edu/read/1626/chapter/8 Cited 2020 Sep 2.

Dawson A, Henriksen B, Cortvriend P. Guideline Implementation in Standardized Office Workflows and Exam Types. J Prim Care Community Heal. 2019;10. Available from: https://pubmed.ncbi.nlm.nih.gov/30900500/ . Cited 2020 Jul 15.

Unverzagt S, Oemler M, Braun K, Klement A. Strategies for guideline implementation in primary care focusing on patients with cardiovascular disease: a systematic review. Fam Pract. 2014;31(3):247–66. Available from: https://academic.oup.com/fampra/article/31/3/247/608680 . Cited 2020 Nov 5.

Article   PubMed   Google Scholar  

Nilsen P. Making sense of implementation theories, models and frameworks. Implement Sci. 2015;10(1):1–13. Available from: https://implementationscience.biomedcentral.com/articles/10.1186/s13012-015-0242-0 . Cited 2022 May 1.

Article   Google Scholar  

Mangana F, Massaquoi LD, Moudachirou R, Harrison R, Kaluangila T, Mucinya G, et al. Impact of the implementation of new guidelines on the management of patients with HIV infection at an advanced HIV clinic in Kinshasa, Democratic Republic of Congo (DRC). BMC Infect Dis. 2020;20(1):N.PAG-N.PAG. Available from: https://search.ebscohost.com/login.aspx?direct=true&db=c8h&AN=146325052&amp .

Browman GP, Levine MN, Mohide EA, Hayward RSA, Pritchard KI, Gafni A, et al. The practice guidelines development cycle: a conceptual tool for practice guidelines development and implementation. 2016;13(2):502–12. https://doi.org/10.1200/JCO.1995.13.2.502 .

Killeen SL, Donnellan N, O’Reilly SL, Hanson MA, Rosser ML, Medina VP, et al. Using FIGO Nutrition Checklist counselling in pregnancy: A review to support healthcare professionals. Int J Gynecol Obstet. 2023;160(S1):10–21. Available from: https://www.scopus.com/inward/record.uri?eid=2-s2.0-85146194829&doi=10.1002%2Fijgo.14539&partnerID=40&md5=d0f14e1f6d77d53e719986e6f434498f .

Bauer MS, Damschroder L, Hagedorn H, Smith J, Kilbourne AM. An introduction to implementation science for the non-specialist. BMC Psychol. 2015;3(1):1–12. Available from: https://bmcpsychology.biomedcentral.com/articles/10.1186/s40359-015-0089-9 . Cited 2020 Nov 5.

Cassetti V, M VLR, Pola-Garcia M, AM G, J JPC, L APDT, et al. An integrative review of the implementation of public health guidelines. Prev Med reports. 2022;29:101867. Available from: http://www.epistemonikos.org/documents/7ad499d8f0eecb964fc1e2c86b11450cbe792a39 .

Eccles MP, Mittman BS. Welcome to implementation science. Implementation Science BioMed Central. 2006. Available from: https://implementationscience.biomedcentral.com/articles/10.1186/1748-5908-1-1 .

Damschroder LJ. Clarity out of chaos: Use of theory in implementation research. Psychiatry Res. 2020;1(283):112461.

Handley MA, Gorukanti A, Cattamanchi A. Strategies for implementing implementation science: a methodological overview. Emerg Med J. 2016;33(9):660–4. Available from: https://pubmed.ncbi.nlm.nih.gov/26893401/ . Cited 2022 Mar 7.

Wang Y, Wong ELY, Nilsen P, Chung VC ho, Tian Y, Yeoh EK. A scoping review of implementation science theories, models, and frameworks — an appraisal of purpose, characteristics, usability, applicability, and testability. Implement Sci. 2023;18(1):1–15. Available from: https://implementationscience.biomedcentral.com/articles/10.1186/s13012-023-01296-x . Cited 2024 Jan 22.

Moullin JC, Dickson KS, Stadnick NA, Albers B, Nilsen P, Broder-Fingert S, et al. Ten recommendations for using implementation frameworks in research and practice. Implement Sci Commun. 2020;1(1):1–12. Available from: https://implementationsciencecomms.biomedcentral.com/articles/10.1186/s43058-020-00023-7 . Cited 2022 May 20.

Glasgow RE, Vogt TM, Boles SM. *Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health. 1999;89(9):1322. Available from: /pmc/articles/PMC1508772/?report=abstract. Cited 2022 May 22.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Asada Y, Lin S, Siegel L, Kong A. Facilitators and Barriers to Implementation and Sustainability of Nutrition and Physical Activity Interventions in Early Childcare Settings: a Systematic Review. Prev Sci. 2023;24(1):64–83. Available from: https://www.scopus.com/inward/record.uri?eid=2-s2.0-85139519721&doi=10.1007%2Fs11121-022-01436-7&partnerID=40&md5=b3c395fdd2b8235182eee518542ebf2b .

Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, et al., editors. Cochrane Handbook for Systematic Reviews of Interventions. version 6. Cochrane; 2022. Available from: https://training.cochrane.org/handbook. Cited 2022 May 23.

Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372. Available from: https://www.bmj.com/content/372/bmj.n71 . Cited 2021 Nov 18.

M C, AD O, E P, JP H, S G. Appendix A: Guide to the contents of a Cochrane Methodology protocol and review. Higgins JP, Green S, eds Cochrane Handb Syst Rev Interv. 2011;Version 5.

Kislov R, Pope C, Martin GP, Wilson PM. Harnessing the power of theorising in implementation science. Implement Sci. 2019;14(1):1–8. Available from: https://implementationscience.biomedcentral.com/articles/10.1186/s13012-019-0957-4 . Cited 2024 Jan 22.

Ouzzani M, Hammady H, Fedorowicz Z, Elmagarmid A. Rayyan-a web and mobile app for systematic reviews. Syst Rev. 2016;5(1):1–10. Available from: https://systematicreviewsjournal.biomedcentral.com/articles/10.1186/s13643-016-0384-4 . Cited 2022 May 20.

JBI. JBI’s Tools Assess Trust, Relevance & Results of Published Papers: Enhancing Evidence Synthesis. Available from: https://jbi.global/critical-appraisal-tools . Cited 2023 Jun 13.

Drisko JW. Qualitative research synthesis: An appreciative and critical introduction. Qual Soc Work. 2020;19(4):736–53.

Pope C, Mays N, Popay J. Synthesising qualitative and quantitative health evidence: A guide to methods. 2007. Available from: https://books.google.com.br/books?hl=pt-PT&lr=&id=L3fbE6oio8kC&oi=fnd&pg=PR6&dq=synthesizing+qualitative+and+quantitative+health+evidence&ots=sfELNUoZGq&sig=bQt5wt7sPKkf7hwKUvxq2Ek-p2Q#v=onepage&q=synthesizing=qualitative=and=quantitative=health=evidence& . Cited 2022 May 22.

Nilsen P, Birken SA, Edward Elgar Publishing. Handbook on implementation science. 542. Available from: https://www.e-elgar.com/shop/gbp/handbook-on-implementation-science-9781788975988.html . Cited 2023 Apr 15.

Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: A consolidated framework for advancing implementation science. Implement Sci. 2009;4(1):1–15. Available from: https://implementationscience.biomedcentral.com/articles/10.1186/1748-5908-4-50 . Cited 2023 Jun 13.

Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health. 2011;38(2):65–76. Available from: https://pubmed.ncbi.nlm.nih.gov/20957426/ . Cited 2023 Jun 11.

Bahtsevani C, Willman A, Khalaf A, Östman M, Ostman M. Developing an instrument for evaluating implementation of clinical practice guidelines: a test-retest study. J Eval Clin Pract. 2008;14(5):839–46. Available from: https://search.ebscohost.com/login.aspx?direct=true&db=c8h&AN=105569473&amp . Cited 2023 Jan 18.

Balbale SN, Hill JN, Guihan M, Hogan TP, Cameron KA, Goldstein B, et al. Evaluating implementation of methicillin-resistant Staphylococcus aureus (MRSA) prevention guidelines in spinal cord injury centers using the PARIHS framework: a mixed methods study. Implement Sci. 2015;10(1):130. Available from: https://pubmed.ncbi.nlm.nih.gov/26353798/ . Cited 2023 Apr 3.

Article   PubMed   PubMed Central   Google Scholar  

Breimaier HE, Heckemann B, Halfens RJGG, Lohrmann C. The Consolidated Framework for Implementation Research (CFIR): a useful theoretical framework for guiding and evaluating a guideline implementation process in a hospital-based nursing practice. BMC Nurs. 2015;14(1):43. Available from: https://search.ebscohost.com/login.aspx?direct=true&db=c8h&AN=109221169&amp . Cited 2023 Apr 3.

Chou AF, Vaughn TE, McCoy KD, Doebbeling BN. Implementation of evidence-based practices: Applying a goal commitment framework. Health Care Manage Rev. 2011;36(1):4–17. Available from: https://pubmed.ncbi.nlm.nih.gov/21157225/ . Cited 2023 Apr 30.

Porritt K, McArthur A, Lockwood C, Munn Z. JBI Manual for Evidence Implementation. JBI Handbook for Evidence Implementation. JBI; 2020. Available from: https://jbi-global-wiki.refined.site/space/JHEI . Cited 2023 Apr 3.

Jeong HJJ, Jo HSS, Oh MKK, Oh HWW. Applying the RE-AIM Framework to Evaluate the Dissemination and Implementation of Clinical Practice Guidelines for Sexually Transmitted Infections. J Korean Med Sci. 2015;30(7):847–52. Available from: https://pubmed.ncbi.nlm.nih.gov/26130944/ . Cited 2023 Apr 3.

GPC G de trabajo sobre implementación de. Implementación de Guías de Práctica Clínica en el Sistema Nacional de Salud. Manual Metodológico. 2009. Available from: https://portal.guiasalud.es/wp-content/uploads/2019/01/manual_implementacion.pdf . Cited 2023 Apr 3.

Australia C of. A guide to the development, implementation and evaluation of clinical practice guidelines. National Health and Medical Research Council; 1998. Available from: https://www.health.qld.gov.au/__data/assets/pdf_file/0029/143696/nhmrc_clinprgde.pdf .

Health Q. Guideline implementation checklist Translating evidence into best clinical practice. 2022.

Google Scholar  

Quittner AL, Abbott J, Hussain S, Ong T, Uluer A, Hempstead S, et al. Integration of mental health screening and treatment into cystic fibrosis clinics: Evaluation of initial implementation in 84 programs across the United States. Pediatr Pulmonol. 2020;55(11):2995–3004. Available from: https://www.embase.com/search/results?subaction=viewrecord&id=L2005630887&from=export . Cited 2023 Apr 3.

Urquhart R, Woodside H, Kendell C, Porter GA. Examining the implementation of clinical practice guidelines for the management of adult cancers: A mixed methods study. J Eval Clin Pract. 2019;25(4):656–63. Available from: https://search.ebscohost.com/login.aspx?direct=true&db=c8h&AN=137375535&amp . Cited 2023 Apr 3.

Yinghui J, Zhihui Z, Canran H, Flute Y, Yunyun W, Siyu Y, et al. Development and validation for evaluation of an evaluation tool for guideline implementation. Chinese J Evidence-Based Med. 2022;22(1):111–9. Available from: https://www.embase.com/search/results?subaction=viewrecord&id=L2016924877&from=export .

Breimaier HE, Halfens RJG, Lohrmann C. Effectiveness of multifaceted and tailored strategies to implement a fall-prevention guideline into acute care nursing practice: a before-and-after, mixed-method study using a participatory action research approach. BMC Nurs. 2015;14(1):18. Available from: https://search.ebscohost.com/login.aspx?direct=true&db=c8h&AN=103220991&amp .

Lai J, Maher L, Li C, Zhou C, Alelayan H, Fu J, et al. Translation and cross-cultural adaptation of the National Health Service Sustainability Model to the Chinese healthcare context. BMC Nurs. 2023;22(1). Available from: https://www.scopus.com/inward/record.uri?eid=2-s2.0-85153237164&doi=10.1186%2Fs12912-023-01293-x&partnerID=40&md5=0857c3163d25ce85e01363fc3a668654 .

Zhao J, Li X, Yan L, Yu Y, Hu J, Li SA, et al. The use of theories, frameworks, or models in knowledge translation studies in healthcare settings in China: a scoping review protocol. Syst Rev. 2021;10(1):13. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7792291 .

Tabak RG, Khoong EC, Chambers DA, Brownson RC. Bridging research and practice: models for dissemination and implementation research. Am J Prev Med. 2012;43(3):337–50. Available from: https://pubmed.ncbi.nlm.nih.gov/22898128/ . Cited 2023 Apr 4.

Phulkerd S, Lawrence M, Vandevijvere S, Sacks G, Worsley A, Tangcharoensathien V. A review of methods and tools to assess the implementation of government policies to create healthy food environments for preventing obesity and diet-related non-communicable diseases. Implement Sci. 2016;11(1):1–13. Available from: https://implementationscience.biomedcentral.com/articles/10.1186/s13012-016-0379-5 . Cited 2022 May 1.

Buss PM, Pellegrini FA. A Saúde e seus Determinantes Sociais. PHYSIS Rev Saúde Coletiva. 2007;17(1):77–93.

Pereira VC, Silva SN, Carvalho VKSS, Zanghelini F, Barreto JOMM. Strategies for the implementation of clinical practice guidelines in public health: an overview of systematic reviews. Heal Res Policy Syst. 2022;20(1):13. Available from: https://health-policy-systems.biomedcentral.com/articles/10.1186/s12961-022-00815-4 . Cited 2022 Feb 21.

Grimshaw J, Eccles M, Tetroe J. Implementing clinical guidelines: current evidence and future implications. J Contin Educ Health Prof. 2004;24 Suppl 1:S31-7. Available from: https://pubmed.ncbi.nlm.nih.gov/15712775/ . Cited 2021 Nov 9.

Lotfi T, Stevens A, Akl EA, Falavigna M, Kredo T, Mathew JL, et al. Getting trustworthy guidelines into the hands of decision-makers and supporting their consideration of contextual factors for implementation globally: recommendation mapping of COVID-19 guidelines. J Clin Epidemiol. 2021;135:182–6. Available from: https://pubmed.ncbi.nlm.nih.gov/33836255/ . Cited 2024 Jan 25.

Lenzer J. Why we can’t trust clinical guidelines. BMJ. 2013;346(7913). Available from: https://pubmed.ncbi.nlm.nih.gov/23771225/ . Cited 2024 Jan 25.

Molino C de GRC, Ribeiro E, Romano-Lieber NS, Stein AT, de Melo DO. Methodological quality and transparency of clinical practice guidelines for the pharmacological treatment of non-communicable diseases using the AGREE II instrument: A systematic review protocol. Syst Rev. 2017;6(1):1–6. Available from: https://systematicreviewsjournal.biomedcentral.com/articles/10.1186/s13643-017-0621-5 . Cited 2024 Jan 25.

Albers B, Mildon R, Lyon AR, Shlonsky A. Implementation frameworks in child, youth and family services – Results from a scoping review. Child Youth Serv Rev. 2017;1(81):101–16.

Download references

Acknowledgements

Not applicable

This study is supported by the Fundação de Apoio à Pesquisa do Distrito Federal (FAPDF). FAPDF Award Term (TOA) nº 44/2024—FAPDF/SUCTI/COOBE (SEI/GDF – Process 00193–00000404/2024–22). The content in this article is solely the responsibility of the authors and does not necessarily represent the official views of the FAPDF.

Author information

Authors and affiliations.

Department of Management and Incorporation of Health Technologies, Ministry of Health of Brazil, Brasília, Federal District, 70058-900, Brazil

Nicole Freitas de Mello & Dalila Fernandes Gomes

Postgraduate Program in Public Health, FS, University of Brasília (UnB), Brasília, Federal District, 70910-900, Brazil

Nicole Freitas de Mello, Dalila Fernandes Gomes & Jorge Otávio Maia Barreto

René Rachou Institute, Oswaldo Cruz Foundation, Belo Horizonte, Minas Gerais, 30190-002, Brazil

Sarah Nascimento Silva

Oswaldo Cruz Foundation - Brasília, Brasília, Federal District, 70904-130, Brazil

Juliana da Motta Girardi & Jorge Otávio Maia Barreto

You can also search for this author in PubMed   Google Scholar

Contributions

NFM and JOMB conceived the idea and the protocol for this study. NFM conducted the literature search. NFM, SNS, JMG and JOMB conducted the data collection with advice and consensus gathering from JOMB. The NFM and JMG assessed the quality of the studies. NFM and DFG conducted the data extraction. NFM performed the analysis and synthesis of the results with advice and consensus gathering from JOMB. NFM drafted the manuscript. JOMB critically revised the first version of the manuscript. All the authors revised and approved the submitted version.

Corresponding author

Correspondence to Nicole Freitas de Mello .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

13012_2024_1389_moesm1_esm.docx.

Additional file 1: PRISMA checklist. Description of data: Completed PRISMA checklist used for reporting the results of this systematic review.

Additional file 2: Literature search. Description of data: The search strategies adapted for the electronic databases.

13012_2024_1389_moesm3_esm.doc.

Additional file 3: JBI’s critical appraisal tools for cross-sectional studies. Description of data: JBI’s critical appraisal tools to assess the trustworthiness, relevance, and results of the included studies. This is specific for cross-sectional studies.

13012_2024_1389_MOESM4_ESM.doc

Additional file 4: JBI’s critical appraisal tools for qualitative studies. Description of data: JBI’s critical appraisal tools to assess the trustworthiness, relevance, and results of the included studies. This is specific for qualitative studies.

13012_2024_1389_MOESM5_ESM.doc

Additional file 5: Methodological quality assessment results for cross-sectional studies. Description of data: Methodological quality assessment results for cross-sectional studies using JBI’s critical appraisal tools.

13012_2024_1389_MOESM6_ESM.doc

Additional file 6: Methodological quality assessment results for the qualitative studies. Description of data: Methodological quality assessment results for qualitative studies using JBI’s critical appraisal tools.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/ .

Reprints and permissions

About this article

Cite this article.

Freitas de Mello, N., Nascimento Silva, S., Gomes, D.F. et al. Models and frameworks for assessing the implementation of clinical practice guidelines: a systematic review. Implementation Sci 19 , 59 (2024). https://doi.org/10.1186/s13012-024-01389-1

Download citation

Received : 06 February 2024

Accepted : 01 August 2024

Published : 07 August 2024

DOI : https://doi.org/10.1186/s13012-024-01389-1

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Implementation
• Practice guideline
• Evidence-Based Practice
• Implementation science

Implementation Science

ISSN: 1748-5908

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

This paper is in the following e-collection/theme issue:

Published on 8.8.2024 in Vol 26 (2024)

Identifying Reddit Users at a High Risk of Suicide and Their Linguistic Features During the COVID-19 Pandemic: Growth-Based Trajectory Model

Authors of this article:

Original Paper

• Yifei Yan 1 , BA   ; 
• Jun Li 2 , MA   ; 
• Xingyun Liu 3 , PhD   ; 
• Qing Li 2 , PhD   ; 
• Nancy Xiaonan Yu 1 , PhD  

1 Department of Social and Behavioural Sciences, City University of Hong Kong, Hong Kong, China (Hong Kong)

2 Department of Computing, The Hong Kong Polytechnic University, Hong Kong, China (Hong Kong)

3 Key Laboratory of Adolescent Cyberpsychology and Behavior, Central China Normal University, Ministry of Education, School of Psychology, Wuhan, China

Corresponding Author:

Nancy Xiaonan Yu, PhD

Department of Social and Behavioural Sciences

City University of Hong Kong

Tat Chee Avenue, Kowloon, HKSAR, P. R. China

Hong Kong, 000

China (Hong Kong)

Phone: 852 34429436

Fax:852 34420283

Email: [email protected]

Background: Suicide has emerged as a critical public health concern during the COVID-19 pandemic. With social distancing measures in place, social media has become a significant platform for individuals expressing suicidal thoughts and behaviors. However, existing studies on suicide using social media data often overlook the diversity among users and the temporal dynamics of suicide risk.

Objective: By examining the variations in post volume trajectories among users on the r/SuicideWatch subreddit during the COVID-19 pandemic, this study aims to investigate the heterogeneous patterns of change in suicide risk to help identify social media users at high risk of suicide. We also characterized their linguistic features before and during the pandemic.

Methods: We collected and analyzed post data every 6 months from March 2019 to August 2022 for users on the r/SuicideWatch subreddit (N=6163). A growth-based trajectory model was then used to investigate the trajectories of post volume to identify patterns of change in suicide risk during the pandemic. Trends in linguistic features within posts were also charted and compared, and linguistic markers were identified across the trajectory groups using regression analysis.

Results: We identified 2 distinct trajectories of post volume among r/SuicideWatch subreddit users. A small proportion of users (744/6163, 12.07%) was labeled as having a high risk of suicide, showing a sharp and lasting increase in post volume during the pandemic. By contrast, most users (5419/6163, 87.93%) were categorized as being at low risk of suicide, with a consistently low and mild increase in post volume during the pandemic. In terms of the frequency of most linguistic features, both groups showed increases at the initial stage of the pandemic. Subsequently, the rising trend continued in the high-risk group before declining, while the low-risk group showed an immediate decrease. One year after the pandemic outbreak, the 2 groups exhibited differences in their use of words related to the categories of personal pronouns; affective, social, cognitive, and biological processes; drives; relativity; time orientations; and personal concerns. In particular, the high-risk group was discriminant in using words related to anger (odds ratio [OR] 3.23, P <.001), sadness (OR 3.23, P <.001), health (OR 2.56, P =.005), achievement (OR 1.67, P =.049), motion (OR 4.17, P <.001), future focus (OR 2.86, P <.001), and death (OR 4.35, P <.001) during this stage.

Conclusions: Based on the 2 identified trajectories of post volume during the pandemic, this study divided users on the r/SuicideWatch subreddit into suicide high- and low-risk groups. Our findings indicated heterogeneous patterns of change in suicide risk in response to the pandemic. The high-risk group also demonstrated distinct linguistic features. We recommend conducting real-time surveillance of suicide risk using social media data during future public health crises to provide timely support to individuals at potentially high risk of suicide.

Introduction

The COVID-19 pandemic has triggered a global increase in mental disorders [ 1 ], with heightened concerns about suicide risk both in the short and in the long term [ 2 , 3 ]. While an interrupted time-series study indicated that most countries or regions have not experienced a significant rise in suicide deaths [ 4 ], the prevalence of suicidal ideation (10.81%-12.10%) and suicide attempts (4.86%) has notably increased compared with prepandemic levels [ 5 , 6 ]. Our recent meta-analysis, utilizing longitudinal data, has further corroborated an increase in the prevalence of suicidal ideation and suicide attempts among both nonclinical and clinical populations following the onset of the pandemic [ 7 ]. Suicide remains a critical public health concern throughout the pandemic, underscoring the need for ongoing monitoring and vigilance as the situation continues to evolve [ 8 , 9 ]. In this study, we utilized posts from a social media forum to examine patterns of change in suicide risk during the COVID-19 pandemic, analyzing their corresponding linguistic features.

Social media and online discussion forums are increasingly recognized as valuable resources for suicide research [ 10 , 11 ]. These virtual platforms offer users open and widely accessible spaces to share experiences, engage in discussions, seek social support, and exchange information anonymously and in real time [ 12 , 13 ]. Compared with traditional clinical data sets, these online data sets offer several advantages: they are publicly and freely available, involve larger sample sizes, provide access to participants who are ordinarily difficult to engage, enable comparisons with historical data, and exhibit high ecological validity through documentation of first-person experiences [ 14 ]. Utilizing social media data from platforms such as Twitter/X (X Corp.), Reddit (Reddit, Inc.), and Weibo (Weibo Corporation), previous studies have detected and predicted suicide risk and developed intervention programs [ 15 - 21 ]. As a result of lockdown measures and heightened concerns during the pandemic, social media data have become even more valuable for suicide research, as these online platforms have become the primary means for many people to receive information and stay connected with the outside world [ 22 , 23 ].

The r/SuicideWatch subreddit is a semianonymous forum that provides “peer support for anyone struggling with suicidal thoughts or worried about someone who may be at risk” [ 24 ]. This makes it an important resource for suicide research, as it offers high-quality, self-reported suicide data. Dominant suicide risk assessments for online posts, such as machine learning models combined with manual coding, typically approach it as a multiclassification task. These models output the post-level suicide risk (ie, the risk associated with a single post) using ordinal data. For example, they might label a post as being at ideation, behavior, or attempt level based on the probability score for each level [ 25 - 27 ]. However, such data cannot capture user-level suicide risk (ie, the risk of a user based on posts during a specific period) and may not conform to the distributional assumptions of many statistical analyses, such as growth-based trajectory models, potentially introducing biases [ 28 , 29 ]. To assess users’ overall risk and track temporal changes in risk, researchers can consider both the quantity and quality of posts published, including methods such as topic modeling or linguistic analysis of content [ 30 , 31 ]. Specifically, the quantitative method focuses on the total number of posts (ie, post volume) within a specific period, which indicates users’ posting activity and social engagement level [ 32 , 33 ]. Changes in users’ post volume can reflect changes in their suicide risk. For instance, the diurnal and weekly patterns of post volume on r/SuicideWatch corresponded to temporal fluctuations in suicide risk, including actual suicide attempts or deaths [ 34 ]. Similarly, Twitter users exhibited an increased volume of suicide-related posts before their suicide attempts [ 35 ]. Furthermore, analyses of both the quantity and quality of posts have demonstrated that higher post volume (quantity) corresponds to active disclosure of suicidal thoughts in post content (quality) [ 30 , 31 ]. Therefore, post volume in online suicide communities can serve as an effective indicator of user-level suicide risk, offering sufficient accessibility and flexibility for statistical analysis.

In this study, we monitored users’ post volume on the r/SuicideWatch subreddit before and during the pandemic to observe changes in their suicide risk. Among active adolescent users during the pandemic (April 2020-September 2021), post volume remained stable compared with prepandemic periods [ 36 ]. However, when examining post volume over shorter intervals, it fluctuated and demonstrated an overall decrease until December 2020 [ 37 , 38 ]. There is currently no documented study on how post volume has evolved beyond 2020. It is important to note that users of r/SuicideWatch may vary in their levels of suicide risk, and the findings mentioned above could be ambiguous without accounting for this heterogeneity among users. Through expert annotation, active r/SuicideWatch users (ie, those with at least 10 total posts) were categorized into 4 risk levels: no risk (36/245, 15%), low risk (50/245, 20%), moderate risk (115/245, 47%), and severe risk (44/245, 18%) of suicide [ 39 ]. However, each user’s risk label was determined based on the highest risk observed in their posts during a 7-year period before the pandemic. As suggested by the fluid vulnerability theory [ 40 ], individual suicide risk is best understood as a temporal process influenced by both baseline and acute risk factors. Environmental stressors or contexts such as sudden outbreaks of infectious disease epidemics or pandemics, social isolation, and fear [ 3 , 7 ] can easily trigger acute suicide risk in individuals predisposed to underlying vulnerabilities (ie, those with a higher baseline risk). Cognitive, emotional, behavioral, and physiological factors interact to either sustain or alleviate their suicide risk [ 41 ]. Considering both stability and dynamism, some individuals exhibit fluctuations in suicide risk, moving between low and high states where suicidal behavior may be more or less likely to emerge, while others display a stable pattern. To address the heterogeneity among users and understand the temporal nature of suicide risk, using trajectory modeling techniques such as group-based trajectory modeling (GBTM) and growth curve modeling can be beneficial. These methods identify subgroups within a population that share similarities in outcomes over time [ 42 ].

The linguistic or language styles used in posts or comments can offer valuable insights into the experiences and perspectives of suicidal individuals. This can assist researchers in understanding the underlying thoughts, emotions, and behaviors that individuals may be unwilling or unable to express explicitly [ 43 ]. Importantly, linguistic features can act as markers that distinguish suicide-related posts from general posts, aiding in the identification of potential high-risk users who may need support [ 17 , 44 , 45 ]. Suicide-related social media posts often exhibit characteristics such as simplicity in words and short sentences, reduced lexical diversity, and language disorganization [ 46 ]. They may also include more statements related to self-destruction, commands, and conflicts [ 47 ], along with increased use of first-person pronouns, adverbs, and multifunctional words. These posts frequently reference death, anger, and the present moment, while showing fewer occurrences of second- and third-person pronouns, nouns, and references to causes and differentiation [ 17 , 48 , 49 ]. However, it remains unclear whether high-risk users have exhibited these linguistic features during the COVID-19 pandemic, which has introduced different stressors potentially influencing suicide risk. Existing studies during the pandemic that utilized psycholinguistic analysis of r/SuicideWatch posts have primarily concentrated on monitoring temporal shifts in these linguistic features. Specifically, studies have identified increased use of words associated with negative emotions and a focus on the past, along with fewer references to positive emotions, social interactions, and leisure activities. References to death and first-person pronouns remained stable [ 36 , 37 ]. However, these findings only encompassed a limited time frame during the pandemic (until September 2021) and did not account for the heterogeneity among users in terms of suicide risk. Different users may exhibit varying linguistic characteristics and trends in linguistic changes over time. Therefore, it is crucial to first identify groups of users exhibiting similar patterns of suicidal behavior throughout the pandemic. Subsequently, analyzing their respective linguistic trends and markers can yield valuable insights for suicide surveillance and targeted interventions among high-risk users.

Utilizing posts from the r/SuicideWatch subreddit before and during the COVID-19 pandemic (March 2019-September 2022), this study aimed to investigate the following: (1) the potential for distinct patterns of change in suicide risk using GBTM of users’ post volumes, (2) the trends and characteristics of linguistic features within posts across each trajectory group, and (3) the linguistic markers associated with each trajectory group. We anticipated that identifying trajectories of post volume on the r/SuicideWatch subreddit would uncover users’ diversity by accounting for the temporal dynamics of suicide risk. Analyzing their associated linguistic features could also help identify users potentially at high risk of suicide. These findings could have significant implications for enhancing suicide screening, monitoring, and interventions during future public health crises.

Data Set and Participants

For this study, we gathered the longitudinal data set using the Reddit application programming interface [ 50 ]. Following a previously established method [ 14 ], we crawled posts from users who contributed to the r/SuicideWatch subreddit between March 1, 2020, and August 31, 2022, resulting in a total of 603,802 posts from 6943 users. We expanded our data set by retrieving historical posts dating back to March 1, 2019, to analyze the trajectory of post volume before and during the pandemic. To streamline data usage in subsequent analyses, we excluded accounts that were canceled and posts with deleted content. The final data set comprised 6163 users and their posts from the r/SuicideWatch subreddit (N=33,714) spanning the period from March 1, 2019, to August 31, 2022, encompassing the COVID-19 pandemic period. As a result of the onset of the COVID-19 pandemic around March 2020, there was a notable increase in discussions related to the pandemic on Reddit [ 37 ]. Therefore, we used March 1, 2020, as a cutoff point and defined 2 prepandemic periods (T1: March 1, 2019-August 31, 2019 and T2: September 1, 2019-February 29, 2020) and 5 peripandemic periods (T3: March 1, 2020-August 31, 2020; T4: September 1, 2020-February 28, 2021; T5: March 1, 2021-August 31, 2021; T6: September 1, 2021-February 28, 2022; and T7: March 1, 2022-August 31, 2022) to track post volume trajectories across these time frames.

Trajectory Variable: Post Volume

To assess changes in suicide risk, we used post volume from the r/SuicideWatch subreddit within each pre- and peripandemic period for each user as a proxy for the trajectory variable [ 30 , 31 , 34 , 35 , 51 ]. We quantified the number of posts made by each user during each specific period, with periods where no posts were made recorded as 0. For users who joined r/SuicideWatch after March 1, 2020 (ie, those who began posting suicide-related content on r/SuicideWatch only after the pandemic outbreak; 5759/6163, 93.44%) [ 48 ], their post counts on r/SuicideWatch during the 2 prepandemic periods and the peripandemic periods before their initial post were recorded as 0. As noted by De Choudhury et al [ 48 ], there are a few suicide-related posts found on subreddits outside of r/SuicideWatch. The transition of these users to r/SuicideWatch may indicate the onset and progression of their suicidal concerns following the pandemic outbreak. Therefore, their 0 post volume during the pre- and peripandemic periods can serve as a proxy for their respective suicide risk trajectories, reflecting changes in their behavioral patterns and suicide risk following the pandemic. According to our eligibility criteria, all users included in the study had posted at least once across all periods.

There might be a concern that some of our collected posts discussed the suicide risk of others rather than the user’s own risk, although previous studies have indicated that posts on r/SuicideWatch primarily focus on self-directed concerns [ 30 , 48 ]. To address this issue in our data set, we randomly selected 10% (3372/33,714) of the total posts and manually screened the content to determine the subject of the posts. Among the 3372 posts, only 18 (0.53%) were found not to be about the user’s own suicide-related issues: 6 discussed others’ suicide risk, 5 provided help to others, 3 were about irrelevant topics, and 4 were unclassified. Thus, the majority of our collected posts accurately reflected users’ own experiences and concerns related to suicide risk.

Linguistic Features

To analyze the linguistic features of posts, we used Linguistic Inquiry and Word Count (LIWC) 2015 [ 52 ], a widely used tool for language analysis. LIWC encompasses more than 80 word categories, each containing hundreds of dictionary words for the identification and analysis of word use patterns related to suicide risk. The primary psycholinguistic categories in LIWC are personal pronouns and words related to affective, social, cognitive, perceptual, and biological processes; drives; time orientations; relativity; and personal concerns. For each Reddit user included in our study, we calculated LIWC measures for these 10 major psycholinguistic categories during each period [ 53 ]. First, we tallied the occurrences of each word within a specific post alongside the post’s length (ie, the total number of words used). Second, we summed the occurrences of each word and the total length of posts for each period. Finally, for each period, we calculated the normalized frequency of word use in each LIWC category by dividing the total count of the LIWC category by the total length of posts in that period. Therefore, each LIWC measure represents the normalized frequency of word use within a specific LIWC category during each period analyzed.

Statistical Analysis

GBTM was used to define trajectory groups based on post volume across the COVID-19 pandemic. As a finite mixture model, GBTM is capable of identifying distinct groups of individuals with similar developmental trajectories in a particular outcome or behavior within a population. It accommodates trajectory variables that adhere to distributions such as censored normal (CNORM), zero-inflated Poisson (ZIP), beta, and Bernoulli distributions [ 54 ]. Within each period, the exploratory analysis revealed that a large number of eligible users had 0 posts, resulting in a skewed and zero-inflated distribution of the trajectory variable (ie, post volume within each period). Among the available models, the ZIP model was selected to fit our data because of its capability to address excessive zeros. The ZIP model combines an inflation model for zeros with a count model for nonzero values, making it suitable for our data set [ 54 , 55 ].

To identify the best-fitting model with the optimal number of trajectory groups, we followed 3 steps [ 56 ]. Initially, we incrementally increased the number of group specifications from 2 to 5 to pinpoint the optimal number of trajectories. Specifically, we selected the model based on 4 commonly used fit statistics in GBTM analysis [ 56 - 58 ]: the Bayesian information criterion (BIC), the Akaike information criterion (AIC), entropy, and group composition. AIC relies on information theory to assess the relative information value of the model by considering the maximum likelihood estimate and the number of parameters within the model [ 59 ]. Similar to AIC, BIC originates from the Bayesian framework and can be interpreted as the posterior probability of a model based on the observed data [ 60 ]. Both statistics aim to identify the most informative model by balancing between goodness-of-fit and model complexity. However, BIC imposes a stronger penalty for model complexity compared with AIC, taking into account the sample size [ 61 ]. The goodness-of-fit and penalty terms are summed to compute AIC and BIC values, where smaller values indicate better-fitting models [ 62 - 64 ]. Additionally, entropy assesses the classification accuracy of the model by summarizing the likelihood of each participant being correctly classified [ 57 ]. With values ranging from 0 to 1, higher entropy values indicate more precise classification, typically considered satisfactory when exceeding 0.8 [ 65 ]. We also analyzed the group composition (ie, the percentage of the population represented in each subgroup), ensuring that each subgroup represented at least 5% of the total sample [ 58 ]. Second, we determined the shapes of each trajectory by specifying their functional forms (eg, linear and cubic). Starting with a cubic specification (up to 3 degrees), we iteratively dropped nonsignificant ( P >.05) polynomial terms until only significant ones remained [ 56 ]. In the count model part, linear terms were retained regardless of their statistical significance. Third, after identifying the optimal number of trajectories and their shapes, we used the average posterior probabilities (APPs) of group membership to validate the selected model. The APP measures the average probability of each participant belonging to their assigned group and should ideally be at least 0.7 for each group to ensure robustness [ 58 ].

After identifying the best-fitting model, we assigned users to their respective trajectory groups. Subsequently, summary descriptive statistics of linguistic features during each period were computed and graphed for each user group. Specifically, we compared linguistic frequencies between groups across different periods. As a result of the skewed and excessively zero-inflated distribution of LIWC frequency, parametric tests such as t tests or ANOVA may not be appropriate, as they violate their assumptions and can reduce the robustness of nonparametric tests such as the median or Wilcoxon-Mann-Whitney test. Therefore, Poisson regression modeling was recommended for its improved interpretability of data and comparability among potential models [ 66 ]. In the Poisson regression model, group membership was included as an independent variable, while the frequency of each linguistic feature in each period served as the dependent variable. The results of the Poisson model provided rate ratios (RRs) along with SEs, indicating the relative changes in counts of the outcomes between the groups.

In the final analysis, our goal was to identify linguistic markers that could differentiate between groups of users exhibiting different trajectories. In the context of suicidal text analysis, linguistic markers, or linguistic distinguishers, are language features (eg, LIWC categories) extracted from texts that substantially distinguish users or posts with varying risk statuses [ 17 , 49 ]. To identify these features, we analyzed the linguistic profiles of high- and low-risk users and examined which words could indicate their respective suicide risk levels. As the markers were not intended to predict users’ group membership, a temporal sequence (eg, baseline linguistic data) was not necessary. We utilized data from the 7 periods and followed these steps to identify potential linguistic markers. First, we identified linguistic features from the periods that exhibited significant ( P ≤.05) between-group differences based on the results of Poisson regression, considering them as potential markers. Second, to prevent duplication, we excluded high-level LIWC categories that have hierarchical relationships with each other (eg, ppron includes I, we, you, she/he, they). Third, we used Lasso logistic regression with cross-validation to determine the optimal penalty parameter, aiming to mitigate collinearity among the remaining linguistic measures [ 67 ]. Given the lack of a theoretical basis for estimating post-selection coefficients with nonlinear Lasso models [ 67 ], we utilized the Lasso model solely for model selection purposes to filter out redundant variables. The remaining variables were then integrated into the best-fitting GBTM, and their associations with group membership were assessed using multivariate logistic regression [ 68 ]. In this analysis, we computed odds ratios (ORs), where potential linguistic features served as independent variables and group memberships as the dependent variable. Significance levels were determined at a P value ≤.05. Data extraction was performed using Python (Python Foundation), while data analysis was conducted using Stata/SE 16.1 (StataCorp LLC) along with the Traj plugin for trajectory analysis.

Ethical Considerations

The data used in this study were obtained from publicly accessible posts on the r/SuicideWatch subreddit through purely observational and nonintrusive means. The raw data did not contain personally identifiable information. To uphold user privacy and confidentiality, selected posts were deidentified before analysis. This involved removing any identifying information such as names, genders, ages, addresses, and links from the post content. We maintained annotated user data separately from the raw data and stored them on secure servers, linked only through anonymous IDs. Furthermore, all examples presented in Multimedia Appendix 1 were anonymized and paraphrased to safeguard user privacy, following the framework outlined by Bruckman [ 69 ]. As publicly available data were utilized, this study fell outside the purview of ethical review by the City University of Hong Kong Research Committee, for which an exemption was obtained.

Post Trajectories on the r/SuicideWatch Subreddit Throughout the COVID-19 Pandemic

To determine post trajectories, we evaluated the model fit statistics for 2- to 5-group solutions of the GBTM to identify the optimal number of trajectory groups ( Table 1 ). As the number of groups increased from 2, we noted that both the AIC and BIC values tended to increase, while entropy decreased. Additionally, starting from the 3-group model, some group compositions did not meet the 5% threshold. Therefore, we selected the 2-group model as the optimal choice, with AIC and BIC values of –39,605.31 and –39,659.12, respectively, and an entropy of 0.96. Further analysis indicated that the 2-group solution, using cubic and quadratic functions in the count model and 2 cubic functions in the inflation model, resulted in all polynomial terms being statistically significant ( Table 2 ). The APPs for groups 1 and 2 were 0.95 and 0.99, respectively, indicating strong alignment between users and their assigned groups within this 2-group ZIP model.

a AIC: Akaike information criterion (a lower value is better).

b BIC: Bayesian information criterion (a lower value is better).

c Entropy (a value >0.8 is better).

d Group composition (the percentage of the population represented in each subgroup should exceed 5%).

Figure 1 depicts the post volume trajectories across the COVID-19 pandemic for the 2 identified groups of users. Group 1, designated as the “high risk of suicide” group, consisted of 744 (12.07%) users. Their post volume on r/SuicideWatch showed a gradual increase during the 2 prepandemic periods, followed by a rapid acceleration after the pandemic began. This trend peaked approximately 1 year after the pandemic outbreak and subsequently declined, returning to its initial level during the second year of the pandemic. Group 2, identified as the “low risk of suicide” group, comprised the majority of users (5419/6163, 87.93%). This group exhibited a slight increase in post volume on r/SuicideWatch following the pandemic outbreak, followed by stabilization and eventual recovery. Throughout the pandemic, group 2 maintained a relatively low post volume on the subreddit.

Linguistic Feature Analysis

The summary distribution of frequency of use in LIWC for the 2 groups can be found in Multimedia Appendix 2 . Similar to the distribution of post volume, we observed a zero-inflated phenomenon for these linguistic features across periods. Therefore, descriptive statistics including the median, first quartile, and third quartile were used. By plotting the median frequency trend for each included LIWC feature throughout the pandemic for the 2 groups, trends by category were illustrated ( Multimedia Appendix 3 ). We observed the following: (1) During the year before the pandemic (T1: March 2019-September 2019 and T2: September 2019-March 2020), word frequency was generally low for both groups. (2) Throughout the pandemic (T3-T6: March 2020-March 2022), words related to cognitive processes, perceptual processes, biological processes, and personal concerns showed relatively lower frequency compared with personal pronouns, affective processes, social processes, drives, relativity, and time orientations. (3) During the first year of the pandemic (T3-T4: March 2020-March 2021), both groups exhibited sharp increases in word frequency. (4) During the second year of the pandemic (T5-T6: March 2021-March 2022), the high-risk group continued to experience a slower increase until reaching a peak and subsequent decrease, while the low-risk group’s frequency decreased. (5) Moving into the third year of the pandemic (T7: March 2022-September 2022), word frequency returned to prepandemic levels in both groups. Despite both groups showing increased use of most word types during the pandemic, the high-risk group exhibited a longer-lasting increase with a peak lagging behind that of the low-risk group. This suggests that the pandemic had a more enduring impact on high-risk users.

The results of between-group comparisons using Poisson regression (with the low-risk group as the reference) are depicted in Figure 2 . In general, the high-risk group utilized most types of words more frequently than the low-risk group both before and during the initial 6 months of the pandemic (illustrated in red for T1-T3). Later, in the second half of the pandemic, their differences narrowed and even reversed (as shown in green during T4), with both groups demonstrating increased word use. Subsequently, the high-risk group once again surpassed the low-risk group, and these differences grew larger in the subsequent periods (as indicated in deeper red from T5 to T7). This pattern corresponded with the plotted trend, where the high-risk group exhibited a prolonged increase and a delayed peak following the rise during T4, whereas the frequency of the low-risk group quickly decreased and returned to its initial level.

Statistical differences in the frequency of word use were primarily observed after the pandemic outbreak ( Figure 2 ). During T3 (March 2020-September 2020), the high-risk group showed significantly more frequent use of words related to personal pronouns (RR 2.09, SE 0.73, P =.03), affective processes (RR 2.11, SE 0.73, P =.03), relativity (RR 2.02, SE 0.55, P =.01), and present focus (RR 2.00, SE 0.54, P =.01) compared with the low-risk group. During T5 (March 2021-September 2021), posts in the high-risk group also exhibited higher frequencies of words related to personal pronouns (RR 1.51, SE 0.26, P =.02), first-person singular (RR 1.54, SE 0.31, P =.03), affective processes (RR 1.60, SE 0.27, P =.005), negative emotions (RR 1.66, SE 0.35, P =.02), relativity (RR 1.48, SE 0.20, P =.005), and present focus (RR 1.53, SE 0.20, P =.001). During T6 (September 2021-March 2022), 21 types of words across categories such as personal pronouns; affective, social, cognitive, and biological processes; drives; relativity; time orientations; and personal concerns showed higher frequencies in the high-risk group compared with the low-risk group. Meanwhile, in T7 (March 2022-September 2022), the high-risk group exhibited higher frequencies of words related to personal pronouns (RR 3.61, SE 2.08, P =.03), first-person singular (RR 3.75, SE 2.38, P =.04), affective processes (RR 3.98, SE 2.22, P =.01), negative emotions (RR 4.3, SE 3.02, P =.04), drives (RR 3.78, SE 2.35, P =.03), relativity (RR 4.01, SE 1.75, P =.001), space (RR 3.88, SE 2.66, P =.04), time (RR 4.33, SE 2.76, P =.02), and present focus (RR 4.19, SE 1.8, P =.001).

To investigate the linguistic markers that could distinguish group membership, we identified 21 word types that significantly ( P ≤.05) differed between the 2 groups during the last 3 periods (T5, T6, and T7). These word types include personal pronouns, first-person singular, affective processes, positive emotions, negative emotions, anger, sadness, social processes, cognitive processes, biological processes, health, drives, achievement, relativity, motion, space, time, past focus, present focus, future focus, and death. The word types with the most observed differences were selected as potential linguistic markers for further examination. Then, we omitted 6 word types (ie, personal pronouns, affective processes, negative emotions, biological processes, drives, and relativity) due to their hierarchical relationship with their subcategory words to avoid duplication. To better fit the multivariate logistic regression, we calculated a binary measure for each of the remaining 15 potential markers, indicating no use (0) or use (1) of the word. We calculated the average frequency of each word across T5, T6, and T7, and then dichotomized these averages. Averaged values of 0 were retained as 0, indicating no use of the word during T5, T6, and T7. For averaged values greater than 0, we recoded the value as 1, indicating that the word was used at least once during T5, T6, and T7, regardless of the actual frequency. To mitigate collinearity among the 15 words, we used lasso regression for variable selection. Ultimately, we omitted 3 word types—specifically, the first-person singular, space, and time—leaving us with 12 linguistic features: positive emotions, anger, sadness, social processes, cognitive processes, health, achievement, motion, past focus, present focus, future focus, and death.

Table 3 presents the results of the multivariate logistic regression, incorporating potential linguistic markers into the 2-group GBTM. This analysis models the odds of being in the high-risk group based on the usage of potential linguistic features, with no use of the word serving as the reference. The final model indicated that 9 linguistic features emerged as significant ( P ≤.05) markers distinguishing the 2 groups. Notably, using words related to cognitive processes and present focus during the later COVID-19 periods had lower odds of being in the high-risk group compared with not using these words (OR cognitive processes 0.06, SE 0.85, P <.001; OR present focus 0.03, SE 0.85, P <.001). This indicates that the use of these words was associated with being in the low-risk group. Contrastingly, the odds of being in the high-risk group were substantially higher when using words related to anger, sadness, health, achievement, motion, future focus, and death, compared with not using these words (OR anger 3.23, SE 0.29, P <.001; OR sadness 3.23, SE 0.25, P <.001; OR health 2.56, SE 0.33, P =.005; OR achievement 1.67, SE 0.26, P =.049; OR motion 4.17, SE 0.37, P <.001; OR future focus 2.86, SE 0.3, P <.001; OR death 4.35, SE 0.26, P <.001). The results illustrated that these 7 words, used 1 year after the pandemic outbreak, were linguistic markers for being in the high-risk group. Multimedia Appendix 1 provides examples of posts that high-risk users published 1 year after the pandemic outbreak.

Principal Findings

To the best of our knowledge, this work is the first to address heterogeneity in suicide risk among social media users by incorporating the temporal characteristics of suicide. Based on the 2 identified trajectories of post volume throughout the COVID-19 pandemic, users on the r/SuicideWatch subreddit were divided into the “high risk of suicide” group (744/6163, 12.07%), characterized by a sharp and lasting increase in post volume, and the “low risk of suicide” group (5419/6163, 87.93%), characterized by a consistently low and mild increase in post volume during the pandemic. In terms of linguistic features, the 2 groups exhibited distinct frequency trends throughout the pandemic. The high-risk group demonstrated longer-lasting increases and lagged peaks in most linguistic frequencies. Contrarily, the low-risk group displayed different trends. Notably, the use of words related to anger, sadness, health, achievement, motion, future focus, and death 1 year after the pandemic outbreak emerged as markers for membership in the high-risk group. Conversely, words associated with cognitive processes and present focus were identified as linguistic markers for the low-risk group.

Across the pre- and peripandemic periods, this study identified 2 distinct patterns of change in suicide risk among r/SuicideWatch users based on trajectory modeling of their post volume. These findings underscore the heterogeneity in suicide risk among r/SuicideWatch users from a longitudinal perspective during the pandemic. Users’ participation in subreddits, including posting frequency, commenting habits, and emotional expression, was influenced by significant pandemic events [ 38 ], particularly its progression in Western countries such as the US, the UK, Canada, Australia, and Germany, where a majority of Redditors originate [ 70 ]. Both groups of users exhibited immediate increases in post volume following the onset of the COVID-19 pandemic in March 2020. However, post volume returned to prepandemic levels in later stages, around September 2021, as many Western countries began to resume normalcy [ 71 ]. According to the fluid vulnerability theory [ 40 ], an environmental stressor can trigger a suicidal response within individuals who have predispositions to such reactions. While the half-year intervals may not fully capture users’ detailed responses to the pandemic or fluctuations in their suicidal episodes, the heightened posting activity observed in both groups following the pandemic’s onset suggests an overall increase in their suicide risk. Therefore, the ongoing pandemic and its repercussions may serve as a persistent environmental stressor for users. Importantly, the high-risk group exhibited significantly greater increases in post volume during the pandemic (T3-T5: March 2020-September 2021) compared with the low-risk group. This suggests that the onset of suicidal episodes was more pronounced among the high-risk group than the low-risk group. The finding of users’ heterogeneity in suicide risk can be explained by the interaction between one’s baseline and acute risk of suicide, as proposed by the fluid vulnerability theory [ 41 ]. Individuals in the high-risk group may have a higher baseline risk due to underlying vulnerabilities, making their suicidal tendencies more readily activated compared with those in the low-risk group, who have fewer vulnerabilities and a lower baseline risk. The higher level of predispositions among high-risk users also renders them more vulnerable to the adverse impacts of the pandemic. This vulnerability activates heightened risks in various domains including cognition (eg, hopelessness), emotion (eg, depression), behavior (eg, social withdrawal), and physiology (eg, sleep disturbances), contributing to their increased acute risk. The higher baseline and acute risks motivate high-risk users to express their heightened concerns, seek support, and exchange information online, leading to a significant increase in social media engagement [ 72 ]. By contrast, the low-risk group, which showed consistently low and mild increases in post volume, likely represents the majority less predisposed to suicide risk, indicating greater resilience to the pandemic. Therefore, they may perceive the pandemic as less threatening and experience fewer burdens related to cognitive, emotional, behavioral, and physiological factors. With fewer concerns to share, they exhibited only a mild and minimal increase in post volume. Our findings underscore the heterogeneity in patterns of suicide risk change during the pandemic within this population, highlighting the importance of considering users’ individual differences and the temporal dynamics of suicide in future studies using social media data.

Additionally, this study observed differences in the trends of linguistic features between the high- and low-risk groups. During the first year of the pandemic (T3-T4: March 2020-March 2021), both groups significantly increased their use of words related to personal pronouns, positive and negative emotions, social processes, drives, relativity, and time orientations compared with other word categories, indicating broader topics of interest during this period [ 73 ]. However, the increase in linguistic frequency continued at a slower pace in the high-risk group before reaching a peak and returning to its original volume (T5-T7: March 2021-September 2022), whereas the low-risk group experienced an early, mild peak followed by an immediate decrease. This divergent trend highlights that most statistical differences in linguistic frequency between the 2 groups became evident 1 year after the outbreak of the pandemic (T5-T7: March 2021-September 2022), indicating that the impact of the pandemic on the high-risk group was more prolonged and delayed compared with the low-risk group. This finding not only underscores the heterogeneity between the 2 groups but also highlights that high-risk users have experienced prolonged stress and heightened sensitivity during the pandemic.

To better identify users at high risk of suicide and understand their underlying concerns, we examined linguistic markers based on several features that showed between-group differences 1 year into the pandemic. Specifically, words related to anger, sadness, health, achievement, motion, future focus, and death were identified as linguistic markers for the high-risk group, which partially aligns with previous findings [ 36 , 49 ]. We delved deeper into the post content of high-risk users to grasp the context in which these linguistic markers were used. Words related to anger and sadness were used by high-risk users to express agitation and hopelessness concerning the overwhelming impact of the pandemic, emotions strongly linked with an increased risk of suicidal thoughts and behaviors [ 74 - 76 ]. When discussing health and motion, high-risk users conveyed heightened concerns about their physical well-being and limitations in movement due to pandemic-related lockdowns [ 37 ]. Additionally, they used achievement-related words to express feelings of failure in meeting their goals and fulfilling their need for social recognition. These users may place high demands on themselves, striving to accomplish difficult tasks and meet high standards, which can increase their vulnerability to depression and suicidal behaviors [ 17 , 77 ]. The widespread economic losses, unemployment, and disruptions in educational settings caused by the pandemic further impeded their ability to achieve success, leading to lowered self-esteem, depressive mood, and heightened suicidal risk [ 78 ]. Additionally, we discovered that words related to future focus served as linguistic markers for the high-risk group. While previous studies have noted that suicidal individuals often emphasize present-focused words, reflecting their hopelessness about the future and acute concerns about their current state [ 49 , 79 ], this pattern may differ during the pandemic. High-risk users articulated their apprehensions about an uncertain and uncontrollable future amid the evolving pandemic, as exemplified in the texts ( Multimedia Appendix 1 ). Additionally, the high-risk group used more words related to death. In addition to referencing suicide or hopelessness, this marker also indicated their perceived threats from virus infections, death cases, or the loss of loved ones during the pandemic [ 37 , 80 ].

Our findings have significant implications for managing suicide issues during future public health crises. By analyzing social media posts, we identified a small percentage of users at high risk of suicide who appear particularly sensitive and vulnerable to pandemic-related events or similar public health crises in the future. Although the majority are at low risk of suicide, these results underscore serious concerns, as high-risk users may be poised to progress to the next stage of suicidal ideation or take action [ 36 ]. Therefore, it is crucial to pay particular attention to this subset of users to alleviate their difficulties in such situations. Moreover, the active posting and disclosure by these high-risk users may lead to “suicidal contagion” affecting low-risk users, potentially propagating suicidal tendencies within online communities [ 81 ]. Therefore, ongoing surveillance, screening, and timely intervention during public health crises are necessary to prevent this issue. Furthermore, the distinct linguistic patterns observed in the 2 groups in this study can serve as a foundation for understanding the underlying concerns contributing to these users’ suicide risk, thereby aiding in the development of targeted interventions. The identified language markers for the high-risk group can also serve as a basis for screening high-risk individuals in future pandemic-like events.

Additionally, this study has several limitations. First, aside from users disclosing their own suicidal issues, r/SuicideWatch includes posts about others’ suicide risk, providing assistance to those in distress, and disseminating research messages [ 24 ]. Although the percentage of these posts was small in our manual screening of selected posts (18/3372, 0.53%, sampled posts), future studies are advised to mitigate this noise or incorporate users’ other online behaviors (eg, commenting frequency and post length) to more accurately assess users’ suicide risk. Moreover, a significant portion of users in our data set transitioned from other subreddits to r/SuicideWatch following the onset of the pandemic, starting with 0 post volume in periods preceding their initial posts (eg, 2 prepandemic periods). Future studies could track users’ earlier psychosocial characteristics on other subreddits to identify indicators that might foreshadow their shift toward actively discussing suicidal concerns on r/SuicideWatch. Second, we utilized seven 6-month intervals as the time frames for capturing post volume and linguistic frequency, which may have been too lengthy to capture specific fluctuations. Nan et al [ 82 ] also utilized 6-month intervals and identified a 2-trajectory model for changes in suicidal ideation throughout the pandemic using scores from multiple-item scales as the trajectory variable. However, using shorter intervals (eg, 2-6 weeks) can reveal more trajectories, as it considers minor but significant differences rather than averaging them in the analysis [ 83 , 84 ]. Given the frequent release of pandemic-related news and information (eg, daily reports), users shared real-time reactions to these updates in their posts, potentially reflecting immediate changes in their suicidal thoughts or behaviors, a nuance that might not have been fully captured in our study [ 38 ]. Future studies could benefit from shorter time intervals to capture more nuanced and continuous changes in suicide risk, potentially revealing diverse trajectories of suicidal ideation. Third, due to the anonymity of Reddit data, our access was restricted to users’ demographics (eg, country or region, age, and sex). Consequently, these factors could not be included as potential covariates for modeling trajectory groups or for comparing the demographic compositions between high- and low-risk user groups. We also acknowledge the potential confounding impact of varying pandemic waves and government control policies across different countries, which we were unable to explore due to the lack of geographical information from users. Future studies should aim to investigate these factors while maintaining the integrity of data characterized by high self-disclosure and authenticity [ 14 ]. Additionally, our analysis focused exclusively on Reddit data from a Western context [ 70 ]. Cross-cultural validation using data from other platforms, such as Weibo, will be crucial to enhance the generalizability of findings and consider cultural and national policy influences.

Conclusions

This study used social media posts to demonstrate the heterogeneous patterns of change in suicide risk during the COVID-19 pandemic. A group of Reddit users at high risk of suicide was identified, characterized by a sharp and sustained increase in post volume. These high-risk users exhibited distinct linguistic patterns, particularly in their use of words related to anger, sadness, health, achievement, motion, future focus, and death during the later stages of the pandemic. Our findings underscore the importance of recognizing users’ heterogeneity in long-term suicide risk. Real-time surveillance of suicide risk using social media data during future public health crises is essential to provide timely support to individuals potentially at high risk of suicide.

Acknowledgments

The study was sponsored by the Research Grants Council of the Hong Kong Special Administrative Region, China (Collaborative Research Fund, Project No. C1031-18G). The sponsors had no further role in study design, in the collection, analysis, and interpretation of data, in the writing of the report, and in the decision to submit the article for publication.

Data Availability

The data set used in this study is available from the corresponding author upon request.

Authors' Contributions

NXY and QL conceptualized the study. YY, JL, and XL further completed the study design. JL and XL contributed to the data collection. YY was responsible for data analysis and interpretation. YY drafted the manuscript. NXY, QL, JL, and XL reviewed and edited the draft. QL and NXY administrated the project and acquired the funding. We thank the 2 anonymous reviewers for their valuable input.

Conflicts of Interest

None declared.

Examples of linguistic markers for users in the high-risk group.

Summary distribution of LIWC frequency for r/SuicideWatch users in the high- and low-risk groups during each period (median [Q1, Q3]). LIWC: Linguistic Inquiry and Word Count.

Trends of LIWC frequency by category for r/SuicideWatch users in the high- and low-risk groups throughout the COVID-19 pandemic (based on the median). LIWC: Linguistic Inquiry and Word Count.

• Riedel-Heller S, Richter D. COVID-19 pandemic and mental health of the general public: is there a tsunami of mental disorders? Psychiatr Prax. Nov 02, 2020;47(8):452-456. [ CrossRef ] [ Medline ]
• Gunnell D, Appleby L, Arensman E, Hawton K, John A, Kapur N, et al. COVID-19 Suicide Prevention Research Collaboration. Suicide risk and prevention during the COVID-19 pandemic. Lancet Psychiatry. Jun 2020;7(6):468-471. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Sher L. The impact of the COVID-19 pandemic on suicide rates. QJM. Oct 01, 2020;113(10):707-712. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Pirkis J, Gunnell D, Shin S, Del Pozo-Banos M, Arya V, Aguilar PA, et al. Suicide numbers during the first 9-15 months of the COVID-19 pandemic compared with pre-existing trends: an interrupted time series analysis in 33 countries. EClinicalMedicine. Sep 2022;51:101573. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Dubé JP, Smith MM, Sherry SB, Hewitt PL, Stewart SH. Suicide behaviors during the COVID-19 pandemic: a meta-analysis of 54 studies. Psychiatry Res. Jul 2021;301:113998. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Pathirathna ML, Nandasena HMRKG, Atapattu AMMP, Weerasekara I. Impact of the COVID-19 pandemic on suicidal attempts and death rates: a systematic review. BMC Psychiatry. Jul 28, 2022;22(1):506. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Yan Y, Hou J, Li Q, Yu NX. Suicide before and during the COVID-19 pandemic: a systematic review with meta-analysis. Int J Environ Res Public Health. Feb 14, 2023;20(4):3346. [ FREE Full text ] [ CrossRef ] [ Medline ]
• John A, Pirkis J, Gunnell D, Appleby L, Morrissey J. Trends in suicide during the covid-19 pandemic. BMJ. Nov 12, 2020;371:m4352. [ CrossRef ] [ Medline ]
• Knipe D, John A, Padmanathan P, Eyles E, Dekel D, Higgins JPT, et al. Suicide and self-harm in low- and middle- income countries during the COVID-19 pandemic: a systematic review. PLOS Glob Public Health. 2022;2(6):e0000282. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Robinson J, Cox G, Bailey E, Hetrick S, Rodrigues M, Fisher S, et al. Social media and suicide prevention: a systematic review. Early Interv Psychiatry. Apr 2016;10(2):103-121. [ CrossRef ] [ Medline ]
• Tadesse MM, Lin H, Xu B, Yang L. Detection of suicide ideation in social media forums using deep learning. Algorithms. Dec 24, 2019;13(1):7. [ CrossRef ]
• Slemon A, McAuliffe C, Goodyear T, McGuinness L, Shaffer E, Jenkins EK. Reddit users' experiences of suicidal thoughts during the COVID-19 pandemic: a qualitative analysis of r/Covid19_support posts. Front Public Health. 2021;9:693153. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Wong K, Chan CS, Chan M, Wong C, Cheng Q, Xiong C, et al. Who seeks help online? Comparing online and offline help-seeking preferences amongst youths with suicidal ideation. J Affect Disord. Sep 01, 2021;292:21-29. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Low DM, Rumker L, Talkar T, Torous J, Cecchi G, Ghosh SS. Natural language processing reveals vulnerable mental health support groups and heightened health anxiety on Reddit during COVID-19: observational study. J Med Internet Res. Oct 12, 2020;22(10):e22635. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Bitew S, Bekoulis G, Deleu J, Sterckx L, Zaporojets K, Demeester T. Predicting suicide risk from online postings in Reddit the UGent-IDLab submission to the CLPysch 2019 shared task A. In: Proceedings of the Sixth Workshop on Computational Linguistics and Clinical Psychology. New York, NY. Association for Computational Linguistics; 2019. Presented at: The Sixth Workshop on Computational Linguistics and Clinical Psychology; June 6, 2019:158-161; Minneapolis, MN. URL: https://aclanthology.org/W19-3019 [ CrossRef ]
• Chatterjee M, Samanta P, Kumar P, Sarkar D. Suicide ideation detection using multiple feature analysis from Twitter data. New York, NY. IEEE; 2022. Presented at: 2022 IEEE Delhi Section Conference (DELCON); February 1-13, 2022:1-6; New Delhi, India. [ CrossRef ]
• Cheng Q, Li TM, Kwok C, Zhu T, Yip PS. Assessing suicide risk and emotional distress in Chinese social media: a text mining and machine learning study. J Med Internet Res. Jul 10, 2017;19(7):e243. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ruiz V, Shi L, Quan W, Ryan N, Biernesser C, Brent D. CLPsych2019 shared task: predicting suicide risk level from reddit posts on multiple forums. In: Proceedings of the Sixth Workshop on Computational Linguistics and Clinical Psychology. New York, NY. Association for Computational Linguistics; 2019. Presented at: The Sixth Workshop on Computational Linguistics and Clinical Psychology; June 6, 2019:162-166; Minneapolis, MN. URL: https://aclanthology.org/W19-3020 [ CrossRef ]
• Shan Y. A promising form of suicide intervention model: distant supervision in social media the application of artificial intelligence in suicide intervention. In: Proceedings of the 2nd International Symposium on Artificial Intelligence for Medicine Sciences. 2021. Presented at: The 2nd International Symposium on Artificial Intelligence for Medicine Sciences; October 29-31, 2021; Beijing, China. [ CrossRef ]
• Tan Z, Liu X, Liu X, Cheng Q, Zhu T. Designing microblog direct messages to engage social media users with suicide ideation: interview and survey study on Weibo. J Med Internet Res. Dec 12, 2017;19(12):e381. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Varathan K, Talib N. Suicide detection system based on Twitter. 2014. Presented at: 2014 Science and Information Conference; August 27-29, 2014; London, UK. [ CrossRef ]
• Goel A, Gupta L. Social media in the times of COVID-19. J Clin Rheumatol. Sep 2020;26(6):220-223. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Hussain W. Role of social media in COVID-19 pandemic. Int J Front Sci. 2020;4(2):59-60. [ FREE Full text ] [ CrossRef ]
• r/SuicideWatch. Reddit. URL: https://www.reddit.com/r/SuicideWatch/ [accessed 2023-03-26]
• Kusuma K, Larsen M, Quiroz JC, Gillies M, Burnett A, Qian J, et al. The performance of machine learning models in predicting suicidal ideation, attempts, and deaths: a meta-analysis and systematic review. J Psychiatr Res. Nov 2022;155:579-588. [ CrossRef ] [ Medline ]
• Sawhney R, Joshi H, Gandhi S, Shah R. Towards ordinal suicide ideation detection on social media. In: Proceedings of the 14th ACM International Conference on Web Search and Data Mining. 2021. Presented at: ACM International Conference on Web Search and Data Mining; March 8-12, 2021; New York, NY. [ CrossRef ]
• Zirikly A, Resnik P, Uzuner O, Hollingshead K. CLPsych 2019 shared task: Predicting the degree of suicide risk in Reddit posts. In: Proceedings of The Sixth Workshop on Computational Linguistics and Clinical Psychology. New York, NY. Association for Computational Linguistics; 2019. Presented at: The Sixth Workshop on Computational Linguistics and Clinical Psychology; June 6, 2019:24-33; Minneapolis, MN. URL: https://aclanthology.org/W19-3003.pdf [ CrossRef ]
• Li Q, Xiong Q, Ji S, Wen J, Gao M, Yu Y, et al. Using fine-tuned conditional probabilities for data transformation of nominal attributes. Pattern Recognition Letters. Dec 2019;128:107-114. [ CrossRef ]
• Zdravevski E, Lameski P, Kulakov A, Kalajdziski S. Transformation of nominal features into numeric in supervised multi-class problems based on the weight of evidence parameter. 2015. Presented at: 2015 Federated Conference on Computer Science and Information Systems (FedCSIS); September 13-16, 2015; Lodz, Poland. [ CrossRef ]
• Kumar M, Dredze M, Coppersmith G, De CM. Detecting changes in suicide content manifested in social media following celebrity suicides. In: HT '15: Proceedings of the 26th ACM Conference on Hypertext & Social Media. New York, NY. Association for Computing Machinery; 2015. Presented at: HT '15: 26th ACM Conference on Hypertext and Social Media; September 1-4, 2015:85-94; Guzelyurt, Northern Cyprus. [ CrossRef ]
• Monselise M, Yang C. "I'm always in so much pain and no one will understand"-Detecting patterns in suicidal ideation on Reddit. In: WWW '22: Companion Proceedings of the Web Conference 2022. New York, NY. Association for Computing Machinery; 2022. Presented at: WWW '22: The ACM Web Conference; April 25-29, 2022:686-691; Virtual Event/Lyon, France. URL: https://dl.acm.org/doi/10.1145/3487553.3524700 [ CrossRef ]
• Lee K, Agrawal A, Choudhary A. Real-time disease surveillance using Twitter data: demonstration on flu and cancer. New York, NY. Association for Computing Machinery; 2013. Presented at: KDD' 13: The 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining; August 11-14, 2013:1474-1477; Chicago, IL. [ CrossRef ]
• Valdez D, Ten Thij M, Bathina K, Rutter LA, Bollen J. Social media insights into US mental health during the COVID-19 pandemic: longitudinal analysis of Twitter data. J Med Internet Res. Dec 14, 2020;22(12):e21418. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Dutta R, Gkotsis G, Velupillai S, Bakolis I, Stewart R. Temporal and diurnal variation in social media posts to a suicide support forum. BMC Psychiatry. May 19, 2021;21(1):259. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Coppersmith G, Ngo K, Leary R, Wood A. Exploratory analysis of social media prior to a suicide attempt. In: Proceedings of the Third Workshop on Computational Linguistics and Clinical Psychology. 2016. Presented at: The Third Workshop On Computational Linguistics and Clinical Psychology; June 16, 2016:106-117; San Diego, CA. URL: https://aclanthology.org/W16-0311 [ CrossRef ]
• Feldhege J, Wolf M, Moessner M, Bauer S. Psycholinguistic changes in the communication of adolescent users in a suicidal ideation online community during the COVID-19 pandemic. Eur Child Adolesc Psychiatry. Jun 2023;32(6):975-985. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Biester L, Matton K, Rajendran J, Provost EM, Mihalcea R. Understanding the impact of COVID-19 on online mental health forums. ACM Trans Manage Inf Syst. Sep 08, 2021;12(4):1-28. [ CrossRef ]
• Zhang S, Liu M, Li Y, Chung JE. Teens' social media engagement during the COVID-19 pandemic: a time series examination of posting and emotion on Reddit. Int J Environ Res Public Health. Sep 25, 2021;18(19):10079. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Shing H, Nair S, Zirikly A, Friedenberg M, Daumé IH, Resnikxpert, Crowdsourced, et al. Expert, crowdsourced, and machine assessment of suicide risk via online postings. In: Proceedings of the Fifth Workshop on Computational Linguistics and Clinical Psychology: From Keyboard to Clinic. New York, NY. Association for Computational Linguistics; 2018. Presented at: The Fifth Workshop on Computational Linguistics and Clinical Psychology; June 5, 2018:25-36; New Orleans, LA. URL: https://aclanthology.org/W18-0603.pdf [ CrossRef ]
• Rudd MD, Mandrusiak M, Joiner TE. The case against no-suicide contracts: the commitment to treatment statement as a practice alternative. J Clin Psychol. Feb 2006;62(2):243-251. [ CrossRef ] [ Medline ]
• Bryan CJ, Butner JE, May AM, Rugo KF, Harris J, Oakey DN, et al. Nonlinear change processes and the emergence of suicidal behavior: a conceptual model based on the fluid vulnerability theory of suicide. New Ideas Psychol. Apr 2020;57:100758. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Nguena Nguefack HL, Pagé MG, Katz J, Choinière M, Vanasse A, Dorais M, et al. Trajectory modelling techniques useful to epidemiological research: a comparative narrative review of approaches. Clin Epidemiol. 2020;12:1205-1222. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Pennebaker JW, Mehl MR, Niederhoffer KG. Psychological aspects of natural language use: our words, our selves. Annu Rev Psychol. 2003;54:547-577. [ CrossRef ] [ Medline ]
• Coppersmith G, Leary R, Crutchley P, Fine A. Natural language processing of social media as screening for suicide risk. Biomed Inform Insights. 2018;10:1178222618792860. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Coppersmith G, Leary R, Whyne E, Wood T. Quantifying suicidal ideation via language usage on social media. In: Joint Statistics Meetings Proceedings, Statistical Computing Section. 2015. Presented at: Joint Statistics Meetings, Statistical Computing Section; August 8-15, 2015:8-15; Seattle, WA. URL: https://api.semanticscholar.org/CorpusID:54026822
• Adil Jaafar E, Abdul-Salam Jasim H. A corpus-based stylistic analysis of online suicide notes retrieved from Reddit. Cogent Arts & Humanities. Mar 06, 2022;9(1):2047434. [ CrossRef ]
• Jaafar EA, Jaafar EA. Studies on linguistic stylistic analysis of suicide notes and suicidal thought posts. IJRSSH. Mar 31, 2022;12(01):100-124. [ CrossRef ]
• De Choudhury M, Kiciman E, Dredze M, Coppersmith G, Kumar M. Discovering shifts to suicidal ideation from mental health content in social media. Proc SIGCHI Conf Hum Factor Comput Syst. May 2016;2016:2098-2110. [ FREE Full text ] [ CrossRef ] [ Medline ]
• O'Dea B, Larsen ME, Batterham PJ, Calear AL, Christensen H. A linguistic analysis of suicide-related Twitter posts. Crisis. Sep 2017;38(5):319-329. [ CrossRef ] [ Medline ]
• Reddit. Reddit API Access. Reddit. 2023. URL: https://www.reddit.com/wiki/api/ [accessed 2023-03-26]
• Liu X, Huang J, Yu NX, Li Q, Zhu T. Mediation effect of suicide-related social media use behaviors on the association between suicidal ideation and suicide attempt: cross-sectional questionnaire study. J Med Internet Res. Apr 28, 2020;22(4):e14940. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Pennebaker JW, Boyd RL, Jordan K, Blackburn K. The Development and Psychometric Properties of LIWC 2015. Austin, TX. University of Texas; 2015. URL: http://hdl.handle.net/2152/31333
• Chen J, Hsieh G, Mahmud J, Nichols J. Understanding individuals' personal values from social media word use. In: Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing;. New York, NY. Association for Computing Machinery; 2014. Presented at: CSCW'14: Computer Supported Cooperative Work; February 15-19, 2014:405-414; Baltimore, MD. [ CrossRef ]
• Jones BL, Nagin DS. A note on a Stata plugin for estimating group-based trajectory models. Sociological Methods & Research. Sep 30, 2013;42(4):608-613. [ CrossRef ]
• Kim SY, Cho S. Developmental trajectories of tobacco use and risk factors from adolescence to emerging young adulthood: a population-based panel study. BMC Public Health. Aug 29, 2022;22(1):1636. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Baumgartner SE, Leydesdorff L. Group‐based trajectory modeling (GBTM) of citations in scholarly literature: dynamic qualities of “transient” and “sticky knowledge claims”. Asso for Info Science & Tech. Nov 20, 2013;65(4):797-811. [ CrossRef ]
• Frankfurt S, Frazier P, Syed M, Jung KR. Using group-based trajectory and growth mixture modeling to identify classes of change trajectories. The Counseling Psychologist. Aug 05, 2016;44(5):622-660. [ CrossRef ]
• Nagin D. Group-Based Modeling of Development. Harvard, MA. Harvard University Press; 2005.
• Akaike H. A new look at the statistical model identification. IEEE Trans Automat Contr. Dec 1974;19(6):716-723. [ CrossRef ]
• Schwarz G. Estimating the dimension of a model. Ann Statist. Mar 1, 1978;6(2):461-464. [ CrossRef ]
• Vrieze SI. Model selection and psychological theory: a discussion of the differences between the Akaike information criterion (AIC) and the Bayesian information criterion (BIC). Psychol Methods. Jun 2012;17(2):228-243. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Cavanaugh JE, Neath AA. The Akaike information criterion: background, derivation, properties, application, interpretation, and refinements. WIREs Computational Stats. Mar 14, 2019;11(3):e1460. [ CrossRef ]
• Neath AA, Cavanaugh JE. The Bayesian information criterion: background, derivation, and applications. WIREs Computational Stats. Dec 14, 2011;4(2):199-203. [ CrossRef ]
• Mohammed E, Naugler C, Far B. Emerging business intelligence framework for a clinical laboratory through big data analytics. In: Tran QN, Arabnia H, Mohammed EA, editors. Emerging Trends in Computational Biology, Bioinformatics and System Biology (1st edition). New York, NY. Elsevier/Morgan Kaufmann; 2015:577-602.
• Chen B, Gong W, Lai AY, Sit SM, Ho S, Wang M, et al. Patterns of perceived harms and benefits of the COVID-19 outbreak in Hong Kong adults: a latent profile analysis. Int J Environ Res Public Health. Apr 05, 2022;19(7):4352. [ FREE Full text ] [ CrossRef ] [ Medline ]
• McElduff F, Cortina-Borja M, Chan S, Wade A. When t-tests or Wilcoxon-Mann-Whitney tests won't do. Adv Physiol Educ. Sep 2010;34(3):128-133. [ FREE Full text ] [ CrossRef ] [ Medline ]
• StataCorp. Stata LASSO reference manual release 17. StataCorp. 2021. URL: https://www.stata.com/manuals/lasso.pdf [accessed 2023-03-26]
• Liang W, Liang H, Ou L, Chen B, Chen A, Li C, et al. Development and validation of a clinical risk score to predict the occurrence of critical illness in hospitalized patients with COVID-19. JAMA Intern Med. Aug 01, 2020;180(8):1081-1089. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Bruckman A. Studying the amateur artist: a perspective on disguising data collected in human subjects research on the internet. Ethics Inf Technol. 2002;4:217-231. [ FREE Full text ] [ CrossRef ]
• Statista. Regional distribution of desktop traffic to Reddit. Statista. 2022. URL: https://www.statista.com/statistics/325144/reddit-global-active-user-distribution/ [accessed 2023-03-26]
• Centers for Disease Control and Prevention (CDC). CDC Museum COVID-19 Timeline. CDC. 2023. URL: https://www.cdc.gov/museum/timeline/covid19.html [accessed 2023-03-26]
• Oh S, Syn SY. Motivations for sharing information and social support in social media: a comparative analysis of Facebook, Twitter, Delicious, YouTube, and Flickr. Asso for Info Science & Tech. Apr 21, 2015;66(10):2045-2060. [ CrossRef ]
• Park C, Shim M, Eo S, Lee S, Seo J, Moon H, et al. Empirical analysis of parallel corpora and in-depth analysis using LIWC. Applied Sciences. May 30, 2022;12(11):5545. [ FREE Full text ] [ CrossRef ]
• Hawkins KA, Hames JL, Ribeiro JD, Silva C, Joiner TE, Cougle JR. An examination of the relationship between anger and suicide risk through the lens of the interpersonal theory of suicide. J Psychiatr Res. Mar 2014;50:59-65. [ CrossRef ] [ Medline ]
• Lightman E, McCarthy P, Dufty D, McNamara D. Using computational text analysis tools to compare the lyrics of suicidal and non-suicidal songwriters. In: Proceedings of the Annual Meeting of the Cognitive Science Society. 2007. Presented at: Annual Meeting of the Cognitive Science Society; August 1-4, 2007:1-4; Nashville, TN. URL: https://escholarship.org/uc/item/0dh4553j
• Pestian J, Nasrallah H, Matykiewicz P, Bennett A, Leenaars A. Suicide note classification using natural language processing: a content analysis. Biomed Inform Insights. Aug 04, 2010;2010(3):19-28. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Farrell DM. Suicide among gifted students. Roeper Review. Mar 1989;11(3):134-139. [ CrossRef ]
• Hull-Blanks EE, Kerr BA, Robinson Kurpius SE. Risk factors of suicidal ideations and attempts in talented, at-risk girls. Suicide Life Threat Behav. 2004;34(3):267-276. [ CrossRef ] [ Medline ]
• Ware ZR. Adolescent Suicide Through the Lens of 13 Reasons Why: A Corpus Based Study. Doctor of Philosophy Dissertation. Oregan State University. 2022. URL: https://ir.library.oregonstate.edu/concern/graduate_thesis_or_dissertations/41687r69m [accessed 2024-07-15]
• Ornell F, Schuch JB, Sordi AO, Kessler FHP. "Pandemic fear" and COVID-19: mental health burden and strategies. Braz J Psychiatry. 2020;42(3):232-235. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Mueller AS, Abrutyn S. Suicidal disclosures among friends: using social network data to understand suicide contagion. J Health Soc Behav. Mar 2015;56(1):131-148. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Nan J, Salina N, Chong ST, Jiang H. Trajectory of suicidal ideation among medical students during the COVID-19 pandemic: the role of childhood trauma. BMC Psychiatry. Feb 06, 2023;23(1):90. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Huang S, Wang D, Zhao J, Chen H, Ma Z, Pan Y, et al. Changes in suicidal ideation and related influential factors in college students during the COVID-19 lockdown in China. Psychiatry Res. Aug 2022;314:114653. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Madsen T, van Spijker B, Karstoft K, Nordentoft M, Kerkhof AJ. Trajectories of suicidal ideation in people seeking web-based help for suicidality: secondary analysis of a Dutch randomized controlled trial. J Med Internet Res. Jun 30, 2016;18(6):e178. [ FREE Full text ] [ CrossRef ] [ Medline ]

Abbreviations

Edited by A Mavragani; submitted 11.05.23; peer-reviewed by E Jaafar, S Li; comments to author 15.03.24; revised version received 05.04.24; accepted 18.04.24; published 08.08.24.

©Yifei Yan, Jun Li, Xingyun Liu, Qing Li, Nancy Xiaonan Yu. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 08.08.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research (ISSN 1438-8871), is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.

ORIGINAL RESEARCH article

Comprehensive insights of pretreatment strategies on the structures and bioactivities variation of lignin-carbohydrate complexes.

• 1 Institute of Chemical Industry of Forest Products, Chinese Academy of Forestry, Nanjing, China
• 2 School of Materials Science and Engineering, Faculty of Information and Engineering Science, Peking University, Beijing, Beijing Municipality, China
• 3 State Key Laboratory of Biobased Material and Green Papermaking, Qilu University of Technology, Jinan, Shandong Province, China

The final, formatted version of the article will be published soon.

Select one of your emails

You have multiple emails registered with Frontiers:

Notify me on publication

Please enter your email address:

If you already have an account, please login

You don't have a Frontiers account ? You can register here

The lignin-carbohydrate complex (LCC) displays great potential in vast industrial applications due to its unique structural features and bioactivities. However, the elucidation of how various pretreatment methods affect the structure and bioactivities remains unaddressed. Herein, the effects of different pretreatment strategies on the structure alterations and bioactivity variations of LCC were comprehensively investigated. The results showed that compared to physical or chemical pretreatments, biological pretreatment was the most effective approach in improving the bioactivities of LCC. The LCC from biological pretreatment (enzymatic hydrolysis, ELCC4) had more functional groups while the lower weight-average molecular weight (Mw) and polydispersity index (PDI) were well-endowed. The highest antioxidant abilities against ABTS and DPPH of ELCC4 were high up to 95% and 84%, respectively. Furthermore, ELCC4 also showed the best ultraviolet (UV)-blocking rate of 96%, which was increased by 6% and 2% compared to LCC8 (physical pretreatment) and LLCC4 (chemical pretreatment). This work prospectively boosts the understanding of pretreatment strategies on the structures and bioactivities variation of LCC and facilitates its utilization as sustainable and biologically active materials in various fields.

Keywords: Lignin-carbohydrate complex, Pretreatment methods, Structure variation, antioxidant, Anti-ultraviolet

Received: 16 Jul 2024; Accepted: 09 Aug 2024.

Copyright: © 2024 Huang, Su, Wang, Deng, Tian and Fang. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY) . The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

* Correspondence: Chen Huang, Institute of Chemical Industry of Forest Products, Chinese Academy of Forestry, Nanjing, China

Disclaimer: All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.

• open search

From the latest big breakthrough to the most influential and inspiring figures on campus to Pitt in the community, Pittwire is your official source for what’s happening now.

• Health and Wellness
• Technology and Science
• Arts and Humanities
• Community Impact
• Diversity, Equity, and Inclusion
• Innovation and Research
• Our City/Our Campus
• Pitt Magazine
• Features & Articles
• Accolades & Honors
• Ones to Watch
• Announcements and Updates
• Life at Pitt
• Arts & Sciences
• Computing & Information
• Dental Medicine
• Engineering
• General Studies
• Health & Rehabilitation
• Honors College
• Public & Intl Affairs
• Public Health
• Social Work
• COVID-19 Response
• Sustainability

The Pittsburgh Supercomputing Center earned an NIH grant to fund a third-generation Anton supercomputer

• Technology & Science
• Propel scholarship, creativity and innovation
• It's Possible at Pitt

Subscribe to Pittwire Today

A third-generation Anton supercomputer (Anton 3) will soon arrive at the Pittsburgh Supercomputing Center (PSC) thanks to a $3.15-million, five-year award from the National Institutes of Health that will fund the system's operations. The grant will make the system available without cost for noncommercial use by biomedical researchers at U.S. universities and other not-for-profit institutions.

The Anton family of supercomputers, developed by D.E. Shaw Research, was specially designed for atomic-level simulation of molecules relevant to biology — for example, DNA, proteins, and drugs. The technology gives scientists the ability to simulate interactions between biomolecules that inform disease research, basic science and drug design two orders of magnitude faster than possible with general-purpose supercomputers. Like its predecessors, the new Anton was designed from the ground up around a new custom chip to best exploit the capabilities offered by new technologies.

Philip Blood and Marcela Madrid will be the project leads at PSC, a joint center of the University of Pittsburgh and Carnegie Mellon University.

“With the latest Anton system, we will be able to provide researchers with a unique resource capable of producing results in days that would take years on any other resource,” said Blood, scientific director and PI of the Anton project at PSC. “The new system will spark innovative studies that will challenge and shift current paradigms in the simulation of biomolecular systems.”

Since the beginning of the Anton project at PSC in 2010, users nationwide have used the system to obtain long-timescale simulations resulting in more than 440 papers with 20,000 citations. Examples of the breakthrough science enabled by Anton at PSC include:

• studies of ion channel opening, a process key to the control of excitations in the central nervous system and immune cells
• the complex lipid-protein interactions in SARS-CoV-2, the virus responsible for COVID-19
• the mechanisms underlying the CRISPR genome-editing machinery
• the molecular mechanisms of amyloid self-assembly, implicated in Alzheimer’s and Parkinson’s disease

Time on the machine will be allotted based on research proposals submitted to an independent expert committee convened by the National Research Council at the National Academy of Sciences.

The system will begin operations at PSC in the spring of 2025. Faculty and staff members at U.S. academic or nonprofit research institutions, including researchers without previous experience on Anton systems, are invited to apply for an allocation. The application deadline is Monday, Oct. 14. More information can be found here .

The Pittsburgh Supercomputing Center is a joint computational research center with Carnegie Mellon University and the University of Pittsburgh. PSC provides university, government and industrial researchers with access to several of the most powerful systems for high-performance computing, communications and data storage available to scientists and engineers nationwide for unclassified research. PSC advances the state of the art in high-performance computing, communications and data analytics and offers a flexible environment for solving the largest and most challenging problems in research.

Apply for a program on developing community engaged scholarship projects

Pitt-bradford has been a college of distinction for a decade, volunteers needed: contribute your time to pitt’s united way campaign.