research article on mathematical modelling

Loading metrics

Open Access

Ten simple rules for tackling your first mathematical models: A guide for graduate students by graduate students

Roles Conceptualization, Investigation, Writing – original draft, Writing – review & editing

* E-mail: [email protected]

Affiliations Department of Biological Sciences, University of Toronto Scarborough, Toronto, Ontario, Canada, Department of Ecology and Evolution, University of Toronto, Toronto, Ontario, Canada

Affiliation Department of Ecology and Evolution, University of Toronto, Toronto, Ontario, Canada

Affiliation Department of Physical and Environmental Sciences, University of Toronto Scarborough, Toronto, Ontario, Canada

Affiliation Department of Biology, Memorial University of Newfoundland, St John’s, Newfoundland, Canada

• Korryn Bodner, 
• Chris Brimacombe, 
• Emily S. Chenery, 
• Ariel Greiner, 
• Anne M. McLeod, 
• Stephanie R. Penk, 
• Juan S. Vargas Soto

Published: January 14, 2021

• https://doi.org/10.1371/journal.pcbi.1008539
• Reader Comments

Citation: Bodner K, Brimacombe C, Chenery ES, Greiner A, McLeod AM, Penk SR, et al. (2021) Ten simple rules for tackling your first mathematical models: A guide for graduate students by graduate students. PLoS Comput Biol 17(1): e1008539. https://doi.org/10.1371/journal.pcbi.1008539

Editor: Scott Markel, Dassault Systemes BIOVIA, UNITED STATES

Copyright: © 2021 Bodner et al. This is an open access article distributed under the terms of the Creative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Funding: The authors received no specific funding for this work.

Competing interests: The authors have declared that no competing interests exist.

Introduction

Biologists spend their time studying the natural world, seeking to understand its various patterns and the processes that give rise to them. One way of furthering our understanding of natural phenomena is through laboratory or field experiments, examining the effects of changing one, or several, variables on a measured response. Alternatively, one may conduct an observational study, collecting field data and comparing a measured response along natural gradients. A third and complementary way of understanding natural phenomena is through mathematical models. In the life sciences, more scientists are incorporating these quantitative methods into their research. Given the vast utility of mathematical models, ranging from providing qualitative predictions to helping disentangle multiple causation (see Hurford [ 1 ] for a more complete list), their increased adoption is unsurprising. However, getting started with mathematical models may be quite daunting for those with traditional biological training, as in addition to understanding new terminology (e.g., “Jacobian matrix,” “Markov chain”), one may also have to adopt a different way of thinking and master a new set of skills.

Here, we present 10 simple rules for tackling your first mathematical models. While many of these rules are applicable to basic scientific research, our discussion relates explicitly to the process of model-building within ecological and epidemiological contexts using dynamical models. However, many of the suggestions outlined below generalize beyond these disciplines and are applicable to nondynamic models such as statistical models and machine-learning algorithms. As graduate students ourselves, we have created rules we wish we had internalized before beginning our model-building journey—a guide by graduate students, for graduate students—and we hope they prove insightful for anyone seeking to begin their own adventures in mathematical modelling.

• PPT PowerPoint slide
• PNG larger image
• TIFF original image

Boxes represent susceptible, infected, and recovered compartments, and directed arrows represent the flow of individuals between these compartments with the rate of flow being controlled by the contact rate, c , the probability of infection, γ , and the recovery rate, θ .

https://doi.org/10.1371/journal.pcbi.1008539.g001

Rule 1: Know your question

“All models are wrong, some are useful” is a common aphorism, generally attributed to statistician George Box, but determining which models are useful is dependent upon the question being asked. The practice of clearly defining a research question is often drilled into aspiring researchers in the context of selecting an appropriate research design, interpreting statistical results, or when outlining a research paper. Similarly, the practice of defining a clear research question is important for mathematical models as their results are only as interesting as the questions that motivate them [ 5 ]. The question defines the model’s main purpose and, in all cases, should extend past the goal of merely building a model for a system (the question can even answer whether a model is even necessary). Ultimately, the model should provide an answer to the research question that has been proposed.

When the research question is used to inform the purpose of the model, it also informs the model’s structure. Given that models can be modified in countless ways, providing a purpose to the model can highlight why certain aspects of reality were included in the structure while others were ignored [ 6 ]. For example, when deciding whether we should adopt a more realistic model (i.e., add more complexity), we can ask whether we are trying to inform general theory or whether we are trying to model a response in a specific system. For example, perhaps we are trying to predict how fast an epidemic will grow based on different age-dependent mixing patterns. In this case, we may wish to adapt our basic SIR model to have age-structured compartments if we suspect this factor is important for the disease dynamics. However, if we are exploring a different question, such as how stochasticity influences general SIR dynamics, the age-structured approach would likely be unnecessary. We suggest that one of the first steps in any modelling journey is to choose the processes most relevant to your question (i.e., your hypothesis) and the direct and indirect causal relationships among them: Are the relationships linear, nonlinear, additive, or multiplicative? This challenge can be aided with a good literature review. Depending on your model purpose, you may also need to spend extra time getting to know your system and/or the data before progressing forward. Indeed, the more background knowledge acquired when forming your research question, the more informed your decision-making when selecting the structure, parameters, and data for your model.

Rule 2: Define multiple appropriate models

Natural phenomena are complicated to study and often impossible to model in their entirety. We are often unsure about the variables or processes required to fully answer our research question(s). For example, we may not know how the possibility of reinfection influences the dynamics of a disease system. In cases such as these, our advice is to produce and sketch out a set of candidate models that consider alternative terms/variables which may be relevant for the phenomena under investigation. As in Fig 2 , we construct 2 models, one that includes the ability for recovered individuals to become infected again, and one that does not. When creating multiple models, our general objective may be to explore how different processes, inputs, or drivers affect an outcome of interest or it may be to find a model or models that best explain a given set of data for an outcome of interest. In our example, if the objective is to determine whether reinfection plays an important role in explaining the patterns of a disease, we can test our SIR candidate models using incidence data to determine which model receives the most empirical support. Here we consider our candidate models to be alternative hypotheses, where the candidate model with the least support is discarded. While our perspective of models as hypotheses is a view shared by researchers such as Hilborn and Mangel [ 7 ], and Penk and colleagues [ 8 ], please note that others such as Oreskes and colleagues [ 9 ] believe that models are not subject to proof and hence disagree with this notion. We encourage modellers who are interested in this debate to read the provided citations.

(A) A susceptible/infected/recovered model where individuals remain immune (gold) and (B) a susceptible/infected/recovered model where individuals can become susceptible again (blue). Arrows indicate the direction of movement between compartments, c is the contact rate, γ is the infection rate given contact, and θ is the recovery rate. The text below each conceptual model are the hypotheses ( H1 and H2 ) that represent the differences between these 2 SIR models.

https://doi.org/10.1371/journal.pcbi.1008539.g002

Finally, we recognize that time and resource constraints may limit the ability to build multiple models simultaneously; however, even writing down alternative models on paper can be helpful as you can always revisit them if your primary model does not perform as expected. Of course, some candidate models may not be feasible or relevant for your system, but by engaging in the activity of creating multiple models, you will likely have a broader perspective of the potential factors and processes that fundamentally shape your system.

Rule 3: Determine the skills you will need (and how to get them)

Equipping yourself with the necessary analytical tools that form the basis of all quantitative techniques is essential. As Darwin said, those that have knowledge of mathematics seem to be endowed with an extra sense [ 10 ], and having a background in calculus, linear algebra, and statistics can go a long way. Thus, make it a habit to set time for yourself to learn these mathematical skills, and do not treat all your methods like a black box. For instance, if you plan to use ODEs, consider brushing up on your calculus, e.g., using Stewart [ 11 ]. If you are working with a system of ODEs, also read up on linear algebra, e.g., using Poole [ 12 ]. Some universities also offer specialized math biology courses that combine topics from different math courses to teach the essentials of mathematical modelling. Taking these courses can help save time, and if they are not available, their syllabi can help focus your studying. Also note that while narrowing down a useful skillset in the early stages of model-building will likely spare you from some future headaches, as you progress in your project, it is inevitable that new skills will be required. Therefore, we advise you to check in at different stages of your modelling journey to assess the skills that would be most relevant for your next steps and how best to acquire them. Hopefully, these decisions can also be made with the help of your supervisor and/or a modelling mentor. Building these extra skills can at first seem daunting but think of it as an investment that will pay dividends in improving your future modelling work.

When first attempting to tackle a specific problem, find relevant research that accomplishes the same tasks and determine if you understand the processes and techniques that are used in that study. If you do, then you can implement similar techniques and methods, and perhaps introduce new methods. If not, then determine which tools you need to add to your toolbox. For instance, if the problem involves a system of ODEs (e.g., SIR models, see above), can you use existing symbolic software (e.g., Maple, Matlab, Mathematica) to determine the systems dynamics via a general solution, or is the complexity too great that you will need to create simulations to infer the dynamics? Figuring out questions like these is key to understanding what skills you will need to work with the model you develop. While there is a time and a place for involving collaborators to help facilitate methods that are beyond your current reach, we strongly advocate that you approach any potential collaborator only after you have gained some knowledge of the methods first. Understanding the methodology, or at least its foundation, is not only crucial for making a fruitful collaboration, but also important for your development as a scientist.

Rule 4: Do not reinvent the wheel

While we encourage a thorough understanding of the methods researchers employ, we simultaneously discourage unnecessary effort redoing work that has already been done. Preventing duplication can be ensured by a thorough review of the literature (but note that reproducing original model results can advance your knowledge of how a model functions and lead to new insights in the system). Often, we are working from established theory that provides an existing framework that can be applied to different systems. Adapting these frameworks can help advance your own research while also saving precious time. When digging through articles, bear in mind that most modelling frameworks are not system-specific. Do not be discouraged if you cannot immediately find a model in your field, as the perfect model for your question may have been applied in a different system or be published only as a conceptual model. These models are still useful! Also, do not be shy about reaching out to authors of models that you think may be applicable to your system. Finally, remember that you can be critical of what you find, as some models can be deceptively simple or involve assumptions that you are not comfortable making. You should not reinvent the wheel, but you can always strive to build a better one.

Rule 5: Study and apply good coding practices

The modelling process will inevitably require some degree of programming, and this can quickly become a challenge for some biologists. However, learning to program in languages commonly adopted by the scientific community (e.g., R, Python) can increase the transparency, accessibility, and reproducibility of your models. Even if you only wish to adopt preprogrammed models, you will likely still need to create code of your own that reads in data, applies functions from a collection of packages to analyze the data, and creates some visual output. Programming can be highly rewarding—you are creating something after all—but it can also be one of the most frustrating parts of your research. What follows are 3 suggestions to avoid some of the frustration.

Organization is key, both in your workflow and your written code. Take advantage of existing software and tools that facilitate keeping things organized. For example, computational notebooks like Jupyter notebooks or R-Markdown documents allow you to combine text, commands, and outputs in an easily readable and shareable format. Version control software like Git makes it simple to both keep track of changes as well as to safely explore different model variants via branches without worrying that the original model has been altered. Additionally, integrating with hosting services such as Github allows you to keep your changes safely stored in the cloud. For more details on learning to program, creating reproducible research, programming with Jupyter notebooks, and using Git and Github, see the 10 simple rules by Carey and Papin [ 13 ], Sandve and colleagues [ 14 ], Rule and colleagues [ 15 ], and Perez-Riverol and colleagues [ 16 ], respectively.

Comment your code and comment it well (see Fig 3 ). These comments can be the pseudocode you have written on paper prior to coding. Assume that when you revisit your code weeks, months, or years later, you will have forgotten most of what you did and why you did it. Good commenting can also help others read and use your code, making it a critical part of increasing scientific transparency. It is always good practice to write your comments before you write the code, explaining what the code should do. When coding a function, include a description of its inputs and outputs. We also encourage you to publish your commented model code in repositories such that they are easily accessible to others—not only to get useful feedback for yourself but to provide the modelling foundation for others to build on.

Two functionally identical codes in R [ 17 ] can look very different without comments (left) and with descriptive comments (right). Writing detailed comments will help you and others understand, adapt, and use your code.

https://doi.org/10.1371/journal.pcbi.1008539.g003

When writing long code, test portions of it separately. If you are writing code that will require a lot of processing power or memory to run, use a simple example first, both to estimate how long the project will take, and to avoid waiting 12 hours to see if it works. Additionally, when writing code, try to avoid too many packages and “tricks” as it can make your code more difficult to understand. Do not be afraid of writing 2 separate functions if it will make your code more intuitive. As with writing, your skill as a writer is not dependent on your ability to use big words, but instead about making sure your reader understands what you are trying to communicate.

Rule 6: Sweat the “right” small stuff

By “sweat the ‘right’ small stuff,” we mean considering the details and assumptions that can potentially make or break a mathematical model. A good start would be to ensure your model follows the rules of mass and energy conservation. In a closed system, mass and energy cannot be created nor destroyed, and thus, the left side of the mathematical equation must equal the right under all circumstances. For example, in Eq 2 , if the number of susceptible individuals decreases due to infection, we must include a negative term in this equation (− cγIS ) to indicate that loss and its conjugate (+ cγIS ) to the infected individuals equation, Eq 3 , to represent that gain. Similarly, units of all terms must also be balanced on both sides of the equation. For example, if we wish to add or subtract 2 values, we must ensure their units are equivalent (e.g., cannot add day −1 and year −1 ). Simple oversights in units can lead to major setbacks and create bizarre dynamics, so it is worth taking the time to ensure the units match up.

Modellers should also consider the fundamental boundary conditions of each parameter to determine if there are some values that are illogical. Logical constraints and boundaries can be developed for each parameter using prior knowledge and assumptions (e.g., Huntley [ 18 ]). For example, when considering an SIR model, there are 2 parameters that comprise the transmission rate—the contact rate, c , and the probability of infection given contact, γ . Using our intuition, we can establish some basic rules: (1) the contact rate cannot be negative; (2) the number of susceptible, infected, and recovered individuals cannot be below 0; and (3) the probability of infection given contact must fall between 0 and 1. Keeping these in mind as you test your model’s dynamics can alert you to problems in your model’s structure. Finally, simulating your model is an excellent method to obtain more reasonable bounds for inputs and parameters and ensure behavior is as expected. See Otto and Day [ 5 ] for more information on the “basic ingredients” of model-building.

Rule 7: Simulate, simulate, simulate

Even though there is a lot to be learned from analyzing simple models and their general solutions, modelling a complex world sometimes requires complex equations. Unfortunately, the cost of this complexity is often the loss of general solutions [ 19 ]. Instead, many biologists must calculate a numerical solution, an approximate solution, and simulate the dynamics of these models [ 20 ]. Simulations allow us to explore model behavior, given different structures, initial conditions, and parameters ( Fig 4 ). Importantly, they allow us to understand the dynamics of complex systems that may otherwise not be ethical, feasible, or economically viable to explore in natural systems [ 21 ].

Gold lines represent the SIR structure ( Fig 2A ) where lifelong immunity of individuals is inferred after infection, and blue lines represent an SIRS structure ( Fig 2B ) where immunity is lost over time. The solid lines represent model dynamics assuming a recovery rate ( θ ) of 0.05, while dotted lines represent dynamics assuming a recovery rate of 0.1. All model runs assume a transmission rate, cγ , of 0.2 and an immunity loss rate, ψ , of 0.01. By using simulations, we can explore how different processes and rates change the system’s dynamics and furthermore determine at what point in time these differences are detectable. SIR, Susceptible-Infected-Recovered; SIRS, Susceptible-Infected-Recovered-Susceptible.

https://doi.org/10.1371/journal.pcbi.1008539.g004

One common method of exploring the dynamics of complex systems is through sensitivity analysis (SA). We can use this simulation-based technique to ascertain how changes in parameters and initial conditions will influence the behavior of a system. For example, if simulated model outputs remain relatively similar despite large changes in a parameter value, we can expect the natural system represented by that model to be robust to similar perturbations. If instead, simulations are very sensitive to parameter values, we can expect the natural system to be sensitive to its variation. Here in Fig 4 , we can see that both SIR models are very sensitive to the recovery rate parameter ( θ ) suggesting that the natural system would also be sensitive to individuals’ recovery rates. We can therefore use SA to help inform which parameters are most important and to determine which are distinguishable (i.e., identifiable). Additionally, if observed system data are available, we can use SA to help us establish what are the reasonable boundaries for our initial conditions and parameters. When adopting SA, we can either vary parameters or initial conditions one at a time (local sensitivity) or preferably, vary multiple of them in tandem (global sensitivity). We recognize this topic may be overwhelming to those new to modelling so we recommend reading Marino and colleagues [ 22 ] and Saltelli and colleagues [ 23 ] for details on implementing different SA methods.

Simulations are also a useful tool for testing how accurately different model fitting approaches (e.g., Maximum Likelihood Estimation versus Bayesian Estimation) can recover parameters. Given that we know the parameter values for simulated model outputs (i.e., simulated data), we can properly evaluate the fitting procedures of methods when used on that simulated data. If your fitting approach cannot even recover simulated data with known parameters, it is highly unlikely your procedure will be successful given real, noisy data. If a procedure performs well under these conditions, try refitting your model to simulated data that more closely resembles your own dataset (i.e., imperfect data). If you know that there was limited sampling and/or imprecise tools used to collect your data, consider adding noise, reducing sample sizes, and adding temporal and spatial gaps to see if the fitting procedure continues to return reasonably correct estimates. Remember, even if your fitting procedures continue to perform well given these additional complexities, issues may still arise when fitting to empirical data. Models are approximations and consequently their simulations are imperfect representations of your measured outcome of interest. However, by evaluating procedures on perfectly known imperfect data, we are one step closer to having a fitting procedure that works for us even when it seems like our data are against us.

Rule 8: Expect model fitting to be a lengthy, arduous but creative task

Model fitting requires an understanding of both the assumptions and limitations of your model, as well as the specifics of the data to be used in the fitting. The latter can be challenging, particularly if you did not collect the data yourself, as there may be additional uncertainties regarding the sampling procedure, or the variables being measured. For example, the incidence data commonly adopted to fit SIR models often contain biases related to underreporting, selective reporting, and reporting delays [ 24 ]. Taking the time to understand the nuances of the data is critical to prevent mismatches between the model and the data. In a bad case, a mismatch may lead to a poor-fitting model. In the worst case, a model may appear well-fit, but will lead to incorrect inferences and predictions.

Model fitting, like all aspects of modelling, is easier with the appropriate set of skills (see Rule 2). In particular, being proficient at constructing and analyzing mathematical models does not mean you are prepared to fit them. Fitting models typically requires additional in-depth statistical knowledge related to the characteristics of probability distributions, deriving statistical moments, and selecting appropriate optimization procedures. Luckily, a substantial portion of this knowledge can be gleaned from textbooks and methods-based research articles. These resources can range from covering basic model fitting, such as determining an appropriate distribution for your data and constructing a likelihood for that distribution (e.g., Hilborn and Mangel [ 7 ]), to more advanced topics, such as accounting for uncertainties in parameters, inputs, and structures during model fitting (e.g., Dietze [ 25 ]). We find these sources among others (e.g., Hobbs and Hooten [ 26 ] for Bayesian methods; e.g., Adams and colleagues [ 27 ] for fitting noisy and sparse datasets; e.g., Sirén and colleagues [ 28 ] for fitting individual-based models; and Williams and Kendall [ 29 ] for multiobject optimization—to name a few) are not only useful when starting to fit your first models, but are also useful when switching from one technique or model to another.

After you have learned about your data and brushed up on your statistical knowledge, you may still run into issues when model fitting. If you are like us, you will have incomplete data, small sample sizes, and strange data idiosyncrasies that do not seem to be replicated anywhere else. At this point, we suggest you be explorative in the resources you use and accept that you may have to combine multiple techniques and/or data sources before it is feasible to achieve an adequate model fit (see Rosenbaum and colleagues [ 30 ] for parameter estimation with multiple datasets). Evaluating the strength of different techniques can be aided by using simulated data to test these techniques, while SA can be used to identify insensitive parameters which can often be ignored in the fitting process (see Rule 7).

Model accuracy is an important metric but “good” models are also precise (i.e., reliable). During model fitting, to make models more reliable, the uncertainties in their inputs, drivers, parameters, and structures, arising due to natural variability (i.e., aleatory uncertainty) or imperfect knowledge (i.e., epistemic uncertainty), should be identified, accounted for, and reduced where feasible [ 31 ]. Accounting for uncertainty may entail measurements of uncertainties being propagated through a model (a simple example being a confidence interval), while reducing uncertainty may require building new models or acquiring additional data that minimize the prioritized uncertainties (see Dietze [ 25 ] and Tsigkinopoulou and colleagues [ 32 ] for a more thorough review on the topic). Just remember that although the steps outlined in this rule may take a while to complete, when you do achieve a well-fitted reliable model, it is truly something to be celebrated.

Rule 9: Give yourself time (and then add more)

Experienced modellers know that it often takes considerable time to build a model and that even more time may be required when fitting to real data. However, the pervasive caricature of modelling as being “a few lines of code here and there” or “a couple of equations” can lead graduate students to hold unrealistic expectations of how long finishing a model may take (or when to consider a model “finished”). Given the multiple considerations that go into selecting and implementing models (see previous rules), it should be unsurprising that the modelling process may take weeks, months, or even years. Remembering that a published model is the final product of long and hard work may help reduce some of your time-based anxieties. In reality, the finished product is just the tip of the iceberg and often unseen is the set of failed or alternative models providing its foundation. Note that taking time early on to establish what is “good enough” given your objective, and to instill good modelling practices, such as developing multiple models, simulating your models, and creating well-documented code, can save you considerable time and stress.

Rule 10: Care about the process, not just the endpoint

As a graduate student, hours of labor coupled with relative inexperience may lead to an unwillingness to change to a new model later down the line. But being married to one model can restrict its efficacy, or worse, lead to incorrect conclusions. Early planning may mitigate some modelling problems, but many issues will only become apparent as time goes on. For example, perhaps model parameters cannot be estimated as you previously thought, or assumptions made during model formulation have since proven false. Modelling is a dynamic process, and some steps will need to be revisited many times as you correct, refine, and improve your model. It is also important to bear in mind that the process of model-building is worth the effort. The process of translating biological dynamics into mathematical equations typically forces us to question our assumptions, while a misspecified model often leads to novel insights. While we may wish we had the option to skip to a final finished product, in the words of Drake, “sometimes it’s the journey that teaches you a lot about your destination”.

There is no such thing as a failed model. With every new error message or wonky output, we learn something useful about modelling (mostly begrudgingly) and, if we are lucky, perhaps also about the study system. It is easy to cave in to the ever-present pressure to perform, but as graduate students, we are still learning. Luckily, you are likely surrounded by other graduate students, often facing similar challenges who can be an invaluable resource for learning and support. Finally, remember that it does not matter if this was your first or your 100th mathematical model, challenges will always present themselves. However, with practice and determination, you will become more skilled at overcoming them, allowing you to grow and take on even greater challenges.

Acknowledgments

We thank Marie-Josée Fortin, Martin Krkošek, Péter K. Molnár, Shawn Leroux, Carina Rauen Firkowski, Cole Brookson, Gracie F.Z. Wild, Cedric B. Hunter, and Philip E. Bourne for their helpful input on the manuscript.

• 1. Hurford A. Overview of mathematical modelling in biology II. 2012 [cite 2020 October 25]. Available: https://theartofmodelling.wordpress.com/2012/01/04/overview-of-mathematical-modelling-in-biology-ii/
• View Article
• PubMed/NCBI
• Google Scholar
• 3. Maki Y, Hirose H, ADSIR M. Infectious Disease Spread Analysis Using Stochastic Differential Equations for SIR Model. International Conference on Intelligent Systems, Modelling and Simulation. IEEE. 2013.
• 5. Otto SP, Day T. A biologist’s guide to mathematical modeling in ecology and evolution. Princeton, NJ: Princeton University Press; 2007.
• 7. Hilborn R, Mangel M. The ecological detective: Confronting models with data. Princeton, NJ: Princeton University Press; 1997.
• 10. Darwin C. The autobiography of Charles Darwin. Darwin F, editor. 2008. Available: https://www.gutenberg.org/files/2010/2010-h/2010-h.htm
• 11. Stewart J. Calculus: Early transcendentals. Eighth. Boston, MA: Cengage Learning; 2015.
• 12. Poole D. Linear algebra: A modern introduction. Fourth. Stamford, CT: Cengage Learning; 2014.
• 17. R Core Team. R: A language and environment for statistical computing (version 3.6.0, R foundation for statistical computing). 2020.
• 18. Huntley HE. Dimensional analysis. First. New York, NY: Dover Publications; 1967.
• 19. Corless RM, Fillion N. A graduate introduction to numerical methods. New York, NY: Springer; 2016.
• 23. Saltelli A, Ratto M, Andres T, Campolongo F, Cariboni J, Gatelli D, et al. Global sensitivity analysis: The primer. Chichester: Wiley; 2008.
• 25. Dietze MC. Ecological forecasting. Princeton, NJ: Princeton University Press; 2017.
• 26. Hobbs NT, Hooten MB. Bayesian models: A statistical primer for ecologists. Princeton, NJ: Princeton University Press

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Open access
• Published: 27 August 2019

A short comment on statistical versus mathematical modelling

• Andrea Saltelli   ORCID: orcid.org/0000-0003-4222-6975 1 , 2  

Nature Communications volume  10 , Article number:  3870 ( 2019 ) Cite this article

42k Accesses

59 Citations

354 Altmetric

Metrics details

• Computational science
• Environmental sciences
• Mathematics and computing

While the crisis of statistics has made it to the headlines, that of mathematical modelling hasn’t. Something can be learned comparing the two, and looking at other instances of production of numbers.Sociology of quantification and post-normal science can help.

While statistical and mathematical modelling share important features, they don’t seem to share the same sense of crisis. Statisticians appear mired in an academic and mediatic debate where even the concept of significance appears challenged, while more sedate tones prevail in the various communities of mathematical modelling. This is perhaps because, unlike statistics, mathematical modelling is not a discipline. It cannot discuss possible fixes in disciplinary fora under the supervision of recognised leaders. It cannot issue authoritative statements of concern from relevant institutions such as e.g., the American Statistical Association or the columns of Nature.

Additionally the practice of modelling is spread among different fields, each characterised by its own quality assurance procedures (see 1 for references and discussion). Finally, being the coalface of research, statistics is often blamed for the larger reproducibility crisis affecting scientific production 2 .

Yet if statistics is coming to terms with methodological abuse and wicked incentives, it appears legitimate to ask if something of the sort might be happening in the multiverse of mathematical modelling. A recent work in this journal reviews common critiques of modelling practices, and suggests—for model validation, to complement a data-driven with a participatory-based approach, thus tackling the dichotomy of model representativeness—model usefulness 3 . We offer here a commentary which takes statistics as a point of departure and comparison.

For a start, modelling is less amenable than statistics to structured remedies. A statistical experiment in medicine or psychology can be pre-registered, to prevent changing the hypothesis after the results are known. The preregistration of a modelling exercise before the model is coded is unheard of, although without assessing model purpose one cannot judge its quality. For this reason, while a rhetorical or ritual use of methods is lamented in statistics 2 , it is perhaps even more frequent in modelling 1 . What is meant here by ritual is the going through the motions of a scientific process of quantification while in fact producing vacuous numbers 1 .

All model-knowing is conditional on assumptions 4 . Techniques for model sensitivity and uncertainty quantification can answer the question of what inference is conditional on what assumption, helping users to understand the true worth of a model. This understanding is identified in ref. 3 as a key ingredient of validation. Unfortunately, most modelling studies don’t bother with a sensitivity analysis—or perform a poor one 5 . A possible reason is that a proper appreciation of uncertainty may locate an output on the right side of Fig. 1 , which is a reminder of the important trade-off between model complexity and model error. Equivalent formulations of Fig. 1 can be seen in many fields of modelling and data analysis, and if the recommendations of the present comment should be limited to one, it would be that a poster of Fig. 1 hangs in every office where modelling takes place.

Model error as ideally resulting from the superposition of two curves: (i) model inadequacy error, due to using too simple a model for the problem at hand. This term goes down by making the model more complex; (ii) error propagation, which results from the uncertainty in the input variables propagating to the model output. This term grows with model complexity. Whenever the system being modelled in not elementary, overlooking important processes leaves us on the left-hand side of the plot, while modelling hubris can take us to the right-hand side

In modelling—as is the case of statistics, one can expect a mix of technical and normative problems—the latter referring to expectations, interests, values and policies being touched by the modelling activity. In cost-benefit analyses an estimate of return giving a range from a large loss to a large gain may not be what the client wishes to hear. The analysts may be tempted to “adjust” the uncertainty in the input until the output range is narrower and conveniently located in friendlier territory. Integrated climate-economy models pretend to show the fate of the planet and its economy several decades ahead, while uncertainty is so wide as to render any expectations for the future meaningless. In economics, models universally known to be wrong continue to play a role in economic policy decisions, while the neologism ‘mathiness’ has been proposed for the use of mathematics in models to veil ideological stances. Disingenuous pricing of opaque financial products is held as partly responsible for the onset of the last recession: modellers chose to calibrate the pricing of bundles of mortgages based on data for the real estate market in an up-swing period. Needless to say, these calibrations conveniently ignored what would happen when the market took a turn for the worse. Transport policy offer a curious example where a model requires as an input how many people will be sitting in a car on average decades from now. See ref. 1 for the references to the cases just described. More examples are described in ref. 6 , portraying flawed models used to justify unwise policies in evaluation of fisheries’ stock, AIDS epidemics, mill tailing, coastal erosion, and so on. Among those, studies for the safety of an underground disposal of radioactive waste stand out for providing what the authors in 6 call “A million years of certainty”, achieved thanks to a huge mathematical model including 286 sub-models.

Modelling hubris may lead to “trans-science”, a practice which lends itself to the language and formalism of science but where science cannot provide answers 7 . Models may be used as a convenient tool of displacement – from what happens in reality to what happens in the model 8 . The merging of algorithms with big data blurs many existing distinctions among different instances of quantification, leading to the question “what qualities are specific to rankings, or indicators, or models, or algorithms?” 9 Thus the problems just highlighted are likely to apply to all of these instances, as shown by the recent alarm about unethical use of algorithms 10 , the disruptive use of artificial intelligence exemplified by Facebook, or the well documented problems with the abuse of metrics 11 , which is now reflected in an increasing militancy against statistical and metrical abuses 12 .

This is not an indictment of mathematical modelling. Modelling is essential to the scientific enterprise. When Steven Shapin, a scholar studying science and technology, talks about “invisible science”—meaning scientific and technological products which improve our life—one chapter could be devoted to “invisible models” underpinning these technologies. The malpractices alluded to above are all different: not only a racist algorithm is different from an audacious cost-benefit analysis, or a low-powered statistical study. Even within modelling, different problems are at play. Modelling hubris has its counterpart in living in an idealised model-land of appealing simplicity but scarce realism 6 .

Hence, recipes cannot be prescriptive or universal. The following could help (see ref. 1 for details):

Memento Fig. 1 .

Mathematical modelling could benefit from structure and standards based on statistical principles including a systemic appraisal of model uncertainties and parametric sensitivities.

Statistics could help by internalising these into its own syllabi and practices.

Models–including algorithms, should be made inherently interpretable.

For key models used in policy, peer review should be extended to include auditing by an extended community involving a plurality of disciplines and interested actors, leading to model pedigrees, as discussed on this journal 3 and more diffusely in ref. 1 .

Audits could be used to uncover a model’s underlying, unspoken, metaphors 1 .

To put the prescriptions into practice a movement of resistance is needed, perhaps along the lines of the so-called statistical activism 12 . This kind of resistance is familiar to scholars gathered around post-normal science (PNS) 13 . The foundational works 14 , 15 of PNS’ fathers Silvio Funtowicz and Jerome R. Ravetz see model quality in terms of fitness for purpose. As noted in ref. 3 this view—with would entail reconsidering the model any time to see whether the purpose or the question put to the model are changed—is still a minority view in the modelling community. PNS suggests an approach to the use of models which is more reflexive—i.e., the analyst is part of the analysis, and participatory—including an extended peer community. While this vision is gaining new traction 3 more could be done. A new ethics of quantification ( https://www.uib.no/en/svt/127044/ethics-quantification ) must be nurtured, which takes inspiration from a long tradition of sociology of numbers; Pierre Bourdieu 12 and Theodor Porter 16 come to mind. What the authors in ref. 3 chose to call the distinction between a positivistic and a relativistic philosophy in model validation needs to be overcome for progress to be achieved.

Saltelli, A. Should statistics rescue mathematical modelling? ArXiv arXiv:1712 (2018).

Stark, P. B. & Saltelli, A. Cargo‐cult statistics and scientific crisis. Significance 15 , 40–43 (2018).

Article   Google Scholar  

Eker, S., Rovenskaya, E., Obersteiner, M. & Langan, S. Practice and perspectives in the validation of resource management models. Nat. Commun. 9 , 5359 (2018).

Article   ADS   Google Scholar  

Saltelli, A., Guimaraes Pereira, Â., van der Sluijs, J. P. & Funtowicz, S. What do I make of your latinorum? Sensitivity auditing of mathematical modelling. Int. J. Foresight Innov. Policy 9 , 213–234 (2013).

Saltelli, A. et al. Why so many published sensitivity analyses are false: a systematic review of sensitivity analysis practices. Environ. Model. Softw. 114 , 29–39 (2019).

Pilkey, O. H. Pilkey-Jarvis, L. Useless Arithmetic: Why Environmental Scientists Can’t Predict the Future (Columbia University Press, 2009).

Weinberg, A. Science and trans-science. Minerva 10 , 209–222 (1972).

Rayner, S. Uncomfortable knowledge: the social construction of ignorance in science and environmental policy discourses. Econ. Soc. 41 , 107–125 (2012).

Popp Berman, E. & Hirschman, D. The Sociology of Quantification: Where Are We Now? Contemp. Sociol. 47 , 257–266 (2018).

Brauneis, R. & Goodman, E. P. Algorithmic Transparency for the Smart City. Yale J. Law Technol. 20 , 103–176 (2018).

Google Scholar  

Muller, J. Z. The Tyranny of Metrics (Princeton University Press, 2018).

Bruno, I., Didier, E. & Prévieux, J. Stat-Activisme . (Comment Lutter Avec Des Nombres, Zones, La Découverte, Paris, 2014).

Funtowicz, S. & Ravetz, J. R. Science for the post-normal age. Futures 25 , 739–755 (1993).

Ravetz, J. R. Scientific Knowledge and Its Social Problems (Oxford University Press, 1971).

Funtowicz, S. & Ravetz, J. R. Uncertainty and Quality in Science for Policy . (Kluwer, Dordrecht, 1990).

Book   Google Scholar  

Porter, T. M. Trust in Numbers: The Pursuit of Objectivity in Science and Public Life (Princeton University Press, 1996).

Download references

Author information

Authors and affiliations.

Centre for the Study of the Sciences and the Humanities (SVT), University of Bergen (UIB), 5020, Bergen, Norway

• Andrea Saltelli

Open Evidence Research, Universitat Oberta de Catalunya (UOC), 08018, Barcelona, Spain

You can also search for this author in PubMed   Google Scholar

Contributions

The author is entirely responsible for the execution of all tasks pertaining to the paper.

Corresponding author

Correspondence to Andrea Saltelli .

Ethics declarations

Competing interests.

The author declares no competing interests.

Additional information

Peer review information : Nature Communications thanks Ragnar Fjelland for their contribution to the peer review of this work.

Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Saltelli, A. A short comment on statistical versus mathematical modelling. Nat Commun 10 , 3870 (2019). https://doi.org/10.1038/s41467-019-11865-8

Download citation

Received : 04 July 2019

Accepted : 08 August 2019

Published : 27 August 2019

DOI : https://doi.org/10.1038/s41467-019-11865-8

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

This article is cited by

The challenge of quantification: an interdisciplinary reading.

• Monica Di Fiore
• Marta Kuc-Czarnecka

Minerva (2023)

Pay No Attention to the Model Behind the Curtain

• Philip B. Stark

Pure and Applied Geophysics (2022)

Quantitative Storytelling in the Making of a Composite Indicator

• Samuele Lo Piano

Social Indicators Research (2020)

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing: AI and Robotics newsletter — what matters in AI and robotics research, free to your inbox weekly.

• Research article
• Open access
• Published: 19 November 2019

Mathematical modelling for health systems research: a systematic review of system dynamics and agent-based models

• Rachel Cassidy   ORCID: orcid.org/0000-0002-4824-0260 1 ,
• Neha S. Singh 1 ,
• Pierre-Raphaël Schiratti 2 , 3 ,
• Agnes Semwanga 4 ,
• Peter Binyaruka 5 ,
• Nkenda Sachingongu 6 ,
• Chitalu Miriam Chama-Chiliba 7 ,
• Zaid Chalabi 8 ,
• Josephine Borghi 1 &
• Karl Blanchet 1  

BMC Health Services Research volume  19 , Article number:  845 ( 2019 ) Cite this article

23k Accesses

50 Citations

5 Altmetric

Metrics details

Mathematical modelling has been a vital research tool for exploring complex systems, most recently to aid understanding of health system functioning and optimisation. System dynamics models (SDM) and agent-based models (ABM) are two popular complementary methods, used to simulate macro- and micro-level health system behaviour. This systematic review aims to collate, compare and summarise the application of both methods in this field and to identify common healthcare settings and problems that have been modelled using SDM and ABM.

We searched MEDLINE, EMBASE, Cochrane Library, MathSciNet, ACM Digital Library, HMIC, Econlit and Global Health databases to identify literature for this review. We described papers meeting the inclusion criteria using descriptive statistics and narrative synthesis, and made comparisons between the identified SDM and ABM literature.

We identified 28 papers using SDM methods and 11 papers using ABM methods, one of which used hybrid SDM-ABM to simulate health system behaviour. The majority of SDM, ABM and hybrid modelling papers simulated health systems based in high income countries. Emergency and acute care, and elderly care and long-term care services were the most frequently simulated health system settings, modelling the impact of health policies and interventions such as those targeting stretched and under resourced healthcare services, patient length of stay in healthcare facilities and undesirable patient outcomes.

Conclusions

Future work should now turn to modelling health systems in low- and middle-income countries to aid our understanding of health system functioning in these settings and allow stakeholders and researchers to assess the impact of policies or interventions before implementation. Hybrid modelling of health systems is still relatively novel but with increasing software developments and a growing demand to account for both complex system feedback and heterogeneous behaviour exhibited by those who access or deliver healthcare, we expect a boost in their use to model health systems.

Peer Review reports

Introduction

Health systems are complex adaptive systems [ 1 ]. As such, they are characterised by extraordinary complexity in relationships among highly heterogeneous groups of stakeholders and the processes they create [ 2 ]. Systems phenomena of massive interdependencies, self-organising and emergent behaviour, non-linearity, time lags, feedback loops, path dependence and tipping points make health system behaviour difficult and sometimes impossible to predict or manage [ 3 ]. Conventional reductionist approaches using epidemiological and implementation research methods are inadequate for tackling the problems health systems pose [ 4 ]. It is increasingly recognised that health systems and policy research need a special set of approaches, methods and tools that derive from systems thinking perspectives [ 5 ]. Health systems encompass a many tiered system providing services to local, district and national populations, from community health centres to tertiary hospitals. Attempting to evaluate the performance of such a multi-faceted organisation presents a daunting task. Mathematical modelling, capable of simulating the behaviour of complex systems, is therefore a vital research tool to aid our understanding of health system functioning and optimisation.

System dynamics model (SDM)

System dynamics models (SDM) and agent-based models (ABM) are the two most popular mathematical modelling methods for evaluating complex systems; while SDM are used to study macro-level system behaviour such as the movement of resources or quantities in a system over time, ABM capture micro-level system behaviour, such as human decision-making and heterogeneous interactions between humans.

While use of SDM began in business management [ 6 , 7 ] it now has wide spread application from engineering to economics, from environmental science to waste and recycling research [ 8 , 9 , 10 , 11 , 12 , 13 ]. A SDM simulates the movement of entities in a system, using differential equations to model over time changes to system state variables. A stock and flow diagram can be used to provide a visual representation of a SDM, describing the relationships between system variables using stocks, rates and influencing factors. The diagram can be interpreted as mimicking the flow of water in and out of a bath tub [ 7 ]; the rates control how much ‘water’ (some quantifiable entity, resource) can leave or enter a ‘bath tub’ (a stock, system variable) which changes over time depending on what constraints or conditions (e.g. environmental or operational) are placed on the system. Often before the formulation of a stock and flow diagram, a causal loop diagram is constructed which can be thought of as a ‘mental model’ of the system [ 14 ], representing key dynamic hypotheses.

Agent-based model (ABM)

Unlike SDM, ABM is a ground-up representation of a system, simulating the changing states of individual ‘agents’ in a system rather than the broad entities or aggregate behaviour modelled in SDM. Aggregate system behaviour can however be inferred from ABM. Use of ABM to model system behaviour has been trans-disciplinary, with application in economics to ecology, from social sciences to engineering [ 15 , 16 , 17 , 18 , 19 ]. There can be multiple types of agent modelled, each assigned their own characteristics and pattern of behaviour [ 20 , 21 ]. Agents can learn from their own experiences, make decisions and perform actions based on set rules (e.g. heuristics), informed by their interactions with other agents, their own assigned attributes or based on their interaction with the modelled environment [ 22 ]. The interactions between agents can result in three levels of communication between agents; one-to-one communication between agents, one-to-many communication between agents and one-to-location communication where an agent can influence other agents contained in a particular location [ 22 ].

Why use SDM and ABM to model health systems?

ABM and SDM, with their ability to simulate micro- and macro-level behaviour, are complementary instruments for examining the mechanisms in complex systems and are being recognised as crucial tools for exploratory analysis. Their use in mapping health systems, for example, has steadily risen over the last three decades. ABM is well-suited to explore systems with dynamic patient or health worker activity, a limitation of other differential equation or event-based simulation tools [ 23 , 24 , 25 ]. Unlike discrete-event simulation (DES) for example, which simulates a queue of events and agent attributes over time [ 26 ], the agents modelled in ABM are decision makers rather than passive individuals. Closer to the true system modelled, ABM can also incorporate ongoing learning from events whereby patients can be influenced by their interactions with other patients or health workers and by their own personal experience with the health system [ 21 ]. SDM has also been identified as a useful tool for simulating feedback and activity across the care continuum [ 27 , 28 , 29 , 30 ] and is highly adept at capturing changes to the system over time [ 31 ]. This is not possible with certain ‘snapshot in time’ modelling approaches such as DES [ 32 ]. SDM is best implemented where the aim of the simulation is to examine aggregate flows, trends and sub-system behaviour as opposed to intricate individual flows of activity which are more suited to ABM or DES [ 33 ].

There are also models that can accommodate two or more types of simulation, known as hybrid models. Hybrid models produce results closer to true system behaviour by drawing on the strengths of one or more modelling methods while reducing the limitations associated with using a single simulation type [ 27 ]. The activity captured in such models emulates the individual variability of patients and health professionals while retaining the complex, aggregate behaviour exhibited in health systems.

Health scientists and policy makers alike have recognised the potential of using SDM and ABM to model all aspects of health systems in support of decision making from emergency department (ED) optimisation [ 34 ] to policies that support prevention or health promotion [ 35 ]. Before implementing or evaluating costly health policy interventions or health service re-structuring in the real world, modelling provides a relatively risk-free and low budget method of examining the likely impact of potential health system policy changes. They allow the simulation of ‘what if’ scenarios to optimise an intervention [ 36 ]. They can help identify sensitive parameters in the system that can impede the success of initiatives and point to possible spill-over effects of these initiatives to other departments, health workers or patients. Perhaps most important of all, these modelling methods allow researchers to produce simulations, results and a graphical-user interface in relation to alternative policy options that are communicable to stakeholders in the health system [ 37 ], those responsible for implementing system-wide initiatives and changes.

Study aim and objectives

Given the increasing amount of literature in this field, the main aim of the study was to examine and describe the use of SDM and ABM to model health systems. The specific objectives were as follows: (1) Determine the geographical, and healthcare settings in which these methods have been used (2) Identify the purpose of the research, particularly the health policies or interventions tested (3) Evaluate the limitations of these methods and study validation, and (4) Compare the use of SDM and ABM in health system research.

Although microsimulation, DES and Markov models have been widely used in disease health modelling and health economic evaluation, our aim in this study was to review the literature on mathematical methods which are used to model complex dynamic systems, SDM and ABM. These models represent two tenants of modelling: macroscopic (top-level) and microscopic (individual-level) approaches. Although microsimulation and DES are individual-based models like ABM, individuals in ABM are “active agents” i.e. decision-makers rather than “passive agents” which are the norm in microsimulation and DES models. Unlike Markov models which are essentially one-dimensional, unidirectional and linear, SDM are multi-dimensional, nonlinear with feedback mechanisms. We have therefore focussed our review on SDM and ABM because they are better suited to characterise the complexity of health systems. This study reviews the literature on the use of SDM and ABM in modelling health systems, and identifies and compares the key characteristics of both modelling approaches in unwrapping the complexity of health systems. In identifying and summarising this literature, this review will shed light on the types of health system research questions that these methods can be used to explore, and what they add to more traditional methods of health system research. By providing an over overview of how these models can be used within health system research, this paper is also expected to encourage wider use and uptake of these methods by health system researchers and policy makers.

The review was conducted in compliance with the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) statement [ 38 ].

Search strategy and information sources

The literature on ABM and SDM of health systems has not been confined to a single research discipline, making it necessary to widen the systematic review to capture peer-reviewed articles found in mathematical, computing, medicine and health databases. Accordingly, we searched MEDLINE, EMBASE, Cochrane Library, MathSciNet, ACM Digital Library, HMIC, Econlit and Global Health databases for literature. The search of health system literature was narrowed to identify articles that were concerned with modelling facility-based healthcare, services and related healthcare financing agreements which had been excluded or were not the focus of previous reviews [ 34 , 35 , 39 , 40 , 41 ]. The search criteria used for MEDLINE was as follows, with full search terms for each database and search terms used to locate SDM and ABM literature found in Additional file  1 :

(health system* OR health care OR healthcare OR health service* OR health polic* OR health facil* OR primary care OR secondary care OR tertiary care OR hospital*).ab,ti. AND (agent-based OR agent based).ab,ti. AND (model*).ab,ti.

In addition, the reference list of papers retained in the final stage of the screening process, and systematic reviews identified in the search, were reviewed for relevant literature.

Data extraction and synthesis

The screening process for the review is given in Fig.  1 (adapted from [ 38 ]). All search results were uploaded to Mendeley reference software where duplicate entries were removed. The remaining records were screened using their titles and abstracts, removing entries based on eligibility criteria given in Table  1 . Post-abstract review, the full text of remaining articles was screened. Papers retained in final stage of screening were scrutinised, with data imported to Excel based on the following categories; publication date, geographical and healthcare setting modelled, purpose of research in addition to any policies or interventions tested, rationale for modelling method and software platform, validation and limitations of model. The results were synthesised using descriptive statistics and analysis of paper content that were used to answer the objectives.

a Flow-chart for systematic review of SDMs and b ABMs of health systems (Database research discipline is identified by colour; mathematical and computing (red), medicine (blue) and health (green) databases). Adapted from PRISMA [ 38 ]

The studies were first described by three characteristics: publication date, geographical setting, and what aspect of the health system was modelled and why. These characteristics were chosen for the following reasons. Publication date (Fig.  2 ) allows us to examine the quantity of SDM and ABM studies over time. Geographical settings (Fig. 2 , top) allows us to see which health systems have been studied, as health systems in LMIC are very different from those in developed countries. Studies are classified as modelling health systems in high, upper middle, lower middle and low income countries as classified by The World Bank based on economy, July 2018 [ 42 ]. Finally, we examined which aspects of the health system have been modelled and the types of research/policy questions that the models were designed to address, to shed light on the range of potential applications of these models, and also potential gaps in their application to date.

Number of articles in the final review by year of publication and economic classification

The analysis of paper content was split into three sections; SDM use in health system research (including hybrid SDM-DES), ABM use in health system research (including hybrid ABM-DES) and hybrid SDM-ABM use in health system research. The quality of selected studies will not be presented as our aim was to compare and summarise the application of SDM and ABM in modelling health systems rather than a quality appraisal of studies.

Study selection

The search initially yielded 535 citations for ABM and 996 citations for SDM of facility-based healthcare and services (see Fig. 1 ). Post-full text screening 11 ABM and 28 SDM papers were retained for analysis, six of which utilised hybrid modelling methods. Three of the hybrid modelling papers integrated SDM with DES [ 43 , 44 , 45 ], two integrated ABM with DES [ 24 , 46 ] and one integrated SDM with ABM [ 47 ]. A summary table of selected papers is given in Table  2 .

Descriptive statistics

Publication date.

The first SDM paper to model health systems was published in 1998 [ 56 ] whilst the first publication [ 66 ] utilising ABM came almost a decade later (Fig. 2 ). We found an increasing trend in publications for both modelling approaches, with 90.9% (10/11) and 71.4% (20/28) of all ABM and SDM articles, respectively, having been published in the last decade. The first hybrid modelling article was published in 2010 [ 43 ], using SDM and DES to model the impact of an intervention to aid access to social care services for elderly patients in Hampshire, England.

Geographical setting

The proportion of papers that modelled health systems in high, upper middle, lower middle and low income countries is presented in Fig. 2 . Eighteen (18/28) papers that employed SDM simulated health systems in high income countries including England [ 33 , 36 , 43 , 45 , 50 , 54 , 56 , 57 ] and Canada [ 28 , 51 , 62 ]. Four SDM papers simulated upper middle income country health systems, including Turkey [ 52 , 59 ] and China [ 64 ], with a nominal number of papers (5/28) focussing on lower middle or low income countries (West Bank and Gaza [ 48 , 55 ], Indonesia [ 37 ], Afghanistan [ 30 ] and Uganda [ 60 ]). Almost all ABM papers (9/11) modelled a high income country health system, including the US [ 20 , 23 , 25 ] and Austria [ 65 ]. Two (2/11) ABM papers described an upper-middle income based health system (Brazil [ 22 , 67 ]). All six articles that implemented a hybrid SDM or ABM simulated health systems based in high income countries, including Germany [ 44 ] and Poland [ 47 ].

Healthcare setting and purpose of research

The healthcare settings modelled in the SDM, ABM and hybrid simulation papers are presented in Fig.  3 . Healthcare settings modelled using SDM included systems that were concerned with delivering emergency or acute care (11/28) [ 28 , 31 , 36 , 45 , 47 , 50 , 56 , 57 , 58 , 61 , 62 ], elderly or long-term care services (LTC)(12/28) [ 28 , 31 , 36 , 43 , 44 , 45 , 49 , 50 , 51 , 54 , 61 , 62 ] and hospital waste management (4/28) [ 37 , 48 , 52 , 55 ]. Twenty of the SDM papers selected in this review assessed the impact of health policy or interventions on the modelled system. Common policy targets included finding robust methods to relieve stretched healthcare services, ward occupancy and patient length of stay [ 28 , 31 , 36 , 43 , 49 , 50 , 54 , 58 , 62 ], reducing the time to patient admission [ 33 , 53 , 61 ], targeting undesirable patient health outcomes [ 47 , 58 , 60 , 63 ], optimising performance-based incentive health system policies [ 30 , 59 ] and reducing the total cost of care [ 33 , 54 , 61 ]. The remaining eight papers explored factors leading to undesirable emergency care system behaviour [ 56 , 57 ], simulating hospital waste management systems and predicting future waste generation [ 37 , 48 , 55 ], estimating future demand for cardiac care [ 44 ], exploring the impact of patient admission on health professionals stress level in an integrated care system [ 45 ], and variation in physician decision-making [ 32 ].

The health system sector locations modelled in the SDM, ABM and hybrid modelling literature. Long-term care (LTC); Accountable care organisation (ACO); Maternal, newborn and child health (MNCH)

ABM papers modelled systems focussed on delivering emergency or acute care (4/11) [ 21 , 22 , 47 , 67 ] and accountable care organisations (ACO) or health insurance reimbursement schemes (3/11) [ 23 , 25 , 65 ]. Nine of the ABM papers assessed the impact of health policy or interventions on the modelled system. Common policy targets included decreasing the time agents spent performing tasks, waiting for a service or residing in parts of the system [ 20 , 22 , 24 , 67 ], reducing undesirable patient outcomes [ 23 , 25 , 47 , 67 ], reducing the number of patients who left a health facility without being seen by a physician [ 22 , 67 ] and optimising resource utility (beds and healthcare staff) [ 46 , 66 , 67 ]. The remaining two papers described simulation tools capable of comparing health insurance reimbursement schemes [ 65 ] and assessing risk, allocation of resources and identifying weaknesses in emergency care services [ 21 ].

Papers that utilised hybrid simulation, combining the strengths of two modelling approaches to capture detailed individual variability, agent-decision making and patient flow, modelled systems focussed on delivering elderly care or LTC services [ 43 , 44 , 45 ] and emergency or acute care [ 45 , 47 ]. Four of the hybrid simulation papers assessed the impact of policy or intervention on the modelled system. Policy targets included improving access to social support and care services [ 43 ], reducing undesirable patient outcomes [ 47 ], decreasing patient waiting time to be seen by a physician [ 24 ] and improving patient flow through the system by optimising resource allocation [ 46 ]. The remaining two papers used hybrid simulation to estimate the future demand for health care from patients with cardiac disease [ 44 ] and model patient flow through an integrated care system to estimate impact of patient admission on health care professionals wellbeing [ 45 ].

SDM use in health systems research (including hybrid SDM-DES)

Rationale for using model.

Gaining a holistic system perspective to facilitate the investigation of delays and bottlenecks in health facility processes, exploring counter-intuitive behaviour and monitoring inter-connected processes between sub-systems was cited frequently as reasons for using SDM to model health systems [ 28 , 36 , 37 , 48 , 56 ]. SDM was also described as a useful tool for predicting future health system behaviour and demand for care services, essential for health resource and capacity planning [ 48 , 60 ]. Configuration of the model was not limited by data availability [ 28 , 52 , 64 ] and could integrate data from various sources when required [ 51 ].

SDM was described as a tool for health policy exploration and optimising system interventions [ 33 , 36 , 51 , 54 , 58 , 64 ], useful for establishing clinical and financial ramifications on multiple groups (such as patients and health care providers) [ 63 ], identifying policy resistance or unintended system consequences [ 59 , 61 ] and quantifying the impact of change to the health system before real world implementation [ 62 ]. The modelling platform also provided health professionals, stakeholders and decision makers with an accessible visual learning environment that enabled engagement with experts necessary for model conception and validation [ 48 , 50 , 55 , 57 ]. The model interface could be utilised by decision makers to develop and test alternative policies in a ‘real-world’ framework that strengthened their understanding of system-wide policy impact [ 31 , 49 , 58 , 61 ].

SDM-DES hybrid models enabled retention of deterministic and stochastic system variability and preservation of unique and valuable features of both methods [ 44 ], capable of describing the flow of entities through a system and rapid insight without the need for large data collection [ 43 ], while simulating individual variability and detailed interactions that influence system behaviour [ 43 ]. SDM-DES offered dual model functionality [ 44 ] vital for simulating human-centric activity [ 45 ], reducing the practical limitations that come with using either SDM or DES to model health systems such as attempting to use SDM to model elements which have non-aggregated values (e.g. patient arrival time) [ 45 ] which is better suited for DES.

Healthcare setting

Sixteen papers that utilised SDM modelled systems that were concerned with the delivery of emergency or acute care, or elderly care or LTC services.

Ten of the reviewed papers primarily modelled sectors of the health system that delivered emergency or acute care Footnote 1 , Footnote 2 . Brailsford et al. [ 50 ], Lane et al. [ 56 ], Lane et al. [ 57 ] and Lattimer et al. [ 36 ] simulated the delivery of emergency care in English cities, specifically in Nottingham and London. Brailsford et al. [ 50 ] and Lattimer et al. [ 36 ] created models that replicated the entire emergency care system for the city of Nottingham, from primary care (i.e. General Practice surgeries) to secondary care (i.e. hospital admissions wards), to aid understanding of how emergency care was delivered and how the system would need to adapt to increasing demand. Lane et al. [ 56 ] and Lane et al. [ 57 ] modelled the behaviour of an ED in an inner-London teaching hospital, exploring the knock on effects of ED performance to hospital ward occupancy and elective admissions. Esensoy et al. [ 28 ] and Wong et al. [ 62 ] both modelled emergency care in Canada, Esensoy et al. [ 28 ] focussing on six sectors of the Ontario health system that cared for stroke patients while Wong et al. [ 62 ] simulated the impact of delayed transfer of General Internal Medicine patients on ED occupancy. Rashwan et al. [ 31 ], Walker et al. [ 61 ] and Mahmoudian-Dehkordi et al. [ 58 ] modelled patient flow through a generic emergency care facility with six possible discharge locations in Ireland, a sub-acute extended care hospital with patient flow from feeder facilities in Australia and an intensive care unit, ED and general wards in a generic facility.

Five of the SDM papers primarily simulated the behaviour of LTC facilities or care services for elderly patients Footnote 3 . Ansah et al. [ 49 ] modelled the demand and supply of general LTC services in Singapore with specific focus on the need for LTC and acute health care professionals. Desai et al. [ 54 ] developed a SDM that investigated future demand of care services for older people in Hampshire, England which simulated patient flow through adult social care services offering 13 different care packages. In modelling complex care service demand, Cepoiu-Martin et al. [ 51 ] explored patient flow within the Alberta continuing care system in Canada which offered supportive living and LTC services for patients with dementia. Brailsford et al. [ 43 ] used a hybrid SDM-DES model to investigate how local authorities could improve access to services and support for older people, in particular the long term impact of a new contact centre for patients. The SDM replicated the whole system for long term care, simulating the future demography and demand for care services and the nested DES model simulated the operational issues and staffing of the call centre in anticipation of growing demand for services. Zulkepli et al. [ 45 ] also used SDM-DES to model the behaviour of an integrated care system in the UK, modelling patient flow (DES) and intangible variables (SDM) related to health professionals such as motivation and stress levels.

Policy impact evaluation/testing

Twenty papers that utilised SDM tested the impact of policy or interventions on key health system performance or service indicators. The intended target of these policies ranged from relieving strained and under resourced healthcare services, decreasing healthcare costs to reducing patient mortality rates.

Ansah et al. [ 49 ], Brailsford et al. [ 50 ] and Desai et al. [ 54 ] aimed to reduce occupancy in acute or emergency care departments through policies that targeted elderly utilisation of these services. While demand for LTC services is expected to exponentially increase in Singapore, focus has been placed on expanding the acute care sector. Ansah et al. [ 49 ] simulated various LTC service expansion policies (static ‘current’ policy, slow adjustment, quick adjustment, proactive adjustment) and identified that proactive expansion of LTC services stemmed the number of acute care visits by elderly patients over time and required only a modest increase in the number of health professionals when compared with other policies. In Brailsford et al. [ 50 ] simulation of the entire emergency care system for Nottingham, England, policy testing indicated that while the emergency care system is operating near full capacity, yearly total occupancy of hospital beds could be reduced by re-directing emergency admissions from patients over 60 years of age (who make up around half of all admissions) to more appropriate services, such as those offered by community care facilities. To explore challenges that accompany providing care for an ageing population subject to budget restraints, Desai et al. [ 54 ] simulated the delivery and demand for social care services in Hampshire over a projected 5 year period. In offering care packages to only critical need clients and encouraging extra care services at home rather than offering residential care, the number of patients accessing acute care services reduced over the observed period.

Desai et al. [ 54 ], in addition to Taylor et al. [ 33 ] and Walker et al. [ 61 ], also examined policies that could reduce the total cost of care. Increasing the proportion of hired unqualified care workers (over qualified care workers who are employed at a higher cost rate) resulted in savings which could be fed back into care funding, although Desai et al. [ 54 ] remarked on the legal and practical limitations to this policy. Taylor et al. [ 33 ] examined the impact of shifting cardiac catheterization services from tertiary to secondary level hospitals for low risk investigations and explored how improvements could be made to services. Significant and stable improvements in service, including reduced waiting list and overall cost of service, were achieved with the implementation of strict (appropriate referral) guidelines for admitting patients. Walker et al. [ 61 ] modelled patient flow from feeder hospitals to a single sub-acute extended care facility in Victoria, Australia, to assess the impact of local rules used by the medical registrar for admission. The local admission policy which prioritised admissions from patients under the care of private doctors pushed the total cost of care over the facility budget by 6% whereas employing no prioritisation rule reduced the total cost of care to 3% under budget.

Semwanga et al. [ 60 ], Mahmoudian-Dehkordi et al. [ 58 ] and Worni et al. [ 63 ] evaluated the impact of health policy on undesirable patient outcomes (mortality and post-treatment complication rates). Semwanga et al. [ 60 ] tested the effectiveness of policies designed to promote maternal and neonatal care in Uganda, established from the literature. Policies that enabled service uptake, such as community health education, free delivery kits and motorcycle coupons were significant in reducing neonatal death over the simulated period. Mahmoudian-Dehkordi et al. [ 58 ] explored the intended and unintended consequences of intensive care unit resource and bed management policies on system performance indicators, including patient mortality. During a simulated crisis scenario, prioritising intensive care unit patient admission to general wards over emergency admissions was found to be the most effective policy in reducing total hospital mortality. Worni et al. [ 63 ] estimated the impact of a policy to reduce venous thromboembolism rates post-total knee arthroplasty surgery and identified unintentional consequences of the strategy. The policy prevented the reimbursement of patient care fees in the event that a patient was not taking the recommended prophylaxis medication and consequently develops venous thromboembolism. Simulation results indicated a positive 3-fold decrease in venous thromboembolism rates but an unintended 6-fold increase in the number of patients who develop bleeding complications as a result of compulsory prophylaxis treatment.

Validation (including sensitivity analysis)

Statistically-based models are usually used in quantitative data rich environments where model parameters are estimated through maximum likelihood or least-squares estimation methods. Bayesian methods can also be used to compare alternative statistical model structures. SDMs and ABMs on the other hand are not fitted to data observations in the traditional statistical sense. The data are used to inform model development. Both quantitative data and qualitative data (e.g. from interviews) can be used to inform the structure of the model and the parameters of the model. Furthermore, model structure and parameter values can also be elicited from expert opinion. This means that the nature of validation of ABMs and SDMs requires more scrutiny than that of other types of models.

With increasing complexity of such models, and to strengthen confidence in their use particularly for decision support, models are often subjected to sensitivity analysis and validation tests. Twenty-two papers that utilised SDM undertook model validation, the majority having performed behavioural validity tests (see Additional file  2 for details of validation methods for each model). Key model output such as bed occupancy [ 36 , 50 ], department length of stay [ 62 ] and number of department discharges [ 31 ] were compared with real system performance data from hospitals [ 32 , 33 , 36 , 48 , 50 , 54 , 58 , 59 , 61 , 62 ], local councils [ 54 ], nationally reported figs [ 31 , 64 ]. as well being reviewed by experts [ 57 , 60 ] as realistic. Others performed more structure orientated validity tests. Model conception [ 28 , 60 ], development [ 30 , 36 , 50 , 53 , 54 , 57 , 62 ] and formulation [ 54 , 56 , 59 ] were validated by a variety of experts including health professionals [ 47 , 53 , 54 , 57 , 59 , 62 ], community groups [ 56 ] and leaders [ 60 ], steering committees [ 36 ], hospital and care representatives [ 50 , 56 , 59 ], patient groups [ 60 ] and healthcare policy makers [ 60 ]. Further tests for structural validity included checking model behaviour when subjected to extreme conditions or extreme values of parameters [ 30 , 31 , 52 , 57 , 59 , 60 , 64 ], model dimensional consistency [ 31 , 52 , 57 , 59 , 60 ], model boundary adequacy [ 31 ] and mass balance [ 54 ] and integration error checks [ 31 , 52 ]. Sensitivity analysis was performed to assess how sensitive model output was to changes in key parameters [ 49 , 51 , 57 , 60 , 64 ], to test the impact of parameters that had been based on expert opinion on model output [ 28 ] and varying key system parameters to test the robustness and effectiveness of policies [ 28 , 30 , 52 , 53 , 58 ] (on the assumption of imperfect policy implementation [ 28 ]).

Limitations of research

Most of the model limitations reported were concerned with missing parameters, feedback or inability to simulate all possible future health system innovations. Mielczarek et al. [ 44 ], Cepoiu-Martin et al. [ 51 ], Ansah et al. [ 49 ] and Rashwan et al. [ 31 ] did not take into account how future improvements in technology or service delivery may have impacted results, such as the possibility of new treatment improving patient health outcomes [ 51 ] and how this could impact the future utilisation of acute care services [ 49 ]. Walker et al. [ 61 ] and Alonge et al. [ 30 ] described how the models may not simulate all possible actions or interactions that occurred in the real system, such as all proactive actions taken by hospital managers to achieve budget targets [ 61 ] or all unintended consequences of a policy on the system [ 30 ]. De Andrade et al. [ 53 ] and Rashwan et al. [ 31 ] discussed the reality of model boundaries, that SDMs cannot encapsulate all health sub-sector behaviour and spill-over effects. Although these have been listed here as limitations, not accounting for possible future improvements in healthcare service or not simulating all possible actions in the modelled system did not prevent authors from fulfilling study objectives. When developing a SDM, it is not possible to account for all possible spill-over effects to other healthcare departments and this should not be attempted; model boundaries are set to only include variables and feedback that are pertinent to exploring the defined problem.

Simplification of model parameters was another common limitation. Wong et al. [ 62 ] stated that this would result in some model behaviour not holding in the real system, such as using weekly hospital admission and discharge averages in place of hourly rates due to the hospital recording aggregated data. This aggregation of model parameters may not have reflected real system complexity; Eleyan et al. [ 55 ] did not differentiate between service level and type of hospital when modelling health care waste production (described as future work) and Worni et al. [ 63 ] refrained from stratifying post-surgery complications by severity, potentially combining lethal and less harmful complications within the same stock (although this did not detract from the study conclusion that the rate of complications would increase as a result of the tested policy).

Data availability, lack of costing analysis and short time horizons were also considered credible limitations. Models that had been calibrated with real data were at risk of using datasets that contained measurement errors or incomplete datasets lacking information required to inform model structure or feedback [ 32 ]. Routine facility data required for model conception and formulation was unavailable which restricted the replication of facility behaviour in the model [ 36 ] and restricted validation of model behaviour [ 59 ], although it should be noted that this is only one method among many for SDM validation and the author was able to use other sources of data for this purpose. Lack of costing or cost effectiveness analysis when testing policies [ 60 ], particularly policies that required significant investment or capacity expansion [ 58 ], limited discussion on their feasibility in the real system. Models that simulated events over short time scales did not evaluate long term patient outcomes [ 33 ] or the long term effects of facility policies on certain groups of patient [ 57 ].

ABM use in health system research (including hybrid ABM-DES)

The model’s ability to closely replicate human behaviour that exists in the real system was frequently cited [ 20 , 21 , 22 , 25 , 66 ], providing a deeper understanding of multiple agent decision-making [ 23 , 67 ], agent networks [ 25 ] and interactions [ 21 , 22 ]. The modelling method was described as providing a flexible framework capable of conveying intricate system structures [ 20 ], where simulations captured agent capacity for learning and adaptive behaviour [ 20 , 25 ] and could incorporate stochastic processes that mimicked agent transition between states [ 25 ]. ABM took advantage of key individual level agent data [ 25 ] and integrated information from various sources including demographic, epidemiological and health service data [ 65 ]. The visualisation of systems and interface available with ABM software packages facilitated stakeholder understanding of how tested policies could impact financial and patient health outcomes [ 23 ], particularly those experts in the health industry with minimal modelling experience [ 67 ].

Integrating DES and ABM within a single model ensured an intelligent and flexible approach for simulating complex systems, such as the outpatient clinic described in Kittipittayakorn et al. [ 24 ]. The hybrid model captured both orthopaedic patient flow and agent decision-making that enabled identification of health care bottlenecks and optimum resource allocation [ 24 ].

Seven papers that utilised ABM modelled systems that were either concerned with delivering emergency or acute care 2 , ACOs or health insurance reimbursement schemes.

Liu et al. [ 21 ] and Yousefi et al. [ 22 ] modelled behaviour in EDs in Spanish and Brazilian tertiary hospitals. Liu et al. [ 21 ] simulated the behaviour of eleven key agents in the ED including patients, admission staff, doctors, triage nurses and auxiliary staff. Patients were admitted to the ED and triaged before tests were requested and a diagnosis issued. Over time, agent states changed based on their interaction with other agents such as when a doctor decided upon a course of action for a patient (sending the patient home, to another ward, or continue with diagnosis and treatment). For further details of agent type and model rules for each paper, see Additional file  3 .

Yousefi et al. [ 22 ] modelled the activities of patients, doctors, nurses and receptionists in a ED. Agents could communicate with each other, to a group of other agents or could send a message to an area of the ED where other agents reside. They made decisions based on these interactions and the information available to them at the time. The main focus of the simulation was on patients who left the ED without being seen by a physician; patients decided whether to leave the ED based on a ‘tolerance’ time extracted from the literature, which changed based on their interaction with other agents. In an additional paper, Yousefi et al. [ 67 ] simulated decision-making by patients, doctors, nurses and lab technicians within a generic ED informed from the literature. Group decision-making was employed, whereby facility staff could interact with each other and reach a common solution for improving the efficacy of the department such as re-allocating staff where needed. Yousefi et al. [ 67 ], Yousefi et al. [ 22 ] and Liu et al. [ 21 ] each used a finite state machine (a computational model which describes an entity that can be in one of a finite number of states) to model interactions between agents and their states.

Liu et al. [ 25 ] and Alibrahim et al. [ 23 ] modelled the behaviour of patients, health providers and payers using series of conditional probabilities, where health providers had participated in an ACO in the United States. Liu et al. [ 25 ] presented a model where health providers within an ACO network worked together to reduce congestive heart failure patient healthcare costs and were consequently rewarded a portion of the savings from the payer agent (hypothetically, the Centers for Medicare and Medicaid Services). Patients were Medicare beneficiaries over the age of 65 who developed diabetes, hypertension and/or congestive heart failure and sought care within the network of health providers formed of three hospitals and 15 primary care physician clinics. Alibrahim et al. [ 23 ] adapted Liu et al. [ 25 ] ACO network model to allow patients to bypass their nearest medical provider in favour of an alternative provider. The decision for a patient to bypass their nearest health centre was influenced by patient characteristics, provider characteristics and the geographical distance between health providers. Providers were also given a choice on whether to participate in an ACO network, where they would then need to implement a comprehensive congestive heart failure disease management programme.

Einzinger et al. [ 65 ] created a tool that could be used to compare different health insurance reimbursement schemes in the Austrian health sector. The ABM utilised anonymous routine data from practically all persons with health insurance in Austria, pertaining to medical services accessed in the outpatient sector. In the simulation, patients developed a chronic medical issue (such as coronary heart disease) that required medical care and led to the patient conducting a search of medical providers through the health market. The patient then accessed care at their chosen provider where the reimbursement system, notified of the event via a generic interface, reimbursed the medical provider for patients care.

Nine papers tested the impact of policy on key health system performance or service indicators. The intended target of these policies ranged from decreasing patient length of stay, to reducing the number of patients who leave without being seen by a physician to reducing patient mortality and hospitalisation rates.

Huynh et al. [ 20 ], Yousefi et al. [ 22 ], Yousefi et al. [ 67 ] and Kittipittayakorn et al. [ 24 ] tested policies to reduce the time agents spent performing tasks, waiting for a service or residing in parts of the system. Huynh et al. [ 20 ] modelled the medication administration workflow for registered nurses at an anonymous medical centre in the United States and simulated changes to the workflow to improve medication administration safety. Two policies were tested; establishing a rigid order for tasks to be performed and for registered nurses to perform tasks in the most frequently observed order (observed in a real medical centre) to see if this improved the average amount of time spent on tasks. Yousefi et al. [ 67 ] modelled the effects of group decision-making in ED compared with the standard approach for resource allocation (where a single supervisor allocates resources) to assess which policy resulted in improved ED performance. Turning ‘on’ group decision-making and starting the simulation with a higher number of triage staff and receptionists resulted in the largest reduction of average patient length of stay and number of patients who left without being seen. This last performance indicator was the subject of an additional paper [ 22 ], with focus on patient-to-patient interactions and how this impacted their decision to leave the ED before being seen by a physician. Four policies adapted from case studies were simulated to reduce the number of patients leaving the ED without being seen and average patient length of stay. The policy of fast-tracking patients who were not acutely unwell during triage performed well as opposed to baseline, where acutely ill patients were always given priority. Kittipittayakorn et al. [ 24 ] used ABM-DES to identify optimal scheduling for appointments in an orthopaedic outpatient clinic, with average patient waiting time falling by 32% under the tested policy.

Liu et al. [ 25 ], Alibrahim et al. [ 23 ] and Yousefi et al. [ 67 ] tested the impact of health policy on undesirable patient outcomes (patient mortality and hospitalisation rates). Liu et al. [ 25 ] modelled health care providers who operated within an ACO network and outside of the network and compared patient outcomes. Providers who operated within the ACO network worked together to reduce congestive heart failure patient healthcare costs and were then rewarded with a portion of the savings. As part of their membership, providers implemented evidence-based interventions for patients, including comprehensive discharge planning with post-discharge follow-up; this intervention was identified in the literature as key to reducing congestive heart failure patient hospitalisation and mortality, leading to a reduction in patient care fees without compromising the quality of care. The ACO network performed well, with a 10% reduction observed in hospitalisation compared with the standard care network. In another study [ 23 ] six scenarios were simulated with combinations of patient bypass capability (turned “on” or “off”) and provider participation in the ACO network (no ACO present, optional participation in ACO or compulsory participation in ACO). Provider participation in the ACO, in agreement with Liu et al. [ 25 ], led to reduced mortality and congestive heart failure patient hospitalisation, with patient bypass capability marginally increasing provider ACO participation. Yousefi et al. [ 67 ] also modelled the impact of group decision-making in ED on the number of patient deaths and number of wrong discharges i.e. patients sent to the wrong sector for care after triage and are then discharged before receiving correct treatment.

Nine of the 11 papers that utilised ABM undertook model validation, consisting almost exclusively of behavioural validity tests. Model output, such as patient length of stay and mortality rates, was reviewed by health professionals [ 46 , 66 ] and compared with data extracted from pilot studies [ 20 ], health facilities (historical) [ 22 , 24 , 46 , 65 , 66 ], national health surveys [ 65 ] and relevant literature [ 23 , 25 ]. Papers presented the results of tests to determine the equivalence of variance [ 20 ] and difference in mean [ 20 , 24 ] between model output and real data. Structural validity tests included extreme condition testing [ 23 , 46 ] and engaging health care experts to ensure the accuracy of model framework [ 22 , 47 ]. Sensitivity analysis was performed to determine how variations or uncertainty in key parameters (particularly where they had not been derived from historical or care data [ 65 ]) affected model outcomes [ 23 , 25 ].

The majority of model limitations reported were concerned the use or availability of real system or case data. Huynh et al. [ 20 ], Yousefi et al. [ 67 ] and Liu et al. [ 25 ] formulated their models using data that was obtainable, such as limited sample data extracted from a pilot study [ 20 ], national average trends [ 25 ] and data from previous studies [ 67 ]. Yousefi et al. [ 22 ] case study dataset did not contain key system feedback, such as the tolerance time of patients waiting to be seen by a physician in the ED, although authors were able to extract this data from a comparable study identified in the literature.

Missing model feedback or parameters, strict model boundaries and simplification of system elements were also considered limitations. Huynh et al. [ 20 ], Hutzschenreuter et al. [ 66 ] and Einzinger et al. [ 65 ] did not model all the realistic complexities of their system, such as all possible interruptions to tasks that occur in patient care units [ 20 ], patient satisfaction of admission processes [ 66 ] (which will be addressed in future work), how treatment influences the course of disease or that morbid patients are at higher risk of developing co-morbidity than healthier patients, which would affect the service needs and consumption needs of the patient [ 65 ]. To improve the accuracy of the model, Huynh et al. stated that further research is taking place to obtain real, clinical data (as opposed to clinical simulation lab results) to assess the impact of interruptions on workflow. Liu et al.’s [ 21 ] model boundary did not include other hospital units that may have been affected by ED behaviour and they identify this as future work, for example to include hospital wards that are affected by ED behaviour. Alibrahim et al. [ 23 ] and Einzinger et al. [ 65 ] made simplifications to the health providers and networks that were modelled, such as assuming equal geographical distances and identical care services between health providers in observed networks [ 23 ], limiting the number of factors that influenced a patients decision to bypass their nearest health provider [ 65 ] and not simulating changes to health provider behaviour based on service utilisation or reimbursement scheme in place [ 23 ]. Alibrahim et al. [ 23 ] noted that although the model was constrained by such assumptions, the focus of future work would be to improve the capability of the model to accurately study the impact of patient choice on economic, health and health provider outcomes.

SDM-ABM use in health system research

A single paper used hybrid SDM-ABM to model health system behaviour. Djanatliev et al. [ 47 ] developed a tool that could be used to assess the impact of new health technology on performance indicators such as patient health and projected cost of care. A modelling method that could reproduce detailed, high granularity system elements in addition to abstract, aggregate health system variables was sought and a hybrid SDM-ABM was selected. The tool nested an agent-based human decision-making module (regarding healthcare choices) within a system dynamics environment, simulating macro-level behaviour such as health care financing and population dynamics. A case study was presented to show the potential impact of Mobile Stroke Units (MSU) on patient morbidity in Berlin, where stroke diagnosis and therapy could be initiated quickly as opposed to standard care. The model structure was deemed credible after evaluation by experts, including doctors and health economists.

Comparison of SDM and ABM papers

The similarities and differences among the SDM and ABM body of literature are described in this section and shown in Table  3 . A high proportion of papers across both modelling methods simulated systems that were concerned with emergency or acute care. A high number of SDM papers (11/28) simulated patient flow and pathways through emergency care [ 28 , 31 , 36 , 45 , 47 , 50 , 56 , 57 , 58 , 61 , 62 ] with a subset evaluating the impact of policies that relieved pressure on at capacity ED’s [ 28 , 36 , 50 , 58 , 62 ]. ABM papers simulated micro-level behaviour associated with emergency care, such as health professional and patient behaviour in EDs and what impact agent interactions have on actions taken over time [ 21 , 22 , 47 , 67 ]. ACOs and health insurance reimbursement schemes, a common modelled healthcare setting among the ABM papers [ 23 , 25 , 65 ] was the focus of a single SDM paper [ 63 ] while health care waste management, a popular healthcare setting for SDM application [ 37 , 48 , 52 , 55 ] was entirely absent among the selected ABM literature. SDM and ABM were both used to test the impact of policy on undesirable patient outcomes, including patient mortality [ 23 , 25 , 58 , 60 , 67 ] and hospitalisation rates [ 23 , 25 ]. Interventions for reducing patient waiting time for services [ 24 , 33 , 53 , 61 , 67 ] and patient length of stay [ 22 , 31 , 67 ] were also tested using these methods, while policy exploration to reduce the total cost of care was more frequent among SDM studies [ 33 , 54 , 61 ].

SDM and ABM software platforms provide accessible, user-friendly visualisations of systems that enable engagement with health experts necessary for model validation [ 48 , 50 , 55 , 57 ] and facilitate stakeholder understanding of how alternative policies can impact health system performance under a range conditions [ 31 , 49 , 58 , 61 ]. The ability to integrate information and data from various sources was also cited as rationale for using SDM and ABM [ 51 ]. Reasons for using SDM to model health systems, as opposed to other methods, included gaining a whole-system perspective crucial for investigating undesirable or counter-intuitive system behaviour across sub-systems [ 28 , 36 , 37 , 48 , 56 ] and identifying unintended consequences or policy resistance with tested health policies [ 59 , 61 ]. The ability to replicate human behaviour [ 20 , 21 , 22 , 25 , 66 ] and capacity for learning and adaptive behaviour [ 20 , 25 ] was frequently cited as rationale for using ABM to simulate health systems.

Validation of SDMs and ABMs consisted mostly of behavioural validity tests where model output was reviewed by experts and compared to real system performance data or to relevant literature. Structural validity tests were uncommon among ABM papers while expert consultation on model development [ 30 , 36 , 50 , 53 , 54 , 57 , 62 , 63 ], extreme condition [ 30 , 31 , 52 , 57 , 59 , 60 , 64 ] and dimensional consistency tests [ 31 , 52 , 57 , 59 , 60 ] were frequently reported in the SDM literature. The inability to simulate all actions or interactions that occur in the real system [ 20 , 30 , 61 , 65 , 66 ] and simplification of model parameters [ 23 , 55 , 62 , 63 , 65 ] were described as limitations in both SDM and ABM papers. Data availability for model conception and formulation [ 20 , 22 , 25 , 32 , 36 , 67 ] and the impact of model boundaries (restricting exploration of interconnected sub-system behaviour [ 21 , 31 , 53 ]) were also cited limitations common to both sets of literature. Lack of costing analysis [ 58 , 60 ], short time horizons [ 33 , 57 ] and an inability to model future improvements in technology or service delivery [ 31 , 44 , 49 , 51 ] were additionally cited among the SDM papers.

Statement of principal findings

Our review has confirmed that there is a growing body of research demonstrating the use of SDM and ABM to model health care systems to inform policy in a range of settings. While the application of SDM has been more widespread (with 28 papers identified) there are also a growing number of ABM being used (11), just over half of which used hybrid simulation. A single paper used hybrid SDM-ABM to model health system behaviour. To our knowledge this is the first review to identify and compare the application of both SDM and ABM to model health systems. The first ABM article identified in this review was published almost a decade after the first SDM paper; this reflects to a certain extent the increasing availability of SDM and ABM dedicated software tools with the developments in ABM software lagging behind their SDM modelling counterparts.

Emergency and acute care, and elderly care and LTC services were the most frequently simulated health system setting. Both sets of services are facing exponential increases in demand with constraints on resources, presenting complex issues ideal for evaluation through simulation. Models were used to explore the impact and potential spill over effects of alternative policy options, prior to implementation, on patient outcomes, service use and efficiency under various structural and financial constraints.

Strengths and weaknesses of the study

To ensure key papers were identified, eight databases across four research areas were screened for relevant literature. Unlike other reviews in the field [ 39 , 40 ], there was no restriction placed on publication date. The framework for this review was built to provide a general overview of the SDM and ABM of healthcare literature, capturing papers excluded in other published reviews as a result of strict inclusion criteria. These include reviews that have focussed specifically on compiling examples of modelled health policy application in the literature [ 35 ] or have searched for papers with a particular health system setting, such as those that solely simulate the behaviour of emergency departments [ 34 ]. One particularly comprehensive review of the literature had excluded papers that simulated hospital systems, which we have explicitly included as part of our search framework [ 39 ].

The papers presented in this review, with selection restricted by search criteria, provide a broad picture of the current health system modelling landscape. The focus of this review was to identify models of facility-based healthcare, purposely excluding literature where the primary focus is on modelling disease progression, disease transmission or physiological disorders which can be found in other reviews such as Chang et al. [ 39 ] and Long et al. [ 41 ]. The data sources or details of how data was used to conceptualise and formulate models are not presented in this paper; this could on its own be the focus of another study and we hope to publish these results as future work. This information would be useful for researchers who want to gain an understanding of the type and format of data used to model health systems and best practice for developing and validating such models.

Literature that was not reported in English was excluded from the review which may have resulted in a small proportion of relevant papers being missed. Papers that described DES models, the other popular modelling method for simulating health system processes, were not included in this review (unless DES methods are presented as part of a hybrid model integrated with SDM or ABM) but have been compiled elsewhere [ 68 , 69 , 70 ]. Finally, the quality of the papers was not assessed.

Implications for future research

A nominal number of SDM papers (9/28), an even lower proportion of ABM papers (2/11) and none of the hybrid methods papers simulated health systems based in low- or middle-income countries (LMICs). The lower number of counterpart models in LMICs can be attributed to a lack of capacity in modelling methods and perhaps the perceived scarcity of suitable data; however, the rich quantitative and qualitative primary data collected in these countries for other types of evaluation could be used to develop such models. Building capacity for using these modelling methods in LMICs should be a priority and generating knowledge of how and which secondary data to use in these settings for this purpose. In this review, we observed that it is feasible to use SDM to model low-income country health systems, including those in Uganda [ 60 ] and Afghanistan [ 30 ]. The need to increase the use of these methods within LMICs is paramount; even in cases where there is an absence of sufficient data, models can be formulated for LMICs and used to inform on key data requirements through sensitivity analysis, considering the resource and healthcare delivery constraints experienced by facilities in these settings. This research is vital for our understanding of health system functioning in LMICs, and given the greater resource constraints, to allow stakeholders and researchers to assess the likely impact of policies or interventions before their costly implementation, and to shed light on optimised programme design.

Health system professionals can learn greatly from using modelling tools, such as ABM, SDM and hybrid models, developed originally in non-health disciplines to understand complex dynamic systems. Understanding the complexity of health systems therefore require collaboration between health scientists and scientists from other disciplines such as engineering, mathematics and computer science. Discussion and application of hybrid models is not a new phenomenon in other fields but their utilisation in exploring health systems is still novel; the earliest article documenting their use in this review was published in 2010 [ 43 ]. Five of the six hybrid modelling papers [ 43 , 44 , 45 , 46 , 47 ] were published as conference proceedings (the exception Kittipittayakorn et al. [ 24 ]), demonstrating the need to include conference articles in systematic reviews of the literature in order to capture new and evolving applications of modelling for health systems research.

The configuration and extent to which two distinct types of models are combined has been described in the literature [ 71 , 72 , 73 , 74 , 75 ]. The hybrid modelling papers selected in this review follow what is described as ‘hierarchical’ or ‘process environment’ model structures, the former where two distinct models pass information to each other and the latter where one model simulates system processes within the environment of another model [ 72 ]. Truly ‘integrated’ models, considered the ‘holy grail’ [ 43 ] of hybrid simulation, where elements of the system are simulated by both methods of modelling with no clear distinction, were not identified in this review and in the wider literature remain an elusive target. In a recent review of hybrid modelling in operational research only four papers were identified to have implemented truly integrated hybrid simulation and all used bespoke software, unrestricted by the current hybrid modelling environments [ 76 ].

Of the six hybrid modelling papers, only Djanatliev et al. [ 47 ] presented a model capable of both ABM and SDM simulation. The crucial macro- and micro- level activity captured in such models represent feedback in the wider, complex system while retaining the variable behaviour exhibited by those who access or deliver healthcare. With increasing software innovation and growing demand for multi-method modelling in not only in healthcare research but in the wider research community, we need to increase their application to modelling health systems and progress towards the ‘holy grail’ of hybrid modelling.

We identified 28 papers using SDM methods and 11 papers using ABM methods to model health system behaviour, six of which implemented hybrid model structures with only a single paper using SDM-ABM. Emergency and acute care, and elderly care and LTC services were the most frequently simulated health system settings, modelling the impact of health policies and interventions targeting at-capacity healthcare services, patient length of stay in healthcare facilities and undesirable patient outcomes. A high proportion of articles modelled health systems in high income countries; future work should now turn to modelling healthcare settings in LMIC to support policy makers and health system researchers alike. The utilisation of hybrid models in healthcare is still relatively new but with an increasing demand to develop models that can simulate the macro- and micro-level activity exhibited by health systems, we will see an increase in their use in the future.

Availability of data and materials

Data sharing is not applicable to this article as no datasets were generated or analysed during the current study.

One of the elderly or LTC services papers also modelled emergency or acute care but it was not the primary focus and is therefore not discussed here.

The single SDM-ABM paper that modelled the delivery of emergency or acute care is discussed in section ‘SDM-ABM use in health system research’.

Six of the emergency or acute care review papers and one of the cardiology care papers also modelled elderly or LTC services but it was not the primary focus and are therefore not discussed here.

Abbreviations

Accountable care organisation

Agent-based model

Discrete-event simulation

Emergency Department

Long-term care

Low- and middle-income countries

System dynamics model

Kitson A, Brook A, Harvey G, Jordan Z, Marshall R, O’Shea R, et al. Using Complexity and Network Concepts to Inform Healthcare Knowledge Translation. Int J Heal Policy Manag. 2017;7:231–43. https://doi.org/10.15171/ijhpm.2017.79 .

Article   Google Scholar  

Paina L, Peters DH. Understanding pathways for scaling up health services through the lens of complex adaptive systems. Health Policy Plan. 2012;27:365–73. https://doi.org/10.1093/heapol/czr054 .

Article   PubMed   Google Scholar  

Lipsitz LA. Understanding health care as a complex system: the foundation for unintended consequences. JAMA. 2012;308:243–4. https://doi.org/10.1001/jama.2012.7551 .

Article   PubMed   PubMed Central   CAS   Google Scholar  

Adam T, de Savigny D. Systems thinking for strengthening health systems in LMICs: need for a paradigm shift. Health Policy Plan. 2012;27:iv1–3. https://doi.org/10.1093/heapol/czs084 .

de Savigny D, Blanchet K, Adam T. Applied systems thinking for health systems research : a methodological handbook: McGraw-Hill Education; 2017.

Shepherd SP. A review of system dynamics models applied in transportation. Transp B Transp Dyn. 2014;2:83–105. https://doi.org/10.1080/21680566.2014.916236 .

Sterman JD. Business dynamics: systems thinking and modeling for a complex world: McGraw-Hill Companies Inc; 2000.

Kunc M, Mortenson MJ, Vidgen R. A computational literature review of the field of system dynamics from 1974 to 2017. J Simul. 2018;12:115–27. https://doi.org/10.1080/17477778.2018.1468950 .

System Dynamics for Engineering Students. Elsevier; 2018. doi: https://doi.org/10.1016/C2011-0-05346-2 .

Radzicki MJ. System Dynamics and Its Contribution to Economics and Economic Modeling. Encycl. Complex. Syst. Sci. New York: Springer New York; 2009. p. 8990–9000. https://doi.org/10.1007/978-0-387-30440-3_539 .

Book   Google Scholar  

Ford A. Global warming and system dynamics. Int Conf Syst Dyn Soc. 2007.

Fiddaman TS. Exploring policy options with a behavioral climate-economy model. Syst Dyn Rev. 2002;18:243–67. https://doi.org/10.1002/sdr.241 .

Popli K, Sudibya GL, Kim S. A review of solid waste management using system dynamics modeling. J Environ Sci Int. 2017;26:1185–200. https://doi.org/10.5322/JESI.2017.26.10.1185 .

Groesser SN, Schaffernicht M. Mental models of dynamic systems: taking stock and looking ahead. Syst Dyn Rev. 2012;28:46–68. https://doi.org/10.1002/sdr.476 .

Epstein JM. Generative social science: studies in agent-based computational modeling: STU-Stud. Princeton University Press; 2006.

Wilensky U, Rand W. An Introduction to Agent-Based Modeling: Mit Press; 2015.

Miller JH, Page SE. Complex Adaptive Systems: An Introduction to Computational Models of Social Life: STU-Stud. Princeton University Press; 2007.

Railsback SF, Grimm V. Agent-Based and Individual-Based Modeling: Princeton University Press; 2011.

Abar S, Theodoropoulos G, Lemarinier P, O’Hare G. Agent based Modelling and simulation tools: a review of the state-of-art software. Comput Sci Rev. 2017;24:13–33. https://doi.org/10.1016/j.cosrev.2017.03.001 .

Huynh N, Snyder R, Vidal J, Tavakoli A, Cai B. Application of computer simulation modeling to medication administration process redesign. J Healthc Eng. 2012;3:649–62. https://doi.org/10.1260/2040-2295.3.4.649 .

Liu Z, Cabrera E, Rexachs D, Luque E. A generalized agent-based model to simulate emergency departments. Sixth Int Conf Adv Syst Simul. 2014:65–70.

Yousefi M, Yousefi M, Fogliatto FS, Ferreira RPM, Kim JH. Simulating the behavior of patients who leave a public hospital emergency department without being seen by a physician: a cellular automaton and agent-based framework. Brazilian J Med Biol Res. 2018;51:e6961. https://doi.org/10.1590/1414-431X20176961 .

Alibrahim A, Wu S. An agent-based simulation model of patient choice of health care providers in accountable care organizations. Health Care Manag Sci. 2018;21:131–43. https://doi.org/10.1007/s10729-016-9383-1 .

Kittipittayakorn C, Ying K-C. Using the integration of discrete event and agent-based simulation to enhance outpatient service quality in an orthopedic department. J Healthc Eng. 2016;2016:4189206. https://doi.org/10.1155/2016/4189206 .

Article   PubMed Central   Google Scholar  

Liu P, Wu S. An agent-based simulation model to study accountable care organizations. Health Care Manag Sci. 2016;19:89–101. https://doi.org/10.1007/s10729-014-9279-x .

Allen M, Spencer A, Gibson A, Matthews J, Allwood A, Prosser S, et al. Right cot, right place, right time: improving the design and organisation of neonatal care networks – a computer simulation study. Heal Serv Deliv Res. 2015;3:9. https://doi.org/10.3310/hsdr03200 .

Brailsford SC. Tutorial: Advances and challenges in healthcare simulation modeling: 2007 Winter Simul. Conf., IEEE; 2007. p. 1436–48. https://doi.org/10.1109/WSC.2007.4419754 .

Esensoy AV, Carter MW. High-Fidelity whole-system patient flow modeling to assess health care transformation policies. Eur J Oper Res. 2018;266:221–37. https://doi.org/10.1016/j.ejor.2017.09.019 .

Homer JB, Hirsch GB. System dynamics modeling for public health: background and opportunities. Am J Public Health. 2006;96:452–8. https://doi.org/10.2105/AJPH.2005.062059 .

Article   PubMed   PubMed Central   Google Scholar  

Alonge O, Lin S, Igusa T, Peters DH. Improving health systems performance in low- and middle-income countries: a system dynamics model of the pay-for-performance initiative in Afghanistan. Health Policy Plan. 2017;32:1417–26. https://doi.org/10.1093/heapol/czx122 .

Article   CAS   PubMed   PubMed Central   Google Scholar  

Rashwan W, Abo-Hamad W, Arisha A. A system dynamics view of the acute bed blockage problem in the Irish healthcare system. Eur J Oper Res. 2015;247:276–93. https://doi.org/10.1016/j.ejor.2015.05.043 .

Ghaffarzadegan N, Epstein AJ, Martin EG. Practice variation, bias, and experiential learning in cesarean delivery: A data-based system dynamics approach. Health Serv Res. 2013;48:713–34. https://doi.org/10.1111/1475-6773.12040 .

Taylor K, Dangerfield B. Modelling the feedback effects of reconfiguring health services. J Oper Res Soc. 2005;56:659–75. https://doi.org/10.1057/palgrave.jors.2601862 .

Mohiuddin S, Busby J, Savovic J, Richards A, Northstone K, Hollingworth W, et al. Patient flow within UK emergency departments: a systematic review of the use of computer simulation modelling methods. BMJ Open. 2017;7:e015007. https://doi.org/10.1136/bmjopen-2016-015007 .

Atkinson J-A, Wells R, Page A, Dominello A, Haines M, Wilson A. Applications of system dynamics modelling to support health policy. Public Heal Res Pract. 2015;25. https://doi.org/10.17061/phrp2531531 .

Lattimer V, Brailsford S, Turnbull J, Tarnaras P, Smith H, George S, et al. Reviewing emergency care systems I: insights from system dynamics modelling. Emerg Med J. 2004;21:685–91. https://doi.org/10.1136/emj.2002.003673 .

Chaerul M, Tanaka M, Shekdar AV. A system dynamics approach for hospital waste management. Waste Manag. 2008;28:442–9. https://doi.org/10.1016/j.wasman.2007.01.007 .

Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ. 2009;339:b2535. https://doi.org/10.1136/bmj.b2535 .

Chang AY, Ogbuoji O, Atun R, Verguet S. Dynamic modeling approaches to characterize the functioning of health systems: A systematic review of the literature. Soc Sci Med. 2017;194:160–7. https://doi.org/10.1016/j.socscimed.2017.09.005 .

Rusoja E, Haynie D, Sievers J, Mustafee N, Nelson F, Reynolds M, et al. Thinking about complexity in health: a systematic review of the key systems thinking and complexity ideas in health. J Eval Clin Pract. 2018;24:600–6. https://doi.org/10.1111/jep.12856 .

Long KM, Meadows GN. Simulation modelling in mental health: a systematic review. J Simul. 2017. https://doi.org/10.1057/s41273-017-0062-0 .

Banco Mundial. World Bank: country and lending groups - DataBank. World Bank Gr. 2018.

Brailsford SC, Desai SM, Viana J. Towards the holy grail: Combining system dynamics and discrete-event simulation in healthcare: Proc. 2010 Winter Simul. Conf., IEEE; 2010. p. 2293–303. https://doi.org/10.1109/WSC.2010.5678927 .

Mielczarek B, Zabawa J. Modeling Healthcare Demand Using a Hybrid Simulation Approach, vol. 2016: Proc. 2016 Winter Simul. Conf., IEEE Press. p. 1535–46.

Zulkepli J. Hybrid simulation for modelling large systems: an example of integrated care model: Proc. 2012 Winter Simul. Conf; 2012.

Viana J, Simonsen TB, Dahl FA, Flo K. A Hybrid Discrete Event Agent Based Overdue Pregnancy Outpatient Clinic Simulation Model. Proc. 2018 Winter Simul. Conf. Piscataway: IEEE Press; 2018. p. 1488–99.

Google Scholar  

Djanatliev A, German R, Kolominsky-Rabas P, Hofmann BM. Hybrid simulation with loosely coupled system dynamics and agent-based models for prospective health technology assessments: Proc. Winter Simul. Conf., Winter Simulation Conference; 2012. p. 69:1–69:12.

Al-Khatib IA, Eleyan D, Garfield J. A system dynamics approach for hospital waste management in a city in a developing country: the case of Nablus, Palestine. Environ Monit Assess. 2016;188:503. https://doi.org/10.1007/s10661-016-5487-9 .

Ansah JP, Eberlein RL, Love SR, Bautista MA, Thompson JP, Malhotra R, et al. Implications of long-term care capacity response policies for an aging population: a simulation analysis. Health Policy. 2014;116:105–13. https://doi.org/10.1016/j.healthpol.2014.01.006 .

Brailsford SC. Emergency and on-demand health care: modelling a large complex system. J Oper Res Soc. 2004;55.

Cepoiu-Martin M, Bischak DP. Policy choices in dementia care - an exploratory analysis of the Alberta Continuing Care System (ACCS) using system dynamics. Spec Issue Complex Forum “real” World Heal Syst Implic Complex. Theor Sci. 2018;24:278–84.

Ciplak N, Barton JR. A system dynamics approach for healthcare waste management: A case study in Istanbul Metropolitan City, Turkey. Waste Manag Res. 2012;30:576–86. https://doi.org/10.1177/0734242X12443405 .

de Andrade L, Lynch C, Carvalho E, Rodrigues CG, Vissoci JRN, Passos GF, et al. System dynamics modeling in the evaluation of delays of care in ST-segment elevation myocardial infarction patients within a tiered health system. PLoS One. 2014;9:e103577. https://doi.org/10.1371/journal.pone.0103577 .

Desai MS, Penn ML, Brailsford S, Chipulu M. Modelling of Hampshire adult services—gearing up for future demands. Health Care Manag Sci. 2008;11:167–76. https://doi.org/10.1007/s10729-007-9049-0 .

Eleyan D, Al-Khatib IA, Garfield J. System dynamics model for hospital waste characterization and generation in developing countries. Waste Manag Res. 2013;31:986–95. https://doi.org/10.1177/0734242X13490981 .

Lane D. Emergency but no accident: a system dynamics study of casualty waiting times in the British NHS. Eurohealth (Lond). 1998;4.

Lane DC. Looking in the wrong place for healthcare improvements: a system dynamics study of an accident and emergency department. J Oper Res Soc. 2000;51.

Mahmoudian-Dehkordi A, Sadat S. Sustaining critical care: using evidence-based simulation to evaluate ICU management policies. Health Care Manag Sci. 2017;20:532–47. https://doi.org/10.1007/s10729-016-9369-z .

Meker T, Barlas Y. Dynamic consequences of performance-based payment Systems in Public Hospitals. Syst Res Behav Sci. 2015;32:459–80. https://doi.org/10.1002/sres.2338 .

Semwanga AR, Nakubulwa S, Adam T. Applying a system dynamics modelling approach to explore policy options for improving neonatal health in Uganda. Heal Res Policy Syst. 2016;14:35. https://doi.org/10.1186/s12961-016-0101-8 .

Walker BC, Haslett T. The dynamics of local rules in hospital admission processes. Aust Health Rev. 2003;26:98–106.

Wong HJ, Wu RC, Caesar M, Abrams H, Morra D. Smoothing inpatient discharges decreases emergency department congestion: a system dynamics simulation model. Emerg Med J. 2010;27:593–8. https://doi.org/10.1136/emj.2009.078543 .

Worni M, Pietrobon R, Zammar GR, Shah J, Yoo B, Maldonato M, et al. System dynamics to model the unintended consequences of denying payment for venous thromboembolism after total knee arthroplasty. PLoS One. 2012;7:e30578. https://doi.org/10.1371/journal.pone.0030578 .

Yu W, Li M, Ge Y, Li L, Zhang Y, Liu Y, et al. Transformation of potential medical demand in China: A system dynamics simulation model. J Biomed Inform. 2015;57:399–414. https://doi.org/10.1016/j.jbi.2015.08.015 .

Einzinger P, Popper N, Breitenecker F, Pfeffer N, Jung R, Endel G. The GAP-DRG Model: Simulation of Outpatient Care for Comparison of Different Reimbursement Schemes. Proc. 2013 Winter Simul. Conf. Simul. Mak. Decis. A complex world. Piscataway: IEEE Press; 2013. p. 2299–308.

Hutzschenreuter AK, Bosman PAN, Blonk-Altena I, van Aarle J, La Poutré H. Agent-based patient admission scheduling in hospitals. Belgian/Netherlands Artif Intell Conf. 2008:315–6. https://doi.org/10.1007/3-540-32062-8 .

Yousefi M, Ferreira R. An agent-based simulation combined with group decision-making technique for improving the performance of an emergency department. Brazilian J Med Biol Res. 2017;50:1–10. https://doi.org/10.1590/1414-431X20175955 .

Günal MM, Pidd M. Discrete event simulation for performance modelling in health care: a review of the literature. J Simul. 2010;4:42–51. https://doi.org/10.1057/jos.2009.25 .

Salleh S, Thokala P, Brennan A, Hughes R, Dixon S. Discrete event simulation-based resource Modelling in health technology assessment. Pharmacoeconomics. 2017;35:989–1006. https://doi.org/10.1007/s40273-017-0533-1 .

Zhang X. Application of discrete event simulation in health care: a systematic review. BMC Health Serv Res. 2018;18:687. https://doi.org/10.1186/s12913-018-3456-4 .

Bennett PG. On linking approaches to decision-aiding: issues and prospects. J Oper Res Soc. 1985;36:659. https://doi.org/10.2307/2582261 .

Chahal K, Eldabi T. Applicability of hybrid simulation to different modes of governance in UK healthcare: 2008 Winter Simul. Conf., IEEE; 2008. p. 1469–77. https://doi.org/10.1109/WSC.2008.4736226 .

Lättilä L, Hilletofth P, Lin B. Hybrid simulation models – when, why, how? Expert Syst Appl. 2010;37:7969–75. https://doi.org/10.1016/j.eswa.2010.04.039 .

Brailsford SC. Hybrid simulation in healthcare: New concepts and new tools: 2015 Winter Simul. Conf., IEEE; 2015. p. 1645–53. https://doi.org/10.1109/WSC.2015.7408284 .

Morgan JS, Howick S, Belton V. A toolkit of designs for mixing discrete event simulation and system dynamics. Eur J Oper Res. 2017;257:907–18. https://doi.org/10.1016/j.ejor.2016.08.016 .

Brailsford SC, Eldabi T, Kunc M, Mustafee N, Osorio AF. Hybrid simulation modelling in operational research: a state-of-the-art review. Eur J Oper Res. 2018. https://doi.org/10.1016/j.ejor.2018.10.025 .

Download references

Acknowledgements

Not applicable.

The work described in this paper was funded by the Health Systems Research Initiative (HSRI). MRC Grant Reference Number: MR/R013454/1.

Author information

Authors and affiliations.

Department of Global Health and Development, London School of Hygiene and Tropical Medicine, 15-17 Tavistock Place, London, WC1H 9SH, UK

Rachel Cassidy, Neha S. Singh, Josephine Borghi & Karl Blanchet

Department of Mathematics, University College London, London, UK

Pierre-Raphaël Schiratti

Sia Partners UK, London, UK

Information Systems Department, College of Computing and Information Sciences, Makerere University, P.O. Box 7062, Kampala, Uganda

Agnes Semwanga

Ifakara Health Institute, PO Box 78373, Dar es Salaam, Tanzania

Peter Binyaruka

Department of Gender Studies, School of Humanities and Social Sciences, University of Zambia, 10101, Lusaka, Zambia

Nkenda Sachingongu

Economic and Business Research Programme, University of Zambia, Institute of Economic and Social Research, P O Box 30900, 10101, Lusaka, Zambia

Chitalu Miriam Chama-Chiliba

Department of Public Health, Environments and Society, London School of Hygiene and Tropical, London, UK

Zaid Chalabi

You can also search for this author in PubMed   Google Scholar

Contributions

RC, KB, ZC specified the search criteria and selection of databases. RC screened titles, evaluated full text articles and was responsible for full text extraction. RC is lead author, NSS, KB, ZC, PS, JB, AS, PB, CC, NS provided guidance on the draft and final manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Rachel Cassidy .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Additional file 1..

Search criteria used for each database.

Additional file 2.

Descriptive table of validation methods used in SDM and ABM literature.

Additional file 3.

Descriptive table of ABM model rules.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

Cassidy, R., Singh, N.S., Schiratti, PR. et al. Mathematical modelling for health systems research: a systematic review of system dynamics and agent-based models. BMC Health Serv Res 19 , 845 (2019). https://doi.org/10.1186/s12913-019-4627-7

Download citation

Received : 14 June 2019

Accepted : 11 October 2019

Published : 19 November 2019

DOI : https://doi.org/10.1186/s12913-019-4627-7

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• System dynamics
• Agent-based
• Health systems
• Systematic review

BMC Health Services Research

ISSN: 1472-6963

• General enquiries: [email protected]

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Wiley Open Access Collection

The use of mathematical modeling studies for evidence synthesis and guideline development: A glossary

Teegwendé v. porgo.

1 Population Health and Optimal Health Practices Research Unit, Department of Social and Preventative Medicine, Faculty of Medicine, Université Laval, Quebec, Canada

2 Department of Information, Evidence and Research, World Health Organization, Geneva, Switzerland

Susan L. Norris

Georgia salanti.

3 Institute of Social and Preventive Medicine, University of Bern, Bern, Switzerland

Leigh F. Johnson

4 Centre for Infectious Disease Epidemiology and Research, University of Cape Town, Cape Town, South Africa

Julie A. Simpson

5 Centre for Epidemiology and Biostatistics, Melbourne School of Population and Global Health, University of Melbourne, Melbourne, Australia

Matthias Egger

Christian l. althaus.

Mathematical modeling studies are increasingly recognised as an important tool for evidence synthesis and to inform clinical and public health decision‐making, particularly when data from systematic reviews of primary studies do not adequately answer a research question. However, systematic reviewers and guideline developers may struggle with using the results of modeling studies, because, at least in part, of the lack of a common understanding of concepts and terminology between evidence synthesis experts and mathematical modellers. The use of a common terminology for modeling studies across different clinical and epidemiological research fields that span infectious and non‐communicable diseases will help systematic reviewers and guideline developers with the understanding, characterisation, comparison, and use of mathematical modeling studies. This glossary explains key terms used in mathematical modeling studies that are particularly salient to evidence synthesis and knowledge translation in clinical medicine and public health.

1. INTRODUCTION

Mathematical models are increasingly used to aid decision making in public health and clinical medicine. 1 , 2 The results of mathematical modeling studies can provide evidence when a systematic review of primary studies does not identify sufficient studies to draw conclusions or to support a recommendation in a guideline, or when the studies that are identified do not apply to the specific populations of interest or do not provide data on long‐term follow‐up or on relevant outcomes. For example, mathematical models have been used to inform guideline recommendations about tuberculosis (TB) control in health care facilities, 3 blood donor suitability with regard to human T‐cell leukemia virus type I (HTLV‐I) infection, 4 and cancer screening. 5 , 6 Mathematical modeling studies are frequently used to synthesize evidence from multiple data sources to address a clinical or public health question not directly addressed by a primary study. For example, a mathematical model was used to synthesize evidence obtained from virological, clinical, epidemiological, and behavioral data to help determine optimal target populations for influenza vaccination programs. 7 Other examples are mathematical modeling studies that aim to predict the real‐world drug effectiveness from randomized controlled trial (RCT) efficacy data (reviewed in Panayidou et al 7 ).

The development of methods for incorporating mathematical modeling studies into evidence syntheses and clinical and public health guidelines is still at an early stage. Systematic reviewers and guideline developers struggle with questions about whether and how to include the results of mathematical modeling studies into a body of evidence. The review of mathematical modeling studies predicting drug‐effectiveness from RCT data identified 12 studies using four different modeling approaches. 7 Because of the varying use of key terminology between studies, and because certain terms can have different meanings in the literature, it was necessary to describe in the review each modeling approach in detail to illustrate the differences between them. This effort highlights an important reason for the challenges in summarizing the results of mathematical modeling studies. Researchers who develop and analyze mathematical models have different theoretical and practical backgrounds from systematic reviewers, guideline developers, and policy makers, which can result in a lack of a common understanding of concepts and terminology. These communication issues might result either in not using the findings of mathematical modeling studies in evidence synthesis and to inform decision making, or accepting these findings without critical assessment. 8 A glossary of commonly used terms in mathematical modeling studies that are relevant to evidence synthesis and to clinical and public health guideline development could improve the use of such studies.

A mathematical model is a “mathematical framework representing variables and their interrelationships to describe observed phenomena or predict future events.” 9 We define a mathematical modeling study as a study that uses mathematical modeling to address specific research questions, for example, the impact of interventions in health care facilities to reduce nosocomial transmission of TB. 10 For the modeling studies that are most relevant to evidence synthesis and clinical and public health decision‐making, the framework of the mathematical model represents interrelationships among exposure risks, interventions, health outcomes, and health costs (all of these are variables ) where their interrelationships are typically described by the parameters of interest. Mathematical modelers can use different methods to specify these parameters; they can use theoretical values, values reported in the scientific literature, or estimate the parameters from data using methods from statistical modeling. There is some overlap between the terms “mathematical model” and “statistical model” and their uses. Contemporary mathematical modeling studies increasingly include one or more statistical modeling parts. In this glossary, we will consider statistical models as a class of mathematical models that are often integrated into complex mathematical modeling studies to relate the model output to data through a statistical framework.

The goal of this glossary is to provide a common terminology for public health specialists who would like to incorporate the results of mathematical modeling studies in systematic reviews and in the development of guidelines. To identify the terms included in this glossary, we first made an exhaustive list of terms related to mathematical models. Terms were then selected based on discussions among experts attending the World Health Organization's (WHO) consultation on the development of guidance on how to incorporate the results of modeling in WHO guidelines (Geneva, Switzerland, 26 April 2016). Experts included epidemiologists, statisticians, mathematical modelers, and public health specialists. The glossary is divided into three sections. In Section 2 , we define some key terms that can be used to characterize the scope of and approach to mathematical models, using examples from the field of infectious disease modeling. In Section 3 , we present a list of terms that are commonly used across different research fields in epidemiology to describe more detailed technical properties and aspects of mathematical models. In Section 4 , we first discuss how knowledge of the terms can help to assess whether a mathematical modeling study is appropriate for providing evidence for a specific question. We then use the example of the World Health Organization (WHO) guidelines for TB control in health care facilities 3 to show how mathematical modeling studies can inform recommendations. For more specific definitions of terms that are primarily used in infectious disease modeling, we refer to the glossary by Mishra et al. 11 Terms appearing in italics are defined in other entries of the glossary.

2. TERMS USED TO DEFINE THE SCOPE OF, AND APPROACHES TO MATHEMATICAL MODELS

Before one starts to assess and compare the results of different mathematical modeling studies with each other, it can be helpful to fit them into a larger picture. Experts in systematic reviews and guideline developers need to be able to sort out which modeling studies are likely to help them draw a conclusion, formulate a recommendation, interpret the findings of another study, or understand the clinical or pathological background to a problem. Mathematical modeling studies can be characterized using several dichotomies that help to describe broad aspects, such as the scope and approach. Table  1 provides a list of some important model dichotomies, together with a brief definition, an example, and their relevance to systematic reviews and guideline development.

Model dichotomies describing the scope of, and approaches to, mathematical models in infectious disease epidemiology

A fundamental distinction can be made between mechanistic and phenomenological models . Mechanistic models use mathematical terms to describe the real‐world interactions among different model variables. The parameters governing these models typically have a physical, biological or behavioral interpretation. Infectious disease models, for example, can describe the movement of individuals within hospital wards, and how infections are transmitted upon physical contact between a susceptible and an infected person. 10 These models have the advantage that specific interventions, such as infection prevention through quarantine or isolation, can be explicitly implemented. Phenomenological models, on the other hand, describe the relationships among different model variables, consistent with fundamental theory, but not derived from first principles. Hence, this type of model does not attempt to describe or explain why and how certain model variables interact, but instead, focuses on the functional relationship that best describes an observed phenomenon. Statistical models, such as regression models, are typically phenomenological and describe the statistical relationship or association between different model variables.

A predictive model can forecast future events, such as the course of an epidemic in a given population under different scenarios, whereas a descriptive model describes and/or explains previously observed phenomena, such as the effectiveness of past interventions. Quantitative models provide a numerical estimation of an intervention effect on model variables, and therefore depend on high‐quality data to inform the model parameters. Qualitative models are usually relatively simple models that only provide insights into the direction of an effect, but not its precise magnitude. Nevertheless, they can be used to thoroughly investigate the interrelationships between model variables and the influence of specific parameters on health outcomes (also see Analytic solution ). Qualitative models can also be useful to explore the potential for unintended consequences of interventions beyond the direct intended effects that might have been observed in RCTs. Finally, an important model dichotomy distinguishes between what drives the results of mathematical modeling studies. Most mathematical models incorporate a combination of some underlying theory, model assumptions, and data. The results of a theory‐driven model are primarily based on a priori knowledge or assumptions about specific interrelationships, such as the effectiveness of a particular intervention, and are not directly inferred from data. Data‐driven models infer their results primarily from data, and are not driven by theory or assumptions that are not well supported.

3. TERMS RELATED TO TECHNICAL PROPERTIES AND ASPECTS OF MATHEMATICAL MODELS

3.1. technical terms related to model development and structure.

Once the mathematical modeling studies have been broadly characterized, and their purpose has been determined, it is important to gain a better understanding about some of the terms used to describe the technical aspects of the model used in a study. For example, has heterogeneity among different individuals been incorporated, or what simulation methods were used to obtain the model results? The following list includes some of the most frequently used terms in mathematical modeling studies in various fields of epidemiology. The terms in Section 3.1 will help in assessing the technical aspects that relate to model development and structure. The terms in Section 3.2 are related to model calibration and validation.

3.1.1. Agent‐based model

See Individual‐based model .

3.1.2. Analytic solution

Relates the health outcomes directly to the model parameters using mathematical formulae. Models that can be solved analytically are usually simple models, while more complex models typically require a computational (numerical) solution .

3.1.3. Assumption

In mathematical modeling studies, assumptions typically relate to the structure of the model and the supposed interrelationships of model variables. An important assumption in infectious disease models concerns the way in which individuals have contacts with each other. This could either be at random or involve some form of heterogeneity. In order to relate the model output to data via a statistical framework, one has to make additional assumptions about the way the data has been gathered and the expected random error.

3.1.4. Compartmental model

This model type stratifies the population into different compartments, such as different health states. Compartments are assumed to represent homogeneous subpopulations within which the entities being modeled–such as individuals or patients–have the same characteristics, for example the same sex, age, risk of infection, or death. The model can account for the transition of entities between compartments (see State‐transition model ).

3.1.5. Computational (numerical) solution

This describes the approach of solving a mathematical model using either deterministic or stochastic (see Monte Carlo methods ) simulation techniques to iteratively calculate the model variables, which are often time‐dependent, for a specific set of parameters. Iteratively calculating the model variables means updating the population characteristics at each time point based on the simulated population characteristics at previous time points. Computational solutions are used when the model is too complicated for deriving an analytic solution.

3.1.6. Continuous‐time model

This is a dynamic model where time is treated as a continuous variable (in contrast to a discrete‐time model ), meaning that the state or value of all other variables (or health outcomes) can be calculated for any time point of interest.

3.1.7. Cycle length

In a discrete‐time model, cycle length represents the interval from one time point to the next, for example a specific number of days, weeks, months, or years. 7

3.1.8. Decision analytic model

This term refers to mathematical models that synthesize available evidence to estimate health outcomes and guide decision making. The term is typically used in health economic analyses.

3.1.9. Deterministic model

This model type typically describes the average behavior of a system (eg, populations or subpopulations) without taking into account stochastic processes or chance events in single entities (eg, individuals). Hence, such models are typically applied to situations with a large number of individuals where stochastic variation becomes less important and heterogeneity can be accounted for using various subpopulations. The parameters of a deterministic model are typically fixed, and a simulation always produces the same result. Deterministic models are typically easier to calibrate to data than stochastic models. 11 , 20

3.1.10. Discrete‐time model

This type of dynamic model treats time as a discrete variable (in contrast to a continuous‐time model ) and other variables (or health outcomes) can only change at specific time points. 7

3.1.11. Dynamic model

A dynamic model contains at least one time‐dependent variable. 11 This type of model is used to describe and predict the course of health outcomes (eg, infection incidence) over time when, for example, the exposure risk (eg, infection prevalence) also changes over time.

3.1.12. Heterogeneity

In mathematical modeling studies, this typically describes the differences among individuals, or the variability across parameter values for a specific group of individuals, because of their demographic, biological, or behavioral characteristics.

3.1.13. Individual‐based model

This is a stochastic model representing individuals as discrete entities with unique characteristics. An individual‐based model can be useful to accommodate heterogeneity in a given population. Individual‐based models are also often referred to as agent‐based or micro‐simulation models . While individual‐based models can provide more realistic representations of a system, they can be difficult to parameterize because they require much more detailed knowledge, or assumptions, of how variables interact. The stochastic nature of these models makes them computationally intensive and challenging to calibrate.

3.1.14. Markov model

A Markov model assumes that the future state of variables depends only on the current state, but not the previous states, of variables. For example, in a discrete‐time Markov model, the number of new infected individuals is calculated based on the total number of infected individuals at the previous time step.

3.1.15. Micro‐simulation model

See Individual‐based model.

3.1.16. Monte Carlo methods

These are a class of computational methods that are based on random sampling. Monte Carlo methods are typically used to simulate stochastic models and are computationally intensive.

3.1.17. Ordinary differential equations

Equations that describe the change of a dependent variable, with respect to an independent variable, based on differential calculus. For example, ordinary differential equations can be used to describe the increase and decrease of infected individuals in continuous time resulting from acquisition or clearance of infection. Ordinary differential equations are typically used for deterministic and compartmental models.

3.1.18. Parameter

A parameter is a quantity used to describe the interrelationships between model variables. For example, parameters can describe how long different individuals reside in different health states, or how likely they are to transmit a disease to another person. There are different methods to specify the value of parameters. Mathematical modelers can either choose theoretical values based on specific assumptions, or set the values based on literature reviews or model calibration.

3.1.19. Parsimonious model

In a parsimonious model , descriptive or predictive, the number of assumptions, parameters and variables is minimized. Parsimonious models are often relatively simple, but they can also become more complex if they achieve the right balance between complexity and explanatory power.

3.1.20. Population‐based model

A type of deterministic or stochastic model where individuals that share the same characteristics, on average, are being grouped into a single population or several subpopulations. In contrast, an individual‐based model treats every individual as a single entity that can have unique characteristics.

3.1.21. State‐transition model

State‐transition models assume that individuals can be in different (health) states and move (transition) between them. 21 They are typically described using the framework of either Markov models or individual‐based models.

3.1.22. Static model

In a static model , all variables are independent of time and constant. A static model typically describes the equilibrium of a system, and relates the model variables for a particular time point only. In contrast to dynamic models, this type of model cannot take into account time‐dependent changes of exposure risks or health outcomes. Decision‐tree models are static models.

3.1.23. Stochastic model

A type of model where the parameters, variables, and/or the change in variables can be described by probability distributions. This type of model can account for process variability by taking into account the random nature of variable interactions, or can accommodate parameter uncertainty, and so may predict a distribution of possible health outcomes. Considering process variability can be particularly important when populations are small or certain events are very rare. Stochastic models are often simulated using Monte Carlo methods.

3.1.24. Time horizon

A time horizon denotes a chosen time at which point the effect of an intervention will be evaluated. The time horizon should reflect the health outcomes and the relevant intermediate and long‐term effects of an intervention. 1

3.1.25. Variable

Variables describe model elements such as exposure risks, interventions, or health outcomes that can vary between settings or over time. The value of a dependent variable (eg, number of infected individuals) changes in relation to an independent variable (eg, time).

3.2. Technical terms related to model calibration and validation

3.2.1. calibration.

Calibration is the process of adjusting model parameters, such that the model output is in agreement with the data that are used for model development. 22 The aim of calibration is to reduce parameter uncertainty in order to achieve high model credibility .

3.2.2. Credibility

The credibility of a model refers to judgments about the degree to which the model provides trustworthy results. Several dimensions of credibility have been described, including validity , design, data analysis, reporting, interpretation, and conflicts of interest. 16

3.2.3. Sensitivity analysis

A range of techniques used to test the impact of the assumptions made about the parameters. The analysis can be done by changing one parameter (one‐way, univariate), or simultaneously changing several parameters (multi‐way, multivariate). The parameters selected for sensitivity analyses are thought to have an impact on the outcome of interest. In a deterministic sensitivity analysis, a parameter is assigned a limited number of values, while in a probabilistic sensitivity analysis, each parameter is assigned a probability distribution, and parameter values are randomly sampled from these distributions. 1 , 11

3.2.4. Uncertainty analysis

A range of techniques to determine the reliability of model results or predictions, accounting for uncertainty in model structure, input parameters, and/or methods used for data analysis. 11 Structural uncertainty relates to the extent to which the structure of the model captures the key features of the system 23 , 24 , 25 and can be analyzed by comparing the results of models with different structures. Parameter uncertainty stems from the model parameters that are used, but whose true values are not known because of measurement error or an absence of evidence. 23 , 25 This uncertainty can be analyzed by examining model outputs for a range of values of the parameter. Methodological uncertainty arises when there are different methods for analyzing or expressing model outputs. This term is used mostly in health economic modeling.

3.2.5. Validation

A term describing processes for assessing how well a model performs and how applicable the results are to a particular situation. 26 There are five main types of validation: face validation (subjective expert judgment about how well the model represents the problem it addresses); internal validation (internal consistency, verification, and addresses whether or not the model behaves as intended and has been implemented correctly); cross validation ( convergent validity, model results are confirmed by other models); external validation (model results predict outcomes obtained in a real world setting or in a data set different from the one used for model development); predictive validation (model‐predicted events are later corroborated by real‐world observations). 7 , 27

4. MATHEMATICAL MODELING STUDIES IN GUIDELINE DEVELOPMENT

In addition to providing a useful common terminology for public health specialists and mathematical modelers, the description of different model types and other terms defined in the glossary facilitate interpretation of the results of mathematical modeling studies and inform their incorporation into the guideline development process. As a first step, one needs to identify whether a particular research question, eg, the evaluation of public health programs, long‐term effectiveness or comparative effectiveness, can be investigated using a model. Next, it will be necessary to assess whether existing mathematical modeling studies are appropriate to inform or support a research question or recommendation. We identified four comprehensive frameworks of good modeling practice. 28 These frameworks cover items such as relevance, conceptualization of the problem, or model structure. Questions such as whether the model population is relevant, the variables represent the desired health outcomes, the necessary heterogeneity is taken into consideration, the time horizon is appropriate, or the assumptions justified can help in the assessment of mathematical modeling studies. Other items concern validity or consistency, ie, the performance of the model according to its specifications. The model should also consider uncertainty with regard to the structure, parameters, and methods. Finally, credibility, which takes a number of these items into account, can then be used as the central concept for guideline developers to address the appropriateness of a mathematical modeling study for providing evidence for a specific question, 29 as illustrated in the following example.

Prevention of TB transmission in hospitals, and particularly of multidrug‐resistant TB, is essential in all countries and requires a combination of strategies. Predicting the spread of TB in a hospital and the surrounding community, and how alternative methods of control might limit the emergence of resistance, are complex nonlinear processes. It is, however, ethically and logistically impossible to conduct RCTs to examine the efficacy of these strategies. Mathematical modeling studies that use observational evidence can therefore play an important role in deciding which strategies are likely to be the most effective. The WHO guideline development group for TB infection control in health care facilities, congregate settings and households assessed systematic reviews of the evidence, which included mathematical modeling studies. 3

One mathematical modeling study that the guideline committee considered, investigated the effects of several different control measures on the spread of extensively drug resistant (XDR) TB in a community in South Africa. 10 The model described the transmission of TB in a complex system that included variables representing or contributing to: both the hospital and the surrounding community; different TB health states such as susceptible, latent, infectious, and recovered; drug resistance; HIV infection; and the effects of different control interventions alone and in combination. Hence, the study considered the transmission setting that was of relevance to the guideline, and the model structure included the desired health outcomes and variables. The authors used a mechanistic approach to make explicit the way in which stages in the transmission and natural history of TB are related. A deterministic, compartmental model , using ordinary differential equations to describe the transitions between different health states in a dynamic way was appropriate because it allowed the right balance between complexity and tractability. Key parameters that described the natural history, such as rate of natural clearance and rate of relapse, were based on the literature, and their influence was assessed in an uncertainty analysis. Parameters such as the transmissibility coefficient were calibrated using longitudinal data of individuals in a South African community, where data on TB were collected. The model outputs provided quantitative predictions about the percentage reduction in XDR‐TB cases over a reasonable time horizon. External validation of the model was performed using cross‐sectional data with information on the prevalence of TB and of drug resistance and the proportion of resistance cases in people with HIV infection. In summary, the mathematical modeling study covered many of the critical items, and we would conclude that the study has a high credibility.

Compared with natural ventilation, the authors found that mechanical ventilation alone would reduce XDR‐TB cases by 12% (range 10%‐25%). The use of respiratory masks by health workers would prevent 2% of all TB cases, but nearly two‐thirds of XDR cases in hospital staff. The guideline development group considered this study, together with other observational and modeling studies identified through the systematic review. Even though the summarized evidence for the use of ventilation systems and particulate respirators was weak, indirect, and of low quality, the studies suggested that these interventions are favorable for TB infection control.

CONFLICT OF INTEREST

CA, ME, and NL received funding through a grant from the Special Programme for Research and Training in Tropical Diseases (TDR) to conduct this study. SLN is a member of the GRADE Working Group which develops processes and methods for guideline development. LFJ, TVP, GS, and JAS have no interests to disclose.

The authors alone are responsible for the views expressed in this article and they do not necessarily represent the views, decisions, or policies of the institutions with which they are affiliated.

ACKNOWLEDGEMENTS

This work was funded through a grant from the Special Programme for Research and Training in Tropical Diseases (TDR). JA Simpson is funded by a NHMRC Senior Research Fellowship (1104975) and her research is supported by two NHMRC Centres of Research Excellence (Victoria Centre for Biostatistics, ViCBiostat; and Policy relevant infectious disease simulation and mathematical modeling, PRISM).

Porgo TV, Norris SL, Salanti G, et al. The use of mathematical modeling studies for evidence synthesis and guideline development: A glossary . Res Syn Meth . 2019; 10 :125–133. 10.1002/jrsm.1333 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]

Advertisement

Mathematical modeling for theory-oriented research in educational technology

• Development Article
• Published: 29 November 2021
• Volume 70 , pages 149–167, ( 2022 )

Cite this article

• Elena Novak   ORCID: orcid.org/0000-0003-0873-2081 1  

868 Accesses

2 Altmetric

Explore all metrics

Mathematical modeling describes how events, concepts, and systems of interest behave in the world using mathematical concepts. This research approach can be applied to theory construction and testing by using empirical data to evaluate whether the specific theory can explain the empirical data or whether the theory fits the data available. Although extensively used in the physical sciences and engineering, as well as some social and behavioral sciences to examine theoretical claims and form predictions of future events and behaviors, theory-oriented mathematical modeling is less common in educational technology research. This article explores the potential of using theory-oriented mathematical modeling for theory construction and testing in the field of educational technology. It presents examples of how this approach was used in social, behavioral, and educational disciplines, and provides rationale for why educational technology research can benefit from a theory-oriented model-testing approach.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save.

• Get 10 units per month
• Download Article/Chapter or eBook
• 1 Unit = 1 Article or 1 Chapter
• Cancel anytime

Price includes VAT (Russian Federation)

Instant access to the full article PDF.

Rent this article via DeepDyve

Institutional subscriptions

Adapted from Bessière et al. ( 2006 )

Adapted from Anderson ( 2011 )

Adapted from Picciano ( 2017 )

Adapted from Means et al. ( 2014 )

Similar content being viewed by others

The Philosophy of Science and Educational Technology Research

The instrumental value of conceptual frameworks in educational technology research.

Educational design research: grappling with methodological fit

Explore related subjects.

• Artificial Intelligence
• Digital Education and Educational Technology

Anderson, T. (2008). The theory and practice of online learning (2nd ed.). AU Press.

Google Scholar  

Anderson, T. (2011). The theory and practice of online learning . AU Press.

Atkinson, R. C., & Schiffrin, R. M. (1971). The control of short-term memory. Scientific American, 225 , 82–90.

Atkinson, R. C., & Shiffrin, R. M. (1968a). Human memory: A proposed system and its control processes. In K. W. Spence & J. T. Spence (Eds.), The psychology of learning and motivation (Volume 2) (pp. 89–195). Academic Press.

Atkinson, R. C., & Shiffrin, R. M. (1968b). Human memory: A proposed system and its control processes. In K. W. Spence & J. T. Spence (Eds.), The psychology of learning and motivation: Advances in research and theory (Vol 2) (pp. 89–195). Academic Press.

Bessière, K., Newhagen, J. E., Robinson, J. P., & Shneiderman, B. (2006). A model for computer frustration: The role of instrumental and dispositional factors on incident, session, and post-session frustration and mood. Computers in Human Behavior, 22 (6), 941–961. https://doi.org/10.1016/j.chb.2004.03.015

Article   Google Scholar  

Blömeke, S., & Kaiser, G. (2011). Homogeneity or heterogeneity? Profiles of opportunities to learn in primary teacher education and their relationship to cultural context and outcomes. ZDM Mathematics Education, 44 (3), 249–264. https://doi.org/10.1007/s11858-011-0378-6

Boland, L. A. (2014). Model building in economics: Its purposes and limitations . Cambridge University Press.

Bulfin, S., Henderson, M., Johnson, N. F., & Selwyn, N. (2014). Methodological capacity within the field of “educational technology” research: An initial investigation. British Journal of Educational Technology, 45 (3), 403–414.

Cilesiz, S., & Spector, J. M. (2014). The philosophy of science and educational technology research. In J. M. Spector, M. D. Merrill, J. Elen, & M. J. Bishop (Eds.), Handbook of research on educational communications and technology (pp. 875–884). New York: Springer.

Cronbach, L. J. (1975). Beyond the two disciplines of scientific psychology. American Psychologist, 20 , 116–117.

Dabbaghian, V., & Mago, V. K. (2014). Theories and simulations of complex social systems . Springer.

Dym, C. (2004). Principles of mathematical modeling . Academic Press.

Ertmer, P. A. (1999). Addressing first- and second-order barriers to change: Strategies for technology integration. Educational Technology Research and Development, 47 (4), 47–61.

Garrison, D. R., Anderson, T., & Archer, W. (2000). Critical inquiry in a text-based environment: Computer conferencing in higher education model. The Internet and Higher Education, 2 (2–3), 87–105.

Goe, L. (2007). The link between teacher quality and student outcomes: A research synthesis . Washington, DC: National Comprehensive Center for Teacher Quality. Retrieved from http://ncctq.learningpt.org/publications/LinkBetweenTQandStudentOutcomes.pdf

Gollwitzer, P. M. (1999). Implementation intentions. Strong effects of simple plans. American Psychologist, 54 (7), 493–503.

Harasim, L. (2012). Learning theory and online technologies . Routledge/Taylor & Francis.

Hew, K. F., & Brush, T. (2007). Integrating technology into K-12 teaching and learning: Current knowledge gaps and recommendations for future research. Education Technology Research and Development, 55 , 223–252. https://doi.org/10.1007/s11423-006-9022-5

Hew, K. F., Lan, M., Tang, Y., Jia, C., & Lo, C. K. (2019). Where is the “theory” within the field of educational technology research? British Journal of Educational Technology . https://doi.org/10.1111/bjet.12770

Hoffman, B. (2010). “I think I can, but I’m afraid to try”: The role of self-efficacy beliefs and mathematics anxiety in mathematics problem-solving efficiency. Learning and Individual Differences, 20 (3), 276–283.

Hoffman, R. (2003). Why buy that theory? In O. Sacks (Ed.), The best American science writing: 2003 (pp. 222–227). Harper-Collins.

Hollander, E. P. (1967). Principles and methods of social psychology . Oxford University Press.

Holmberg (1985).  The feasibility of t heory of teaching for distance education and a proposed theory (ZIFF Paiere 60). Hagen, West Germany: Fern Universitat, Zentrales Institute fur Fernstudienforscgung Arbeitsbereich. (ERIC Document Reproduction Service No. ED290013).

Huang, W.-H., Huang, W.-Y., & Tschopp, J. (2010). Sustaining iterative game playing processes in DGBL: The relationship between motivational processing and outcome processing. Computers & Education, 55 (2), 789–797.

Jaccard, J., & Jacoby, J. (2009). Theory construction and model building skills: A practical guide for social scientists . Guilford Press.

Johns, G. (2006). The essential impact of context on organizational behavior. Academy of Management Review, 31 (2), 386–408.

Jones, C., & Czerniewicz, L. (2011). Theory in learning technology. Research in Learning Technology, 19 (3), 173–177.

Keller, J. M. (1999). Using the ARCS motivational process in computer-based instruction and distance education. New Directions for Teaching and Learning, 78 , 39–48.

Keller, J. M. (2008). An integrative theory of motivation, volition, and performance. Technology, Instruction, Cognition and Learning, 16 , 79–104.

Knowles, M. S., Holton, E. F., & Swanson, R. A. (1998). The adult learner (5th ed.). Butterworth-Heinemann Publishers.

Kuhl, J. (1987). Action control: The maintenance of motivational states. In F. Halisch & J. Kuhl (Eds.), Motivation, intention and volition (pp. 279–291). Springer.

Kuhn, T. S. (1996). The structure of scientific revolutions . University of Chicago Press.

Malone, T. W. (1985). Designing organizational interfaces. In L. Borman & R. Smith (Eds.), Proceedings of the CHI’85 Conference on Human Factors in Computing Systems (pp. 66–71). New York, NY: ACM Press.

Markauskaite, L., & Reimann, P. (2014). Editorial: e-Research for education: Applied, methodological and critical perspectives. British Journal of Educational Technology, 45 (3), 385–391.

Mayer, R. E. (2001). Multimedia learning . Cambridge University Press.

McDonnell, L. M. (1995). Opportunity to learn as a research concept and a policy instrument. Educational Evaluation and Policy Analysis, 17 (3), 305–322.

McKenney, S., & Reeves, T. C. (2014). Educational design research. In J. M. Spector, M. D. Merrill, J. Elen, & M. J. Bishop (Eds.), Handbook of research on educational communications and technology (pp. 131–140). Springer.

Means, B., Bakia, M., & Murphy, R. (2014). Learning online: What research tells us about whether, when and how . Routledge.

Means, B., Toyama, Y., Murphy, R., & Baki, M. (2013). The effectiveness of online and blended learning: A meta-analysis of the empirical literature. Teachers College Record, 115 , 1–47.

Mintzberg, H. (2005). Developing theory about the development of theory. In M. Hitt & K. Smith (Eds.), Minds in management: The process of theory development (pp. 355–372). Oxford University Press.

Moore, M. G. (1997). Theory of transactional distance. In D. Keegan (Ed.), Theoretical principles of distance education (pp. 22–38). Routledge.

Moore, M. G., & Diehl, W. C. (2018). Handbook of distance education . Routledge.

Morgan, C., & Wildemuth, B. M. (2009). Questions related to theory. Applications of social research methods to questions in information and library science (pp. 40–50). Libraries Unlimited.

Nelson, R., & Winter, S. (1974). Neoclassical vs. evolutionary theories of economic growth: Critique and prospectus. Economic Journal, 84 (336), 886–905.

Novak, E. (2014). Toward a mathematical model of motivation, volition, and performance. Computers & Education, 74 , 73–80. https://doi.org/10.1016/j.compedu.2014.01.009

Novak, E., Daday, J., & McDaniel, K. (2018). Using a mathematical model of motivation, volition, and performance to examine students’ e-text learning experiences. Educational Technology Research & Development, 66 (5), 1189–1209. https://doi.org/10.1007/s11423-018-9599-5

Novak, E., McDaniel, K., Daday, J., & Soyturk, I. (2021). Understanding student frustration with e-learning materials: Development and validation of an E-Text Frustration scale . Featured Research Paper presented at the Association for Educational Communications and Technology (AECT), Chicago, IL. November 2021.

Opp, K.-D. (1970). Theories of the middle range as a strategy for the construction of a general sociological theory. Quality and Quantity, 4 (2), 243–253. https://doi.org/10.1007/BF00199565

Pearl, J. (2000). Causality: Models, reasoning, and inference . Cambridge University Press.

Picciano, A. G. (2017). Theories and frameworks for online education: Seeking an integrated model. Online Learning, 21 (3), 166–190. https://doi.org/10.24059/olj.v21i3.1225

Puntambekar, S., et al. (2018). Design-based research. In F. Fisher (Ed.), International handbook of the learning sciences (pp. 383–392). Routledge.

Qian, H., & Youngs, P. (2016). The effect of teacher education programs on future elementary mathematics teachers’ knowledge: A five-country analysis using TEDS-M data. Journal of Mathematics Teacher Education, 19 (4), 371–396. https://doi.org/10.1007/s10857-014-9297-0

Reeves, T. C. (2006). Design research from the technology perspective. In J. V. Akker, K. Gravemeijer, S. McKenney, & N. Nieveen (Eds.), Educational design research (pp. 86–109). Routledge.

Rodgers, J. L. (2003). EMOSA sexuality models, memes, and the tipping point: Policy and program implications. In D. Romer (Ed.), Reducing adolescent risk: Toward an integrated approach (pp. 185–192). Sage.

Rodgers, J. L. (2010). The epistemology of mathematical and statistical modeling: A quiet methodological revolution. American Psychologist, 65 (1), 1–12.

Rodgers, J. L., & Doughty, D. (2001). Does having boys or girls run in the family? Chance, 14 , 8–13.

Rodgers, J. L., & Rowe, D. C. (1993). Social contagion and adolescent sexual behavior: A developmental EMOSA model. Psychological Review, 100 , 479–510. https://doi.org/10.1037/0033-295X.100.3.479

Rowe, D. C., & Rodgers, J. L. (1991). Adolescent smoking and drinking: Are they epidemics? Journal of Studies on Alcohol, 52 , 110–117.

Scheerens, J., & Blömeke, S. (2016). Integrating teacher education effectiveness research into educational effectiveness models. Educational Research Review, 18 , 70–87. https://doi.org/10.1016/j.edurev.2016.03.002

Seel, N. M. (2009). Bonjour tristesse: Why don’t we research as we have been taught? Methodological considerations on instructional technology research. Technology, Instrustion, Cognition and Learning, 6 , 151–176.

Shadish, W., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference . Houghton Mifflin.

Shepard, R. N. (1982). Geometrical approximations to the structure of musical pitch. Psychological Review, 89 , 305–333.

Siemens, G. (2004). Connectivism: A learning theory for the digital age. International Journal of Instructional Technology and Distance Learning, 2 , 1–8.

Simon, H. A., & Newell, A. (1956). Models: Their uses and limitations. In D. White (Ed.), The state of the social sciences (pp. 61–83). University of Chicago Press.

Straub, D. W. (2009). Editor’s comments: Why top journals accept your paper. MIS Quarterly, 33 (3), iii–x.

Suppes, P. (1978). Impact of research on education: Some case studies . National Academy of Education.

Tatto, M. T., Schwille, J., Senk, S., Ingvarson, L., Peck, R., & Rowley, G. (2008). Teacher education and development study in mathematics (TEDS-M): Policy, practice, and readiness to teach primary and secondary mathematics. Conceptual framework . Teacher Education and Development International, Study Center, College of Education, Michigan State University.

Tipton, E., & Olsen, R. B. (2018). A review of statistical methods for generalizing from evaluations of educational interventions. Educational Researcher, 47 (8), 516–524. https://doi.org/10.3102/0013189x18781522

Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003). User acceptance of information technology: Toward a unified view. MIS Quarterly, 27 (3), 425–478.

Venkatesh, V., Thong, J. Y. L., & Xu, X. (2016). Unified theory of acceptance and use of technology: A synthesis and the road ahead. Journal of the Association for Information Systems, 17 (5), 328–376.

Wedemeyer, C. A. (1981). Learning at the back door: Reflections on non-traditional learning in the lifespan . The University of Wisconsin Press.

Wenger, E. (1998). Communities of practice: Learning, meaning, and identity . Cambridge University Press.

Wenger, E., & Lave, J. (1991). Learning in doing: Social, cognitive and computational perspectives. Situated learning: Legitimate peripheral participation . Cambridge University Press.

Willingham, D. (2008). What is developmentally appropriate? American Educator, 32 (2), 34–39.

Zawacki-Richter, O., Bäcker, E., & Vogt, S. (2009). Review of distance education research (2000 to 2008): Analysis of research areas, methods, and authorship patterns. International Review of Research in Open and Distributed Learning, 10 (6), 21–50. https://doi.org/10.19173/irrodl.v10i6.741

Zimmerman, B. J. (2001). Theories of self-regulated learning and academic achievement: An overview and analysis. In B. J. Zimmerman & D. H. Schunk (Eds.), Self-regulated learning and academic achievement. Theoretical perspectives (pp. 1–38). Mahwah.

Download references

Author information

Authors and affiliations.

School of Teaching, Learning and Curriculum Studies, Kent State University, 150 Terrace Drive, P.O. Box 5190, Kent, OH, 44242-0001, USA

Elena Novak

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Elena Novak .

Ethics declarations

Conflict of interest.

The author declares that she has no conflict of interest.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Novak, E. Mathematical modeling for theory-oriented research in educational technology. Education Tech Research Dev 70 , 149–167 (2022). https://doi.org/10.1007/s11423-021-10069-6

Download citation

Accepted : 21 November 2021

Published : 29 November 2021

Issue Date : February 2022

DOI : https://doi.org/10.1007/s11423-021-10069-6

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Model-testing
• Mathematical modeling
• Educational technology
• Find a journal
• Publish with us
• Track your research

Information

• Author Services

Initiatives

You are accessing a machine-readable page. In order to be human-readable, please install an RSS reader.

All articles published by MDPI are made immediately available worldwide under an open access license. No special permission is required to reuse all or part of the article published by MDPI, including figures and tables. For articles published under an open access Creative Common CC BY license, any part of the article may be reused without permission provided that the original article is clearly cited. For more information, please refer to https://www.mdpi.com/openaccess .

Feature papers represent the most advanced research with significant potential for high impact in the field. A Feature Paper should be a substantial original Article that involves several techniques or approaches, provides an outlook for future research directions and describes possible research applications.

Feature papers are submitted upon individual invitation or recommendation by the scientific editors and must receive positive feedback from the reviewers.

Editor’s Choice articles are based on recommendations by the scientific editors of MDPI journals from around the world. Editors select a small number of articles recently published in the journal that they believe will be particularly interesting to readers, or important in the respective research area. The aim is to provide a snapshot of some of the most exciting work published in the various research areas of the journal.

Original Submission Date Received: .

• Active Journals
• Find a Journal
• Proceedings Series
• For Authors
• For Reviewers
• For Editors
• For Librarians
• For Publishers
• For Societies
• For Conference Organizers
• Open Access Policy
• Institutional Open Access Program
• Special Issues Guidelines
• Editorial Process
• Research and Publication Ethics
• Article Processing Charges
• Testimonials
• Preprints.org
• SciProfiles
• Encyclopedia

Article Menu

• Subscribe SciFeed
• Recommended Articles
• Author Biographies
• Google Scholar
• on Google Scholar
• Table of Contents

Find support for a specific problem in the support section of our website.

Please let us know what you think of our products and services.

Visit our dedicated information section to learn more about MDPI.

JSmol Viewer

Comparative evaluation of model accuracy for predicting selected attributes in agile project management.

1. Introduction

2. project management with ai.

• Gathering requirements and planning: AI utilises user data to formulate comprehensive plans, considering schedules, potential risks, and alternative solutions to navigate challenges effectively.
• Analysis and design: AI facilitates optimal resource allocation during the design phase, guiding designers to adopt more accurate and efficient methodologies based on past experiences.
• Implementation: in the implementation stage, AI aids decision-making by assisting project managers in selecting the right individuals for specific tasks within a given environment, ensuring faster and more secure outcomes.
• Testing and delivery: AI continues to play a pivotal role in the testing and delivery phases, supporting the identification of potential risks and contributing to decision-making processes.

2.1. AI Techniques

2.2. types of deep neural networks, 2.2.1. recurrent neural networks (rnns), 2.2.2. long short-term memory (lstm), 2.2.3. convolutional neural networks (cnns), 2.2.4. cnn-lstm hybrid model, 2.2.5. gated recurrent units (grus), 2.2.6. multilayer perceptron (mlp), 2.3. comparison between models, 2.4. evaluation metrics.

• Mean Absolute Error:
• n : is the number of observations;
• y i : is the actual value;
• y ^ i : is the predicted value.
• Mean Squared Error:
• Mean Absolute Percentage Error:

3. Research Objective

• To precisely assess the performance of LSTM, CNN, CNN-LSTM, GRU, MLP, and RNN models in predicting project completion time, required personnel, and estimated costs. This involves utilising a comprehensive dataset that reflects the complexities of real-world projects to determine how well each model can forecast these crucial project parameters.
• To compare the accuracy of these models using specific evaluation metrics: MAE, MSE, and MAPE. This comparison aims to provide a detailed understanding of each model’s predictive precision, aiding in selecting the most accurate model for project management.
• To determine which machine learning architecture (LSTM, CNN, CNN-LSTM, GRU, MLP, RNN) best predicts project parameters. The goal is to identify the most effective model for different project scenarios and data patterns, thereby supporting more reliable forecasting.
• To elucidate each model’s strengths and limitations, focusing on its ability to handle various project types and complexities. This will help us understand which models are best suited for different project characteristics and provide insights into potential areas for improvement.
• To offer actionable insights for refining predictive models and exploring advanced techniques, such as ensemble methods and hyperparameter optimisation. This objective aims to improve prediction accuracy and robustness in project management.
• To assist students and project managers in ensuring the success of their projects by providing accurate predictions for project completion times and required resources. This includes offering guidance on measuring, estimating, and managing timeframes effectively, enhancing project planning and execution.
• To evaluate the practical implications of the resulting model predictions for real-world project management. This involves understanding how these predictive models can be applied to improve decision-making, resource allocation, and overall project success.

4. Related Work

5. study methodology, 5.1. data collection, 5.1.1. data collection process.

• We distributed the questionnaire on electronic platforms associated with the academic bodies mentioned above.
• We obtained responses from students, totaling 190 responses.
• We performed data checking and cleaning, including removing incomplete or incomprehensible responses.
• We classified the collected data into two main groups: students following the waterfall methodology and those adopting the agile method.
• Data classification was based on project parameters such as the project type, team size, expected completion time, and project overview.

5.1.2. Respondent Selection

• Affiliation: Only students from the Department of Computers and Informatics at the Technical University of Košice were selected.
• Academic Standing: Respondents needed to be in their final classes, ensuring they had sufficient knowledge and experience in informatics and software development.
• Experience with Project Management: Students who had completed at least one project management course or had practical experience were targeted.
• Project Methodology: Respondents were selected based on their familiarity with the waterfall or agile project management methodologies.

5.2. Consent and Personal Data Protection

• One of the project team members provided the students with a detailed explanation of the study. This session included information about the study’s objectives, the nature of the data being collected, and how the data would be used.
• The study was also distributed on one of the university’s documented platforms, ensuring that all information was accessible to the students.
• Participants were assured that their responses would be kept confidential and that any published results would not include identifiable information.
• All collected data were anonymised to ensure that individual respondents could not be identified from the dataset.

5.3. Machine Learning Models

5.4. training and evaluation process for each model.

• Data splitting: The dataset was split into training and testing sets to assess the models’ generalisation performance accurately. An 80–20 split was adopted, allocating 80% of the data for training and 20% for testing.
• Model curation: Distinct models were used for each machine learning architecture, incorporating unique layer configurations. For instance, the CNN model featured convolutional layers for spatial feature extraction, while the LSTM model emphasised sequential data understanding.
• Training: Models underwent an iterative training process using the training set. The training involved minimising a predefined loss function, adjusting model weights through back-propagation, and optimising performance over multiple epochs.
• Prediction: After training, each model was evaluated on the testing set to simulate real-world predictive scenarios, and predictions were generated for project completion times based on the patterns learned from the training data.
• Evaluation metrics: Each model’s performance was quantified using standard evaluation metrics such as MAE, MSE, and MAPE. These metrics provided insights into the accuracy and reliability of the models in predicting project completion times.

6. Results and Analysis

6.1. “predicted completion time” results, 6.2. “persons needed” results, 6.3. “estimated cost” results, 6.4. comparison of metrics across models, 6.4.1. mean absolute error.

• The RNN stands out as the top-performing model, with the lowest MAE of 0.0056. The MLP and GRU models also demonstrate excellent performance, with MAE values of 0.0175 and 0.0202, respectively.
• The CNN and LSTM models exhibit slightly higher MAE values but remain competitive, showcasing their competence in minimising prediction errors.
• CNN-LSTM hybrid, while still effective, shows a comparatively higher MAE.

6.4.2. Mean Squared Error

• The RNN again leads with the lowest MSE of 0.0001, emphasising its superior ability to minimise squared errors.
• The MLP and GRU models follow closely with impressive MSE values of 0.000580942 and 0.000505865, respectively.
• The CNN model, LSTM model, and CNN-LSTM hybrid model display higher MSE values but maintain effectiveness in reducing squared errors.

6.4.3. Mean Absolute Percentage Error

• The RNN exhibits exceptional accuracy, with the lowest MAPE of 1.39%, indicating its proficiency in providing precise percentage errors.
• MLP and CNN-LSTM hybrid also perform well, demonstrating MAPE values of 4.36% and 9.21%, respectively.
• While slightly higher in MAPE, the CNN, GRU, and LSTM models effectively minimise percentage errors.

7. Conclusions

Author contributions, data availability statement, conflicts of interest.

• Yang, Y.; Xia, X.; Lo, D.; Bi, T.; Grundy, J.; Yang, X. Predictive Models in Software Engineering: Challenges and Opportunities. arXiv 2020 , arXiv:2008.03656. Available online: http://arxiv.org/abs/2008.03656 (accessed on 10 August 2024). [ CrossRef ]
• Wang, Q. How to apply AI technology in Project Management. PM World J. 2019 , VIII , 1–12. [ Google Scholar ]
• Chicco, D.; Warrens, M.J.; Jurman, G. The coefficient of determination R-squared is more informative than SMAPE, MAE, MAPE, MSE and RMSE in regression analysis evaluation. PeerJ Comput. Sci. 2021 , 7 , e623. [ Google Scholar ] [ CrossRef ]
• Auth, G.; Jokisch, O.; Dürk, C. Revisiting automated project management in the digital age—A survey of AI approaches. Online J. Appl. Knowl. Manag. 2019 , 7 , 27–39. [ Google Scholar ] [ CrossRef ]
• Martínez-Fernández, S.; Bogner, J.; Franch, X.; Oriol, M.; Siebert, J.; Trendowicz, A.; Vollmer, A.M.; Wagner, S. Software Engineering for AI-Based Systems: A Survey. ACM Trans. Softw. Eng. Methodol. 2022 , 31 , 37e. [ Google Scholar ] [ CrossRef ]
• McGrath, J.; Kostalova, J. Project Management Trends and New Challenges 2020+. Hradec Econ. Days 2020 , 10 , 534–542. [ Google Scholar ] [ CrossRef ]
• Bodimani, M. AI and Software Engineering: Rapid Process Improvement through Advanced Techniques. J. Sci. Technol. 2021 , 2 , 95–119. [ Google Scholar ]
• Hofmann, P.; Jöhnk, J.; Protschky, D.; Urbach, N. Developing Purposeful AI Use Cases—A Structured Method and Its Application in Project Management. In Proceedings of the 15th International Conference on Wirtschaftsinformatik (WI) (WI2020), Postdam, Germany, 9–11 March 2020; pp. 33–49, ISBN 978-3-95545-335-0. [ Google Scholar ]
• Khan, A. Artificial Intelligence (AI) Techniques. Available online: https://intellipaat.com/blog/artificial-intelligence-techniques/ (accessed on 5 March 2024).
• Anitha, N.N.S.; Reddymalla, N.R.; Karri, V.R. Introduction of Artificial Intelligence techniques and approaches. Asian J. Multidiscip. Stud. 2020 , 8 , 15–24. [ Google Scholar ]
• Sarker, I.H. AI-Based Modeling: Techniques, Applications and Research Issues Towards Automation, Intelligent and Smart Systems. SN Comput. Sci. 2022 , 3 , 158. [ Google Scholar ] [ CrossRef ]
• Bouwmans, T.; Javed, S.; Sultana, M.; Jung, S.K. Deep neural network concepts for background subtraction: A systematic review and comparative evaluation. Neural Netw. 2019 , 117 , 8–66. [ Google Scholar ] [ CrossRef ]
• Yu, Y.; Si, X.; Hu, C.; Zhang, J. A Review of Recurrent Neural Networks: LSTM Cells and Network Architectures. Neural Comput. 2019 , 31 , 1235–1270. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Schmidt, R.M. Recurrent Neural Networks (RNNs): A Gentle Introduction and Overview. arXiv 2019 , arXiv:1912.05911. Available online: http://arxiv.org/abs/1912.05911 (accessed on 29 July 2024).
• Bianchi, F.M.; Maiorino, E.; Kampffmeyer, M.C.; Rizzi, A.; Jenssen, R. An Overview and Comparative Analysis of Recurrent Neural Networks for Short Term Load Forecasting. arXiv 2017 , arXiv:1705.04378. [ Google Scholar ]
• Muhammad, A.; Külahcı, F.; Salh, H.; Hama Rashid, P.A. Long Short Term Memory networks (LSTM)-Monte-Carlo simulation of soil ionization using radon. J. Atmos. Sol. Terr. Phys. 2021 , 221 , 105688. [ Google Scholar ] [ CrossRef ]
• Staudemeyer, R.C.; Morris, E.R. Understanding LSTM—A Tutorial into Long Short-Term Memory Recurrent Neural Networks. arXiv 2019 , arXiv:1909.09586. Available online: http://arxiv.org/abs/1909.09586 (accessed on 29 July 2024).
• Lynn, H.M.; Pan, S.B.; Kim, P. A Deep Bidirectional GRU Network Model for Biometric Electrocardiogram Classification Based on Recurrent Neural Networks. IEEE Access 2019 , 7 , 145395–145405. [ Google Scholar ] [ CrossRef ]
• Li, Z.; Liu, F.; Yang, W.; Peng, S.; Zhou, J. A Survey of Convolutional Neural Networks: Analysis, Applications, and Prospects. IEEE Trans. Neural Netw. Learn. Syst. 2022 , 33 , 6999–7019. [ Google Scholar ] [ CrossRef ]
• Rosebrock, A. Deep Learing for Computer Vision. PYIMAGESEARCH. 2017. Available online: https://pyimagesearch.com/deep-learning-computer-vision-python-book/ (accessed on 9 July 2024).
• Mandal, M. Introduction to Convolutional Neural Networks (CNN), Analytics Vidhya. 2021. Available online: https://www.analyticsvidhya.com/blog/2021/05/convolutional-neural-networks-cnn/ (accessed on 5 March 2024).
• Abdallah, M.; An Le Khac, N.; Jahromi, H.; Delia Jurcut, A. A Hybrid CNN-LSTM Based Approach for Anomaly Detection Systems in SDNs. In Proceedings of the 16th International Conference on Availability, Reliability and Security, Vienna, Austria, 17–20 August 2021; ACM: New York, NY, USA, 2021; pp. 1–7. [ Google Scholar ]
• Zargar, S.A. Introduction to Sequence Learning Models: RNN, LSTM, GRU. Agric. Philos. 2021 . [ Google Scholar ] [ CrossRef ]
• Astawa, I.N.G.A.; Pradnyana, I.P.B.A.; Suwintana, I.K. Comparison of RNN, LSTM, and GRU Methods on Forecasting Website Visitors. J. Comput. Sci. Technol. Stud. 2022 , 4 , 11–18. [ Google Scholar ] [ CrossRef ]
• Li, D.; Wang, H.; Li, Z. Accurate and Fast Wavelength Demodulation for Fbg Reflected Spectrum Using Multilayer Perceptron (Mlp) Neural Network. In Proceedings of the 2020 12th International Conference on Measuring Technology and Mechatronics Automation (ICMTMA), Phuket, Thailand, 28–29 February 2020; IEEE: Piscataway, NJ, USA, 2020; pp. 265–268. [ Google Scholar ]
• Abiodun, O.I.; Kiru, M.U.; Jantan, A.; Omolara, A.E.; Dada, K.V.; Umar, A.M.; Linus, O.U.; Arshad, H.; Kazaure, A.A.; Gana, U. Comprehensive Review of Artificial Neural Network Applications to Pattern Recognition. IEEE Access 2019 , 7 , 158820–158846. [ Google Scholar ] [ CrossRef ]
• Shiri, F.M.; Perumal, T.; Mustapha, N.; Mohamed, R. A Comprehensive Overview and Comparative Analysis on Deep Learning Models: CNN, RNN, LSTM, GRU. arXiv 2023 . [ Google Scholar ] [ CrossRef ]
• De Myttenaere, A.; Golden, B.; Le Grand, B.; Rossi, F. Mean Absolute Percentage Error for regression models. Neurocomputing 2016 , 192 , 38–48. [ Google Scholar ] [ CrossRef ]
• Prayudani, S.; Hizriadi, A.; Lase, Y.Y.; Fatmi, Y.; Al-Khowarizmi. Analysis Accuracy of Forecasting Measurement Technique on Random K-Nearest Neighbor (RKNN) Using MAPE And MSE. J. Phys. Conf. Ser. 2019 , 1361 , 012089. [ Google Scholar ] [ CrossRef ]
• Filippetto, A.S.; Lima, R.; Barbosa, J.L.V. A risk prediction model for software project management based on similarity analysis of context histories. Inf. Softw. Technol. 2021 , 131 , 106497. [ Google Scholar ] [ CrossRef ]
• Malgonde, O.; Chari, K. An ensemble-based model for predicting agile software development effort. Empir. Softw. Eng. 2019 , 24 , 1017–1055. [ Google Scholar ] [ CrossRef ]
• Pasuksmit, J.; Thongtanunam, P.; Karunasekera, S. Towards Reliable Agile Iterative Planning via Predicting Documentation Changes of Work Items. In Proceedings of the 19th International Conference on Mining Software Repositories, Pittsburgh, PA, USA, 23–24 May 2022; ACM: New York, NY, USA, 2022; pp. 35–47. [ Google Scholar ]
• Licorish, S.A.; Savarimuthu, B.T.R.; Keertipati, S. Attributes that Predict which Features to Fix: Lessons for App Store Mining. In Proceedings of the Proceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering, Karlskrona, Sweden, 15–16 June 2017; ACM: New York, NY, USA, 2017; pp. 108–117. [ Google Scholar ]
• Alzeyani, E.M.M.; Szabo, C. A Study on the Methodology of Software Project Management Used by Students whether They are Using an Agile or Waterfall Methodology. In Proceedings of the 2022 20th International Conference on Emerging eLearning Technologies and Applications (ICETA), Stary Smokovec, Slovakia, 20–21 October 2022; IEEE: Piscataway, NJ, USA, 2022; pp. 22–27. [ Google Scholar ] [ CrossRef ]

Click here to enlarge figure

Share and Cite

Alzeyani, E.M.M.; Szabó, C. Comparative Evaluation of Model Accuracy for Predicting Selected Attributes in Agile Project Management. Mathematics 2024 , 12 , 2529. https://doi.org/10.3390/math12162529

Alzeyani EMM, Szabó C. Comparative Evaluation of Model Accuracy for Predicting Selected Attributes in Agile Project Management. Mathematics . 2024; 12(16):2529. https://doi.org/10.3390/math12162529

Alzeyani, Emira Mustafa Moamer, and Csaba Szabó. 2024. "Comparative Evaluation of Model Accuracy for Predicting Selected Attributes in Agile Project Management" Mathematics 12, no. 16: 2529. https://doi.org/10.3390/math12162529

Article Metrics

Article access statistics, further information, mdpi initiatives, follow mdpi.

Subscribe to receive issue release notifications and newsletters from MDPI journals

• Clinical trials and evidence

Getty Images/iStockphoto

Alternatives to Animal Testing Models in Clinical and Biomedical Research

Over the past several years, a growing number of alternative techniques have been developed and used to replace animal testing models in clinical and biomedical research..

• Alivia Kaylor, MSc

Before the FDA Modernization Act 2.0 was passed in December 2022, the US government required that all investigational drugs be tested on animals before they could advance to human trials. Although this act does not ban animal testing, it allows researchers to use scientifically proven, non-animal testing methods, such as cell-based assays, microfluidic chips, tissue models, computer models, and human volunteers, when possible.

Animal Testing

Animal testing models have been used throughout history, dating back to 500 BC in ancient Greece. Researching animals is essential for translating discoveries and observations in the laboratory or clinic into new treatments.

Today, standard animal models usually include mice and rats due to their anatomical, genetic, and physiological similarity to humans and their ease of maintenance and size, short life cycle, and abundant genetic resources.

In several research fields, non-human primates (NHPs) — a group of hominins, apes, and monkeys that are biologically and evolutionally similar to humans — are used for scientific, educational, and exhibition purposes. Because NHPs have short life spans and are susceptible to many of the same health conditions that affect humans, they serve as ideal research subjects for studying entire life cycles or several generations.

NHPs are vital for researching a wide array of diseases:

• central nervous system diseases (Alzheimer’s, Parkinson’s, and Huntington’s)
• cancer (liver, renal, gastric)
• metabolic disorders (diabetes and obesity)
• cardiovascular diseases (atherosclerosis and cardiac arrhythmias)
• infectious diseases (HIV/AIDS, malaria, SARS, COVID-19)
• ocular diseases (dry eyes, cataract glaucoma, age-related macular degeneration)
• reproductive conditions (endometriosis, polycystic ovarian syndrome, pelvic inflammatory diseases)

However, as NHP suppliers struggle to meet an unprecedented surge in global demand to study these types of diseases, research institutions have found it increasingly challenging to obtain NHPs in the past several months.

Animal Welfare

Although federal organizations have developed clinical laws and regulations — such as the  Animal Welfare Act  (AWA) and its succeeding policies — to regulate the treatment of research animals, this practice has been under intense scrutiny by animal rights advocates and policymakers for decades.

According to AWA regulations, principal investigators (PIs) must seek alternatives to painful or distressing procedures. If no other options are viable, researchers must submit a written notice to their Institutional Animal Care and Use Committee (IACUC) that outlines how the team determined that there are no alternatives available. To take it a step further, PIs must also supply written assurance that their research activities do not duplicate prior experiments unnecessarily.

While animal testing has helped humans advance our scientific understanding and develop new medicines and treatments, some experts who study animal testing argue that it can be expensive and ineffective.

“We have many important drugs that have been developed using animal tests. But as we get into some of these more difficult diseases, especially neurological diseases, the animal models aren't serving us as well,” Paul Locke, MPH, JD, DrPH, a scientist and lawyer at Johns Hopkins University, told Wired . “Researchers need new ways to unlock the molecular mechanisms causing these diseases, and the alternatives hold great promise.”

Advocates like Locke highlight studies that demonstrate animal testing can be an  unreliable predictor of toxicity  in the human body. For example, fialuridine , a drug developed for treating hepatitis B, causes liver failure and is toxic to humans but not mice.

Because  90% of general drug candidates in clinical trials never reach the market , drugs that target the brain typically have an  even higher failure rate . These proven inconsistencies and associated time, cost, and ethical concerns have encouraged scientists to develop alternative testing methods that better recapitulate human physiology. 

What Are Alternatives to Animal Testing?

Using alternatives to animal testing in clinical research when suitable does not put patients at risk or delay medical progress. Instead, non-animal testing methods such as human cell- and tissue-based testing, human volunteer testing, and computational and mathematical models can be more accurate, cost-effective, and quicker than traditional animal models.

Human Cell- and Tissue-Based Testing

Miniature cellular and tissue models, such as organs-on-a-chip and 3D bioprinting , use human cells to mimic organ functions and structures to screen treatments and test drugs. This allows researchers to simplify a system, limiting the number of variables. Instead of animals, these human-based models can be used to study biological and disease processes and drug metabolism.

Organs-on-a-Chip

Microfluidic organs-on-a-chip are small, clear, flexible polymer devices that comprise human cells and push fluid through tiny channels to imitate blood flow. In 2010, a team at Harvard University’s Wyss Institute developed the first successful human-cell chip. The first of its kind, the lung-on-a-chip, carried out basic lung functions, like respiration. And now, researchers have expanded upon this concept by successfully creating chips that mimic the liver, stomach, intestine, brain, and skin, among others.

Because roughly 30% of medications fail in human clinical trials due to toxicity — despite pre-clinical data using animal and cell models — tissue chips function as new human cell-based approaches that help researchers accurately determine how effective a therapeutic candidate would be in clinical studies.

By eliminating toxic or ineffective drugs earlier in development, drug manufacturers can save valuable time and money. These chips also could teach scientists a great deal about disease progression, leading to better prevention, diagnosis, and treatment approaches.

Because many industry experts recognize the widespread benefits of human-specific chips, this method is becoming popular in drug discovery and development.

Organs-on-a-chip technology allows scientists to easily replicate human tissue and organ functions to assess the safety and efficacy of new drugs. For instance, the Liver-Chip, a liver-on-a-chip device , can detect drug-induced liver injury missed by animal testing models.

Tissue Bioprinting

Three-dimensional (3D) tissue bioprinting is a revolutionary scientific advancement in drug discovery and development that uses new assay models to predict drug impacts on humans better. These tissue models mimic characteristics of live human tissues and are developed on microplates to test the toxicity and efficacy of small molecules or other therapeutics.

By leveraging tissue engineering, stem cell research, disease biology, and in situ  detection devices for tissue characterization and drug development, 3D tissue bioprinting produces disease-relevant tissue models that can reduce the predictability gap between the results from current 2D cell-based assays and the results from testing in humans.

Human Volunteer Testing

Thanks to numerous technological advances, new and sophisticated scanning devices and recording methods can now be used to study human volunteers safely.

For example, advancements in brain imaging techniques allow researchers to see inside the brain to monitor the progression and treatment of certain brain diseases. Researchers use these approaches to better understand diseases by comparing their results with the results of healthy volunteers.

In other research areas, such as nutrition, substance use, and pain management, consenting humans can help replace animal testing models. Compared to animal subjects, human volunteers provide a significant advantage by having the ability to speak with researchers and offer additional information during the study.

As opposed to animal testing, human volunteers that donate healthy and compromised tissues via surgery provide a more appropriate way of studying human biology and disease. For example, skin and eye models made from reconstituted human skin and other tissues have been developed to replace rabbit irritation tests.

By donating tissue, alive and deceased donors increase the number of human samples available for research and reduce the number of animal subjects needed. In the past, post-mortem brain tissue has provided important breakthroughs in understanding brain regeneration and the effects of multiple sclerosis and Parkinson’s disease.

Computational and Mathematical Models

With the growing capabilities of computers and computer programs, the ability to model certain aspects of the human body has become easier than ever.

Current computer models of the heart, lungs, kidneys, skin, and digestive and musculoskeletal systems have been developed to conduct virtual experiments based on existing mathematical data and information. Additionally, data mining tools assist researchers in making predictions about one substance based on existing data from similar substances.

Many alternatives to animal testing methods aim to overcome translational barriers toward developing urgently needed treatments for unmet medical needs. As a result, using non-animal models for research could save the lives of more humans and animals, time, and money. And without sacrificing quality and safety, alternatives to animal testing could improve the quality of society while improving health outcomes.

• The Fundamentals of Animal Testing in Clinical Research
• Understanding the Value, Complexity of Clinical Trials in the US
• How Brain-in-a-Dish Technology Can Impact the Treatment of MDD

Dig Deeper on Clinical trials and evidence

Managing bird flu risks with advances in prevention

Researchers identify microplastics in human testicular tissue

Geroscience: targeting aging mechanisms for preventing chronic disease

Machine learning characterizes cardiac function, drug response

A national survey revealed that while many Americans are comfortable with artificial intelligence in healthcare, some have ...

AI revealed insights into the relationship between lung cancer therapy and heart complications, which could help minimize cardiac...

Houston Methodist has successfully piloted ambient clinical intelligence and AI tools to bolster efficiency and cost savings ...

Understanding the quality bonus payment program is critical to appreciating Medicare Advantage and discussing the calls for ...

Only 45% of individuals who received a surprise bill challenged it and only 43% of those who were denied coverage appealed the ...

Medicare includes four different segments that each insure a wide variety of services and supplies for enrollees.

Ambient AI technology is helping Ochsner Health improve patient-provider relationships by allowing clinicians to spend ...

Epic's goal to bring customers live on TEFCA by the end of 2025 complements Carequality's plan to align with TEFCA as the ...

Key considerations for selecting an EHR vendor include assessing practice needs, conducting a thorough market scan and evaluating...

NIST encouraged organizations to implement its three post-quantum cryptography standards to prepare for the emergence of powerful...

A healthcare data breach at Alabama Cardiovascular Group affected upwards of 280,000 individuals after an unauthorized party ...

Lawsuits often follow a healthcare data breach, but understanding what drives litigation trends can help healthcare organizations...