reference in education research must be evaluated as per

• Resources Home
• All Publications
• Digital Resources
• The BERA Podcast
• Research Intelligence
• BERA Journals
• Ethics and guidance
• BERA Blog Home
• About the blog
• Blog special issues
• Blog series
• Submission policy
• Opportunities Home
• Awards and Funding
• Communities Home
• Special Interest Groups
• Early Career Researcher Network
• Events Home
• BERA Conference 2024
• Upcoming Events
• Past Events
• Past Conferences
• BERA Membership

Publication

Ethical Guidelines for Educational Research, fourth edition (2018)

13 Aug 2019

• Letter from the president
• Introduction

Transparency

Right to withdraw, harm arising from participation in research, privacy and data storage, responsibilities to the community of educational researchers, scope and format, responsibilities for researchers’ wellbeing and development.

• Historical note

Jump to guideline: 1 | 10 | 20 | 30 | 40 | 50 | 60 | 70 | 80

LETTER FROM THE PRESIDENT

Dear colleague,

On behalf of the Council of the British Educational Research Association (BERA), I am very pleased to present to you our fourth edition (2018) of the Ethical Guidelines for Educational Research .

Research related to education is varied and complex, rarely amenable to precise measurement or given to all-encompassing solutions to its many challenges. Nevertheless, the continued pursuit of improved knowledge and understanding of all aspects of education is vital for our democracy and social wellbeing. To this end, these guidelines are designed to support educational researchers in conducting research to the highest ethical standards in any and all contexts.

BERA’s guidelines unequivocally recognise and celebrate the diversity of approaches in educational research, and promote respect for all those who engage with it: researchers and participants, academics and professional practitioners, commissioning bodies, and those who read and utilise the research. They are not rules and regulations, but they do represent the tenets of best ethical practice that have served our community of researchers well in the past and will continue to do so in the future.

The new guidelines are greatly extended in length in comparison with their predecessors. This has been found appropriate in order to take account of the many changes that have taken place in the last few years, including the rise of social media and online communities, new legislative requirements, and the growing impact on our research of internationalisation and globalisation. As time goes on, BERA Council will continue to review the guidelines and will update them as necessary. I hope that you will find them of assistance in your work, and that you will commend them to everyone who carries out, participates in or makes use of educational research.

With best wishes,

Gary McCulloch President, British Educational Research Association June 2018

^Back to top

INTRODUCTION

The intended audience for these guidelines is anyone undertaking educational research – be they people whose job description includes research, or others who, for a variety of reasons (including studying for a qualification or with the intention of improving practice), conduct research within the field. This includes both independent researchers and those based in educational institutions of any kind (including but not limited to early years settings, schools, colleges and universities).

The Association expects its members to conduct themselves in a way that reflects its vision, aims and ethical values (as stipulated in the BERA code of conduct 1 ). For this reason, the British Educational Research Association (BERA) recommends that members make use of these guidelines (and/or other ethical guidelines, where relevant or required), and expects that they will adhere to their spirit and underlying principles (described later) and apply them with integrity in their research activities so that their actions can be seen to be ethical, justifiable and sound. It is the hope of the Association that these guidelines will attract widespread consideration and use by those engaged in carrying out, sponsoring or using educational research who are not BERA members.

For a great deal of educational research activity, the application of these guidelines will not be problematic, but in some cases dilemmas may arise. We recognise that since few ethical dilemmas have obvious or singular solutions, researchers will take different and creative approaches to resolving them. Certain dilemmas are flagged up within these guidelines, but others that cannot be covered here will also arise. Guidelines that state what action ‘should’ be taken may not be appropriate to all circumstances; in particular, different cultural contexts are likely to require situated judgments. Furthermore, some kinds of research may require ethical clearance from other bodies, such as the National Health Service (NHS), which commit researchers to acting in accordance with their guidelines. In sum, and for each research project, researchers will need to devise specific ethical courses of action which may incorporate elements from more than one set of guidelines – those of both the NHS and BERA, for example . To do this, they may draw on ethical approaches that reflect a range of philosophical orientations (virtue ethics, or deontological ethics, for example). It is adherence to the spirit of the guidelines that we consider most vital to protect all who are involved in or affected by a piece of research. In addition to these guidelines, support and links to related resources are offered wherever possible. 2

We recommend that at all stages of a project – from planning through conduct to reporting – educational researchers undertake wide consultation to identify relevant ethical issues, including listening to those in the research context/site(s), stakeholders and sponsors. This means that ethical decision-making becomes an actively deliberative, ongoing and iterative process of assessing and reassessing the situation and issues as they arise.

BERA recommends that researchers bring these guidelines to the attention of those they work with – including, for example, participants, stakeholders, sponsors and commissioners of research, schools and other organisations – and encourage and support those contacts to engage with them. BERA hopes that these guidelines will inform the training of students enrolled on education and research degrees, and recommends that local ethical review procedures make use of them in support of their own work.

It is recommended that, in addition to vetting applications, committees should consider how to foster opportunities for follow-up dialogue to reveal whether and how researchers have acted in consideration of BERA and/or local ethical principles throughout an entire study.

The guidelines are intended to promote active and concrete responses following from a deliberation of the issues. Researchers and their students and collaborators should – in their research proposals, reports, funding applications, work with schools and so on – explicitly indicate how they are adhering to those points included in these guidelines that are salient to their work.

It is recognised that educational researchers whose work is conducted under the auspices of an educational institution will be required to seek ethical review and clearance from that institution. These guidelines are, therefore, intended to inform and support researchers as they develop their ethical thinking and practice.

Aspirations of educational researchers

Educational researchers aim to extend knowledge and understanding in all areas of educational activity and from all perspectives, including those of learners, educators, policymakers and the public. The Association recognises that the community of educational researchers is multidisciplinary and diverse in its application of research approaches and philosophical positions. Concepts such as ‘data’, ‘reliability’, ‘validity’, ‘credibility’, ‘trustworthiness’, ‘subjectivity’ and ‘objectivity’ may therefore be understood and legitimately applied in different ways. Ethical review processes thus need to be conducted in an open-minded and inclusive manner.

These guidelines do not pre-empt, judge or constrain, directly or indirectly, anyone’s choice of research approach.

Principles underpinning the guidelines

The Association endorses the set of ethical principles agreed in 2015 by the Academy of Social Sciences (AcSS) 3 through in-depth consultation with its member learned societies (including BERA). These principles are as follows.

• Social science is fundamental to a democratic society, and should be inclusive of different interests, values, funders, methods and perspectives.
• All social science should respect the privacy, autonomy, diversity, values and dignity of individuals, groups and communities.
• All social science should be conducted with integrity throughout, employing the most appropriate methods for the research purpose.
• All social scientists should act with regard to their social responsibilities in conducting and disseminating their research.
• All social science should aim to maximise benefit and minimise harm. 4

All five of the AcSS’s principles above are reflected (explicitly or implicitly) in the various sections of the BERA guidelines that follow, and they are consistent with the ethical principles of respect that have been developed through previous iterations of BERA’s guidelines. The Association believes that all educational research should be conducted within an ethic of respect for: the person; knowledge; democratic values; the quality of educational research; and academic freedom. Trust is a further essential element within the relationship between researcher and researched, as is the expectation that researchers will accept responsibility for their actions. These are the collective principles that we ask members and those using the guidelines to commit to and engage with when making decisions in their research.

Applying an ethic of respect may reveal tensions or challenges. For example, there will usually be a need to balance research aspirations, societal concerns, institutional expectations and individual rights. It is recommended that researchers undertake a risk-benefit analysis, beginning at the earliest stage of research planning, to reflect on how different stakeholder groups and the application of this ethic of respect can be considered in the research design.

In guiding researchers on their conduct within this framework, the Association sets out its guidelines under the following five headings:

• responsibilities to participants
• responsibilities to sponsors, clients and stakeholders in research
• responsibilities to the community of educational researchers
• responsibilities for publication and dissemination
• responsibilities for researchers’ wellbeing and development.

Responsibilities to participants

The British Educational Research Association (BERA) believes that educational researchers should operate within an ethic of respect for any persons – including themselves – involved in or touched by the research they are undertaking. Individuals should be treated fairly, sensitively, and with dignity and freedom from prejudice, in recognition of both their rights and of differences arising from age, gender, sexuality, ethnicity, class, nationality, cultural identity, partnership status, faith, disability, political belief or any other significant characteristic.

The Association expects researchers to be mindful of the ways in which structural inequalities – those, for example, associated with ‘race’, gender, LBGT+ issues and socio-economic status – affect all social relationships, including those that are formed in the course of research. Where relevant, attention should be paid to the ways in which such inequalities specifically affect children and young people, and their relationships. Sensitivity and attentiveness towards such structural issues are important aspects of researchers’ responsibilities to participants at all stages of research, including reporting and publication.

Participants in research may be actively or passively involved in such processes as observation, experiment, auto/biographical reflection, survey or test. They may be collaborators or colleagues in the research process, or they may simply be implicated in the context in which a research project takes place. (For example, in a teacher or lecturer’s research into their own professional practice, students or colleagues will be part of the context, but will not themselves be the focus of that research.) It is important for researchers to take account of the rights and interests of those indirectly affected by their research, and to consider whether action is appropriate – for example, they should consider whether it is necessary to provide information or obtain informed consent. In rare cases – for instance, in some politically volatile settings, or where researchers are investigating illegal activity, including suspected abuse – covert research may be defensible. In such cases approval must be obtained from an institutional ethics review committee.

Where research draws on social media and online communities, it is important to remember that digital information is generated by individuals. Researchers should not assume that the name given and/or identity presented by participants in online fora or media is a ‘real’ name: it might be an avatar. This avatar could represent a human or a bot, but behind either will be one or more human creators responsible for it, who could therefore be regarded as participants; whether and how these potential participants might be traceable should be considered. Where an organisation shares its data with researchers, those researchers have a responsibility to account for how and with what consent that data was gathered; they must also consider the authorship of that data and, consequently, whether it is necessary to independently approach the relevant individuals for consent concerning its use. Researchers should keep up to date with changes in data use regulations and advice.

Researchers have a responsibility to consider what the most relevant and useful ways are of informing participants about the outcomes of the research in which they were or are involved. They could consider whether and how to engage with participants at the conclusion of the research by, for example, debriefing them in an audience-friendly format, or by eliciting feedback on the findings. Should conflicting interpretations arise, researchers should normally reflect participants’ views when reporting the research. Researchers may wish to offer them copies of any publications arising from projects in which they have participated, or to produce reports specially tailored for the research context, taking into consideration potential subsequent uses of this material, including by the participants’ institutions. Where the scale of the research makes such a consideration impractical, alternative means such as a website could be used to ensure that participants are informed of the outcomes and the ways in which they are able to engage with them.

Researchers also have a responsibility to consider how to balance maximising the benefits and minimising any risk or harm to participants, sponsors, the community of educational researchers and educational professionals more widely – while again recognising that irresolvable tensions may need to be addressed. At times, some benefits to participants may be compromised in order to achieve other gains or goals, but these compromises should be justifiable and, where possible, explicitly accounted for.

Researchers should not undertake work for which they are not competent.

Adherence to an ethic of respect implies the following responsibilities on the part of researchers.

It is normally expected that participants’ voluntary informed consent to be involved in a study will be obtained at the start of the study, and that researchers will remain sensitive and open to the possibility that participants may wish, for any reason and at any time, to withdraw their consent. The Association takes voluntary informed and ongoing consent to be the condition by which participants understand and agree to their participation, and the terms and practicalities of it, without any duress, prior to the research getting underway. It should be made clear to participants that they can withdraw at any point without needing to provide an explanation – this is detailed in sections 31 and 32 below.

Researchers should do everything they can to ensure that all potential participants understand, as well as they can, what is involved in a study. They should be told why their participation is necessary, what they will be asked to do, what will happen to the information they provide, how that information will be used and how and to whom it will be reported. They also should be informed about the retention, sharing and any possible secondary uses of the research data. Where appropriate, researchers who are BERA members will include a declaration of membership in information sheets and consent forms, to make explicit the fact that members are expected to follow BERA guidance as part of the Association’s code of conduct 5 (which contains a complaints procedure that may be helpful 6 ).

Participants may be willing to take part in research even though they are unable to be fully informed about the implications of their participation – perhaps due to their unfamiliarity with research, a lack of ability to understand, or their circumstances. In these situations, researchers and participants should negotiate consent within relationships of mutual trust, the credibility of which depends upon the integrity and trustworthiness of the researcher.

The institutions and settings within which the research is set also have an interest in the research, and ought to be considered in the process of gaining consent. Researchers should think about whether they should approach gatekeepers before directly approaching participants, and about whether they should adopt an institution’s own ethical approval and safeguarding procedures; this is usually a requirement. (Furthermore, in some circumstances researchers may have a statutory duty to disclose confidential information to relevant authorities; see section 52.) Particularly when researching in more than one language or culture, researchers should consider the effects of translation and/or interpretation on participants’ understandings of what is involved.

In many cases the producers of publicly accessible data may not have considered the fact that it might be used for research purposes, and it should not be assumed that such data is available for researchers to use without consent. Researchers should be attuned to differences between, for example, policy documents, governing body minutes and charitable trust reports that are written with the expectation that they are available for public use or accountability, and data that appear to be in the public domain yet are produced for a range of purposes (in blogs, social media, online discussion forums, face-to-face presentations or meetings, for example). Seeking consent would not normally be expected for data that have been produced expressly for public use. There is no consensus, however, as to whether those in online communities perceive their data to be either public or private, even when copyrights are waived. Therefore, consent is an issue to be addressed with regard to each and any online data-source, with consideration given to the presumed intent of the creators of online content, the extent to which it identifies individuals or institutions, and the sensitivity of the data. See the sources listed in the below footnote for further guidance. 7

Consideration should be given to whether and how best to approach online communities (through members, gatekeepers or moderators, for example), or those involved in face-to-face public events and spaces, in order to inform them about the intended research.

When working with secondary or documentary data, the sensitivity of the data, who created it, the intended audience of its creators, its original purpose and its intended uses in the research are all important considerations. If secondary data concerning participants are to be reused, ownership of the datasets should be determined, and the owners consulted to ascertain whether they can give consent on behalf of the participants. Sometimes it may be deemed appropriate to accept consent from hosts of the data such as a depository on behalf of contributors.

It is accepted that, sometimes, gaining consent from all concerned in public spaces (face-to-face or virtual, past or present) will not be feasible; however, attempts to make contact should be documented. In the event of a secondary source being untraceable, researchers should be able to evidence their attempts to gain consent.

In circumstances in which some members of a group (such as students in a class or their parents/guardians) have not given consent to participate, researchers should decide whether this was an active refusal of consent, in which case they would need to respect this and find a practical solution. For those who it is not possible to contact, a decision should be taken as to how it might be appropriate to proceed, in conjunction with gatekeepers or other stakeholders.

In ethnographic and observational studies, the level of analysis in group-focussed research should be taken into account where some members of the group refuse consent. If the research aims to understand the roles of individuals within the group then these members must not be included in the research. However, to the extent that the research is concerned with the group dynamic as a whole (for example, within a classroom), consenting individuals’ interactions with the non-consenting individuals may still be significant to the research.

Specific issues also arise with respect to consent within large-scale randomised control trials across research settings. Institutional leaders may agree to take part, acting as gatekeepers on behalf of members (such as teachers and students in schools). In order to ensure that all participants are as fully informed as possible about the costs and benefits of the study, researchers should offer both information and support. This may result in participants exercising their right to opt out within the parameters of the intervention. Where stratified random sampling is used, it may be appropriate to select more participants than necessary so that where institutions or individuals drop out, they can be replaced from the randomised pool. However, the potential explanations for dropout would normally be discussed with the research sponsor and the implications addressed in the report.

An important consideration is the extent to which a researcher’s reflective research into their own practice impinges upon others – for example, in the case of power relationships arising from the dual roles of teacher/lecturer/manager and researcher, and their impact on students and colleagues. Dual roles may also introduce explicit tensions in areas such as confidentiality. These may be addressed appropriately by, for example, making the researcher role very explicit; involving an independent third party in the research process; seeking agreement for politically controversial research; and ensuring that researcher identity remains confidential. Researchers who are researching their own practice should also consider how to address any tensions arising between collecting data for different purposes – for example, using for research purposes data collected for evaluation purposes, or vice versa.

In some cases, potential participants may not be in a social position vis a vis the researcher that enables them to easily give unrestrained informed consent. This can occur when the researcher and potential participant are family members, or if the researcher is the participant’s teacher/lecturer. Researchers need to carefully consider how to deal with such situations and, if they can, should reassure such potential participants that non-participation is acceptable.

Researchers using auto/biographical approaches and autoethnography need to consider how their work implicates other people, and what the consequences may be for individuals who, although not directly involved in a study, may be identifiable through their relationship with the researcher or other participants; consent may need to be sought from these individuals in some cases.

BERA expects that the same ethical principles will be applied to research undertaken by UK researchers outside of the UK as to research undertaken by them within the UK. The application of these principles in different social, cultural and political contexts requires careful negotiation, adaptation and sensitivity, and there is ultimately no substitute for the good conscience and ethical code of the individual researcher. In some countries it is advisable to work with a local person as co-researcher/co-investigator in order to establish adequate levels of trust with prospective local participants. Appropriate permission should be sought from relevant authorities (such as community or religious leaders or local government officials) in cultures that adopt a collective approach to consent. However, cultural sensitivity should not extend to excluding the individuals concerned from making their own informed decisions to take part in the research.

BERA’s principles of consent apply to children and young people as well as to adults. However, researchers may make different decisions as they deem appropriate for children and young people of different ages and capacities. BERA endorses the United Nations Convention on the Rights of the Child (UNCRC) 8 ; the best interests of the child are the primary consideration, and children who are capable of forming their own views should be granted the right to express those views freely in all matters affecting them, commensurate with their age and maturity.

Researchers following the UNCRC will take into account the rights and duties of those who have legal responsibility for children, such as those who act in guardianship (parents, for example) or as ‘responsible others’ (that is, those who have responsibility for the welfare and wellbeing of the participants, such as social workers). This may involve gaining the consent of those responsible for children, such as a parent or guardian.

In the case of participants whose capacity, age or other vulnerable circumstance may limit the extent to which they can be expected to understand or agree voluntarily to participate, researchers should fully explore ways in which they can be supported to participate with assent in the research. In such circumstances, researchers should also seek the collaboration and approval of those responsible for such participants.

Opt-in or opt-out procedures of gaining consent could be considered, as appropriate for the context; however, researchers have a responsibility to consult local legislation and laws, 9 particularly with regard to whether gaining opt-out consent is permissible. Participants’ trust in the wider value of the research beyond the researcher’s personal interests might be gained by including an endorsement from a senior leader within the institution/organisation where research is being carried out. (Researchers may need to weigh up competing ethical and methodological considerations – for example, taking steps to maximise opportunities to opt out where that method is selected, in order to reduce otherwise unavoidable sampling bias.)

Researchers should aim to be open and honest with participants and other stakeholders, avoiding non-disclosure unless their research design specifically requires it in order to ensure that the appropriate data are collected, or that the welfare of the researchers is not put in jeopardy. Decisions to use non-disclosure in research should be the subject of full, principled deliberation and subsequent disclosure in reporting, and institutional ethical clearance should be obtained before using it. BERA recommends that if researchers are not employed or enrolled in settings where they are subject to institutional procedures they should seek to gain approval for any course of action involving non-disclosure by approaching a local or institutional ethics body and asking if their work can be reviewed. In any event, if it is possible to do so, researchers should seek consent on a post-hoc basis in cases in which it was not desirable to seek it before undertaking the research.

Principles of consent also apply to possible reuse of data. This covers two different possible future uses: secondary data analysis by the same research team to address new research questions, or the sharing of the dataset for use by other researchers. In both cases, if data are to be reused, this should be made clear as a possibility when gaining initial consent. It is recommended that only anonymised and disaggregated data should be archived for sharing with other researchers beyond the original research team, and that researchers minimise the possibility that traces of identity retained within anonymised digital data can lead to the identification of participants. Researchers should explain to potential participants how long data will be stored for if it is to be reused.

Where research has been sponsored or commissioned this should be made explicit to potential participants and other stakeholders, and in reports of the research and other publications, in the interests of both transparency and acknowledgement.

Researchers should not undertake work in which they can be perceived to have a conflict of interest, or in which self-interest or commercial gain might compromise the objectivity of the research.

Researchers should recognise the right of all participants to withdraw from the research for any or no reason, and at any time, and participants should be informed of this right. Researchers should always provide their own contact details to participants. In all such circumstances researchers should examine their own actions to assess whether they have contributed to the decision to withdraw, and whether a change of approach might persuade the participants to re-engage. In most cases the appropriate course of action will be for the researchers to accept the participant’s decision to withdraw. Decisions to persuade them to re-engage should be taken with care, and payment, coercion or duress of any form must not be used. However, in cases in which participants are required by a contractual obligation to participate (for example, when mandated as part of their employment to facilitate an evaluation study) researchers may have proper recourse to a third party (the employing authority in this example) to request compliance.

In online research contexts, if authors of postings or other material withdraw or delete data then that data should not be used in research. However, since it will not be possible for researchers to identify such withdrawals after data has been harvested, a proviso could be offered that the data were ‘as made available to the public at the [stated] date of harvesting’.

Researchers’ use of incentives to encourage participation should be commensurate with good sense, such that the level of incentive does not impinge on the free decision to participate. Payment for participation in educational research is generally discouraged, not least because of the extra burden of cost that the extension of this practice would place on the practice of research. 10 The use of incentives should be acknowledged in any reporting of the research.

Ethical research design and execution aim to both put participants at their ease and avoid making excessive demands on them. In advance of data collection, researchers have a responsibility to think through their duty of care in order to recognise potential risks, and to prepare for and be in a position to minimise and manage any distress or discomfort that may arise. Researchers should immediately reconsider any actions occurring during the research process that appear to cause emotional or other harm, in order to minimise such harm. The more vulnerable the participants, the greater the responsibilities of the researcher for their protection.

Researchers should make known to the participants (or their guardians or responsible others) any predictable disadvantage or harm potentially arising from the process or reporting of the research. Any unexpected harm to participants that arises during the research should be brought immediately to their attention, or to the attention of their guardians or responsible others as appropriate.

Researchers should take steps to minimise the effects of research designs that advantage or are perceived to advantage one group of participants over others. For example, in an experimental design (including a randomised control study), the intervention made available to one group, while being unavailable to the control or comparison group, may be viewed as desirable. In mitigation, for example, an intervention emerging as effective can typically be offered to control groups after the end of a trial.

The rights of individuals should be balanced against any potential social benefits of the research, and the researcher’s right to conduct research in the service of public understanding. The researcher’s obligations to the wider research community and to public understanding may, in some circumstances, outweigh the researcher’s obligations to act in accordance with the wishes of those in positions of economic, legal or political authority over the participants (such as employers, headteachers or government officials in oppressive regimes).

Researchers should recognise concerns relating to the time and effort that participation in some research can require – the long-term involvement of participants in ethnographic studies, for example, and the repeated involvement of particular participants in survey research or in testing for research or evaluation purposes. Researchers should consider the impact of their research on the lives and workloads of participants, particularly when researching vulnerable or over-researched populations.

During the research process (especially in longitudinal or ethnographic studies), if unforeseen consequences arise – in terms of human relationships or life experiences, for example – it may be appropriate to go back to the participants, gatekeepers or sponsors in order to renegotiate consent.

The confidential and anonymous treatment of participants’ data is considered the norm for the conduct of research. Researchers should recognise the entitlement of both institutions and individual participants to privacy, and should accord them their rights to confidentiality and anonymity. This could involve employing ‘fictionalising’ approaches when reporting, and where using such approaches researchers should fully explain how and why they have done so. However, in some circumstances individual participants, or their guardians or responsible others, may want to specifically and willingly waive their right to confidentiality and anonymity: researchers should recognise participants’ rights to be identified in any publication of their original works or other inputs if they so wish.

It is also acknowledged that anonymity may not be possible in some contexts and cases. For example, if conducting fieldwork within a small, close-knit community, it may be impossible to prevent some members of that community becoming aware – either through observation or because participants discuss it with them – of some details about the research that is being conducted. Similarly, when researching a very well-known institution, it may be possible for some readers to infer the identity of that institution even from a fully anonymised account of that research. Furthermore, approaches to this issue differ according to the type of research being undertaken: for instance, the maintenance of confidentiality and anonymity is not considered the norm for research using historical or archival data, nor is it achievable for autoethnographic work published under the author/researcher’s name.

Anonymity may also need to be reconsidered in the context of some visual methodologies and participatory methods. For instance, the study of facial expressions and gestures and the increasing prevalence of video and multimodal data raise questions about whether concealing identities is always appropriate. Researchers may need to negotiate an ethical course of action here – one that secures very clear agreement about anonymity and about subsequent use of the data. Researchers need to be aware that visual material could be misused by others (for example, as an example of poor practice), and should take steps to prevent this as far as possible.

Any changes to the degree of anonymity afforded to participants should be considered in the light of potential harm that may be caused by doing so and, in particular, the rights to confidentiality of other individual participants or institutions. In some circumstances, potential changes may require renegotiation of consent, or may be decided against if they would impinge on the rights of others. Where this happens, researchers will need to lodge the fact that there has been a change of circumstances with their institutional ethics committee, and seek updated clearance. It is in the researcher’s interests to obtain any waiver of anonymity, or request for identification, in writing.

While many sponsors require researchers to make anonymised versions of data available for secondary analysis, this situation is rapidly changing. In future, sponsors may expect researchers to share with them un-anonymised, fully identifiable data that can be linked with other data that they hold. It is thus extremely important that in seeking consent from participants, researchers make it very explicit what kinds of data (if any) are later to be shared.

Researchers need to be aware of the possible consequences to participants should it prove possible for them to be identified by association or inference. They should take all reasonable precautions to avoid identification – for example, by fictionalising or by changing identifying features that may leave participants in peril when the researcher has departed (from overseas or sensitive settings in particular).

Anonymity is much harder to guarantee in digital contexts. The policies of some social media sites which require identification at signup may exacerbate this. Researchers need to be aware that participants’ understandings of their level of privacy in a particular online space may be inaccurate. Ambiguity about privacy within some online communities in which sensitive or illegal topics are being discussed, or material shared, raise further ethical concerns. Relatedly, researchers should consider the question of what online content, in what circumstances, they would be obligated to report to relevant authorities and/or online service providers, bearing in mind any agreements entered into regarding confidentiality and anonymity (see paragraphs 52 and 53, on disclosure). Researchers using data gathered in such contexts should inform the community concerned about how the data will be used.

Tensions may be raised between a participant’s voice and authentic response on the one hand, and anonymity on the other hand where, for instance, participants take photographs or video recordings in the context of visual ethnography. Researchers need to use their judgment about the appropriateness of anonymity in such circumstances.

Researchers must comply with the legal requirements in relation to the storage and use of personal data as stipulated in the UK by the Data Protection Act (1998) and any subsequent similar acts, including, from May 2018, its replacement: the much stricter General Data Protection Regulation (GDPR). 11 In essence, citizens are entitled to know how and why their personal data is being stored, to what uses it is being put and to whom it may be made available. Researchers must have participants’ explicit permission to disclose personal information to third parties, and are required to ensure that such parties are permitted to have access to that information. They are also required to independently confirm the identity of such persons to their own satisfaction, and must keep a record of any disclosures. The GDPR defines personal data more broadly as ‘any information relating to an identified or identifiable person’, and requires that sensitive personal data is given additional protection. Record-keeping and reporting of breaches are mandatory and compliance must be proven. Organisations such as schools may require a data protection policy and a named data protection officer.

The UK Data Protection Act (1998) and the GDPR that supersedes it also confer the right to private citizens to have access to any personal data that is stored, and which relates to them. Researchers seeking to exploit legal exclusions to these rights must have a clear justification. The Freedom of Information Act (2000) is applicable to requests for access to data held by public authorities, including state schools, but research data in these settings would be exempt from such requests where explicit confidentiality arrangements are in place. The release of such information would be a breach of personal confidence. 12

Researchers should ensure that data are kept securely, and that the form of any publication (including those published online) does not directly or indirectly lead to a breach of agreed confidentiality and anonymity. Measures recommended by the National Foundation for Educational Research (and in some cases required by GDPR) include, for example: the use of secure computer networks; ensuring that data is stored on secure premises; the use of password protection and data encryption; avoiding portable data storage devices such as laptops and USB sticks; using courier or secure electronic transfer when moving data; anonymising records; and ensuring that any third-party users of the data agree to a data-sharing agreement so that the same assurances are given for the protection of data. It is also advisable to avoid sharing data via email and other media that are vulnerable to hacking.

In an international context, researchers should be aware that it will not be possible to protect data stored within some political jurisdictions from scrutiny within that jurisdiction – and should take appropriate steps to ensure its security elsewhere.

Researchers who judge that the agreements they have made with participants about confidentiality and anonymity will allow the continuation of illegal behaviour which has come to light in the course of the research should carefully consider making disclosure to the appropriate authorities. If behaviour reported by participants is likely to be harmful to the participants or to others, the researchers must also consider disclosure. In some cases, such as revelations of abuse or proposed acts of terror, researchers may be under statutory duty to disclose confidential information to relevant authorities, and they must be aware of these responsibilities. Researchers should seek advice from a relevant responsible person before proceeding to disclosure if and when appropriate (students should seek advice from supervisors). Insofar as it does not undermine or obviate the disclosure, or jeopardise researcher safety, researchers should inform the participants, or their guardians or responsible others, of their intentions and reasons for disclosure. In some parts of the world low-level corruption is so endemic that it may be encountered very often. In such contexts, researchers will have to make a situated judgement as to what, if anything, to report, what to describe and what to accept.

At all times, the decision to override agreements on confidentiality and anonymity should be taken after careful and thorough deliberation. In such circumstances it is in the researcher’s interests to make contemporaneous notes on decisions and the reasoning behind them, in case a misconduct complaint or other serious consequence arises. The researcher should also consider very carefully whether overriding confidentiality and anonymity compromises the integrity and/or usefulness of data, and withdraw any compromised data from the study.

Responsibilities to sponsors, clients and stakeholders in research

A stakeholder of research is considered to be any person or body who has a direct interest in its framing and success. A sponsor of research is considered to be a stakeholder that funds or commissions research (such as a research charity or philanthropic foundation, a national research council or other government body, or a non-governmental organisation), or that facilitates it by allowing and enabling access to resources needed to carry out the research, such as data and participants (an examinations body, for example).

Written contracts are considered the norm for funded or commissioned research. Such agreements should, wherever possible and especially in the case of publicly funded research, take into account the rights of the public within a democracy to have open access to the results of research. They should minimally cover the purpose of the research, the research methods to be used, any conditions of access to data or participants, ownership of data, the researcher’s right to publish, requirements for reporting and dissemination (including the need for transparency), and deadlines for completion of the work and the accounting for the use of funds. In recognition of the dynamics of research, agreements should also include provision for negotiating changes sought by either the researchers or the sponsors.

It is in researchers’ interests that respective responsibilities and entitlements should be agreed with sponsors at the outset of the research. Where the sponsor acts essentially as a host or facilitator for research, researchers should, out of courtesy, inform them of the work they propose to undertake. For example, a group of teachers engaging in a process of action research as part of curriculum renewal should inform the school management of their intentions.

In negotiating sponsorship for their research, researchers should provide honest and complete details of their competence and capacity to undertake the research that is proposed. Researchers are encouraged to think carefully about how they position themselves and their research design, analysis and interpretation in relation to the interests of their sponsors and stakeholders. Any conflicts of interest or compromises to the integrity of the research must be made clear and open to scrutiny.

Researchers should acknowledge sponsors of and participants in their studies in any publications or dissemination activities.

These guidelines should not be interpreted as privileging particular research approaches or methods over others: the Association respects the diverse range of possible approaches. Those researchers who prefer or promote specific methods, theories or philosophies of research should have knowledge of alternative approaches sufficient to assure sponsors that they have considered these, and that the needs of the research are being properly addressed. Sponsors should be offered a full, honest and accessible justification for the final choice of methods.

Researchers should, within the context and boundaries of their chosen methods, theories and philosophies of research, communicate the extent to which their data collection and analysis techniques, and the inferences to be drawn from their findings, are robust and can be seen to meet the criteria and markers of quality and integrity applied within different research approaches.

The ‘community of educational researchers’ is considered to mean all those engaged in educational research – including, for example, students following research-based programmes of study and independent researchers, as well as staff who conduct educational research in their employment within organisations such as universities, schools, local and national government, charities and commercial bodies.

All educational researchers should aim to protect the integrity and reputation of educational research by ensuring that they conduct their research to the highest standards. Researchers should contribute to the community spirit of critical analysis and constructive criticism that generates improvement in practice and enhancement of knowledge.

Educational researchers should not criticise their peers in a defamatory or unprofessional manner, in any medium.

It is recommended that researchers, in communications or published information about research projects, identify an appropriate contact whom participants or other research stakeholders can contact in order to raise questions or concerns, including those concerning formal complaints procedures.

Where researchers, participants or other stakeholders become aware of examples of malpractice or potential malpractice by a researcher, they are advised to contact the appropriate individual, organisation or authority and raise their concern, following an established complaints procedure. If there is no established complaints procedure, the complainant should respect the researcher’s right to respond and, with due consideration of the important principle of the public’s right to know, they should avoid bringing the community into disrepute through public accusations or allegations. This is relevant, for example, in the case of potential social media trolling as it relates to commenting on research.

In any instance in which a stakeholder or member of the public raises a concern or makes a complaint, researchers have a duty to consider how to respond with appropriate action.

Plagiarism is the unattributed use of text and/or data, presented as if they were by the plagiarist. The (2008) Committee on Publication Ethics (COPE) guidelines 13 (to which all BERA journals subscribe) stipulate that all sources should be disclosed, and if large amounts of other people’s (or the researcher’s own) written or illustrative material are to be used, permission must be sought and acknowledgement made. In clear cases of plagiarism, the author should be contacted in writing, ideally enclosing documentary evidence. If no response is received, the COPE advice is to ‘contact the author’s institution requesting your concern is passed to the author’s superior and/or person responsible for research governance’. 14

Attribution should include explicitly recognising authors of digital content, in all cases in which an author or creator can be identified. As well as text, this includes images, diagrams, presentations, multimedia content and other forms of content. Researchers need to be aware that a great deal of digital content is subject to copyright, and cannot be freely reused or modified unless it is explicitly licensed as such – for example by means of one of the ‘Creative Commons’ (CC) licences. 15 Authors retain copyright of CC-licensed material (which may be published in print or digitally), but choose to permit reuse, distribution and sometimes adaptation, depending on the licence terms; any copies or modifications have to be made available under the original licence terms and must link to that license. Researchers have the responsibility of checking the conditions for reuse, and for attributing the author(s) in all cases. 16

Subject to any limitations imposed by agreements to protect confidentiality and anonymity, researchers should endeavour to make their data and methods amenable to reasonable external scrutiny. Ideally, researchers will make shareable anonymised versions of data available for secondary analysis. They should be fully aware (and make participants aware) of when funding bodies require this (as is typically the case with UK government funding).

Assessment of the quality of the evidence supporting any inferences is an especially important feature of any research, and should be open to scrutiny. Where sponsors initiate a request for scrutiny, and disclosure of aspects of the data may be injurious to participants, researchers should consider assuring the sponsor of the integrity of the work through the scrutiny of a mutually acceptable third party, who would also be bound by any existing non-disclosure agreements.

Negative results of interventions and evaluations should be reported. Evaluations should be registered beforehand with an official body that maintains a platform for this purpose 17 (many sponsors require this in the UK). A condition of registration is that researchers report the results of their research – including negative results – in full at the specified end-date of a project. It should therefore allay any concerns that negative results will be withheld.

Responsibilities for publication and dissemination

Educational researchers should communicate their findings, and the practical significance of their research, in a clear, straightforward fashion, and in language judged appropriate to the intended audience(s). Researchers have a responsibility to make the results of their research public for the benefit of educational professionals, policymakers and the wider public, subject only to the provisos indicated in subsequent paragraphs. They should not accept contractual terms that obstruct their exercise of this responsibility.

Where research is conducted in international settings in which English is not the prevalent language, researchers should seek to make the fruits of their research available in a language that makes it locally as well as internationally accessible.

To assist researchers in making the results of their research public, consideration should be given to providing open access (without a paywall) to published research findings. In the UK, researchers can use the UK Scholarly Communications Licence 18 to make peer-reviewed manuscripts publicly available using a Creative Commons licence (see section 68 above). Public sponsors of research, such as the UK Research Councils, may also require research to be published open-access. Mindful of the potential impact of research findings outside of academia or specific educational institutions and organisations, researchers should think carefully about the implications of publishing in outlets that restrict public access to their findings.

Researchers acting as consultants should be particularly aware of potential constraints upon publishing findings from projects which their institutions, sponsors, partners or publishers may consider to be commercially sensitive, and whose findings may, in whole or in part, need to remain confidential for that reason.

In some circumstances, research findings will be regarded as sensitive information by sponsors, commissioners or other research stakeholders (because they raise politically or culturally controversial issues, for example, or because they may result in negative publicity for an organisation). When researchers become aware that research findings are likely to be controversial, they should aim to inform stakeholders prior to publication, and negotiate with those stakeholders a fair publication strategy that takes into consideration the public interest in the findings, the researcher’s need to publish, and the stakeholders’ concerns.

Researchers must not bring research into disrepute by in any way falsifying, distorting, suppressing, selectively reporting or sensationalising their research evidence or findings, either in publications based on that material, or as part of efforts to disseminate or promote that work.

The authorship of publications normally comprises a list of everyone who has made a substantive and identifiable contribution to the research being reported. Examples of this include: contributing generative ideas, conceptual schema or analytic categories; writing first drafts or substantial portions of text; significant rewriting or editing; contributing significantly to relevant literature reviewing; and contributing to data collection and analysis, and to judgements and interpretations made in relation to it. Where research has involved collaboration across different roles or professions – between education researchers who are academics and those who are teachers or other practitioners, for example – then anyone who has made a substantive contribution should be credited as a co-author.

Academic status or any other indicator of seniority does not determine first authorship. Rather, the order of authorship should reflect relative leadership and contributions made. Alternatively, co-authors may agree to a simple alphabetic listing of their names. Consensual agreement on authorship should be gained as early as possible in the writing process.

Researchers should not use research carried out with co-researchers as the basis of individual outputs without the agreement of the co-researchers concerned.

Researchers and sponsors have the right to dissociate themselves publicly from accounts of research in which they were involved, but that are authored by others, where they consider the presentation and/or content of those accounts to be misleading or unduly selective. Arbitration may be useful in order to reach agreement before such dissemination.

For contracted and sponsored research, the contract will normally cover: methodologies, reporting processes and publication and dissemination strategies, including how the researcher’s name will appear and whether the researcher may independently publish the findings. Research outcomes are unpredictable, and discussions to resolve sensitive issues are both to be expected and advisable prior to publication of findings.

The format(s) in which research is published, and the means by which those publications are disseminated, should take into account the needs and interests of the communities that were involved in the research. Researchers have a responsibility to share their findings with participants and their wider social groups as fully as possible, while maintaining confidentiality.

Safeguarding the physical and psychological wellbeing of researchers is part of the ethical responsibility of employing institutions and sponsors, as well as of researchers themselves. Safety is a particular concern in qualitative research, as researchers may be conducting fieldwork in situations that are potentially risky. Researchers should be aware of the legal responsibilities as well as the moral duty of institutions towards the safety of staff and students. Institutions, sponsors and independent researchers should consider whether an in-depth risk assessment form and ongoing monitoring of researcher safety is appropriate, especially for those undertaking fieldwork, working abroad and/or investigating sensitive issues; this may be mandatory for postgraduate students. Researchers, principal investigators, students and their supervisors should ideally be offered training on researcher safety. Specialist training should be made available to researchers entering conflict or post-conflict settings internationally, or areas with high risk of disease. 19

Employers and sponsors need to avoid exploiting differences in the conditions of work and roles of other researchers, including student researchers and those on time-limited contracts. Employers are also responsible for supporting researchers’ personal and professional career development. The BERA Charter for Research Staff in Education provides helpful guidance on these issues. 20 Researchers employed in higher education institutions in the UK are covered by the Concordat to Support the Career Development of Researchers, 21 which stipulates the standards that research staff can expect from the institution, as well as their responsibilities as researchers.

If you have any feedback or queries about these ethical guidelines, please contact [email protected] .

While BERA cannot provide ethical guidance further to these guidelines, or comment on individual cases, we value the feedback of the educational research community, and will endeavour to address any and all points and concerns raised in subsequent editions of these guidelines.

HISTORICAL NOTE

The provenance of these guidelines can be traced back to a BERA invitational seminar convened by John Elliott and held at Homerton College, Cambridge in March 1988. The seminar led to a report published in Research Intelligence 31 (February 1989), which called for a code of practice to be drawn up. In 1991, BERA Council invited Caroline Gipps and Helen Simons to formulate a set of guidelines, drawing with permission from the Elliott report and the ethical guidelines recently published by the American Educational Research Association. They published these for members’ comment in Research Intelligence 43 (Summer 1992), and later that year they were formally adopted.

As a code of practice the guidelines were universally welcomed; however, they also attracted a degree of criticism in relation to their scope and application. An example of this was the critique presented by Peter Foster at the 1996 BERA conference. Following Peter Foster’s death in 1999, his paper was reproduced in Research Intelligence 67 as a tribute to his work. Michael Bassey, the then academic secretary of BERA, used the paper to promote debate in the BERA Council and, at the beginning of her presidency in September 2001, Anne Edwards announced her intention to update the 1992 guidelines.

In the spring of 2002, a working group comprising John Gardner (chair), Ann Lewis and Richard Pring began the task of revising the guidelines. The revision built on the 1992 statement to recognise the academic tensions that a multidisciplinary community generates when dealing with the complex research issues that characterise education contexts, and to include the field of action research. Over the next 18 months several consultative exercises were carried out, and in the spring of 2004 the final draft of the Revised Ethical Guidelines for Educational Research 22 was moved by John Furlong (president) and formally adopted by Council. These guidelines stood unchanged until concerns began to be raised about aspects of some contemporary research contracts running contrary to the Association’s declared principles and ethical code – for example, those that prevent or inhibit publication.

Therefore, in late 2008 Pamela Munn (president) set up a working group to examine and make recommendations on these issues. The subsequent report, from David Bridges (chair), Sean Hayes, Jeremy Hoad, Saville Kushner, Olwen McNamara, Ian Menter and Nigel Norris, came to Council in November 2009. This report refined and strengthened the Association’s position on the rights of researchers in commissioned research contexts. It recommended a number of further changes and updates, including the need for updated guidelines on culturally sensitive issues. Council accepted the majority of the changes, and asked another small group, comprising Uvanney Maylor, Pat Thompson and David Bridges, to develop the final amendments on cultural sensitivity. The new guidelines were then moved by John Gardner (president) and formally adopted by Council in June 2011. 23

In 2015 BERA Council and the Academic Publications Committee (APC) convened a subcommittee to review the 2011 ethical guidelines and suggest what may need updating, particularly with regards to how the guidelines accommodate and facilitate practitioner research, how they integrate technological development, and any other pertinent issues arising since the last review. The group was chaired by Anna Mountford-Zimdars and included Rachel Brooks, Alison Fox and David Lundie. The recommendations of the subcommittee were reported to APC and BERA Council, leading to the establishment of an Ethical Guidelines Review Working Group in 2016, chaired by Sara Hennessy (BERA Council member and vice-chair of APC). This group included Ruth Boyask, Alison Fox, David Lundie, Marilyn Leask and Lesley Saunders, assisted by Jodie Pennacchia.

The working group oversaw the review of the guidelines and engaged in consultation with BERA members and a wide range of experts, learned societies and stakeholders, 24 as well as reviewing key publications. Significant revisions were made in order to update the guidelines to incorporate new concerns such as those raised by online and social media research. The consultation process led the review to consider more explicitly the range of contexts for educational research, in particular: research by organisations outside higher education; school-based and practitioner research; studies carried out in international contexts; online and social media-related research. These revised guidelines are the result of considered deliberation about the ethical issues associated with changes in society and technological advances as they affect educational research. In response to our consultations, the guidelines themselves now take a more deliberative and less prescriptive approach in their language.

The working group reported to Council and APC during the spring of 2017. As part of the discussion on the draft, ways in which the guidelines could be made more accessible were endorsed. This included an interactive digital version with links where appropriate, and the curation of a number of illustrative case studies. These will be developed to sit alongside the full downloadable guidelines. After independent peer review, the updated draft was considered by BERA’s General Purposes Committee (GPC) in November 2017 and then passed to Pat Sikes who worked with Gary McCulloch in preparing an updated draft for consideration by Council in January 2018. This published version has then been edited further in the light of Council discussion, and was endorsed by GPC prior to publication. As well as developing additional resources to support these guidelines, BERA Council is committed to a regular review and updating of these guidelines. Any changes made to the text of these guidelines will be described and dated in an appendix to any subsequently published versions of this document.

Acknowledgments

In the putting together of this version of the guidelines, BERA would like to acknowledge the contribution of the original review group of Anna Mountford-Zimdars, Rachel Brooks, Alison Fox and David Lundie. We also acknowledge the efforts of the working group that developed the draft – Sara Hennessy, Ruth Boyask, Alison Fox, David Lundie, Marilyn Leask and Lesley Saunders, assisted by Jodie Pennacchia. Thanks also to those who advised and worked on the development of the final guidelines – Alis Oancea, Pat Sikes, Gary McCulloch and Ross Fulton, as well as all members of BERA Council.

1 British Educational Research Association [BERA] (2017) ‘BERA Handbook: Member Code of Conduct’, London. https://www.bera.ac.uk/about/bera-handbook

2 Furthermore, it is BERA’s ambition to produce and publish a series of case studies that illustrate how researchers have put ethical guidelines and principles into practice within specific projects and contexts. This document, and its associated webpages, will be updated with details of these case study publications as and when they are published.

3 Academy of Social Sciences [AcSS] (2015) ‘Five Ethics Principles for Social Science Research’, London. https://www.acss.org.uk/wp-content/uploads/2016/06/5-Ethics-Principles-for-Social-Science-Research-Flyer.pdf

4 Reproduced from AcSS 2015.

5 BERA (2017).

6 BERA (2017): paragraphs 19-29.

7 Economic and Social Research Council (no date) ‘Internet-mediated research’, webpage. https://esrc.ukri.org/funding/guidance-for-applicants/research-ethics/frequently-raised-topics/internet-mediated-research/

Hewson C, Buchanan T, Brown I, Coulson N, Hagger-Johnson G, Joinson A, Krotoski A and Oates J (2013) Ethics Guidelines for internet-mediated research , London: British Psychological Society. https://www.bps.org.uk/news-and-policy/ethics-guidelines-internet-mediated-research-2017

Markham A and Buchanan E (2012) ‘Ethical decision-making and Internet research 2.0: Recommendations from the AoIR ethics working committee (Version 2.0)’, Chicago: Association of Internet Researchers. http://aoir.org/reports/ethics2.pdf

Townsend L and Wallace C (2016) Social Media Research: A Guide to Ethics , Aberdeen: University of Aberdeen and Economic and Social Research Council. http://www.dotrural.ac.uk/social-media-research-ethics/

Zevenbergen B (ed) (2016) ‘Networked Systems Ethics’, webpage. http://networkedsystemsethics.net

8 See Office for Standards in Education, Children’s Services and Skills [Ofsted] (2012) ‘Young People’s Guide to The United Nations Convention on the Rights of the Child (UNCRC)’, London. https://www.childrenscommissioner.gov.uk/publication/guide-to-the-uncrc/

9 In the context of the European Union, ‘Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016’ (General Data Protection Regulation, or GDPR) is enforceable as of 25 May 2018, and is ‘designed to harmonize data privacy laws across Europe, to protect and empower all EU citizens data privacy and to reshape the way organizations across the region approach data privacy’ ( https://www.eugdpr.org/ ). It defines ‘consent’ and conditions for it, and describes its implications for the lawfulness of processing personal data.

The UK Information Commissioner’s Office, which publishes a guide to the GDPR, suggests that the provisions in this legislation directly relevant to the issue of consent are articles 4(11), 6(1)(a), 7, 8 and 9(2)(a), and recitals 32, 38, 40, 42, 43 and 171 (see https://ico.org.uk/for-organisations/guide-to-the-general-data-protection-regulation-gdpr/lawful-basis-for-processing/consent/ ).

For articles: Council of the European Union and European Parliament (2016) ‘Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (General Data Protection Regulation) (Text with EEA relevance)’, Brussels. https://publications.europa.eu/en/publication-detail/-/publication/3e485e15-11bd-11e6-ba9a-01aa75ed71a1/language-en

For recitals: Intersoft Consulting (no date) ‘Recitals’, General Data Protection Regulation (GDPR), online database. https://gdpr-info.eu/recitals/

See paragraphs 60-62 of these guidelines for further discussion of GDPR.

10 The Association notes that incentives to participate in research may be more commonly offered, and may not necessarily be considered bad practice, in disciplines other than educational research.

11 Council of the European Union and European Parliament (2016).

12 See the advice on the Freedom of Information Act (2000) published by the Information Commissioner’s Office: https://ico.org.uk/for-organisations/guide-to-freedom-of-information/what-is-the-foi-act/ .

13 See https://publicationethics.org/resources/guidelines .

14 Committee on Publication Ethics (2008) ‘What to do if you suspect plagiarism: (b) Suspected plagiarism in a published article’, Eastleigh. https://publicationethics.org/files/u2/02B_Plagiarism_Published.pdf

15 See https://creativecommons.org/licenses/ .

16 For helpful guidance for schools on ‘open educational resources’ (OER) – learning materials published under an open licence that allows anyone to ‘use, share and build on’ those resources free of charge – see the resources published by Leicester City Council: https://schools.leicester.gov.uk/services/planning-and-property/building-schools-for-the-future-bsf/open-education-for-schools/ .

17 Examples of such registries include the American Economic Association’s registry for randomised controlled trials (see https://www.socialscienceregistry.org/ ), and the World Health Organization’s International Clinical Trials Registry Platform (see http://www.who.int/ictrp/en/ ).

18 See http://ukscl.ac.uk/ .

19 See the recommendations of the ESRC National Centre for Research Methods qualitative research node (Qualiti’s) 2006 inquiry, reported in Fincham B, Bloor M and Sampson H (2007) ‘Qualiti (NCRM) Commissioned Inquiry into the risk to well-being of researchers in qualitative research’, Qualitative Researcher 6: 2-4. http://www.cardiff.ac.uk/socsi/qualiti/QualitativeResearcher/QR_Issue6_Sep07.pdf

20 British Educational Research Association (2013) The BERA Charter for Research Staff in Education , London. https://www.bera.ac.uk/researchers-resources/publications/the-bera-charter-for-research-staff-in-education

21 Vitae (2008) The Concordat to Support the Career Development of Researchers , London. https://www.vitae.ac.uk/policy/vitae-concordat-vitae-2011.pdf

22 Archived at https://www.bera.ac.uk/researchers-resources/publications/revised-ethical-guidelines-for-educational-research-2004 .

23 The most recent version of the 2011 edition of BERA’s Ethical Guidelines for Educational Research remains archived online at https://www.bera.ac.uk/researchers-resources/publications/ethical-guidelines-for-educational-research-2011 .

24 Among the experts who gave substantial responses to our consultation were representatives from the National Education Union, teaching school alliances, Chartered College of Teaching, Higher Education Funding Council for England, National Foundation for Educational Research, Social Policy Association, Social Research Association, Centre for the Use of Research and Evidence in Education, and the Education Achievement Service for South East Wales.

More related content

Resources for research 5 Sep 2024

BERA in the news 6 Sep 2024

News 30 Aug 2024

News 12 Aug 2024

News 2 Aug 2024

• Tools and Resources
• Customer Services
• Original Language Spotlight
• Alternative and Non-formal Education 
• Cognition, Emotion, and Learning
• Curriculum and Pedagogy
• Education and Society
• Education, Change, and Development
• Education, Cultures, and Ethnicities
• Education, Gender, and Sexualities
• Education, Health, and Social Services
• Educational Administration and Leadership
• Educational History
• Educational Politics and Policy
• Educational Purposes and Ideals
• Educational Systems
• Educational Theories and Philosophies
• Globalization, Economics, and Education
• Languages and Literacies
• Professional Learning and Development
• Research and Assessment Methods
• Technology and Education
• Share This Facebook LinkedIn Twitter

Article contents

Evidence-based educational practice.

• Tone Kvernbekk Tone Kvernbekk University of Oslo
• https://doi.org/10.1093/acrefore/9780190264093.013.187
• Published online: 19 December 2017

Evidence-based practice (EBP) is a buzzword in contemporary professional debates, for example, in education, medicine, psychiatry, and social policy. It is known as the “what works” agenda, and its focus is on the use of the best available evidence to bring about desirable results or prevent undesirable ones. We immediately see here that EBP is practical in nature, that evidence is thought to play a central role, and also that EBP is deeply causal: we intervene into an already existing practice in order to produce an output or to improve the output. If our intervention brings the results we want, we say that it “works.”

How should we understand the causal nature of EBP? Causality is a highly contentious issue in education, and many writers want to banish it altogether. But causation denotes a dynamic relation between factors and is indispensable if one wants to be able to plan the attainment of goals and results. A nuanced and reasonable understanding of causality is therefore necessary to EBP, and this we find in the INUS-condition approach.

The nature and function of evidence is much discussed. The evidence in question is supplied by research, as a response to both political and practical demands that educational research should contribute to practice. In general, evidence speaks to the truth value of claims. In the case of EBP, the evidence emanates from randomized controlled trials (RCTs) and presumably speaks to the truth value of claims such as “if we do X, it will lead to result Y.” But what does research evidence really tell us? It is argued here that a positive RCT result will tell you that X worked where the RCT was conducted and that an RCT does not yield general results.

Causality and evidence come together in the practitioner perspective. Here we shift from finding causes to using them to bring about desirable results. This puts contextual matters at center stage: will X work in this particular context? It is argued that much heterogeneous contextual evidence is required to make X relevant for new contexts. If EBP is to be a success, research evidence and contextual evidence must be brought together.

• effectiveness
• INUS conditions
• practitioner perspective

Introduction

Evidence-based practice, hereafter EBP, is generally known as the “what works” agenda. This is an apt phrase, pointing as it does to central practical issues: how to attain goals and produce desirable results, and how we know what works. Obviously, this goes to the heart of much (but not all) of the everyday activity that practitioners engage in. The “what works” agenda is meant to narrow the gap between research and practice and be an area in which research can make itself directly useful to practice. David Hargreaves, one of the instigators of the EBP debate in education, has stated that the point of evidence-based research is to gather evidence about what works in what circumstances (Hargreaves, 1996a , 1996b ). Teachers, Hargreaves said, want to know what works; only secondarily are they interested in understanding the why of classroom events. The kind of research we talk about is meant to be relevant not only for teachers but also for policymakers, school developers, and headmasters. Its purpose is to improve practice, which largely comes down to improving student achievement. Hargreaves’s work was supported by, for example, Robert Slavin, who stated that education research not only can address questions about “what works” but also must do so (Slavin, 2004 ).

All the same, despite the fact that EBP, at least at the outset, seems to speak directly to the needs of practitioners, it has met with much criticism. It is difficult to characterize both EBP and the debate about it, but let me suggest that the debate branches off in different but interrelated directions. We may roughly identify two: what educational research can and should contribute to practice and what EBP entails for the nature of educational practice and the teaching profession. There is ample space here for different definitions, different perspectives, different opinions, as well as for some general unclarity and confusions. To some extent, advocates and critics bring different vocabularies to the debate, and to some extent, they employ the same vocabulary but take very different stances. Overall in the EBP conceptual landscape we find such concepts as relevance, effectiveness, generality, causality, systematic reviews, randomized controlled trials (RCTs), what works, accountability, competences, outcomes, measurement, practical judgment, professional experience, situatedness, democracy, appropriateness, ends, and means as constitutive of ends or as instrumental to the achievement of ends. Out of this tangle we shall carefully extract and examine a selection of themes, assumptions, and problems. These mainly concern the causal nature of EBP, the function of evidence, and EBP from the practitioner point of view.

Definition, History, and Context

The term “evidence-based” originates in medicine—evidence-based medicine—and was coined in 1991 by a group of doctors at McMaster University in Hamilton, Ontario. Originally, it denoted a method for teaching medicine at the bedside. It has long since outgrown the hospital bedside and has become a buzzword in many contemporary professions and professional debates, not only in education, but also leadership, psychiatry, and policymaking. The term EBP can be defined in different ways, broadly or more narrowly. We shall here adopt a parsimonious, minimal definition, which says that EBP involves the use of the best available evidence to bring about desirable outcomes, or conversely, to prevent undesirable outcomes (Kvernbekk, 2016 ). That is to say, we intervene to bring about results, and this practice should be guided by evidence of how well it works. This minimal definition does not specify what kinds of evidence are allowed, what “based” should mean, what practice is, or how we should understand the causality that is inevitably involved in bringing about and preventing results. Minimal definitions are eminently useful because they are broad in their phenomenal range and thus allow differing versions of the phenomenon in question to fall under the concept.

We live in an age which insists that practices and policies of all kinds be based on research. Researchers thus face political demands for better research bases to underpin, inform and guide policy and practice, and practitioners face political demands to make use of research to produce desirable results or improve results already produced. Although the term EBP is fairly recent, the idea that research should be used to guide and improve practice is by no means new. To illustrate, in 1933 , the School Commission of Norwegian Teacher Unions (Lærerorganisasjonenes skolenevnd, 1933 ) declared that progress in schooling can only happen through empirical studies, notably, by different kinds of experiments and trials. Examples of problems the commission thought research should solve are (a) in which grade the teaching of a second language should start and (b) what the best form of differentiation is. The accumulated evidence should form the basis for policy, the unions argued. Thus, the idea that pedagogy should be based on systematic research is not entirely new. What is new is the magnitude and influence of the EBP movement and other, related trends, such as large-scale international comparative studies (e.g., the Progress in International Reading Literacy Study, PIRLS, and the Programme for International Student Assessment, PISA). Schooling is generally considered successful when the predetermined outcomes have been achieved, and education worldwide therefore makes excessive requirements of assessment, measurement, testing, and documentation. EBP generally belongs in this big picture, with its emphasis on knowing what works in order to maximize the probability of attaining the goal. What is also new, and quite unprecedented, is the growth of organizations such as the What Works Clearinghouses, set up all around the world. The WWCs collect, review, synthesize, and report on studies of educational interventions. Their main functions are, first, to provide hierarchies that rank evidence. The hierarchies may differ in their details, but they all rank RCTs, meta-analyses, and systematic reviews on top and professional judgment near the bottom (see, e.g., Oancea & Pring, 2008 ). Second, they provide guides that offer advice about how to choose a method of instruction that is backed by good evidence; and third, they serve as a warehouse, where a practitioner might find methods that are indeed backed by good evidence (Cartwright & Hardie, 2012 ).

Educationists today seem to have a somewhat ambiguous relationship to research and what it can do for practice. Some, such as Robert Slavin ( 2002 ), a highly influential educational researcher and a defender of EBP, think that education is on the brink of a scientific revolution. Slavin has argued that over time, rigorous research will yield the same step-by-step, irreversible progress in education that medicine has enjoyed because all interventions would be subjected to strict standards of evaluation before being recommended for general use. Central to this optimism is the RCT. Other educationists, such as Gert Biesta ( 2007 , 2010 ), also a highly influential figure in the field and a critic of EBP, are wary of according such weight to research and to the advice guides and practical guidelines of the WWCs for fear that this might seriously restrict, or out and out replace, the experience and professional judgment of practitioners. And there matters stand: EBP is a huge domain with many different topics, issues, and problems, where advocates and critics have criss-crossing perspectives, assumptions, and value stances.

The Causal Nature of Evidence-Based Practice

As the slogan “what works” suggests, EBP is practical in nature. By the same token, EBP is also deeply causal. Works is a causal term, as are intervention, effectiveness , bring about , influence , and prevent . In EBP we intervene into an already existing practice in order to change its outcomes in what we judge to be a more desirable direction. To say that something (an intervention) works is roughly to say that doing it yields the outcomes we want. If we get other results or no results at all, we say that it does not work. To put it crudely, we do X, and if it leads to some desirable outcome Y, we judge that X works. It is the ambition of EBP to provide knowledge of how intervention X can be used to bring about or produce Y (or improvements in Y) and to back this up by solid evidence—for example, how implementing a reading-instruction program can improve the reading skills of slow or delayed readers, or how a schoolwide behavioral support program can serve to enhance students’ social skills and prevent future problem behavior. For convenience, I adopt the convention of calling the cause (intervention, input) X and the effect (result, outcome, output) Y. This is on the explicit understanding that both X and Y can be highly complex in their own right, and that the convention, as will become clear, is a simplification.

There can be no doubt that EBP is causal. However, the whole issue of causality is highly contentious in education. Many educationists and philosophers of education have over the years dismissed the idea that education is or can be causal or have causal elements. In EBP, too, this controversy runs deep. By and large, advocates of EBP seem to take for granted that causality in the social and human realm simply exists, but they tend not to provide any analysis of it. RCTs are preferred because they allow causal inferences to be made with a high degree of certainty. As Slavin ( 2002 ) put it, “The experiment is the design of choice for studies that seek to make causal conclusions, and particularly for evaluations of educational innovations” (p. 18). In contrast, critics often make much of the causal of nature of EBP, since for many of them this is reason to reject EBP altogether. Biesta is a case in point. For him and many others, education is a moral and social practice and therefore non causal. According to Biesta ( 2010 ):

The most important argument against the idea that education is a causal process lies in the fact that education is not a process of physical interaction but a process of symbolic or symbolically mediated interaction. (p. 34)

Since education is noncausal and EBP is causal, on this line of reasoning, it follows that EBP must be rejected—it fundamentally mistakes the nature of education.

Such wholesale dismissals rest on certain assumptions about the nature of causality, for example, that it is deterministic, positivist, and physical and that it essentially belongs in the natural sciences. Biesta, for example, clearly assumes that causality requires a physical process. But since the mid-1900s our understanding of causality has witnessed dramatic developments; arguably the most important of which is its reformulation in probabilistic terms, thus making it compatible with indeterminism. A quick survey of the field reveals that causality is a highly varied thing. The concept is used in different ways in different contexts, and not all uses are compatible. There are several competing theories, all with counterexamples. As Nancy Cartwright ( 2007b ) has pointed out, “There is no single interesting characterizing feature of causation; hence no off-the-shelf or one-size-fits-all method for finding out about it, no ‘gold standard’ for judging causal relations” (p. 2).

The approach to causality taken here is twofold. First, there should be room for causality in education; we just have to be very careful how we think about it. Causality is an important ingredient in education because it denotes a dynamic relationship between factors of various kinds. Causes make their effects happen; they make a difference to the effect. Causality implies change and how it can be brought about, and this is something that surely lies at the heart of education. Ordinary educational talk is replete with causal verbs, for example, enhance, improve, reduce, increase, encourage, motivate, influence, affect, intervene, bring about, prevent, enable, contribute. The short version of the causal nature of education, and so EBP, is therefore that EBP is causal because it concerns the bringing about of desirable results (or the preventing of undesirable results). We have a causal connection between an action or an intervention and its effect, between X and Y. The longer version of the causal nature of EBP takes into account the many forms of causality: direct, indirect, necessary, sufficient, probable, deterministic, general, actual, potential, singular, strong, weak, robust, fragile, chains, multiple causes, two-way connections, side-effects, and so on. What is important is that we adopt an understanding of causality that fits the nature of EBP and does not do violence to the matter at hand. That leads me to my second point: the suggestion that in EBP causes are best understood as INUS conditions.

The understanding of causes as INUS conditions was pioneered by the philosopher John Mackie ( 1975 ). He placed his account within what is known as the regularity theory of causality. Regularity theory is largely the legacy of David Hume, and it describes causality as the constant conjunction of two entities (cause and effect, input and output). Like many others, Mackie took (some version of) regularity theory to be the common view of causality. Regularities are generally expressed in terms of necessity and sufficiency. In a causal law, the cause would be held to be both necessary and sufficient for the occurrence of the effect; the cause would produce its effect every time; and the relation would be constant. This is the starting point of Mackie’s brilliant refinement of the regularity view. Suppose, he said, that a fire has broken out in a house, and that the experts conclude that it was caused by an electrical short circuit. How should we understand this claim? The short circuit is not necessary, since many other events could have caused the fire. Nor is it sufficient, since short circuits may happen without causing a fire. But if the short circuit is neither necessary nor sufficient, then what do we mean by saying that it caused the fire? What we mean, Mackie ( 1975 ) suggests, is that the short circuit is an INUS condition: “an insufficient but necessary part of a condition which is itself unnecessary but sufficient for the result” (p. 16), INUS being an acronym formed of the initial letters of the italicized words. The main point is that a short circuit does not cause a fire all by itself; it requires the presence of oxygen and combustible material and the absence of a working sprinkler. On this approach, therefore, a cause is a complex set of conditions, of which some may be positive (present), and some may be negative (absent). In this constellation of factors, the event that is the focus of the definition (the insufficient but necessary factor) is the one that is salient to us. When we speak of an event causing another, we tend to let this factor represent the whole complex constellation.

In EBP, our own intervention X (strategy, method of instruction) is the factor we focus on, the factor that is salient to us, is within our control, and receives our attention. I propose that we understand any intervention we implement as an INUS condition. Then it immediately transpires that X not only does not bring about Y alone, but also that it cannot do so.

Before inquiring further into interventions as INUS -conditions, we should briefly characterize causality in education more broadly. Most causal theories, but not all of them, understand causal connections in terms of probability—that is, causing is making more likely. This means that causes sometimes make their effects happen, and sometimes not. A basic understanding of causality as indeterministic is vitally important in education, for two reasons. First, because the world is diverse, it is to some extent unpredictable, and planning for results is by no means straightforward. Second, because we can here clear up a fundamental misunderstanding about causality in education: causality is not deterministic and the effect is therefore not necessitated by the cause. The most common phrase in causal theory seems to be that causes make a difference for the effect (Schaffer, 2007 ). We must be flexible in our thinking here. One factor can make a difference for another factor in a great variety of ways: prevent it, contribute to it, enhance it as part of a causal chain, hinder it via one path and increase it via another, delay it, or produce undesirable side effects, and so on. This is not just conceptual hair-splitting; it has great practical import. Educational researchers may tell us that X causes Y, but what a practitioner can do with that knowledge differs radically if X is a potential cause, a disabler, a sufficient cause, or the absence of a hindrance.

Interventions as INUS Conditions

Human affairs, including education, are complex, and it stands to reason that a given outcome will have several sources and causes. While one of the factors in a causal constellation is salient to us, the others jointly enable X to have an effect. This enabling role is eminently generalizable and crucial to understanding how interventions bring about their effects. As Mackie’s example suggests, enablers may also be absences—that is vital to note, since absences normally go under our radar.

The term “intervention” deserves brief mention. To some it seems to denote a form of practice that is interested only (or mainly) in producing measurable changes on selected output variables. It is not obvious that there is a clear conception of intervention in EBP, but we should refrain from imposing heavy restrictions on it. I thus propose to employ the broad understanding suggested by Peter Menzies and Huw Price ( 1993 )—namely, interventions as a natural part of human agency. We all have the ability to intervene in the world and influence it; that is, to act as agents. Educational interventions may thus take many forms and encompass actions, strategies, programs and methods of instruction. Most interventions will be composites consisting of many different activities, and some, for instance, schoolwide behavioral programs, are meant to run for a considerable length of time.

When practitioners consider implementing an intervention X, the INUS approach encourages them to also consider what the enabling conditions are and how they might allow X to produce Y (or to contribute to its production). Our general knowledge of house fires and how they start prompts us to look at factors such as oxygen, materials, and fire extinguishers. In other cases, we might not know what the enabling conditions are. Suppose a teacher observes that some of his first graders are reading delayed. What to do? The teacher may decide to implement what we might call “Hatcher’s method” (Hatcher et al., 2006 ). This “method” focuses on letter knowledge, single-word reading, and phoneme awareness and lasts for two consecutive 10-week periods. Hatcher and colleagues’ study showed that about 75% of the children who received it made significant progress. So should our teacher now simply implement the method and expect the results with his own students to be (approximately) the same? As any teacher knows, what worked in one context might not work in another context. What we can infer from the fact that the method, X, worked where the data were collected is that a sufficient set of support factors were present to enable X to work. That is, Hatcher’s method serves as an INUS condition in a larger constellation of factors that together are sufficient for a positive result for a good many of the individuals in the study population. Do we know what the enabling factors are—the factors that correspond to presence of oxygen and inflammable material and absence of sprinkler in Mackie’s example? Not necessarily. General educational knowledge may tell us something, but enablers are also contextual. Examples of possible enablers include student motivation, parental support (important if the method requires homework), adequate materials, a separate room, and sufficient time. Maybe the program requires a teacher’s assistant? The enablers are factors that X requires to bring about or improve Y; if they are missing, X might not be able to do its work.

Understanding X as an INUS condition adds quite a lot of complexity to the simple X–Y picture and may thus alleviate at least some of the EBP critics’ fear that EBP is inherently reductionist and oversimplified. EBP is at heart causal, but that does not entail a deterministic, simplistic or physical understanding. Rather, I have argued, to do justice to EBP in education its causal nature must be understood to be both complex and sophisticated. We should also note here that X can enter into different constellations. The enablers in one context need not be the same as the enablers in another context. In fact, we should expect them to be different, simply because contexts are different.

Evidence and Its Uses

Evidence is an epistemological concept. In its immediate surroundings we find such concepts as justification, support, hypotheses, reasons, grounds, truth, confirmation, disconfirmation, falsification, and others. It is often unclear what people take evidence and its function to be. In epistemology, evidence is that which serves to confirm or disconfirm a hypothesis (claim, belief, theory; Achinstein, 2001 ; Kelly, 2008 ). The basic function of evidence is thus summed up in the word “support”: evidence is something that stands in a relation of support (confirmation, disconfirmation) to a claim or hypothesis, and provides us with good reason to believe that a claim is true (or false). The question of what can count as evidence is the question of what kind of stuff can enter into such evidential relations with a claim. This question is controversial in EBP and usually amounts to criticism of evidence hierarchies. The standard criticisms are that such hierarchies unduly privilege certain forms of knowledge and research design (Oancea & Pring, 2008 ), undervalue the contribution of other research perspectives (Pawson, 2012 ), and undervalue professional experience and judgment (Hammersley, 1997 , 2004 ). It is, however, not of much use to discuss evidence in and of itself—we must look at what we want evidence for . Evidence is that which can perform a support function, including all sorts of data, facts, personal experiences, and even physical traces and objects. In murder mysteries, bloody footprints, knives, and witness observations count as evidence, for or against the hypothesis that the butler did it. In everyday life, a face covered in ice cream is evidence of who ate the dessert before dinner.

There are three important things to keep in mind concerning evidence. First, in principle, many different entities can play the role of evidence and enter into an evidentiary relation with a claim (hypothesis, belief). Second, what counts as evidence in each case has everything to do with the type of claim we are interested in. If we want evidence that something is possible, observation of one single instance is sufficient evidence. If we want evidence for a general claim, we at least need enough data to judge that the hypothesis has good inductive support. If we want to bolster the normative conclusion that means M1 serves end E better than means M2, we have to adduce a range of evidences and reasons, from causal connections to ethical considerations (Hitchcock, 2011 ). If we want to back up our hypothesis that the butler is guilty of stealing Lady Markham’s necklace, we have to take into consideration such diverse pieces of evidence as fingerprints, reconstructed timelines, witness observations and alibis. Third, evidence comes in different degrees of trustworthiness, which is why evidence must be evaluated—bad evidence cannot be used to support a hypothesis and does not speak to its truth value; weak evidence can support a hypothesis and speak to its truth value, but only weakly.

The goal in EBP is to find evidence for a causal claim. Here we meet with a problem, because causal claims come in many different shapes: for example, “X leads to Y,” “doing X sometimes leads to Y and sometimes to G,” “X contributes moderately to Y” and “given Z, X will make a difference to Y.” On the INUS approach the hypothesis is that X, in conjunction with a suitable set of support factors, in all likelihood will lead to Y (or will contribute positively to Y, or make a difference to the bringing about of Y). The reason why RCTs are preferred is precisely that we are dealing with causal claims. Provided that the RCT design satisfies all requirements, it controls for confounders, and makes it possible to distinguish correlations from causal connections and to draw causal inferences with a high degree of confidence. In RCTs we compare two groups, the study group and the control group. Random assignment is supposed to ensure that the groups have the same distribution of causal and other factors, save one—namely, the intervention X (but do note that the value of randomization has recently been problematized, most notably by John Worrall ( 2007 ). The standard result from an RCT is a treatment effect, expressed in terms of an effect size. An effect size is a statistical measure denoting average effect in the treatment group minus average effect in the control group (to simplify). We tend to assume that any difference between the groups requires a causal explanation. Since other factors and confounders are (assumed to be) evenly distributed and thus controlled for, we infer that the treatment, whatever it is, is the cause of the difference. Thus, the evidence-ranking schemes seem to have some justification, despite Cartwright’s insistence that there is no gold standard for drawing causal inferences. We want evidence for causal claims, and RCTs yield highly trustworthy evidence and, hence, give us good reason to believe the causal hypothesis. In most cases the causal hypothesis is of the form “if we do X it will lead to Y.”

Effectiveness

Effectiveness is much sought after in EBP. For example, Philip Davies ( 2004 ) describes the role of the Campbell Collaboration as helping both policymakers and practitioners make good decisions by providing systematic reviews of the effectiveness of social and behavioral interventions in education. The US Department of Education’s Identifying and Implementing Educational Practices Supported by Rigorous Evidence: A User Friendly Guide ( 2003 ) provides an example of how evidence, evidence hierarchies, effectiveness, and “what works” are tied together. The aim of the guide is to provide practitioners with the tools to distinguish practices that are supported by rigorous evidence from practices that are not. “Rigorous evidence” is here identical to RCT evidence, and the guide devotes an entire chapter to RCTs and why they yield strong evidence for the effectiveness of some intervention. Thus:

The intervention should be demonstrated effective, through well-designed randomized controlled trials, in more than one site of implementation;

These sites should be typical school or community settings, such as public school classrooms taught by regular teachers; and

The trials should demonstrate the intervention’s effectiveness in school settings similar to yours, before you can be confident that it will work in your schools/classrooms (p. 17).

Effectiveness is clearly at the heart of EBP, but what does it really mean? “Effectiveness” is a complex multidimensional concept containing causal, normative, and conceptual dimensions, all of which have different sides to them. Probabilistic causality comes in two main versions, one concerning causal strength and one concerning causal frequency or tendency (Kvernbekk, 2016 ). One common interpretation says that effectiveness concerns the relation between input and output—that is, the degree to which an intervention works. Effect sizes would seem to fall into this category, expressing as they do the magnitude of the effect and thereby the strength of the cause.

But a large effect size is not the only thing we want; we also want the cause to make its effect happen regularly across different contexts. In other words, we are interested in frequency . A cause may not produce its effect every time but often enough to be of interest. If we are to be able to plan for results, X must produce its effect regularly. Reproducibility of desirable results thus depends crucially of the tendency of the cause to produce its effect wherever and whenever it appears. Hence, the term “effectiveness” signals generality. In passing, the same generality hides in the term “works”—if an intervention works, it can be relied on to produce its desired results wherever it is implemented. The issue of scope also belongs to this generality picture: for which groups do we think our causal claim holds? All students of a certain kind, for example, first graders who are responsive to extra word and phoneme training? Some first graders somewhere in the world? All first graders everywhere?

The normative dimension of “what works,” or effectiveness, is equally important, also because it demonstrates so well that effectiveness is a judgment we make. We sometimes gauge effectiveness by the relation between desired output and actual output; that is, if the correlation between the two is judged to be sufficiently high, we conclude that the method of instruction in question is effective. In such cases, the result (actual or desired) is central to our judgment, even if the focus of EBP undeniably lies on the means, and not on the goals. In a similar vein, to conclude that X works, you must judge the output to be satisfactory (enough), and that again depends on which success criteria you adopt (Morrison, 2001 ). Next, we have to consider the temporal dimension: how long must an effect linger for us to judge that X works? Three weeks? Two months? One year? Indefinitely? Finally, there is a conceptual dimension to judgments of effectiveness: the judgment of how well X works also depends on how the target is defined. For example, an assessment of the effectiveness of reading-instruction methods depends on what it means to say that students can read. Vague target articulations give much leeway for judgments of whether the target (Y) is attained, which, in turn, opens the possibility that many different Xs are judged to lead to Y.

Given the different dimensions of the term “effectiveness,” we should not wonder that effectiveness claims often equivocate on whether they mean effectiveness in terms of strength or frequency or perhaps both. The intended scope is often unclear, the target may be imprecise and the success criteria too broad or too narrow or left implicit altogether. However, since reproducibility of results is vitally important in EBP, it stands to reason that generality—external validity—should be of the greatest interest. All the strategies Dean, Hubbell, Pitler, and Stone ( 2012 ) discuss in their book about classroom instruction that works are explicitly general, for example, that providing feedback on homework assignments will benefit students and help enhance their achievements. This is generality in the frequency and (large) scope sense. It is future oriented: we expect interventions to produce much the same results in the future as they did in the past, and this makes planning possible.

The evidence for general causal claims is thought to emanate from RCTs, so let us turn again to RCTs to see whether they supply us with evidence that can support such claims. It would seem that we largely assume that they do. The Department of Education’s guide, as we have seen, presupposes that two RCTs are sufficient to demonstrate general effectiveness. Keith Morrison ( 2001 ) thinks that advocates of EBP simply assume that RCTs ensure generalizability, which is, of course, exactly what one wants in EBP—if results are generalizable, we may assume that the effect travels to other target populations so that results are reproducible and we can plan for their attainment. But does RCT evidence tell us that a cause holds widely? No, Cartwright ( 2007a ) argued, RCTs require strong premises, and strong premises do not hold widely. Because of design restrictions, RCT results hold formally for the study group (the sample) and only for that group, she insists. Methods that are strong on internal validity are correspondingly weak on external validity. RCTs establish efficacy, not effectiveness. We tend to assume without question, Cartwright argues, that efficacy is evidence for effectiveness. But we should not take this for granted—either it presumes that the effect depends exclusively on the intervention and not on who receives it, or it relies on presumed commonalities between the study group and the target group. This is a matter of concern to EBP and its advocates, because if Cartwright is correct, RCT evidence does not tell us what we think it tells us. Multiple RCTs will not solve this problem; the weakness of enumerative induction—inferences from single instances to a general conclusion—is well known. So how then can we ground our expectation that results are reproducible and can be planned for?

The Practitioner Perspective

EBP, as it is mostly discussed, is researcher centered. The typical advice guides, such as that of the What Works Clearinghouse, tend to focus on the finding of causes and the quality of the evidence produced. Claims and interventions should be rigorously tested by stringent methods such as RCTs and ranked accordingly. The narrowness of the kind of evidence thus admitted (or preferred) is pointed out by many critics, but it is of equal importance that the kind of claims RCT evidence is evidence for is also rather narrow. Shifting the focus from research to practice significantly changes the game. And bring in the practitioners we must—EBP is eminently practical in nature, concerning as it does the production of desirable results. Putting practice center stage means shifting from finding causes and assessing the quality of research evidence to using causes to produce change. In research we can control for confounders and keep variables fixed. In practice we can do no such thing; hence the significant change of the game.

The claim a practitioner wants evidence for is not the same claim that a researcher wants evidence for. The researcher wants evidence for a causal hypothesis, which we have seen can be of many different kinds, for example, the contribution of X to Y. The practitioner wants evidence for a different kind of claim—namely, whether X will contribute positively to Y for his students, in his context. This is the practitioner’s problem: the evidence that research provides, rigorous as it may be, does not tell him whether a proposed intervention will work here , for this particular target group. Something more is required.

Fidelity is a demand for faithfulness in implementation: if you are to implement an intervention that is backed by, say, two solid RCTs, you should do it exactly as it was done where the evidence was collected. The minimal definition of EBP adopted here leaves it open whether fidelity should be included or not, but there can be no doubt that both advocates and critics take it that it is—making fidelity one of the most controversial issues in EBP. The advocate argument centers on quality of implementation (e.g., Arnesen, Ogden, & Sørlie, 2006 ). It basically says that if X is implemented differently than is prescribed by researchers or program developers, we can no longer know exactly what it is that works. If unfaithfully implemented, the intervention might not produce the expected results, and the program developers cannot be held responsible for the results that do obtain. Failure to obtain the expected results is to be blamed on unsystematic or unfaithful implementation of a program, the argument goes. Note that the results are described as expected.

The critics, on the other hand, interpret fidelity as an attempt to curb the judgment and practical knowledge of the teachers; perhaps even as an attempt to replace professional judgment with research evidence. Biesta ( 2007 ), for example, argues that in the EBP framework the only thing that remains for practitioners to do is to follow rules for action. These rules are thought to be somehow directly derived from the evidence. Biesta is by no means the only EBP critic to voice this criticism; we find the same view in Bridges, Smeyers, and Smith ( 2008 ):

The evidence-based policy movement seems almost to presuppose an algorithm which will generate policy decisions: If A is what you want to achieve and if research shows R1, R2 and R3 to be the case, and if furthermore research shows that doing P is positively correlated with A, then it follows that P is what you need to do. So provided you have your educational/political goals sorted out, all you need to do is slot in the appropriate research findings—the right information—to extract your policy. (p. 9)

No consideration of the concrete situation is deemed necessary, and professional judgment therefore becomes practically superfluous. Many critics of EBP make the same point: teaching should not be a matter of following rules, but a matter of making judgments. If fidelity implies following highly scripted lessons to the letter, the critics have a good point. If fidelity means being faithful to higher level principles, such as “provide feedback on home assignments,” it becomes more open and it is no longer clear exactly what one is supposed to be faithful to, since feedback can be given in a number of ways. We should also note here that EBP advocates, for example, David Hargreaves ( 1996b ), emphatically insist that evidence should enhance professional judgment, not replace it. Let us also note briefly the usage of the term “evidence,” since it deviates from the epistemological usage of the term. Biesta (and other critics) picture evidence as something from which rules for action can be inferred. But evidence is (quantitative) data that speak to the truth value of a causal hypothesis, not something from which you derive rules for action. Indeed, the word “based” in evidence-based practice is misleading—practice is not based on the RCT evidence; it is based on the hypothesis (supposedly) supported by the evidence. Remember that the role of evidence can be summed up as support . Evidence surely can enhance judgment, although EBP advocates tend to be rather hazy about how this is supposed to happen, especially if they also endorse the principle of fidelity.

Contextual Matters

If we hold that causes are easily exportable and can be relied on to produce their effect across a variety of different contexts, we rely on a number of assumptions about causality and about contexts. For example, we must assume that the causal X–Y relation is somehow basic, that it simply holds in and of itself. This assumption is easy to form; if we have conducted an RCT (or several, and pooled the results in a meta-analysis) and found a relation between an intervention and an effect of a decent magnitude, chances are that we conclude that this relation simply exists. Causal relations that hold in and of themselves naturally also hold widely; they are stable, and the cause can be relied on as sufficient to bring about its effect most of the time, in most contexts, if not all. This is a very powerful set of assumptions indeed—it underpins the belief that desirable results are reproducible and can be planned for, which is exactly what not only EBP wants but what practical pedagogy wants and what everyday life in general runs on.

The second set of assumptions concerns context. The US Department of Education guide ( 2003 ) advises that RCTs should demonstrate the intervention’s effectiveness in school settings similar to yours, before you can be confident that it will work for you. The guide provides no information about what features should be similar or how similar those features should be; still, a common enough assumption is hinted at here: if two contexts are (sufficiently) similar (on the right kind of features) the cause that worked in one will also work in the other. But as all teachers know, students are different, teachers are different, parents are different, headmasters are different, and school cultures are different. The problem faced by EBP is how deep these differences are and what they imply for the exportability of interventions.

On the view taken here, causal relations are not general, not basic, and therefore do not hold in and of themselves. Causal relations are context dependent, and contexts should be expected to be different, just as people are different. This view poses problems for the practitioner, because it means that an intervention that is shown by an RCT to work somewhere (or in many somewheres ) cannot simply be assumed to work here . Using causes in practice to bring about desirable changes is very different from finding them, and context is all-important (Cartwright, 2012 ).

All interventions are inserted into an already existing practice, and all practices are highly complex causal/social systems with many factors, causes, effects, persons, beliefs, values, interactions and relations. This system already produces an output Y; we are just not happy with it and wish to improve it. Suppose that most of our first graders do learn to read, but that some are reading delayed. We wish to change that, so we consider whether to implement Hatcher’s method. We intervene by changing the cause that we hold to be (mainly) responsible for Y—namely, X—or we implement a brand-new X. But when we implement X or change it from x i to x j (shifting from one method of reading instruction to another), we generally thereby also change other factors in the system (context, practice), not just the ones causally downstream from X. We might (inadvertently) have changed both A, B, and C—all of which may have an effect on Y. Some of these contextual changes might reinforce the effect of X; others might counteract it. For example, in selecting the group of reading-delayed children for special treatment, we might find that we change the interactional patterns in the class, and that we change the attitudes of parents toward their children’s education and toward the teacher or the school. With the changes to A, B, and C, we are no longer in system g but in system h . The probability of Y might thereby change; it might increase or it might decrease. Hence, insofar as EBP focuses exclusively on the X–Y relation, natural as this is, it tells only half the story. If we take the context into account, it transpires that if X is going to be an efficacious strategy for changing (bringing about, enhancing, improving, preventing, reducing) Y, then it is not the relation between X and Y that matters the most. What matters instead is that the probability of Y given X-in-conjunction-with-system is higher than the probability of Y given not-X-in-conjunction-with-system. But what do we need to know to make such judgments?

Relevance and Evidence

On the understanding of EBP advanced here, fidelity is misguided. It rests on causal assumptions that are at least problematic; it fails to distinguish between finding causes and using causes; and it fails to pay proper attention to contextual matters.

What, then, should a practitioner look for when trying to make a decision about whether to implement X or not? X has worked somewhere ; that has been established by RCTs. But when is the fact that X has worked somewhere relevant to a judgment that X will also work here ? If the world is diverse, we cannot simply export a causal connection, insert it into a different context, and expect it to work there. The practitioner will need to gather a lot of heterogeneous evidence, put it together, and make an astute all-things-considered judgment about the likelihood that X will bring about the desired results here were it to be implemented. The success of EBP depends not only on rigorous research evidence but also on the steps taken to use an intervention to bring about desirable changes in a context where the intervention is as yet untried.

What are the things to be considered for an all-things-considered decision about implementing X? First, the practitioner already knows that X has worked somewhere ; the RCT evidence tells him or her that. Thus, we do know that X played a positive causal role for many of the individuals in the study group (but not necessarily all of them; effect sizes are aggregate results and thus compatible with negative results for some individuals).

Second, the practitioner must think about how the intervention might work if it were implemented. RCTs run on an input–output logic and do not tell us anything about how the cause is thought to bring about its effect. But a practitioner needs to ask whether X can play a positive causal role in his or her context, and then the question to ask is how , rather than what .

Third, given our understanding of causes as INUS conditions, the practitioner will have to map the contextual factors that are necessary for X to be able to do its work and bring about Y. What are the enabling factors? If they are not present, can they be easily procured? Do they outweigh any disabling factors that may be present? It is important to remember that enablers may be absences of hindrances. Despite their adherence to the principle of fidelity, Arnesen, Ogden, and Sørlie ( 2006 ) acknowledge the importance of context for bringing about Y. For example, they point out that there must be no staff conflicts if the behavioural program is to work. Such conflicts would be a contextual disabler, and their absence is necessary. If you wish to implement Hatcher’s method, you have to look at your students and decide whether you think this will suit them, whether they are motivated, and how they might interact with the method and the materials. As David Olson ( 2004 ) points out, the effect of an intervention depends on how it is “taken” or understood by the learner. But vital contextual factors also include mundane things such as availability of adequate materials, whether the parents will support and help if the method requires homework, whether you have a suitable classroom and sufficient extra time, whether a teacher assistant is available, and so on. Hatcher’s method is the INUS condition, the salient factor, but it requires a contextual support team to be able to do its work.

Fourth, the practitioner needs to have some idea of how the context might change as a result of implementing X. Will it change the interactions among the students? Create jealousy? Take resources meant for other activities? The stability of the system into which an intervention is inserted is generally of vital importance for our chances of success. If the system is shifting and unstable X may never be able to make its effect happen. The practitioner must therefore know what the stabilizing factors are and how to control them (assuming they are within his or her control).

In sum, the INUS approach to causality and the all-important role of contextual factors and the target group members themselves in bringing about results strongly suggest that fidelity is misguided. The intervention is not solely responsible for the result; one has to take both the target group (whatever the scope) and contextual factors into consideration. On the other hand, similarity of contexts loses its significance because an intervention that worked somewhere can be made to be relevant here—there is no reason to assume that one needs exactly the same contextual support factors. The enablers that made X work there need not the same enablers that will make X work here . What is important is that the practitioner carefully considers how X can be made to work in his or her context.

EBP is a complex enterprise. The seemingly simple question of using the best available evidence to bring about desirable results and prevent undesirable ones branches out in different directions to involve problems concerning what educational research can and should contribute to practice, the nature of teaching, what kind of knowledge teachers need, what education should be all about, how we judge what works, the role of context and the exportability of interventions, what we think causality is, and so on. We thus meet both ontological, epistemological, and normative questions.

It is important to distinguish between the evidence and the claim which it is evidence for . Evidence serves to support (confirm, disconfirm) a claim, and strictly speaking practice is based on claims, not on evidence. Research evidence (as well as everyday types of evidence) should always be evaluated for its trustworthiness, its relevance, and its scope.

EBP as it is generally discussed emphasizes research at the expense of practice. The demands of rigor made on research evidence are very high. There is a growing literature on implementation and a growing understanding of the importance of quality of implementation, but insofar as this focuses on fidelity, it is misguided. Fidelity fails to take into account the diversity of the world and the importance of the context into which an intervention is to be inserted. It is argued here that implementation centers on the matter of whether an intervention will work here and that a reasonable answer to that question requires much local, heterogeneous evidence. The local evidence concerning target group and context must be provided by the practitioner. The research evidence tells only part of the story.

If EBP is to be a success, the research story and the local-practice story must be brought together, and this is the practitioner’s job. The researcher does not know what is relevant in the concrete context faced by the practitioner; that is for the practitioner to decide.

EBP thus demands much knowledge, good thinking, and astute judgments by practitioners.

As a recommendation for future research, I would suggest inquiries into how the research story and the contextual story come together; how practitioners understand the causal systems they work within, how they understand effectiveness, and how they adapt or translate generalized guidelines into concrete local practice.

• Achinstein, P. (2001). The book of evidence . Oxford: Oxford University Press.
• Arnesen, A. , Ogden, T. , & Sørlie, M.-A. (2006). Positiv atferd og støttende læringsmiljø i skolen . Oslo: Universitetsforlaget.
• Biesta, G. (2007). Why “what works” won’t work: Evidence-based practice and the democratic deficit in educational research. Educational Theory , 57 , 1–22.
• Biesta, G. (2010). Good education in an age of measurement: Ethics, politics, democracy . Boulder, CO: Paradigm.
• Bridges, D. , Smeyers, P. , & Smith, R. (2008). Educational research and the practical judgment of policy makers. Journal of Philosophy of Education , 42 (Suppl. 1), 5–11.
• Cartwright, N. (2007a). Are RCTs the gold standard? BioSocieties , 2 , 11–20.
• Cartwright, N. (2007b). Hunting causes and using them: Approaches in philosophy and e conomics . Cambridge, U.K.: Cambridge University Press.
• Cartwright, N. (2012). Will this policy work for you? Predicting effectiveness better: How philosophy helps. Philosophy of Science , 79 , 973–989.
• Cartwright, N. , & Hardie, J. (2012). Evidence-based policy: A practical guide to doing it better . Oxford: Oxford University Press.
• Davies, P. (2004). Systematic reviews and the Campbell Collaboration. In G. Thomas & R. Pring (Eds.), Evidence-based practice in education (pp. 21–33). Maidenhead, U.K.: Open University Press.
• Dean, C. B. , Hubbell, E. R. , Pitler, H. , & Stone, Bj. (2012). Classroom instruction that works: Research-based strategies for increasing student achievement (2d ed.). Denver, CO: Mid-continent Research for Education and Learning.
• Hammersley, M. (1997). Educational research and teaching: A response to David Hargreaves’ TTA lecture. British Educational Research Journal , 23 , 141–161.
• Hammersley, M. (2004). Some questions about evidence-based practice in education. In G. Thomas & R. Pring (Eds.), Evidence-based practice in education (pp. 133–149). Maidenhead, U.K.: Open University Press.
• Hargreaves, D. (1996a). Educational research and evidence-based educational practice: A response to critics. Research Intelligence , 58 , 12–16.
• Hargreaves, D. (1996b). Teaching as a research-based profession: Possibilities and prospects. Teacher Training Agency Annual Lecture, London. Retrieved from https://eppi.ioe.ac.uk/cms/Portals/0/PDF%20reviews%20and%20summaries/TTA%20Hargreaves%20lecture.pdf .
• Hatcher, P. , Hulme, C. , Miles, J. N. , Caroll, J. M. , Hatcher, J. , Gibbs, S. , . . . Snowling, M. J. (2006). Efficacy of small group reading intervention for readers with reading delay: A randomised controlled trial. Journal of Child Psychology and Psychiatry , 47 (8), 820–827.
• Hitchcock, D. (2011). Instrumental rationality. In P. McBurney , I. Rahwan , & S. Parsons (Eds.), Argumentation in multi-agent systems: Proceedings of the 7th international ArgMAS Workshop (pp. 1–11). New York: Springer.
• Kelly, T. (2008). Evidence. In E. Zalta (Ed.), Stanford encyclopedia of philosophy . Retrieved from http://plato.stanford.edu/entries/evidence/ .
• Kvernbekk, T. (2016). Evidence-based practice in education: Functions of evidence and causal presuppositions . London: Routledge.
• Lærerorganisasjonenes skolenevnd . (1933). Innstilling . Oslo: O. Fredr. Arnesens Bok- og Akcidenstrykkeri.
• Mackie, J. L. (1975). Causes and conditions. In E. Sosa (Ed.), Causation and conditionals (pp. 15–38). Oxford: Oxford University Press.
• Menzies, P. , & Price, H. (1993). Causation as a secondary quality. British Journal for the Philosophy of Science , 44 , 187–203.
• Morrison, K. (2001). Randomised controlled trials for evidence-based education: Some problems in judging “what works.” Evaluation and Research in Education , 15 (2), 69–83.
• Oancea, A. , & Pring, R. (2008). The importance of being thorough: On systematic accumulation of “what works” in education research. Journal of Philosophy of Education , 42 (Suppl. 1), 15–39.
• Olson, D. R. (2004). The triumph of hope over experience in the search for “what works”: A response to Slavin. Educational Researcher , 33 , 24–26.
• Pawson, R. (2012). Evidence-based policy: A realist perspective . Los Angeles: SAGE.
• Schaffer, J. (2007). The metaphysics of causation. In E. Zalta (Ed.), Stanford encyclopedia of philosophy . Retrieved from http://plato.stanford.edu/entries/causation-metaphysics/ .
• Slavin, R. E. (2002). Evidence-based education policies: Transforming educational practice and research. Educational Researcher , 31 , 15–21.
• Slavin, R. E. (2004). Education research can and must address “what works” questions. Educational Researcher , 33 , 27–28.
• U.S. Department of Education . (2003). Identifying and implementing educational practices supported by rigorous evidence: A user friendly guide . Washington, DC: Coalition for Evidence-Based Policy. Retrieved from http://www2.ed.gov/rschstat/research/pubs/rigorousevid/rigorousevid/pdf .
• Worrall, J. (2007). Why there’s no cause to randomize. British Journal for the Philosophy of Science , 58 , 451–488.

Bibliography

• Biesta, G. (2010). Why “what works” still won’t work: From evidence-based education to value-based education. Studies in Philosophy and Education , 29 , 491–503.
• Bridges, D. , & Watts, M. (2008). Educational research and policy: Epistemological considerations. Journal of Philosophy of Education , 42 (Suppl. 1), 41–62.
• Cartwright, N. (2013). Knowing what we are talking about: Why evidence doesn’t always travel. Evidence and Policy , 9 , 97–112.
• Cartwright, N. , & Munro, E. (2010): The limitations of randomized controlled trials in predicting effectiveness. Journal of Evaluation in Clinical Practice , 16 , 260–266.
• Cartwright, N. , & Stegenga, J. (2011). A theory of evidence for evidence-based policy. Proceedings of the British Academy , 171 , 289–319.
• Hammersley, M. (Ed.). (2007). Educational research and evidence-based practice . Los Angeles: SAGE.
• Hattie, J. (2009). Visible learning: A synthesis of over 800 meta-analyses relating to achievement . London: Routledge.
• Hattie, J. (2012). Visible learning for teachers: Maximizing impact on learning . London: Routledge.
• Kvernbekk, T. (2013). Evidence-based practice: On the functions of evidence in practical reasoning. Studier i Pædagogisk Filosofi , 2 (2), 19–33.
• Pearl, J. (2009). Causality: Models, reasoning, and inference . Cambridge, U.K.: Cambridge University Press.
• Phillips, D. C. (2007). Adding complexity: Philosophical perspectives on the relationship between evidence and policy. In P. Moss (Ed.), Evidence and decision making . Yearbook of the National Society for the Study of Education, 106 (pp. 376–402). Malden, MA: Blackwell.
• Psillos, S. (2009). Regularity theories. In H. Beebee , C. Hitchcock , & P. Menzies (Eds.), The Oxford handbook of causation (pp. 131–157). Oxford: Oxford University Press.
• Reiss, J. (2009). Causation in the social sciences: Evidence, inference, and purpose. Philosophy of the Social Sciences , 39 , 20–40.
• Rosenfield, S. , & Berninger, V. (Eds.). (2009). Implementing evidence-based academic interventions in school settings . Oxford: Oxford University Press.
• Sanderson, I. (2003). Is it “what works” that matters? Evaluation and evidence-based policy- making. Research Papers in Education , 18 , 331–345.
• Sloman, S. (2005). Causal models: How people think about the world and its alternatives . Oxford: Oxford University Press.
• Smeyers, P. , & Depaepe, M. (Eds.). (2006). Educational research: Why “what works” doesn’t work . Dordrecht, The Netherlands: Springer.
• Thomas, G. , & Pring, R. (Eds.). (2004). Evidence-based practice in education . Maidenhead, U.K.: Open University Press.
• Williamson, J. (2009). Probabilistic theories. In H. Beebee , C. Hitchcock , & P. Menzies (Eds.), The Oxford handbook of causation (pp. 185–212). Oxford: Oxford University Press.
• Woodward, J. (2003). Making things happen: A theory of causal explanation . Oxford: Oxford University Press.

Related Articles

• Rethinking Curriculum and Teaching
• Accountabilities in Schools and School Systems

Printed from Oxford Research Encyclopedias, Education. Under the terms of the licence agreement, an individual user may print out a single article for personal use (for details see Privacy Policy and Legal Notice).

date: 12 September 2024

• Cookie Policy
• Privacy Policy
• Legal Notice
• Accessibility
• [81.177.182.174]
• 81.177.182.174

Character limit 500 /500

What Is in a Reference? Theoretically Understanding the Uses of Evidence in Education Policy

• In book: Evidence and Expertise in Nordic Education Policy (pp.33-57)

• Columbia University

Discover the world's research

• 25+ million members
• 160+ million publication pages
• 2.3+ billion citations
• John Devlin
• Keri Hopeward
• James Hopeward

• Simona Bernotaite

• Chenyu Wang

• Sverre Tveit

• Shiralee Poed
• Christina Gowlett
• Michelle Ocriciano
• César Guadalupe
• Chanwoong Baek

• Jesús Manso
• Bianca Thoilliez

• Saija Volmari

• Alec Fraser
• Sandra Nutley
• Michael Gibbons
• Camille Limoges

• Martin Trow

• Erika Kessler
• Oren Pizmony-Levy
• Katherine E. Smith
• Justyna Bandola-Gill
• Richard Watermeyer
• Recruit researchers
• Join for free
• Login Email Tip: Most researchers use their institutional email address as their ResearchGate login Password Forgot password? Keep me logged in Log in or Continue with Google Welcome back! Please log in. Email · Hint Tip: Most researchers use their institutional email address as their ResearchGate login Password Forgot password? Keep me logged in Log in or Continue with Google No account? Sign up

Want to create or adapt books like this? Learn more about how Pressbooks supports open publishing practices.

Chapter 7: Synthesizing Sources

Learning objectives.

At the conclusion of this chapter, you will be able to:

• synthesize key sources connecting them with the research question and topic area.

7.1 Overview of synthesizing

7.1.1 putting the pieces together.

Combining separate elements into a whole is the dictionary definition of synthesis.  It is a way to make connections among and between numerous and varied source materials.  A literature review is not an annotated bibliography, organized by title, author, or date of publication.  Rather, it is grouped by topic to create a whole view of the literature relevant to your research question.

Your synthesis must demonstrate a critical analysis of the papers you collected as well as your ability to integrate the results of your analysis into your own literature review.  Each paper collected should be critically evaluated and weighed for “adequacy, appropriateness, and thoroughness” ( Garrard, 2017 ) before inclusion in your own review.  Papers that do not meet this criteria likely should not be included in your literature review.

Begin the synthesis process by creating a grid, table, or an outline where you will summarize, using common themes you have identified and the sources you have found. The summary grid or outline will help you compare and contrast the themes so you can see the relationships among them as well as areas where you may need to do more searching. Whichever method you choose, this type of organization will help you to both understand the information you find and structure the writing of your review.  Remember, although “the means of summarizing can vary, the key at this point is to make sure you understand what you’ve found and how it relates to your topic and research question” ( Bennard et al., 2014 ).

As you read through the material you gather, look for common themes as they may provide the structure for your literature review.  And, remember, research is an iterative process: it is not unusual to go back and search information sources for more material.

At one extreme, if you are claiming, ‘There are no prior publications on this topic,’ it is more likely that you have not found them yet and may need to broaden your search.  At another extreme, writing a complete literature review can be difficult with a well-trod topic.  Do not cite it all; instead cite what is most relevant.  If that still leaves too much to include, be sure to reference influential sources…as well as high-quality work that clearly connects to the points you make. ( Klingner, Scanlon, & Pressley, 2005 ).

7.2 Creating a summary table

Literature reviews can be organized sequentially or by topic, theme, method, results, theory, or argument.  It’s important to develop categories that are meaningful and relevant to your research question.  Take detailed notes on each article and use a consistent format for capturing all the information each article provides.  These notes and the summary table can be done manually, using note cards.  However, given the amount of information you will be recording, an electronic file created in a word processing or spreadsheet is more manageable. Examples of fields you may want to capture in your notes include:

• Authors’ names
• Article title
• Publication year
• Main purpose of the article
• Methodology or research design
• Participants
• Measurement
• Conclusions

  Other fields that will be useful when you begin to synthesize the sum total of your research:

• Specific details of the article or research that are especially relevant to your study
• Key terms and definitions
• Strengths or weaknesses in research design
• Relationships to other studies
• Possible gaps in the research or literature (for example, many research articles conclude with the statement “more research is needed in this area”)
• Finally, note how closely each article relates to your topic.  You may want to rank these as high, medium, or low relevance.  For papers that you decide not to include, you may want to note your reasoning for exclusion, such as ‘small sample size’, ‘local case study,’ or ‘lacks evidence to support assertion.’

This short video demonstrates how a nursing researcher might create a summary table.

7.2.1 Creating a Summary Table

  Summary tables can be organized by author or by theme, for example:

For a summary table template, see http://blogs.monm.edu/writingatmc/files/2013/04/Synthesis-Matrix-Template.pdf

7.3 Creating a summary outline

An alternate way to organize your articles for synthesis it to create an outline. After you have collected the articles you intend to use (and have put aside the ones you won’t be using), it’s time to identify the conclusions that can be drawn from the articles as a group.

  Based on your review of the collected articles, group them by categories.  You may wish to further organize them by topic and then chronologically or alphabetically by author.  For each topic or subtopic you identified during your critical analysis of the paper, determine what those papers have in common.  Likewise, determine which ones in the group differ.  If there are contradictory findings, you may be able to identify methodological or theoretical differences that could account for the contradiction (for example, differences in population demographics).  Determine what general conclusions you can report about the topic or subtopic as the entire group of studies relate to it.  For example, you may have several studies that agree on outcome, such as ‘hands on learning is best for science in elementary school’ or that ‘continuing education is the best method for updating nursing certification.’ In that case, you may want to organize by methodology used in the studies rather than by outcome.

Organize your outline in a logical order and prepare to write the first draft of your literature review.  That order might be from broad to more specific, or it may be sequential or chronological, going from foundational literature to more current.  Remember, “an effective literature review need not denote the entire historical record, but rather establish the raison d’etre for the current study and in doing so cite that literature distinctly pertinent for theoretical, methodological, or empirical reasons.” ( Milardo, 2015, p. 22 ).

As you organize the summarized documents into a logical structure, you are also appraising and synthesizing complex information from multiple sources.  Your literature review is the result of your research that synthesizes new and old information and creates new knowledge.

7.4 Additional resources:

Literature Reviews: Using a Matrix to Organize Research / Saint Mary’s University of Minnesota

Literature Review: Synthesizing Multiple Sources / Indiana University

Writing a Literature Review and Using a Synthesis Matrix / Florida International University

 Sample Literature Reviews Grid / Complied by Lindsay Roberts

Select three or four articles on a single topic of interest to you. Then enter them into an outline or table in the categories you feel are important to a research question. Try both the grid and the outline if you can to see which suits you better. The attached grid contains the fields suggested in the video .

Literature Review Table  

Test Yourself

• Select two articles from your own summary table or outline and write a paragraph explaining how and why the sources relate to each other and your review of the literature.
• In your literature review, under what topic or subtopic will you place the paragraph you just wrote?

Image attribution

Literature Reviews for Education and Nursing Graduate Students Copyright © by Linda Frederiksen is licensed under a Creative Commons Attribution 4.0 International License , except where otherwise noted.

Share This Book

Evaluating and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics (2003)

Chapter: references.

Abd-El-Khalick, F., Bell, R.L., and Lederman, N.G. (1998). The nature of science and instructional practice: Making the unnatural natural. Science Education 82 (4): 417–437.

Abrami, P.C., Leventhal, L., and Perry, R.P. (1982). Educational seduction. Review of Educational Research 52 , 446-464.

Allen, D.E., and Duch, B. (1998). Thinking towards solutions: Problem-based learning activities for general biology . Philadelphia, PA: Saunders College.

Allen, D., Groh, S.E., and Allen, D.E. (Eds.). (2001). The power of problem-based learning: A practical “how-to” for teaching undergraduate courses in any discipline . Sterling, VA: Stylus.

Alverno College Faculty. (1994) . Student assessment-as-learning at Alverno College (3rd Ed.). Milwaukee, WI: Alverno College.

Ambady, N., and Rosenthal, R. (1993). Half a minute: Predicting teacher evaluations from thin slices of nonverbal behavior and physical attractiveness. Journal of Personality and Social Psychology 64 (3), 431-441.

American Association for the Advancement of Science. (1989). Science for all Americans . Washington, DC: Author.

American Association for the Advancement of Science. (1990). The liberal art of science . Washington, DC: Author.

American Association for the Advancement of Science. (1993) . Benchmarks for science literacy . New York: Oxford University Press.

American Association for Higher Education. (1993). Making teaching community property: A menu for peer collaboration and peer review . Washington, DC: Author.

American Association for Higher Education. (1995). From idea to prototype: The peer review of teaching—A project workbook . Washington, DC: Author.

American Association for Higher Education. (1996). Teaching, learning and technology (TLT) roundtable workbook . Washington, DC: Author.

American Educational Research Association, American Psychological Association, and National Council on Measurement in Education. (1999). Standards for educational and psychological measurement . Washington, DC: American Educational Research Association.

American Psychological Association. (2002). National guidelines and suggested learning outcomes for the undergraduate psychology major (draft). Washington, DC: Author.

Anderson, E., (Ed.). (1993). Campus use of the teaching portfolio: Twenty-five profiles . Washington, DC: American Association for Higher Education.

Anderson, L.W., Krathwohl, D.R., and Bloom, B.S. (Eds.). (2001). A taxonomy for learning, teaching, and assessing: A revision of Bloom’s taxonomy of educational objectives, 2001 . Boston: Allyn and Bacon.

Angelo, T.A. (1995, November). Assessing (and defining) assessment. AAHE Bulletin 48 (2).

Angelo, T.A. (1999). The campus as a learning community: Seven promising shifts and seven powerful levers. In B.A. Pescosolido, and R. Aminzade (Eds), The social worlds of higher

education: Handbook for teaching in a new century (pp. 110-116). Thousand Oaks, CA: Pine Forge Press.

Angelo, T.A., and Cross, K.P. (1993). Classroom assessment techniques: A handbook for college teachers (2nd ed.) . San Francisco: Jossey-Bass.

Armstrong, L. (2000). Distance learning: An academic leader’s perspective on a disruptive product. Change 32 (6), 20-27.

Astin, A.W., Banta, T.W., Cross, K.P., El-Khawas, E., Ewell, P.T., Hutchings, P., Marchese, T.J., McClenney, M., Mentkowski, M., Miller, M.A., Moran, E.T., and Wright, B.D. (1996). Assessment forum: 9 principles of good practice for assessing student learning . Washington, DC: American Association for Higher Education. See < http://www.aahe.org/principl.htm >.

Baker, P. (1999). Creating learning communities: The unfinished agenda. In B.A. Pescosolido, and R. Aminzade (Eds), The social worlds of higher education: Handbook for teaching in a new century (pp. 95-109). Thousand Oaks, CA: Pine Forge Press.

Banta, T.W. (2000). That second look. Assessment Update 10 (4), 3-14.

Banta, T.W., Lund, J.P., Black, K.E., and Oblander, F. W. (1996). Assessment in practice: Putting principles to work on college campuses . San Francisco: Jossey-Bass.

Barr, R.B., and Tagg, J. (1999). From teaching to learning: A new paradigm for undergraduate education. In B.A. Pescosolido, and R. Aminzade (Eds.), The social worlds of higher education: Handbook for teaching in a new century (pp. 565-581). Thousand Oaks, CA: Pine Forge Press.

Basinger, J. (2000, August 7). Carnegie issues broad changes in system of classifying colleges. Chronicle of Higher Education . See < http://chronicle.com/daily/2000/08/2000080701n.htm >.

Baugher, K. (1992). LEARN: The student quality team manual . Nashville, TN: LEARN.

Bernstein, D.J. (1996, Spring). A departmental system for balancing the development and evaluation of college teaching: A commentary on Cavanaugh. Innovative Higher Education 20 (4), 241-248.

Bernstein, D.J., and Quinlan, K.M. (Eds.). (1996, Spring). The peer review of teaching [Special issue]. Innovative Higher Education 20(4).

Bleak, J., Neiman, H., Sternman, C., and Trower, C. (2000). Faculty recruitment study: Statistical analysis report, August 2000 . Cambridge, MA: Harvard Graduate School of Education.

Bloom, B.S. (Ed.). (1956). Taxonomy of educational objectives: The classification of educational goals. Handbook I: Cognitive domain . New York: Longmans, Green.

Boice, R. (1992). The new faculty member . San Francisco: Jossey-Bass.

Borgman, C.L., Bates, M.J., Cloonan, M.V., Efthimiadis, E.N., Gilliland-Swetland, A.J., Kafai, Y.B., Leazer, G.H., and Maddox, A.B. (1996). Social aspects of digital libraries. In Proceedings of the University of California Los Angeles-National Science Foundation Social Aspects of Digital Libraries Workshop , February 15-17, 1996. See < http://is.geis.ucla.edu/research/dl/index.html >.

Boyer, E.L. (1990). Scholarship reconsidered: Priorities of the professoriate . Princeton, NJ: Carnegie Foundation for the Advancement of Teaching.

Boyer Commission on Educating Undergraduates in the Research University. (1998). Reinventing undergraduate education: A blueprint for America’s research universities . Menlo Park, CA: Carnegie Foundation for the Advancement of Teaching. See < http://notes.cc.sunysb.edu/Pres/boyer.nsf >.

Brand, M. (2000). Changing faculty roles in research universities: Using the pathways strategy. Change 32 (6), 42-45.

Braskamp, L., and Ory, J. (1994). Assessing faculty work: Enhancing individual and institutional performance . San Francisco: Jossey-Bass.

Brinko, K.T. (1993). The practice of giving feedback to improve teaching: What is effective? Journal of Higher Education 64 (5), 574-593.

Brookfield, S.D. (1995). Becoming a critically reflective teacher . San Francisco: Jossey-Bass.

Buck, G.A., Hehn, J.G., and Leslie-Pelecky, D.L. (Eds.). (2000). The role of physics departments in preparing k-12 teachers . College Park, MD: American Institute of Physics.

Cambridge, B.L. (1996, Spring). The paradigm shifts: Examining quality of teaching through assessment of student learning. Innovative Higher Education 20 (4), 287-298.

Cambridge, B.L. (Ed.). (1997). Assessing impact: Evidence and action . Washington, DC: American Association of Higher Education.

Cambridge, B.L. (1999, December). The scholarship of teaching and learning: Questions and answers from the field. AAHE Bulletin . See < http://www.aahe.org/Bulletin/dec99f2.htm >.

Cambridge, B.L. (Ed.). (2001). Electronic portfolios: Emerging practices in student, faculty, and institutional learning . Washington, DC: American Association for Higher Education.

Capelli, P. (Ed.). (1997). Change at work: Trends that are transforming the business of business . Washington, DC: National Policy Association.

Carnegie Foundation for the Advancement of Teaching and Pew Forum on Undergraduate Learning. (2002). National survey of student engagement: The college student report . Menlo Park, CA: Author. See < http://www.indiana.edu/~nsse/l >.

Cashin, W.E. (1990). Students do rate different academic fields differently. In M. Theall, and J. Franklin (Eds.), Student ratings of instruction: Issues for improving practice . San Francisco: Jossey-Bass.

Cashin, W.E., and Downey, R.G. (1999). Using global student rating items for summative evaluation: Convergence with an overall instructor evaluation criteria . Paper presented at American Educational Research Association, April, Washington, DC.

Centra, J.A. (1973). Item reliabilities, the factor structure, comparison with alumni ratings . (Student Instructional Report, No. 3). Princeton, NJ: Educational Testing Service.

Centra, J.A. (1974). The relationship between student and alumni ratings of teachers. Educational and Psychological Measurement 34 (2), 321-326.

Centra, J.A. (1975). Colleagues as raters of classroom instruction. Journal of Higher Education 46 , 327-337.

Centra, J.A. (1979). Determining faculty effectiveness . San Francisco: Jossey-Bass.

Centra, J.A. (1987). Faculty evaluation: Past practices, future direction s. Manhattan, KS: Kansas State University, Center for Faculty Evaluation and Development.

Centra, J.A. (1993). Reflective faculty evaluation: Enhancing teaching and determining faculty effectiveness . San Francisco: Jossey-Bass.

Centra, J.A. (1994). The use of the teaching portfolio and student evaluations for summative evaluation. Journal of Higher Education 65 (5), 555-570.

Centra, J.A. (1998). The development of the student instructional report II, Higher Education Assessment Program . Princeton, NJ: Educational Testing Service.

Centra, J.A. (2001). A model for assessing the scholarship of teaching . Presentation at the 9th Annual American Association of Higher Education Conference on Faculty Roles and Rewards, February, Tampa, FL.

Centra, J.A., and Creech, F.R. (1976). The relationship between student, teacher, and course characteristics and student ratings of teacher effectiveness . (Report No. PR-76-1). Princeton, NJ: Educational Testing Service.

Centra, J.A., and Gaubatz, N.B. (2000a). Is there gender bias in student evaluations of teaching? Journal of Higher Education 70 (1), 17-33.

Centra, J.A., and Gaubatz, N.B. (2000b). Student perceptions of learning and instructional effectiveness in college courses, Higher Education Assessment Program . (Report No. 9). Princeton, NJ: Educational Testing Service.

Chandler, S. (1991) . Issues in evaluating multiinstitutional models . Paper presented at the American Educational Research Association Symposium, Chicago.

Chickering, A.W., and Gamson, Z.F. (1987). Seven principles for good practice in undergraduate education. AAHE Bulletin 39 , 3-7.

Chism, N.V.N. (1999). Peer review of teaching: A sourcebook . Boston: Anker.

Clark, J., and Redmond, M. (1982). Small group instructional diagnosis . (ERIC Ed. 217954). Seattle: University of Washington, Department of Biology Education.

Cochran, K.F. (1997). Pedagogical content knowledge: Teachers’ integration of subject matter, pedagogy, students, and learning environments. Research Matters-to the Science Teacher , #9702.

Cohen, P.A. (1981). Student ratings of instruction and student achievement: A meta-analysis of multisection validity studies. Review of Educational Research 51 , 281-309.

Collis, B., and Moonen, J. (2001). Flexible learning in a digital world: Experiences and expectations . London, UK: Kogan Page.

Cooper, J., and Robinson, P. (1998). Small-group instruction: An annotated bibliography of science, mathematics, engineering and technology resources in higher education (Occasional Paper #6). Madison: University of Wisconsin-Madison, National Institute for Science Education.

Coppola, B.P., and Smith, D.H. (1996). A case for ethics. Journal of Chemical Education 73 , 33-34.

Coppola, B.P., Ege, S.N., and Lawton, R.G. (1997). The University of Michigan undergraduate chemistry curriculum 2— Instructional strategies and assessment. Journal of Chemical Education 74 , 84-94.

Coppola. B.P., and Jacobs, D. (2002). Is the scholarship of teaching and learning new to chemistry? In M.T. Huber, and S. Morreale (Eds.), Disciplinary styles in the scholarship of teaching and learning: Exploring common ground . Washington, DC: American Association for Higher Education and the Carnegie Foundation for the Advancement of Teaching.

Cornell University. (1993). Teaching evaluation handbook . Ithaca, NY: Cornell University, Office of Instructional Support.

Cross, K.P., and Steadman, M.H. (1996). Classroom research: Implementing the scholarship of teaching . San Francisco: Jossey-Bass.

Daffinrud, S.M., and Herrera, O.L. (2000). Evaluation methods and findings: The Field-Tested Learning Assessment Guide (FLAG) final report . Madison, WI: University of Wisconsin, LEAD Center.

Darling-Hammond, L. (Ed.). (1997). Doing what matters most: Investing in quality teaching . New York: National Commission on Teaching and America’s Future.

Davis, B.G. (1988). Sourcebook for evaluating teaching . Berkeley: University of California at Berkeley, Office of Educational Development.

Davis, B.G. (1993). Tools for teaching . San Francisco: Jossey-Bass.

Diamond, R.M., and Adams, B.E. (Eds.). (1995). The disciplines speak: Rewarding the scholarly, professional, and creative work of faculty . Washington, DC: American Association for Higher Education.

Diamond, R.M., and Adams, B.E. (Eds.). (2000). The disciplines speak II: More statements on rewarding the scholarly, professional, and creative work of faculty . Washington, DC: American Association for Higher Education.

Diamond, R.M., and Gray, P.J. (1998). 1997 National Study of Teaching Assistants . Syracuse, NY: Syracuse University Center for Instructional Development.

Doherty, A., Riordan, T., and Roth, J. (Eds.). (2002). Student learning: A central focus for institutions of higher education . (A report and collection of institutional practices of the Student Learning Initiative.) Milwaukee, WI: Alverno College Institute.

Dougherty, A. (1999). Just-in-time, teaching using Web feedback to guide learning and teaching . See < http://ww2.lafayette.edu/~doughera/talks/ets9912/ >.

Doyle, M.P. (Ed.). (2000). Academic excellence: The role of research in the physical sciences at undergraduate institutions . Tucson, AZ: Research Corporation.

Drucker, A.J., and Remmers, H.H. (1951). Do alumni and students differ in their attitudes toward instructors? Journal of Educational Psychology 42 (3), 129-143.

Ebert-May, D., Brewer, C.A., and Allred, S. (1997). Innovation in large lectures: Teaching for active learning through inquiry. BioScience 47 (9), 601-607.

Edgerton, R., Hutchings, P., and Quinlan, K. (1991). The teaching portfolio: Capturing the scholarship in teaching . Washington, DC: The American Association for Higher Education.

Ehrmann, S.C. (2000, November/December). On the necessity of grassroots evaluation of educational technology: Recommendations for higher education . See < http://horizon.unc.edu/TS/assessment/2000-11.asp >.

Ehrmann, S.C. (In press). Technology and educational revolution: Ending the cycle of failure. Liberal Education . See draft < http://www.tltgroup.org/resources/V_Cycle_of_Failure.html >.

Emerson, J.D., Mosteller, F., and Youtz, C. (2000). Students can help improve college teaching: A review and an agenda for the statistics profession. In C. R. Rao, and G. Szekely (Eds.), Statistics for the 21st century . New York: Marcel Dekker.

Emert, J. W., and Parish, C. R. (1996). Assessing concept attainment in undergraduate core courses in mathematics. In T.W. Banta, J. P. Lund, K.E. Black, and F.W. Oblander (Eds.), Assessment in practice . San Francisco: Jossey-Bass.

Ewell, P.T. (1991). To capture the ineffable: New forms of assessment in higher education. Review of Research in Education 17 , 75-125.

Feldman, K.A. (1978). Course characteristics and college students’ ratings of their teachers and courses: What we know and what we don’t. Research in Higher Education 9 , 199-242.

Feldman, K.A. (1984). Class size and college students’ evaluations of teachers and courses: A closer look. Research in Higher Education 21(1):45-116, September.

Feldman, K.A. (1989). Instructional effectiveness of college teachers as judged by teachers themselves, current and former students, colleagues, administrators and external (neutral) observers. Research in Higher Education 30 , 137-189.

Feldman, K.A. (1993). College students’ views of male and female college teachers: Part II-Evidence from students’ evaluations of their classroom teachers. Research in Higher Education 34 (2), 151-211.

Fisch, L. (1998). AD REM: Spotters. The national teaching and learning forum . Phoenix, AZ: Oryx Press.

Fosnot, C. (Ed.). (1996). Constructivism: Theory, perspectives, and practice . New York: Teachers College Press.

Freedman, R.L.H. (1994). Open-ended questioning . New York: Addison Wesley.

French-Lazovik, G. (1981). Documentary evidence in the evaluation of teaching. In J. Millman (Ed.), Handbook of teacher evaluation (pp. 73-89) . Newbury Park, CA: Sage.

Gabel, D.L. (Ed.) (1994). Handbook of research on science teaching and learning . New York: Macmillan.

Gabelnick, F., MacGregor, J., Matthews, R., and Smith, B. (1990). Learning communities: Creating connections among students, faculty, and disciplines . San Francisco: Jossey-Bass.

Gaff, J.G., Pruitt-Logan, A.S., and Weibl, R.A. (2000). Building the faculty we need: Colleges and universities working together . Washington, DC: Association of American Colleges and Universities.

Gardiner, L., Anderson, C., and Cambridge, B. (Eds.). (1997). Learning through assessment: A resource guide for higher education . Washington, DC: American Association for Higher Education.

Gavin, R. (2000). The role of research at undergraduate institutions: Why is it necessary to defend it? In M.P. Doyle (Ed.), Academic excellence: The role of research in the physical sciences at undergraduate institutions (pp. 9-17) . Tucson, AZ: Research Corporation.

Gilmore, G.M., Kane, M.T., and Naccarato, R.W. (1978). The generalizability of student ratings of instruction: Estimation of teacher and course components. Journal of Educational Measurement 15 (1), 1-13.

Glassick, C.E., Huber, M.T., and Maeroff, G.I. (1997). Scholarship assessed: Evaluation of the professoriate . San Francisco: Jossey-Bass.

Goheen, R.F. (1969). The human nature of a university . Princeton, NJ: Princeton University Press.

Golde, C.M., and Dore, T.M. (2001). At cross purposes: What the experiences of today’s doctoral students reveal about doctoral education . Philadelphia, PA: Pew Charitable Trusts.

Gray, J., Diamond, R.M., and Adam, B.E. (1996). A national study on the relative importance of research and undergraduate teaching at colleges and universities . Syracuse, NY: Syracuse University Center for Instructional Development.

Greene, E. (2000, June 23). Some colleges pay students to go to class—to evaluate teaching. The Chronicle of Higher Education , A18.

Greenspan, A. (2000). The economic importance of improving math-science education . Testimony to the U.S. House of Representatives Committee on Education and the Workforce, Washington, DC. See < http://www.federalreserve.gov/boarddocs/testimony/2000/20000921.htm >.

Grouws, D.A. (1992). Handbook of research on mathematics teaching and learning . New York: Macmillan.

Herron, J.D. (1996). The chemistry classroom: Formulas for successful teaching . Washington, DC: American Chemical Society.

Hestenes, D. (1987). Toward a modeling theory of physics instruction. American Journal of Physics 55 , 440-454.

Hestenes, D., and Halloun, I. (1995). Interpreting the force concept inventory. Physics Teacher 33 (8), 502-506.

Heterick, R., and Twigg, C. (1999). Lectures are not cheap! See < http://www.center.rpi.edu/LForum/LM/Sept99.html >.

Huber, M.T. (1999). Disciplinary styles in the scholarship of teaching: Reflections on the Carnegie Academy for the Scholarship of Teaching and Learning . Paper presented at the 7th International Improving Student Learning Symposium Improving Student Learning Through the Disciplines, Menlo Park, CA.

Huber, M.T. (2001, July/August). Balancing acts: Designing careers around the scholarship of teaching. Change , 21-29.

Huber, M.T., and Morreale, S. (Eds.). (2002). Disciplinary styles in the scholarship of teaching and learning: Exploring common ground . Washington, DC: American Association for Higher Education and the Carnegie Foundation for the Advancement of Teaching.

Hutchings, P. (Ed.). (1995). From idea to prototype: The peer review of teaching, a project workbook . Washington, DC: American Association for Higher Education.

Hutchings, P. (Ed.). (1996). Making teaching community property: A menu for peer collaboration and peer review . Washington, DC: American Association for Higher Education.

Hutchings, P. (Ed.). (1998). The course portfolio: How faculty can examine their teaching to advance practice and improve student learning . Washington, DC: American Association for Higher Education.

Hutchings, P. (Ed.). (2000). Opening lines: Approaches to the scholarship of teaching and learning . Menlo Park, CA: Carnegie Foundation for the Advancement of Teaching.

Indiana University. (2000). The college student experiences questionnaire . See < http://www.indiana.edu/~cseq/cseq_content.htm >.

International Technology Education Association. (2000). Standards for technological literacy: Content for the study of technology . Reston, VA: Author. See < http://www.iteawww.org/TAA/STLstds.htm >.

Ireton, M.F.W., Manduco, C.A., and Mogk, D.W. (1996). Shaping the future of undergraduate earth science education: Innovation and change using an earth system approach . Washington, DC: American Geophysical Union.

Johnson, D.W., Johnson, R.T., and Smith, K. (1998). Active learning: Cooperation in the college classroom . Edina, MN: Interaction Books.

Joint Policy Board for Mathematics. (1994). Recognitions and rewards in the mathematical sciences . Washington, DC: American Mathematical Society.

Katz, J., and Henry, M. (1988). Turning professors into teachers: A new approach to faculty development and student learning . New York: Macmillan.

Keig, L., and Waggoner, M.D. (1994). Collaborative peer review: The role of faculty in improving college teaching . (ASHE-ERIC Higher Education Report #2). Washington, DC: George Washington University.

Kellogg Commission on the Future of State and Land-Grant Universities. (1997). Returning to our roots: The student experience . Washington, DC: National Association of State Universities and Land-Grant Colleges.

Kennedy, D. (1997). Academic duty . Cambridge, MA : Harvard University Press.

King, P.M., and Kitchener, K.S. (1994). Developing reflective judgment : Understanding and promoting intellectual growth and critical thinking in adolescents and adults . San Francisco: Jossey-Bass.

Koon, J., and Murray, H.G. (1995). Using multiple outcomes to validate student ratings of overall teacher effectiveness. Journal of Higher Education 66 (1), 61-81.

Kremer, J. (1990). Constant validity of multiple measures in teaching, research, and service and reliability of peer ratings. Journal of Educational Psychology 82 , 213-218.

Lambert, L.M., and Tice, S.L. (Eds.). (1992). Preparing graduate students to teach: A guide to programs that improve undergraduate education and develop tomorrow’s faculty . Washington, DC: American Association for Higher Education.

Landis, C.R., Ellis, A.B., Lisenky, G.C., Lorenz, J.K., Meekder, K., and Wamser, C.C. (Eds.). (2001). Chemistry concept tests: A pathway to interactive classrooms . Upper Saddle River, NJ: Prentice Hall.

Lederman, N.G., and O’Malley, M. (1990). Students’ perceptions of tentativeness in science: Development, use, and sources of change. Science Education 74 , 225-239.

Lederman, N.G., Schwartz, R.S., Abd-El-Khalick, F., and Bell, R.L. (In press). Preservice teachers’ understanding and teaching of the nature of science: an intervention study. Canadian Journal of Science, Mathematics, and Technology Education .

Licata, C.M., and Morreale, J.C. (1997). Posttenure review: Policies, practices, precautions . Washington, DC: American Association for Higher Education.

Licata, C.M, and Morreale, J.C. (2002). Posttenure faculty review and renewal: Experienced voices . Washington, DC: American Association for Higher Education.

Liebowitz, W.R. (1999, October 22). Course evaluations proliferate on the Web—to the chagrin of many professors. Chronicle of Higher Education , A59.

Light, R.L. (2001). Making the most of college: Students speak their minds . Cambridge, MA: Harvard University Press.

Lloyd-Jones, R. (1977). Primary trait scoring. In C. Cooper, and L. Odell (Eds.), Evaluating writing: Describing, measuring, judging . Urbana, IL: National Council of Teachers of English.

Loacker, G. (Ed.). (2001). Self-assessment at Alverno College by Alverno College faculty . Milwaukee, WI: Alverno College.

Lopez, R.E., and Schultz, T. (2001). Two revolutions in k-8 science education. Physics Today 54 (9), 45-49. See < http://www.physicstoday.org/pt/vol-54/iss-9/current.html >.

Lovitts, B., and Nelson, C. (2000). The hidden crisis in graduate education: Attrition from Ph.D. programs. Academe 86(6), 44-50.

Lowman, J. (1995). Mastering the techniques of teaching . (2nd Ed.). San Francisco: Jossey-Bass.

MacGregor, J., Cooper, J., Smith, K., and Robinson, P. (2000). Strategies for energizing large classes: From small groups to learning communities . (No. 81). San Francisco: Jossey-Bass.

Marsh, H.W. (1982). Validity of students’ evaluations of college teaching: A multitrait-multimethod analysis. Journal of Educational Psychology 74 , 264-279.

Marsh, H.W. (1987). Student evaluations of university teaching: Research findings, methodological issues, and directions for future research. International Journal of Educational Research 11 , 253-388.

Marsh, H.W., and Roche, L.A. (1993). The use of student evaluations and an individually structured intervention to enhance university teaching effectiveness. American Educational Research Journal 30 (1), 217-251.

Marsh, H.W., and Roche, L.A. (2000). Effects of grading leniency and low workload on students’ evaluations of teaching: Popular myth, bias, validity or innocent bystander? The Journal of Educational Psychology 92 (1), 202-228.

Mathematical Association of America. (2000). Guidelines for programs and departments in undergraduate mathematical sciences . Washington, DC: Author. See < http://www.maa.org/guidelines/guidelines.html >.

Mazur, E. (1997). Peer instruction: A user’s manual . Upper Saddle River, NJ: Prentice Hall.

McCormick, A.C. (2001). (Ed.). The Carnegie classification of institutions of higher education, 2000 edition . Menlo Park, CA: Carnegie.

McKeachie, W.J. (1979, October). Student ratings of faculty: A reprise. Academe 384-397.

McKeachie, W.J., Pintrich, P.R., Lin, Y., Smith, D.A.F., and Sharma, R. (1990). Teaching and learning in the college classroom: A review of the research literature . Ann Arbor, MI: National Center for Research to Improve Post-Secondary Teaching and Learning.

McKeachie, W.J. (1999). McKeachie’s teaching tips, strategies, research, and theory for college and university teachers . (10th Ed.). New York: Houghton-Mifflin.

McNeal, A.P., and D’Avanzo, C.D. (Eds.) (1997). Student active science: Models of innovation in college science teaching . Fort Worth, TX: Harcourt Brace.

Menges, R., and Svinicki, M. (Eds.). (1991). College teaching: From theory to practice . San Francisco: Jossey-Bass.

Millar, S.B. (Ed.). (1998). Indicators of success in postsecondary SMET education: Shapes of the future . (Synthesis and Proceedings of the Third Annual National Institute for Science Education Forum). Madison: University of Wisconsin-Madison, Wisconsin Center for Education Research.

Mintzes, J.J., Wandersee, J.H., and Novak, J.D. (Eds.). (2000). Assessing science understand

ing: A human constructivist view . San Diego, CA: Academic Press.

Mullin, R. (2001). The undergraduate revolution: Change the system or give incrementalism another 30 years? Change 33(5), 54-58.

Murnane, R.J., and Levy, F. (1996). Teaching the new basic skills: Principles for educating children to thrive in a changing economy . New York: Free Press.

Murray, H.G. (1983). Low inference classroom teaching behaviors and student ratings of college teaching effectiveness. Journal of Educational Psychology 71 , 856-865.

Murray, H.G., Rushton, P.J., and Paunonen, S.V. (1990). Teacher personality traits and student instructional ratings in six types of university courses. Journal of Educational Psychology 82 , 250-261.

Murray, H.G, Gillese, E., Lennon, M., Mercer, P., and Robinson, M. (1996). Ethical principles in university teaching . Vancouver, BC: The Society for Teaching and Learning in Higher Education. See < http://www.umanitoba.ca/academic_support/uts/stlhe/Ethical.html >.

Naftulin, D.H., Ware, J.E., and Donnelly, F.A. (1973). The Doctor Fox lecture: A paradigm of educational seduction. Journal of Medical Education 48 , 630-635.

Narum, J. (1995). Structures for science: A handbook on planning facilities for undergraduate natural science communities (Vol. III). Washington, DC: Project Kaleidoscope.

National Academy of Sciences. (1997). Preparing for the 21st century: The education imperative . Washington, DC: National Academy Press. See < http://www.nap.edu/catalog/9537.html >.

National Center for Education Statistics. (1999). Teacher quality: A report on the preparation and qualification of public school teachers . Washington, DC: U.S. Department of Education.

National Center for Public Policy and Higher Education. (2001). Measuring up 2000: The state-by-state report card for higher education . San Jose, CA: Author. See < http://www.highereducation.org >.

National Council of Teachers of Mathematics. (1989). Curriculum and evaluation standards for school mathematics . Reston, VA: Author.

National Council of Teachers of Mathematics. (2000). Principles and standards for school mathematics . Reston, VA: Author.

National Institute for Science Education. (2001a). Learning through technolo gy. Madison, WI: Author. See < http://www.wcer.wisc.edu/nise/cl1/ilt/ >.

National Institute for Science Education. (2001b). Field-tested learning assessment guide . Madison, WI: Author. See < http://www.wcer.wisc.edu/nise/cl1/flag/ >.

National Institute for Science Education. (2001c). Cooperative learning . See < http://www.wcer.wisc.edu/nise/CL1/CL/ >.

National Research Council. (1991). Moving beyond myths: Revitalizing undergraduate mathematics . Washington, DC: National Academy Press. See < http://books.nap.edu/catalog/1782.html >.

National Research Council. (1995a). Engineering education: Designing an adaptive system . Washington, DC: National Academy Press. See < http://books.nap.edu/catalog/4907.html >.

National Research Council. (1995b). Reshaping the graduate education of scientists and engineers . Washington, DC: National Academy Press. See < http://books.nap.edu/catalog/4935.html >.

National Research Council. (1996a). National science education standards . Washington, DC: National Academy Press. See < http://books.nap.edu/catalog/4962.html >.

National Research Council. (1996b). From analysis to action . Washington, DC: National Academy Press. See < http://books.nap.edu/catalog/9128.html >.

National Research Council. (1996c). The role of scientists in the professional development of science teachers . Washington, DC: National Academy Press. See < http://books.nap.edu/catalog/2310.html >.

National Research Council. (1997a). Science teaching reconsidered: A handbook . Washington, DC: National Academy Press. See < http://books.nap.edu/catalog/5287.html >.

National Research Council. (1997b). Adviser, teacher, role model, friend: On being a mentor to students in science and engineering . Washington, DC: National Research Council. See < http://books.nap.edu/catalog/5789.html >.

National Research Council. (1998a). High stakes: Testing for tracking, promotion, and graduation . Washington, DC: National Academy Press. See < http://books.nap.edu/catalog/6336.html >.

National Research Council. (1998b). Developing a digital national library for undergraduate science, mathematics, engineering, and technology education: Report of a workshop . Washington, DC: National Academy Press. See < http://books.nap.edu/catalog/5952.html >.

National Research Council. (1999a). Transforming undergraduate education in science, mathematics, engineering, and technology . Washington, DC: National Academy Press. See < http://www.nap.edu/catalog/6453.html >.

National Research Council. (1999b). Improving student learning: A strategic plan for education research and its utilization . Washington, DC: National Academy Press. See < http://books.nap.edu/catalog/6488.html >.

National Research Council. (1999c). Global perspectives for local action: Using TIMSS to improve U.S. mathematics and science education . Washington, DC: National Academy Press. See < http://books.nap.edu/catalog/9723.html >.

National Research Council. (1999d). Myths and tradeoffs: The role of tests in undergraduate admissions . Washington, DC: National Academy Press. See < http://books.nap.edu/catalog/9632.html >.

National Research Council. (2000a). Building a workforce for the information economy . Washington, DC: National Academy Press. See < http://books.nap.edu/catalog/9830.html >.

National Research Council. (2000b). Educating teachers of science, mathematics, and technology: New practices for the new millennium . Washington, DC: National Academy Press. See < http://books.nap.edu/catalog/9832.html >.

National Research Council. (2000c). How people learn: Brain, mind, experience, and school: Expanded edition . Washington, DC: National Academy Press. See < http://www.nap.edu/catalog/9853.html >.

National Research Council. (2000d). Inquiry and the national science education standards: A guide for teaching and learning . Washington, DC: National Academy Press. See < http://www.nap.edu/catalog/9596.html >.

National Research Council. (2000e). LC21: A digital strategy for the Library of Congress . Washington, DC: National Academy Press. See < http://books.nap.edu/catalog/9940.html >.

National Research Council. (2001). Knowing what students know: The science and design of educational assessment . Washington, DC: National Academy Press. See < http://www.nap.edu/catalog/10019.html >.

National Research Council. (2002a). Learning and understanding: Improving advanced study of science and mathematics in U.S. high schools . Washington, DC: National Academy Press. See < http://www.nap.edu/catalog/10129.html .

National Research Council. (2002b). Scientific research in education . Washington, DC: National Academy Press. See < http://www.nap.edu/catalog/10236.html >.

National Science Board. (2000). Science and engineering indicators—2000 . Arlington, VA: Author. See < http://www.nsf.gov/search97cgi/vtopic >.

National Science Foundation. (1996). Shaping the future: New expectations for undergraduate education in science, mathematics, engineering, and technology . (NSF 96-139). Arlington, VA: Author. See < http://www.nsf.gov/cgi-bin/getpub?nsf96139 >.

National Science Foundation. (1998). Information technology: Its impact on undergraduate education in science, mathematics, engineering, and technology . (NSF 98-82). Arlington, VA: Author. See < http://www.nsf.gov/cgi-bin/getpub?nsf9882 >.

Neff, R.A., and Weimer, M. (Eds.). (1990). Teaching college: Collected readings for new instructors . Madison, WI: Magna.

Novak, J. (1998). Learning, creating, and using knowledge: Concept maps as facilitative tools in schools and corporations . Mahwah, NJ: Lawrence Erlbaum.

Nyquist, J.D., Abbott, R.D., Wulff, D.H., and Sprague, J. (Eds.). (1991). Preparing the professoriate of tomorrow to teach: Selected readings in TA training . Dubuque, IA: Kendall/Hunt.

Ory, J.C. (2000). Teaching evaluation: Past, present, and future. In K.E. Ryan (Ed.), Evaluating teaching in higher education: A vision for the future : New directions in teaching and learning (pp. 13-18) . San Francisco, CA: Jossey-Bass.

Osterlind, S.J. (1989). Constructing test item s. Boston: Kluwer Academic.

Overall, J.U., and Marsh, H.W. (1980). Students’ evaluations of instruction: A longitudinal study of their stability. Journal of Educational Psychology 72 , 321-325.

Palomba, C.A., and Banta, T.W. (1999). Assessment essentials, planning, implementing, and improving assessment in higher education . San Francisco: Jossey-Bass.

Perry, R.P., and Smart, J.C. (Eds.). (1997). Effective teaching in higher education: Research and practice . New York: Agathon Press.

Pescosolido, B.A., and Aminzade, R. (Eds.). (1999). The social worlds of higher education: Handbook for teaching in a new century . Thousand Oaks, CA: Pine Forge Press.

Pike, G.R. (1995). The relationship between self-reports of college experiences and test scores. Journal of Research in Higher Education 36 (1), 1-21.

Project Kaleidoscope. (1991). What works, building natural science communities: A plan for strengthening undergraduate science and mathematics (Vol 1) . Washington, DC: Author.

Project Kaleidoscope. (1994). What works, leadership: Challenges for the future (Vol. II). Washington, DC: Author.

Project Kaleidoscope. (1998). Shaping the future of undergraduate science, mathematics, engineering and technology education: Proceedings and recommendations from the PKAL day of dialogue . Washington, DC: Author. See < http://www.pkal.org/template2.cfm?2c_id=301 >.

Reis, R.M (1997). Tomorrow’s professor: Preparing for academic careers in science and engineering . Piscataway, NJ: IEEE Press.

Remmers, H.H. (1934). Reliability and halo effect on high school and college students’ judgments of their teachers. Journal of Applied Psychology 18 , 619-630.

Rice, R.E., Sorcinelli, M.D., and Austin, A.E. (2000). Heeding new voices: Academic careers for a new generation . Washington, DC: American Association for Higher Education.

Root, L.S. (1987). Faculty evaluation: Reliability of peer assessments of research, teaching, and service. Research in Higher Education 26 , 71-84.

Rosenthal, R. (1976). Experimenter effects in behavioral research . New York: Appleton-Century-Croft.

Rothman, F.G., and Narum, J.L. (1999). Then, now, and in the next decade: A commentary on strengthening undergraduate science, mathematics, engineering and technology education . Washington, DC: Project Kaleidoscope. See < http://www.pkal.org/news/thennow100.html >.

Rust, E. (1998). Business cares about math and science achievement. In Business Coalition for Education Reform, The formula for success: A business leader’s guide to supporting math and science achievement (pp. 11-14). Washington, DC: National Alliance for Business.

Sanderson, A., Phua, V.C., and Herda, D. (2000). The American faculty poll . Chicago: National Opinion Research Center.

Sands, R.G., Parson, L.A., and Duane, J. (1991). Faculty mentoring faculty in a public university. Journal of Higher Education 62 , 174-193.

Schwartz, C. (1983 ). ABC’s of teaching with excellence: A Berkeley compendium for teaching with excellence . See < http://teaching.berkeley.edu/compendium/ >.

Scriven, M. (1981). Summative teacher evaluation. In J. Millman (Ed.), Handbook of teacher evaluation . Beverly Hills, CA: Sage.

Scriven, M. (1993). Hard-won lessons in program evaluation. New Directions for Program Evaluation 58 , summer.

Seldin, P. (1991). The teaching portfolio . Boston: Anker.

Seldin, P. (1998, March). How colleges evaluate teaching: 1988 vs. 1998. AAHE Bulletin 3-7.

Seymour, E. (In press). Tracking the processes of change in U.S. undergraduate education in science, mathematics, engineering, and technology. Science Education .

Seymour, E., and Hewitt, N.M. (1997). Talking about leaving: Why undergraduates leave the sciences . Boulder, CO: Westview Press.

Shapiro, N.S., and Levine, J.H. (1999). Creating learning communities: A practical guide to winning support, organizing for change, and implementing programs . San Francisco, CA: Jossey-Bass.

Shipman, H.L. (2001). Hands-on science, 680 hands at a time. Journal of College Science Teaching 30 (5), 318-321.

Shore, B.M., et al. (1986). The teaching dossier: A guide to its preparation and use . Montreal: Canadian Association of University Teachers.

Shulman, L. (1986). Those who understand: Knowledge growth in teaching . Educational Researcher 15 , 4-14.

Shulman, L.S. (1993). Teaching as community property: Putting an end to pedagogical solitude. Change 25 (6), 6-7.

Shulman, L. (1995). Faculty hiring: The pedagogical colloquium—three models. AAHE Bulletin 47 (9), 6-9.

Siebert, E.D., and McIntosh, W.J. (Eds.). (2001). College pathways to the science education standards . Arlington, VA: National Science Teachers Association.

Sorcinelli, M.D. (1999). The evaluation of teaching: The 40-year debate about student, colleague, and self-evaluations. In B.A. Pescosolido, and R. Aminzade (Eds.), The social worlds of higher education: Handbook for teaching in a new century (pp. 195-205). Thousand Oaks, CA: Pine Forge Press.

Sorcinelli, M.D. (2000). Principles of good practice: Supporting early career faculty. Guidance for deans, department chairs, and other academic leaders . New Pathways II Project Forum on Faculty Roles and Rewards. Washington, DC: American Association for Higher Education. See < http://www.aahe.org/ffrr/principles_brochure.htm >.

Springer, L., Stanne, M.E., and Donovan, S.S. (1998). Effects of small-group learning on undergraduates in science, mathematics, engineering, and technology: A meta-analysis (Research Monograph No. 11). Madison: University of Wisconsin-Madison, National Institute for Science Education.

Suskie, L. (Ed.). (2000). Assessment to promote deep learning . Washington, DC: American Association for Higher Education. See < http://www.aahe.org/catalog/iteminfo.cfm?itemid=1&itemid=127&g=t >.

Suter, L., and Frechtling, J. (2000). Guiding principles for mathematics and science education research methods: Report of a workshop . (NSF 00-113). Arlington, VA: National Science Foundation. See < http://nsf.gov/cgi-bin/getpub?nsf00113 >.

Svinicki, M., and Menges, R. (Eds.). (1996). Honoring exemplary teaching . New directions for teaching and learning . (No. 65). San Francisco: Jossey-Bass.

Uno, G.E. (1997). Handbook on teaching undergraduate science classes: A survival manual . Norman, OK: University of Oklahoma Press.

Walvoord, B.F., and Anderson, V.J. (1998). Effective grading: A tool for learning and assessment . San Francisco: Jossey-Bass.

Wergin, J. (1994). The collaborative department: How five campuses are inching toward cultures of collective responsibility . Washington, DC: American Association for Higher Education.

Wergin, J., and Swingen, J.N. (2000). Departmental assessment: How some campuses are effectively evaluating the collective work of faculty . Washington, DC: American Association for Higher Education.

Wiggins, G. (1998). Educative assessment: Designing assessments to inform and improve student performance . San Francisco: Jossey-Bass.

Wright, J.C., Millar, S.B., Kosciuk, S.A., Penberthy, D.L., Williams, P.H., and Wampold, B.E. (1998). A novel strategy for assessing the effects of curriculum reform on student competence. Journal of Chemical Education 75 , 986-992.

Wyckoff. S. (2001). Changing the culture of undergraduate science teaching. Journal of College Science Teaching 30 (5), 306-312.

Zimpher, N. (1998) . Ten changing demands on college teachers in the future . Presented at Changing Demands on College Teachers: A Conference for Teaching Support Providers, April 27, Columbus, Ohio. See < http://www.acs.ohio-state.edu/education/ftad/Publications/ten-nancy.html >.

Economic, academic, and social forces are causing undergraduate schools to start a fresh examination of teaching effectiveness. Administrators face the complex task of developing equitable, predictable ways to evaluate, encourage, and reward good teaching in science, math, engineering, and technology.

Evaluating, and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics offers a vision for systematic evaluation of teaching practices and academic programs, with recommendations to the various stakeholders in higher education about how to achieve change.

What is good undergraduate teaching? This book discusses how to evaluate undergraduate teaching of science, mathematics, engineering, and technology and what characterizes effective teaching in these fields.

Why has it been difficult for colleges and universities to address the question of teaching effectiveness? The committee explores the implications of differences between the research and teaching cultures-and how practices in rewarding researchers could be transferred to the teaching enterprise.

How should administrators approach the evaluation of individual faculty members? And how should evaluation results be used? The committee discusses methodologies, offers practical guidelines, and points out pitfalls.

Evaluating, and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics provides a blueprint for institutions ready to build effective evaluation programs for teaching in science fields.

READ FREE ONLINE

Welcome to OpenBook!

You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

Do you want to take a quick tour of the OpenBook's features?

Show this book's table of contents , where you can jump to any chapter by name.

...or use these buttons to go back to the previous chapter or skip to the next one.

Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

Switch between the Original Pages , where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

To search the entire text of this book, type in your search term here and press Enter .

Share a link to this book page on your preferred social network or via email.

View our suggested citation for this chapter.

Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

Get Email Updates

Do you enjoy reading reports from the Academies online for free ? Sign up for email notifications and we'll let you know about new publications in your areas of interest when they're released.

University Libraries Research Guides Course Guides Evaluation Methods in Education Refereed Journals Evaluation methods in education: refereed journals. Finding Refereed Articles Web Evaluation A refereed journal, or peer reviewed journal, is a specific type of publication that meets the high standards and rigor expected with academic publishing. Refereed articles within the journal have been reviewed by a blind editorial panel for rigor in research and appropriateness of conclusions.  Refereed sources enable you to start with the assumption that the information within is vetted, reducing the need to evaluate how factual and accurate the article is. The Anatomy of a Scholarly Source Most refereed articles contain the following sections. Some include all, others combine some, and yet other articles might include only a few. Look for these sections when judging whether an article is scholarly. Introduction or literature review Purpose of the study or problem statement Methodology, procedures, or research design Major findings, results, analysis, or discussion Summary, conclusion, implications, or  ideas for future studies Works cited or references Tables, charts, figures, and statistical data (throughout the article) Peer Review in 3 Minutes This video is provided courtesy of North Carolina State University Libraries. Qualities of a Scholarly Source Most refereed articles can be identified by the qualities you note throughout the writing. Here are a few to look for: Audience : The target audience is other researchers, colleagues, and specialists in the field. Research articles are written for the scholarly audience, rather than a general one. You may need to read the article a few times to fully understand it. Authors : Research articles may have several authors. You will easily be able to determine the academic affiliations of the authors, as well as their role within the organization. Language : Refereed articles are formal, generally do not use the first person, and include jargon used in the field.  Length : Refereed articles can vary in length, but are typically ten to fifty pages long. Topic : Research articles are specific in nature, relate to a particular field, or specialty within a field. Ask a Librarian Ask a Librarian There are many ways to get research help! Overview of Virginia Tech University Libraries Virginia Tech Univeristy Libraries Virginia Tech University Libraries Homepage Maps of the Libraries Library Hours Accessing Materials Interlibrary Loan Off-Campus Access Scholarly Workflows APA Citation Style Information Zotero (Bibliographic Management) Writing Center (English Department) Comm Lab (CLAHS) << Previous: Overview Next: Finding Refereed Articles >> Last Updated: Oct 27, 2023 1:53 PM A Criterion-Referenced Approach to Student Ratings of Instruction Published: 13 September 2016 Volume 58 , pages 545–567, ( 2017 ) Cite this article J. Patrick Meyer   ORCID: orcid.org/0000-0002-0044-7959 1 , Justin B. Doromal 1 , Xiaoxin Wei 1 & Shi Zhu 1   1343 Accesses 5 Citations 5 Altmetric Explore all metrics We developed a criterion-referenced student rating of instruction (SRI) to facilitate formative assessment of teaching. It involves four dimensions of teaching quality that are grounded in current instructional design principles: Organization and structure, Assessment and feedback, Personal interactions, and Academic rigor. Using item response theory and Wright mapping methods, we describe teaching characteristics at various points along the latent continuum for each scale. These maps enable criterion-referenced score interpretation by making an explicit connection between test performance and the theoretical framework. We explain the way our Wright maps can be used to enhance an instructor’s ability to interpret scores and identify ways to refine teaching. Although our work is aimed at improving score interpretation, a criterion-referenced test is not immune to factors that may bias test scores. The literature on SRIs is filled with research on factors unrelated to teaching that may bias scores. Therefore, we also used multilevel models to evaluate the extent to which student and course characteristic may affect scores and compromise score interpretation. Results indicated that student anger and the interaction between student gender and instructor gender are significant effects that account for a small amount of variance in SRI scores. All things considered, our criterion-referenced approach to SRIs is a viable way to describe teaching quality and help instructors refine pedagogy and facilitate course development. This is a preview of subscription content, log in via an institution to check access. Access this article Subscribe and save. Get 10 units per month Download Article/Chapter or eBook 1 Unit = 1 Article or 1 Chapter Cancel anytime Price includes VAT (Russian Federation) Instant access to the full article PDF. Rent this article via DeepDyve Institutional subscriptions Similar content being viewed by others Student Ratings of Instruction: Updating Measures to Reflect Recent Scholarship Student Ratings of Teaching Quality Dimensions: Empirical Findings and Future Directions An integrated strategy for the analysis of student evaluation of teaching: from descriptive measures to explanatory models. The complete measure is available upon request. For brevity, we did not include it in this paper. An unpublished manuscript about the original study is available upon request. A Wright map is also referred to as an item map. Aleamoni, L. M. (1999). Student rating myths versus research facts from 1924 to 1998. Journal of Personnel Evaluation in Education, 13 (2), 153–166. Article   Google Scholar   Anderson, K., & Miller, E. D. (1997). Gender and student evaluations of teaching. PS: Political Science and Politics, 30 (2), 216–219. Google Scholar   Arreola, R. A. (2007). Developing a comprehensive faculty evaluation system: A guide to designing, building, and operating large-scale faculty evaluation systems (3rd ed.). Bolton, MA: Anker Publishing Company Inc. Basow, S. A. (1995). Student evaluations of college professors: When gender matters. Journal of Educational Psychology, 87 (4), 656–665. Bates, D., Maechler, M., Bolker, B., & Walker, S. (2015). Fitting linear mixed effects models using lme4. Journal of Statistical Software, 67 (1), 1–48. doi: 10.18637/jss.v067.i01 . Bensley, D. A. (2010). A brief guide for teaching and assessing critical thinking in psychology. Observer , 23 (10). Retrieved from http://www.psychologicalscience.org/index.php/publications/observer/2010/december-10/a-brief-guide-for-teaching-and-assessing-critical-thinking-in-psychology.html . Benson, J. (1998). Developing a strong program of construct validation: A test anxiety example. Educational Measurement: Issues & Practice, 17 , 10–22. Benton, S. L., & Cashin, W. E. (2014). Student ratings of instruction in college and university courses. In M. B. Paulsen (Ed.), Higher education: Handbook of theory & research (Vol. 29, pp. 279–326). Dordrecht, The Netherlands: Springer. Chapter   Google Scholar   Bond, T. G., & Fox, C. M. (2007). Applying the Rasch model: Fundamental measurement in the human sciences (2nd ed.). New York: Routledge. Braga, M., Paccagnella, M., & Pellizzari, M. (2014). Evaluating students’ evaluations of professors. Economic of Education Review, 41 , 71–88. Bransford, J. D., Brown, A. L., & Cocking, R. R. (Eds.). (1999). How people learn: Brain, mind, experience, and school . Washington, DC: National Academy Press. Brophy, J. E. (1999). Teaching (educational practices series—1) . Geneva, Switzerland: International Academy of Education and International Bureau of Education, UNESCO. Carrell, S. E., & West, J. E. (2010). Does professor quality matter? Evidence from random assignment of students to professors. Journal of Political Economy, 118 (3), 409–432. Cizek, G. J., & Bunch, M. B. (2007). Standard setting: A guide to establishing and evaluating performance standards for tests . Thousand Oaks, CA: Sage. Book   Google Scholar   Clark, R. E. (1983). Reconsidering research on learning from media. Review of Educational Research, 53 , 445–459. doi: 10.3102/00346543053004445 . Clark, R. E. (2009). Translating research into new instructional technologies for higher education: The active ingredient process. The Journal of Computing in Higher Education, 21 , 4–18. doi: 10.1007/s12528-009-9013-8 . Clayson, D. E. (2009). Student evaluations of teaching: Are they related to what students learn? A meta-analysis and review of the literature. Journal of Marketing Education, 31 , 16–30. Cohen, P. A. (1981). Student ratings of instruction and student achievement: A meta-analysis of multisection validity studies. Review of Educational Research, 51 , 281–309. Feldman, K. A. (1992). College students’ views of male and female college teachers: Part I—evidence from the social laboratory and experiments. Research in Higher Education, 33 (3), 317–375. Feldman, K. A. (1993). College students’ views of male and female college teachers: Part II—evidence from students’ evaluations of their classroom teachers. Research in Higher Education, 34 (2), 151–211. Ferguson, R. F. (2012). Can student surveys measure teaching quality. The Phi Delta Kappan, 94 (3), 24–28. Fink, L. D. (2013). Creating significant learning experiences: An integrated approach to designing college courses . San Franciso: Jossey-Bass. Glaser, R. (1963). Instructional technology and the measurement of learning outcomes. American Psychologist, 18 , 519–521. Greenwald, A. G., & Gillmore, G. M. (1997). Grading leniency is a removable contaminant of student ratings. American Psychologist, 52 , 1209–1217. Hamre, B. K., Pianta, R. C., Downer, J. T., DeCoster, J., Mashburn, A. J., Jones, S. M., et al. (2013). Teaching through interactions: Testing a developmental framework of teacher effectiveness in over 4000 classrooms. The Elementary School Journal, 113 (4), 461–487. Hattie, J. (2009). Visible learning: A synthesis of over 800 meta-analyses relating to achievement . London: Routledge. Huynh, H. (1998). On score locations of binary and partial credit items and their applications to item mapping and criterion-referenced interpretation. Journal of Educational and Behavioral Statistics, 23 , 35–56. Huynh, H., & Meyer, J. P. (2003). Maximum information approach to scale description for affective measures based on the Rasch model. Journal of Applied Measurement, 4 , 101–110. Johnson, V. E. (2003). Grade inflation: A crisis in college education . New York: Springer. Kennedy, M. J., Thomas, C. N., Aronin, S., Newton, J. R., & Lloyd, J. W. (2014). Improving teacher candidate knowledge using content acquisition podcasts. Computers & Education, 70 , 116–127. doi: 10.1016/j.compedu.2013.08.010 . Kuznetsova, A., Brockhoff, P. B., & Christensen, R. H. B. (2015). Package ‘lmerTest’ [computer software] version 2.0–29. Retrieved from https://cran.r-project.org/web/packages/lmerTest/index.html . Linacre, J. M. (2006). A user’s guide to WINSTEPS Rasch-model computer programs . Chicago, IL: Author. Lüdtke, O., Robitzsch, A., Trautwein, U., & Kunter, M. (2009). Assessing the impact of learning environments: How to use student ratings of classroom or school characteristics in multilevel modeling. Contemporary Educational Psychology, 34 , 120–131. Marsh, H. W. (1987). Students’ evaluations of university teaching: Research findings, methodological issues, and directions for future research. International Journal of Educational Research, 11 , 253–388. Marsh, H. W., & Roche, L. A. (1997). Making students’ evaluations of teaching effectiveness effective: The critical issues of validity, bias, and utility. American Psychologist, 52 (11), 1187–1197. Marsh, H. W., & Roche, L. A. (2000). Effects of grading leniency and low workload on students’ evaluations of teaching: Popular myths, bias, validity, or innocent bystanders? Journal of Educational Psychology, 92 (1), 202–228. Masters, G. N. (1982). A Rasch model for partial credit scoring. Psychometrika, 47 (2), 149–174. Mayer, R. E. (2008). Applying the science of learning: Evidence-based principles for the design of multimedia instruction. American Psychologist, 63 , 760–769. doi: 10.1037/0003-066X.63.8.760 . Mayer, R. E. (2009). Multimedia learning (2nd ed.). New York: Cambridge University Press. McCaffrey, D. F., Lockwood, J. R., Koretz, D. M., & Hamilton, D. S. (2003). Evaluating value-added models for teacher accountability . Santa Monica, CA: Rand. McKeachie, W. J., & Svinicki, M. (2006). McKeachie’s teaching tips: Strategies, research, and theory for college and university teachers (12th ed.). Boston: Houghton Mifflin. Meyer, J. P. (2014). Applied measurement with jMetrik . New York: Routledge. Ory, J. C., & Ryan, K. (2001). How do student ratings measure up to a new validity framework? New Directions for Institutional Research, 109 , 27–44. doi: 10.1002/ir.2 . Pianta, R. C., & Hamre, B. K. (2009). Conceptualization, measurement, and improvement of classroom processes: Standardized observation can leverage capacity. Educational Researcher, 38 (2), 109–119. Pyc, M. A., Agarwal, P. K., & Roediger, H. L., III (2014). Test-enhanced learning. In V. A. Benassi, C. E. Overson, & C. M. Hakala (Eds.) Applying science of learning in education: Infusing psychological science into the curriculum . Retrieved from the Society for the Teaching of Psychology website http://teachpsych.org/ebooks/asle2014/index.php . Raudenbush, S. W., & Jean, M. (2014). To what extent do student perceptions of classroom quality predict teacher value added? In T. J. Kane, K. A. Kerr, & R. C. Pianta (Eds.), Designing teacher evaluation systems . San Franciso, CA: Jossey-Bass. Wiggins, G., & McTighe, J. (2011). Understanding by design (2nd ed.). Alexandria, VA: Association for Supervision and Curriculum Development. Willingham, D. T. (2007). Critical thinking: Why is it so hard to teach? American Educator . Washington, D.C.: American Federation of Teachers. http://www.aft.org/newspubs/periodicals/ae . Wilson, M. (2005). Constructing measures: An item response modeling approach . Mahwah, NJ: Lawrence Erlbaum. Download references Acknowledgments We thank Emily Bowling, Fares Karam, Bo Odom, and Laura Tortorelli for their work on the original version of this measure. They developed the original teaching framework and wrote the initial pool of items as part of a course project. Author information Authors and affiliations. Curry School of Education, University of Virginia, P.O. Box 400265, 405 Emmet Street South, Charlottesville, VA, 22904, USA J. Patrick Meyer, Justin B. Doromal, Xiaoxin Wei & Shi Zhu You can also search for this author in PubMed   Google Scholar Corresponding author Correspondence to J. Patrick Meyer . Rights and permissions Reprints and permissions About this article Meyer, J.P., Doromal, J.B., Wei, X. et al. A Criterion-Referenced Approach to Student Ratings of Instruction. Res High Educ 58 , 545–567 (2017). https://doi.org/10.1007/s11162-016-9437-8 Download citation Received : 09 February 2016 Published : 13 September 2016 Issue Date : August 2017 DOI : https://doi.org/10.1007/s11162-016-9437-8 Share this article Anyone you share the following link with will be able to read this content: Sorry, a shareable link is not currently available for this article. Provided by the Springer Nature SharedIt content-sharing initiative Criterion-referenced assessment Teaching quality Student ratings of instruction Item response theory Find a journal Publish with us Track your research 10 September 2024: Due to technical disruption, we are experiencing some delays to publication. We are working to restore services and apologise for the inconvenience. For further updates please visit our website: https://www.cambridge.org/universitypress/about-us/news-and-blogs/cambridge-university-press-publishing-update-following-technical-disruption We use cookies to distinguish you from other users and to provide you with a better experience on our websites. Close this message to accept cookies or find out how to manage your cookie settings . Login Alert > Understanding Education and Educational Research > References Book contents Frontmatter Acknowledgements Introduction 1 Education and its research 2 The nature of social science 3 The idea of method 4 The nature of philosophy 5 The art of research 6 Language, truth and meaning 7 On the dominant nature of educational research and its shortcomings 8 Research, policy and practical reasoning 9 The limits of measurement 10 Parenting and government intervention in the family (case study I) 11 Researching happiness and well-being (case study II) 12 Philosophy and research Published online by Cambridge University Press:  05 November 2014 Access options Save book to kindle. To save this book to your Kindle, first ensure [email protected] is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle . Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply. Find out more about the Kindle Personal Document Service . Paul Smeyers , Universiteit Gent, Belgium , Richard Smith , University of Durham Book: Understanding Education and Educational Research Online publication: 05 November 2014 Chapter DOI: https://doi.org/10.1017/CBO9780511920714.015 Save book to Dropbox To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Dropbox . Save book to Google Drive To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Google Drive . Skip to main content Keyboard shortcuts for audio player NPR fact-checked the Harris-Trump presidential debate. Here's what we found Vice President and Democratic presidential candidate Kamala Harris and former President and Republican presidential candidate Donald Trump speak during a presidential debate. Saul Loeb/AFP via Getty Images hide caption Vice President Harris and former President Donald Trump faced off Tuesday in their first — and possibly only — debate of the 2024 campaign, taking questions on key issues like the border, the economy and abortion. With the candidates virtually tied in the polls, and just 55 days until Election Day, Trump and Harris sought to define their visions for America in front of a national audience and deflect attacks from the other side. NPR reporters fact-checked the candidates' claims in real time . Here's what they found: NPR's White House Correspondent Discusses The Debate | Morning Edition | NPR TRUMP: "I had no inflation, virtually no inflation. They had the highest inflation, perhaps in the history of our country, because I've never seen a worse period of time. People can't go out and buy cereal or bacon or eggs or anything else." Inflation soared to a four-decade high of 9.1% in 2022, according to the consumer price index. While inflation has since fallen to 2.9% (as of July), prices — particularly food prices — are still higher than many Americans would like. Other countries have also faced high inflation in the wake of the pandemic, as tangled supply chains struggled to keep pace with surging demand. Russia’s invasion of Ukraine also fueled inflation by driving up energy and food prices worldwide. Government spending in the U.S. under both the Biden-Harris administration and Trump also may have contributed, putting more money in people’s pockets and enabling them to keep spending in the face of high prices. While high prices are a source of frustration for many Americans, the average worker has more buying power today than she did before the pandemic. Since February 2020 (just before the pandemic took hold in the U.S.), consumer prices have risen 21.6% while average wages have risen 23%. Many prices were depressed early in the pandemic, however, so the comparison is less flattering if you start the clock when President Biden and Vice President Harris took office. Since early 2021, consumer prices have risen 19.6%, while average wages have risen 16.9%. Wage gains have been outpacing price increases for over a year, so that gap should eventually close. — NPR economics correspondent Scott Horsley 2024 Election Taylor swift endorses kamala harris in instagram post after the debate. HARRIS: "Donald Trump left us the worst unemployment since the Great Depression." At the height of the Great Depression in 1933, the national unemployment rate was near 25%, according to the Franklin D. Roosevelt Presidential Library. At the start of the COVID pandemic, the unemployment rate peaked at 14.8% in April 2020, a level not seen since 1948, according to the Congressional Research Service. But by the time Trump left office, unemployment had fallen to a lower, but still elevated, level. The January 2021 unemployment rate was 6.3%. — NPR producer Lexie Schapitl Immigration TRUMP: "You see what's happening with towns throughout the United States. You look at Springfield, Ohio, you look at Aurora in Colorado. They are taking over the towns. They're taking over buildings. They're going in violently. These are the people that she and Biden let into our country, and they're destroying our country. They're dangerous. They're at the highest level of criminality, and we have to get them out." Trump attacked Harris and Biden's records on immigration, arguing that they're failing to stem people from other countries from entering the U.S. and causing violence. In the last two years, more than 40,000 Venezuelan immigrants have arrived in the Denver metro area. And it is true that many now live in Aurora. A few weeks ago, a video of gang members in an Aurora, Colo., apartment building had right-wing media declaring the city's takeover by Venezuelan gangs. NPR looked into these claims . Indian migrants drive surge in northern U.S. border crossings Shortly after the video appeared, Colorado's Republican Party sent a fundraising letter claiming the state is under violent attack, and Venezuelan gangs have taken over Aurora. It's also true Aurora police have recently arrested 10 members of a Venezuelan gang called Tren de Aragua. But Aurora's interim police chief, Heather Morris, says there's no evidence of a gang takeover of apartment buildings in her city. What's more, violent crime — including murder, robbery and rape — is way down nationwide, according to the most recent data from the FBI . Notably, analysts predict violent crime rates this year will fall back down to where they were before they surged during the pandemic and may even approach a 50-year low. Trump also claims that migrants are driving up crime rates in the U.S. That is not true. Researchers from Stanford University found that since the 1960s, immigrants have been 60% less likely to be incarcerated than people born in the U.S. The Cato Institute, a libertarian think tank, found undocumented immigrants in Texas were 37% less likely to be convicted of a crime. — NPR immigration correspondent Jasmine Garsd and criminal justice reporter Meg Anderson TRUMP: "In Springfield, they're eating the dogs. The people that came in, they're eating the cats. They're eating the pets of the people that live there." This remark refers to a debunked, dehumanizing claim that Haitian migrants living in Springfield, Ohio, are abducting pets and eating them . Untangling Disinformation Jd vance spreads debunked claims about haitian immigrants eating pets. The claim, which local police say is baseless, first circulated among far-right activists, local Republicans and neo-Nazis before being picked up by congressional leaders, vice presidential candidate JD Vance and others. A well-known advocate for the Haitian community says she received a wave of racist harassment after Vance shared the theory on social media. The Springfield News-Sun reported that local police said that incidents of pets being stolen or eaten were "not something that's on our radar right now." The paper said the unsubstantiated claim seems to have started with a post in a Springfield Facebook group that was widely shared across social media. The claim is the latest example of Trump leaning into anti-immigrant rhetoric. Since entering the political arena in 2015, Trump accused immigrants of being criminals, rapists, or "poisoning the blood of our nation." — NPR immigration correspondent Jasmine Garsd TRUMP: "A lot of these illegal immigrants coming in, [Democrats] are trying to get them to vote." It is illegal for noncitizens to vote in federal elections, and there is no credible evidence that it has happened in significant numbers, or that there is an effort underway to illegally register undocumented immigrants to vote this election. Voter registration forms require voters to sign an oath — under penalty of perjury — that they are U.S. citizens. If a noncitizen lies about their citizenship on a registration form and votes, they have created a paper trail of a crime that is punishable with jail time and deportation. “The deterrent is incredibly strong,” David Becker, executive director of the Center for Election Innovation and Research, told NPR. Illegal crossings hit Biden-era low as migrants wait longer for entry Election officials routinely verify information on voter registration forms, which ask registrants for either a driver’s license number or the last four digits of Social Security numbers. In 2016, the Brennan Center for Justice surveyed local election officials in 42 jurisdictions with high immigrant populations and found 30 cases of suspected noncitizens voting out of 23.5 million votes cast, or 0.0001%. Georgia Secretary of State Brad Raffensperger launched an audit in 2022 that found fewer than 1,700 suspected noncitizens had attempted to register to vote over the past 25 years. None were able to vote. — NPR disinformation reporter Jude Joffe-Block TRUMP: "[Harris] was the border czar. Remember that she was the border czar." Republicans have taken to calling Harris the "border czar" as a way to blame her for increased migration to the U.S. and what they see as border security policy failures of the Biden administration. There is no actual "border czar" position. In 2021, President Biden tasked Harris with addressing the root causes of migration from Central America. As Republicans attack Harris on immigration, here’s what her California record reveals The "root causes strategy ... identifies, prioritizes, and coordinates actions to improve security, governance, human rights, and economic conditions in the region," the White House said in a statement. "It integrates various U.S. government tools, including diplomacy, foreign assistance, public diplomacy, and sanctions." While Harris has been scrutinized on the right, immigration advocates have also criticized Harris, including for comments in 2021 where she warned prospective migrants, "Do not come." TRUMP: "You could do abortions in the seventh month, the eighth month, the ninth month, and probably after birth." As ABC News anchor Linsey Davis mentioned during her real-time fact check, there is no state where it is legal to kill a baby after birth (Trump called it "execution"). A report from KFF earlier this year also noted that abortions “after birth” are illegal in every state. According to the Pew Research Center, the overwhelming majority of abortions — 93% — take place during the first trimester. Pew says 1% take place after 21 weeks. Most of those take place before 24 weeks, the approximate timeline for fetal viability, according to a report by KFF Health News. Trump repeats the false claim that Democrats support abortion 'after birth' in debate A separate analysis from KFF earlier this year noted that later abortions are expensive to obtain and offered by relatively few providers, and often occur because of medical complications or because patients face barriers earlier in their pregnancies. “Nowhere in America is a woman carrying a pregnancy to term and asking for an abortion. That isn’t happening; it’s insulting to the women of America,” Harris said. Harris also invoked religion in her response, arguing that “one does not have to abandon their faith” to agree that the government should not control reproductive health decisions. As Davis also noted, Trump has offered mixed messages about abortion over the course of the campaign. He has bragged about his instrumental role in overturning Roe v. Wade , while appearing to backpedal on an issue that polling makes clear is a liability for Republicans. — NPR political correspondent Sarah McCammon Afghanistan TRUMP: The U.S. withdrawal from Afghanistan "was one of the most incompetently handled situations anybody has ever seen." Trump and Republicans in Congress say President Biden is to blame for the fall of Kabul to the Taliban three years ago, and the chaotic rush at the airport where 13 U.S. troops died in a suicide bomb attack that killed nearly 200 Afghan civilians trying to flee. Of late, Republicans have been emphasizing Harris’ role . But the Afghanistan war spanned four U.S. presidencies , and it's important to note that it was the Trump administration that signed a peace deal that was basically a quick exit plan. Trump regularly claims there were no casualties in Afghanistan for 18 months under his administration, and it’s not true, according to Pentagon records. — NPR veterans correspondent Quil Lawrence Military policy HARRIS: “There is not one member of the military who is in active duty in a combat zone in any war zone around the world for the first time this century.” This is a common administration talking point, and it's technically true. But thousands of troops in Iraq and on the Syrian border are still in very dangerous terrain. U.S. troops died in Jordan in January on a base that keeps watch over the war with ISIS in Syria. HARRIS: "I will not ban fracking. I have not banned fracking as vice president United States, and in fact, I was the tie-breaking vote on the inflation Reduction Act which opened new leases for fracking." When she first ran for president in 2019, Harris had said she was firmly in favor of banning fracking — a stance she later abandoned when she joined President Biden’s campaign as his running mate. In an interview with CNN last month, Harris attempted to explain why her position has changed from being against fracking to being in favor of it. “What I have seen is that we can grow, and we can increase a clean energy economy without banning fracking,” Harris told CNN’s Dana Bash. Harris says she won't ban fracking. What to know about the controversial topic Under the Biden-Harris administration, the U.S. produced a record amount of oil last year — averaging 12.9 million barrels per day. That eclipsed the previous record of 12.3 million barrels per day, set under Trump in 2019. 2023 was also a record year for domestic production of natural gas . Much of the domestic boom in oil and gas production is the result of hydraulic fracturing or “fracking” techniques . In addition to record oil and gas production, the Biden-Harris administration has also coincided with rapid growth of solar and wind power . Meanwhile, coal has declined as a source of electricity. Health care TRUMP: "I had a choice to make: Do I save [the Affordable Care Act] and make it as good as it can be, or do I let it rot? And I saved it." During his presidency, Trump undermined the Affordable Care Act in many ways — for instance, by slashing funding for advertising and free "navigators" who help people sign up for a health insurance plan on HealthCare.gov. And rather than deciding to "save" the ACA, he tried hard to get Congress to repeal it, and failed. When pushed Tuesday on what health policy he would put in its place, he said he has "concepts of a plan." Shots - Health News Amid medicaid's 'unwinding,' many states work to expand health care access. The Biden administration has reversed course from Trump's management of the Affordable Care Act. Increased subsidies have made premiums more affordable in the marketplaces, and enrollment has surged. The uninsurance rate has dropped to its lowest point ever during the Biden administration. The Affordable Care Act was passed in 2010 and is entrenched in the health care system. Republicans successfully ran against Obamacare for about a decade, but it has faded as a campaign issue this year. — NPR health policy correspondent Selena Simmons-Duffin 337 IMAGES Citing Online Articles in APA Format How to Do a Reference Page for a Research Paper: 11 Steps General rules: Reference list What Is a Citation? References in Research Research Papers in Education Referencing Guide · Research Papers in VIDEO Research, Educational research Lecture- 63 Meaning and Scope of educational research (शैक्षिक अनुसंधान का अर्थ व क्षेत्र) Empowering schools and policy institutions through a culture of research engagement HOW TO CITE AND REFERENCE IN YOUR PROJECT AND SEMINAR 1 (TIPS ON CITATION AND BIBLIOGRAPHY) Educational research Engaging with education research: With a little help from the system COMMENTS PDF Common Guidelines for Education Research and Development Research contribute Type seek to test, to improved. #1: Foundational of systems or learning processes. Research. in methodologies and/o technologies of teaching provides or education the fundamental that will influence and infor outcomes. and may knowledge development in contexts. research and innovations of. Ethical Guidelines for Educational Research, fourth edition (2018 The 'community of educational researchers' is considered to mean all those engaged in educational research - including, for example, students following research-based programmes of study and independent researchers, as well as staff who conduct educational research in their employment within organisations such as universities, schools ... References Research and education reform: Roles for the Office of Educational Research and Improvement. Committee on the Federal Role in Education Research. R.C. Atkinson and G.B. Jackson, Eds. Commission on Behavioral and Social Sciences and Education. Washington, DC: National Academy Press. National Research Council. (1998). Achieving Better Educational Practices Through Research Evidence: A From our examination of contemporary policy and research literature, one major finding is that despite the growing attention being given to evidence, consumers of educational products (i.e., superintendents, principals, and procurement officers) report making only limited use of research evidence in selecting products, instead preferring peer ... Quality of Research Evidence in Education: How Do We Know? American Educational Research Association & National Academy of Education. (2020, September 24). ... Carter P. L. (2016). Educational equality is a multifaceted issue: Why we must understand the school's sociocultural context for student achievement. ... Download to reference manager. Assessing the Quality of Education Research Through Its Relevance to RPPs increase local education leaders' access to research and bolster the use of research. RPPs may also strengthen the alignment between education research and the public good. Notwithstanding, employing RPPs as a vehicle to assess research quality has its challenges. Evidence-Based Educational Practice Evidence-based practice (EBP) is a buzzword in contemporary professional debates, for example, in education, medicine, psychiatry, and social policy. It is known as the "what works" agenda, and its focus is on the use of the best available evidence to bring about desirable results or prevent undesirable ones. Sage Research Methods The SAGE Encyclopedia of Educational Research, Measurement, and Evaluation. In an era of curricular changes and experiments and high-stakes testing, educational measurement and evaluation is more important than ever. In addition to expected entries covering the basics of traditional theories and methods, other entries discuss important ... Evaluations of Educational Practice, Programs, Projects ... Program evaluation involves a systematic effort to report and explain what has happened, or what is happening, as a result of particular educational policies, practices, and instructional interventions (Spector, 2013).Introducing and integrating an innovation such as an educational technology or new instructional approach is a situation that is typically evaluated in order to determine to what ... What Is in a Reference? Theoretically Understanding the Uses of The chapter deals with the reference—the unit of analysis for our bibliometric analyses—and examines what it stands for in the policy process. We found Paul Cairney's ( The Politics of ... Chapter 7: Synthesizing Sources A literature review is not an annotated bibliography, organized by title, author, or date of publication. Rather, it is grouped by topic to create a whole view of the literature relevant to your research question. Figure 7.1. Your synthesis must demonstrate a critical analysis of the papers you collected as well as your ability to integrate the ... References Ewell, P.T. (1991). To capture the ineffable: New forms of assessment in higher education. Review of Research in Education 17, 75-125. Feldman, K.A. (1978). Course characteristics and college students' ratings of their teachers and courses: What we know and what we don't. Research in Higher Education 9, 199-242. Feldman, K.A. (1984). How to Design and Evaluate Research in Education An expectancy table is a two-way chart used to evaluate criterion-related evidence of validity. Reliability. The term "reliability," as used in research, refers to the consistency of scores or answers provided by an instrument. Errors of measurement refer to variations in scores obtained by the same individuals on the same instrument. Considerations for Evidence Frameworks in Education Research In this chapter, we describe and compare the standards for evidence used by three entities that review studies of education interventions: Blueprints for Healthy Youth Development, Social Programs that Work, and the What Works Clearinghouse. A framework for understanding the quality of evidence use in education Purpose. With these issues in mind, this article presents a conceptual framework to define and elaborate what quality use of research evidence might mean in relation to education. This work was undertaken as part of an ongoing study in Australia, the Monash Q Project, to understand and improve research use in Australian schools (see Rickinson et al. Citation 2020b, Citation 2020a). Evaluation Methods in Education: Refereed Journals Refereed Journals. A refereed journal, or peer reviewed journal, is a specific type of publication that meets the high standards and rigor expected with academic publishing. Refereed articles within the journal have been reviewed by a blind editorial panel for rigor in research and appropriateness of conclusions. Educational research Educational research refers to the systematic collection and analysis of evidence and data related to the field of education. Research may involve a variety of methods [1] [2] [3] and various aspects of education including student learning, interaction, teaching methods, teacher training, and classroom dynamics. [4]Educational researchers generally agree that research should be rigorous and ... How about the evidence assessment tools used in education and 3.3. Evidence assessment tools used in systematic reviews. We identified a total of 66 assessment tools from the 104 systematic reviews. The most frequently used tool was the Risk of Bias (ROB) and its updated version (16/104, 15.4%), followed by the Medical Education Research Quality Instrument (MERSQI) (9/104, 8.7%). A Criterion-Referenced Approach to Student Ratings of Instruction We developed a criterion-referenced student rating of instruction (SRI) to facilitate formative assessment of teaching. It involves four dimensions of teaching quality that are grounded in current instructional design principles: Organization and structure, Assessment and feedback, Personal interactions, and Academic rigor. Using item response theory and Wright mapping methods, we describe ... Educational Research and Evaluation Educational Research and Evaluation is a fully refereed international journal publishing original academic articles on all aspects of education. It appeals to an international readership and accepts articles that are well-informed and evidence-based. It is indexed by Scopus and attracts a strong number of citations.. The journal is deliberately multi-disciplinary and broad in its scope, and ... Research Worth Using: (Re)Framing Research Evidence Quality for Anchored in scholarship on research utilization and methodological critiques, the chapter introduces a research quality framework that integrates relevance and rigor through five key dimensions of Research Worth Using: (1) relevance of question: alignment of research topics to practical priorities; (2) theoretical credibility: explanatory ... References London: Routledge Google Scholar. Eisenhart, M. 2005. Hammers and saws for the improvement of educational research, Educational Theory 55: 245-261 Google Scholar. Eisner, E. 1993. Forms of understanding and the future of educational research, Educational Researcher 22 (7): 5-11 CrossRef Google Scholar. Evaluation & Research in Education Educational Research and Evaluation is a fully refereed international journal publishing original academic articles on all aspects of education. It appeals to an international readership and accepts articles that are well-informed and evidence-based. It is indexed by Scopus and attracts a strong number of citations.. The journal is deliberately multi-disciplinary and broad in its scope, and ... Fact check of the presidential debate between Kamala Harris and Donald According to the Pew Research Center, ... last year — averaging 12.9 million barrels per day. That eclipsed the previous record of 12.3 million barrels per day, set under Trump in 2019. essay homework paper phd project resume review speech study thesis