Improving the use of research evidence in guideline development: 16. Evaluation.


BACKGROUND:The World Health Organization (WHO), like many other organisations around the world, has recognised the need to use more rigorous processes to ensure that health care recommendations are informed by the best available research evidence. This is the last of a series of 16 reviews that have been prepared as background for advice from the WHO Advisory Committee on Health Research to WHO on how to achieve this. OBJECTIVES:We reviewed the literature on evaluating guidelines and recommendations, including their quality, whether they are likely to be up-to-date, and their implementation. We also considered the role of guideline developers in undertaking evaluations that are needed to inform recommendations. METHODS:We searched PubMed and three databases of methodological studies for existing systematic reviews and relevant methodological research. We did not conduct systematic reviews ourselves. Our conclusions are based on the available evidence, consideration of what WHO and other organisations are doing and logical arguments. KEY QUESTIONS AND ANSWERS:Our answers to these questions were informed by a review of instruments for evaluating guidelines, several studies of the need for updating guidelines, discussions of the pros and cons of different research designs for evaluating the implementation of guidelines, and consideration of the use of uncertainties identified in systematic reviews to set research priorities. How should the quality of guidelines or recommendations be appraised? WHO should put into place processes to ensure that both internal and external review of guidelines is undertaken routinely. A checklist, such as the AGREE instrument, should be used. The checklist should be adapted and tested to ensure that it is suitable to the broad range of recommendations that WHO produces, including public health and health policy recommendations, and that it includes questions about equity and other items that are particularly important for WHO guidelines. When should guidelines or recommendations be updated? Processes should be put into place to ensure that guidelines are monitored routinely to determine if they are in need of updating. People who are familiar with the topic, such as Cochrane review groups, should do focused, routine searches for new research that would require revision of the guideline. Periodic review of guidelines by experts not involved in developing the guidelines should also be considered. Consideration should be given to establishing guideline panels that are ongoing, to facilitate routine updating, with members serving fixed periods with a rotating membership. How should the impact of guidelines or recommendations be evaluated? WHO headquarters and regional offices should support member states and those responsible for policy decisions and implementation to evaluate the impact of their decisions and actions by providing advice regarding impact assessment, practical support and coordination of efforts. Before-after evaluations should be used cautiously and when there are important uncertainties regarding the effects of a policy or its implementation, randomised evaluations should be used when possible. What responsibility should WHO take for ensuring that important uncertainties are addressed by future research when the evidence needed to inform recommendations is lacking? Guideline panels should routinely identify important uncertainties and research priorities. This source of potential priorities for research should be used systematically to inform priority-setting processes for global research.


Health Res Policy Syst


Oxman AD,Schünemann HJ,Fretheim A




Has Abstract


2006-12-08 00:00:00










  • Is U.S. health care an appropriate system? A strategic perspective from systems science.

    abstract:CONTEXT:Systems science provides organizational principles supported by biologic findings that can be applied to any organization; any incongruence indicates an incomplete or an already failing system. U.S. health care is commonly referred to as a system that consumes an ever- increasing percentage of the gross domesti...

    journal_title:Health research policy and systems

    pub_type: 杂志文章


    authors: Janecka IP

    更新日期:2009-01-02 00:00:00

  • Identifying research priorities for public health research to address health inequalities: use of Delphi-like survey methods.

    abstract:BACKGROUND:In the funding of health research and public health research it is vital that research questions posed are important and that funded research meets a research need or a gap in evidence. Many methods are used in the identification of research priorities, however, these can be resource intensive, costly and lo...

    journal_title:Health research policy and systems

    pub_type: 杂志文章


    authors: Turner S,Ollerhead E,Cook A

    更新日期:2017-10-09 00:00:00

  • The influence of cost-per-DALY information in health prioritisation and desirable features for a registry: a survey of health policy experts in Vietnam, India and Bangladesh.

    abstract:BACKGROUND:Economic evaluation has been implemented to inform policy in many areas, including coverage decisions, technology pricing, and the development of clinical practice guidelines. However, there are barriers to evidence-based policy in low- and middle-income countries (LMICs) that include limited stakeholder awa...

    journal_title:Health research policy and systems

    pub_type: 杂志文章


    authors: Teerawattananon Y,Tantivess S,Yamabhai I,Tritasavit N,Walker DG,Cohen JT,Neumann PJ

    更新日期:2016-12-03 00:00:00

  • INA-RESPOND: a multi-centre clinical research network in Indonesia.

    abstract::Nationally representative observational and translational research is needed to address the public health challenges in Indonesia due to the geographic disparity, recently decentralized health system, and diverse infectious disease priorities. To accomplish this, the Indonesian Ministry of Health in collaboration with...

    journal_title:Health research policy and systems

    pub_type: 杂志文章


    authors: Karyana M,Kosasih H,Samaan G,Tjitra E,Aman AT,Alisjahbana B,Fatmawati,Gasem MH,Arif M,Sudarmono P,Suharto,Merati TP,Lane C,Siswanto,Siddiqui S

    更新日期:2015-07-29 00:00:00

  • Priority-setting in health research in Iran: a qualitative study on barriers and facilitators.

    abstract:BACKGROUND:Priority-setting is a complicated and time-consuming process; however, if appropriately conducted, it could efficiently divert resources to the most important studies. A considerable body of evidence indicates that priority-setting measures in health research taken so far in Iran have not satisfied decision-...

    journal_title:Health research policy and systems

    pub_type: 杂志文章


    authors: Badakhshan A,Arab M,Rashidian A,Mehrdad N,Zendehdel K

    更新日期:2018-07-02 00:00:00

  • Regulatory scientists' work has important ramifications for public health and should be open to public scrutiny.

    abstract::The Swedish Medical Products Agency (MPA) objects to the fact that we occasionally refer to one of its senior ex-employees by name. However, names of individual MPA assessors, Food and Drug Administration (FDA) reviewers, and European Medicines Agency rapporteurs and co-rapporteurs are cited in regulatory documents an...

    journal_title:Health research policy and systems

    pub_type: 评论,信件


    authors: Mulinari S,Davis C

    更新日期:2018-11-15 00:00:00

  • Developing pathways for community-led research with big data: a content analysis of stakeholder interviews.

    abstract:BACKGROUND:Big data (BD) informs nearly every aspect of our lives and, in health research, is the foundation for basic discovery and its tailored translation into healthcare. Yet, as new data resources and citizen/patient-led science movements offer sites of innovation, segments of the population with the lowest health...

    journal_title:Health research policy and systems

    pub_type: 杂志文章


    authors: Grayson S,Doerr M,Yu JH

    更新日期:2020-07-08 00:00:00

  • Public funding of health at the district level in Indonesia after decentralization-sources, flows and contradictions.

    abstract:BACKGROUND:During the Suharto era public funding of health in Indonesia was low and the health services were tightly controlled by the central government; district health staff had practically no discretion over expenditure. Following the downfall of President Suharto there was a radical political, administrative and f...

    journal_title:Health research policy and systems

    pub_type: 杂志文章


    authors: Heywood P,Harahap NP

    更新日期:2009-04-16 00:00:00

  • Strange bedfellows: bridging the worlds of academia, public health and the sex industry to improve sexual health outcomes.

    abstract::The public health response to sexually transmitted infections, particularly HIV, has been and continues to be overwhelmingly focused on risk, disease and negative outcomes of sex, while avoiding discussion of positive motivations for sex (e.g. pleasure, desire, love). Recent advocacy efforts have challenged this appro...

    journal_title:Health research policy and systems

    pub_type: 杂志文章


    authors: Knerr W,Philpott A

    更新日期:2011-06-16 00:00:00

  • How do we define the policy impact of public health research? A systematic review.

    abstract:BACKGROUND:In order to understand and measure the policy impact of research we need a definition of research impact that is suited to the task. This article systematically reviewed both peer-reviewed and grey literature for definitions of research impact to develop a definition of research impact that can be used to in...

    journal_title:Health research policy and systems

    pub_type: 杂志文章,评审


    authors: Alla K,Hall WD,Whiteford HA,Head BW,Meurk CS

    更新日期:2017-10-02 00:00:00

  • New directions in evidence-based policy research: a critical analysis of the literature.

    abstract::Despite 40 years of research into evidence-based policy (EBP) and a continued drive from both policymakers and researchers to increase research uptake in policy, barriers to the use of evidence are persistently identified in the literature. However, it is not clear what explains this persistence - whether they represe...

    journal_title:Health research policy and systems

    pub_type: 杂志文章


    authors: Oliver K,Lorenc T,Innvær S

    更新日期:2014-07-14 00:00:00

  • Development of rapid guidelines: 2. A qualitative study with WHO guideline developers.

    abstract:BACKGROUND:Situations such as public health emergencies and outbreaks necessitate the development and publication of high-quality recommendations within a condensed timeframe. For example, WHO has produced examples of and guidance for the development of rapid guidelines (RGs). However, more information is needed to und...

    journal_title:Health research policy and systems

    pub_type: 杂志文章


    authors: Florez ID,Morgan RL,Falavigna M,Kowalski SC,Zhang Y,Etxeandia-Ikobaltzeta I,Santesso N,Wiercioch W,Schünemann HJ

    更新日期:2018-07-13 00:00:00

  • Knowledge translation for realist reviews: a participatory approach for a review on scaling up complex interventions.

    abstract:BACKGROUND:Knowledge syntheses that use a realist methodology are gaining popularity. Yet, there are few reports in the literature that describe how results are summarised, shared and used. This paper aims to inform knowledge translation (KT) for realist reviews by describing the process of developing a KT strategy for...

    journal_title:Health research policy and systems

    pub_type: 杂志文章


    authors: Boyko JA,Riley BL,Willis CD,Stockton L,Zummach D,Kerner J,Robinson K,Chia M

    更新日期:2018-10-22 00:00:00

  • Four centuries on from Bacon: progress in building health research systems to improve health systems?

    abstract::In 1627, Francis Bacon's New Atlantis described a utopian society in which an embryonic research system contributed to meeting the needs of the society. In this editorial, we use some of the aspirations described in New Atlantis to provide a context within which to consider recent progress in building health research ...

    journal_title:Health research policy and systems

    pub_type: 社论


    authors: Hanney SR,González-Block MA

    更新日期:2014-09-23 00:00:00

  • Setting priorities in health research using the model proposed by the World Health Organization: development of a quantitative methodology using tuberculosis in South Africa as a worked example.

    abstract:BACKGROUND:Setting priorities is important in health research given the limited resources available for research. Various guidelines exist to assist in the priority setting process; however, priority setting still faces significant challenges such as the clear ranking of identified priorities. The World Health Organiza...

    journal_title:Health research policy and systems

    pub_type: 杂志文章


    authors: Hacking D,Cleary S

    更新日期:2016-02-09 00:00:00

  • An academic perspective of participation in healthcare redesign.

    abstract::Healthcare redesign, based on building collaborative capacity between academic and clinical partners, should create a method to facilitate flow between the key elements of health service improvement. However, utilising the skills and resources of an organisation outside of the health facility may not always have the d...

    journal_title:Health research policy and systems

    pub_type: 信件


    authors: Prior SJ,Mather C,Miller A,Campbell S

    更新日期:2019-11-20 00:00:00

  • Conceptual frameworks and empirical approaches used to assess the impact of health research: an overview of reviews.

    abstract:BACKGROUND:How to assess the impact of research is of growing interest to funders, policy makers and researchers mainly to understand the value of investments and to increase accountability. Broadly speaking the term "research impact" refers to the contribution of research activities to achieve desired societal outcome...

    journal_title:Health research policy and systems

    pub_type: 杂志文章


    authors: Banzi R,Moja L,Pistotti V,Facchini A,Liberati A

    更新日期:2011-06-24 00:00:00

  • Continuation rates and reasons for discontinuation of intra-uterine device in three provinces of Pakistan: results of a 24-month prospective client follow-up.

    abstract:BACKGROUND:Long-acting reversible contraceptives, such as the intrauterine device (IUD), remain underutilised in Pakistan with high discontinuation rates. Based on a 24-month prospective client follow-up (nested within a larger quasi-experimental study), this paper presents the comparison of two intervention models, on...

    journal_title:Health research policy and systems

    pub_type: 杂志文章


    authors: Hameed W,Azmat SK,Ishaque M,Hussain W,Munroe E,Mustafa G,Khan OF,Abbas G,Ali S,Asghar QJ,Ali S,Ahmed A,Hamza HB

    更新日期:2015-11-25 00:00:00

  • Factors affecting engagement between academic faculty and decision-makers: learnings and priorities for a school of public health.

    abstract:BACKGROUND:Schools of public health (SPHs) are increasingly being recognised as important contributors of human, social and intellectual capital relevant to health policy and decision-making. Few studies within the implementation science literature have systematically examined knowledge exchange experiences within this...

    journal_title:Health research policy and systems

    pub_type: 杂志文章


    authors: Jessani NS,Siddiqi SM,Babcock C,Davey-Rothwell M,Ho S,Holtgrave DR

    更新日期:2018-07-25 00:00:00

  • Role and use of evidence in policymaking: an analysis of case studies from the health sector in Nigeria.

    abstract:BACKGROUND:Health policymaking is a complex process and analysing the role of evidence is still an evolving area in many low- and middle-income countries. Where evidence is used, it is greatly affected by cognitive and institutional features of the policy process. This paper examines the role of different types of evid...

    journal_title:Health research policy and systems

    pub_type: 杂志文章


    authors: Onwujekwe O,Uguru N,Russo G,Etiaba E,Mbachu C,Mirzoev T,Uzochukwu B

    更新日期:2015-10-24 00:00:00

  • Learning sites for health system governance in Kenya and South Africa: reflecting on our experience.

    abstract:BACKGROUND:Health system governance is widely recognised as critical to well-performing health systems in low- and middle-income countries. However, in 2008, the Alliance for Health Policy and Systems Research identified governance as a neglected health systems research issue. Given the demands of such research, the Al...

    journal_title:Health research policy and systems

    pub_type: 信件


    authors: RESYST\/DIAHLS learning site team.

    更新日期:2020-05-11 00:00:00

  • The Paris Declaration in practice: challenges of health sector aid coordination at the district level in Zambia.

    abstract:BACKGROUND:The increasing resources available for and number of partners providing health sector aid have stimulated innovations, notably, the Paris Declaration on Aid Effectiveness, which aim to improve aid coordination. In this, one of the first studies to analyse implementation of aid coordination below national lev...

    journal_title:Health research policy and systems

    pub_type: 杂志文章


    authors: Sundewall J,Forsberg BC,Jönsson K,Chansa C,Tomson G

    更新日期:2009-06-08 00:00:00

  • Implementing without guidelines, learning at the coalface: a case study of health promoters in an era of community health workers in South Africa.

    abstract:BACKGROUND:Internationally, there has been renewed focus on primary healthcare (PHC). PHC revitalisation is one of the mechanisms to emphasise health promotion and prevention. However, it is not always clear who should lead health promotion activities. In some countries, health promotion practitioners provide health pr...

    journal_title:Health research policy and systems

    pub_type: 杂志文章


    authors: Rwafa-Ponela T,Eyles J,Christofides N,Goudge J

    更新日期:2020-05-14 00:00:00

  • Evidence for Health II: Overcoming barriers to using evidence in policy and practice.

    abstract::Even the highest quality evidence will have little impact unless it is incorporated into decision-making for health. It is therefore critical to overcome the many barriers to using evidence in decision-making, including (1) missing the window of opportunity, (2) knowledge gaps and uncertainty, (3) controversy, irrelev...

    journal_title:Health research policy and systems

    pub_type: 杂志文章,评审


    authors: Andermann A,Pang T,Newton JN,Davis A,Panisset U

    更新日期:2016-03-14 00:00:00

  • The development of ORACLe: a measure of an organisation's capacity to engage in evidence-informed health policy.

    abstract:BACKGROUND:Evidence-informed policymaking is more likely if organisations have cultures that promote research use and invest in resources that facilitate staff engagement with research. Measures of organisations' research use culture and capacity are needed to assess current capacity, identify opportunities for improve...

    journal_title:Health research policy and systems

    pub_type: 杂志文章


    authors: Makkar SR,Turner T,Williamson A,Louviere J,Redman S,Haynes A,Green S,Brennan S

    更新日期:2016-01-14 00:00:00

  • Research dissemination workshops: observations and implications based on an experience in Burkina Faso.

    abstract:BACKGROUND:In Burkina Faso, malaria remains the primary cause of healthcare use, morbidity and child mortality. Therefore, efforts are needed to support the knowledge transfer and application of the results of numerous studies to better formulate and implement programs in the fight against the malaria pandemic. To this...

    journal_title:Health research policy and systems

    pub_type: 杂志文章


    authors: Mc Sween-Cadieux E,Dagenais C,Somé PA,Ridde V

    更新日期:2017-06-02 00:00:00

  • Indicators of sustainable capacity building for health research: analysis of four African case studies.

    abstract:BACKGROUND:Despite substantial investment in health capacity building in developing countries, evaluations of capacity building effectiveness are scarce. By analysing projects in Africa that had successfully built sustainable capacity, we aimed to identify evidence that could indicate that capacity building was likely ...

    journal_title:Health research policy and systems

    pub_type: 杂志文章


    authors: Bates I,Taegtmeyer M,Squire SB,Ansong D,Nhlema-Simwaka B,Baba A,Theobald S

    更新日期:2011-03-28 00:00:00

  • Impact of NIHR HTA Programme funded research on NICE clinical guidelines: a retrospective cohort.

    abstract:BACKGROUND:It is vitally important that there is a connection between health research and clinical practice. Indications as to the impact of the research on evidence-based practice and policy can be obtained by tracking the use of outputs of health research, especially its use in clinical guidelines (CGs). This study a...

    journal_title:Health research policy and systems

    pub_type: 杂志文章


    authors: Turner S,Bhurke S,Cook A

    更新日期:2015-08-22 00:00:00

  • An internal health systems research portfolio assessment of a low-income country research institution.

    abstract:BACKGROUND:In order to determine the type and amount of health systems research being conducted within ICDDR,B (also known as the Centre), a leading research institution in Bangladesh, an internal review of all on-going research protocols was conducted in September 2007. METHODS:A review of all ongoing research protoc...

    journal_title:Health research policy and systems

    pub_type: 杂志文章


    authors: Koehlmoos TP,Walker DG,Gazi R

    更新日期:2010-04-06 00:00:00

  • Health systems and policy research evidence in health policy making in Israel: what are researchers' practices in transferring knowledge to policy makers?

    abstract:BACKGROUND:Ensuring the use of research evidence in health system management and policy decisions is an important challenge in this century. Knowledge transfer and exchange (KTE) has emerged as a paradigm to address the challenges and start closing the 'know-do' gap. This area of work is gaining momentum in most develo...

    journal_title:Health research policy and systems

    pub_type: 杂志文章


    authors: Ellen ME,Lavis JN,Sharon A,Shemer J

    更新日期:2014-12-10 00:00:00