GRADE Guidance for Complex Social Interventions

This project, funded under the NCRM Methodological Research Projects funding call, brings together an international collaboration that aims to develop a methodology for rating the certainty in the estimates of effect of complex interventions.

Principal Investigator: Professor Paul Montgomery

International Co-Investigators: Dr Sean Grant – RAND Corporation, Santa Monica, USA, Dr Jane Dennis – Research Synthesis Ltd, Bristol, UK, Dr Erik von Elm – Institut Universitaire de Médecine Sociale et Preventive (IUMSP), Lausanne, Switzerland, Dr Eva Rehfuess – Institute of Medical Informatics, Biometry and Epidemiology Ludwig-Maximilians – University, Munich, Germany, Prof Geraldine Macdonald – University of Bristol, Bristol, UK, Dr Susan Norris – World Health Organization (WHO), Geneva, Switzerland

This project, funded under the NCRM Methodological Research Projects funding call, brings together an international collaboration that aims to develop a methodology for rating the certainty in the estimates of effect of complex interventions.

It is now standard practice for policy makers to draw on systematic reviews as the superior source of evidence to inform decision-making regarding effective interventions. Social interventions, commonly applied within the disciplines of international development, psychology, education, social work and welfare, criminology and public health, are frequently conceptualised as complex. This means they might involve a number of interacting components, multiple outcomes, levels of target (such as communities and entire populations) and diverse delivery formats. In addition, they might be more amenable to contextual factors and intervention implementation issues. For the research synthesis endeavours to be effective in guiding policy and practice, they must utilise adequate methodology that reflects the unique features of these interventions.

The most prominent system to guide evidence-informed decision-making has been developed by the Grading of Recommendations Assessment, Development and Evaluation Working Group (GRADE) in clinical medicine.

  • The GRADE approach provides a systematic and transparent process of rating the "best-available evidence" to inform recommendations for practice.
  • Having been endorsed by more than 80 organisations worldwide, including the Cochrane Collaboration and the World Health Organisation (WHO), the applicability of the GRADE approach outside of clinical practice has been questioned, in part because of differences of the evidence base in biomedical and social practice domains.  

  • Many researchers argue that GRADE does not adequately address the range of methods available for assessing the effectiveness of social interventions, such as different quasi-experimental study designs. In many cases this results in downgrading the quality of the “best evidence possible” for these interventions. 

Project aims

The project aims to:

  • advance and disseminate the GRADE guidance for complex social interventions
  • with the coordination of a Steering Committee of leading experts in the field, the project team will first organise an international online panel involving multidisciplinary expertise to identify aspects that need to be modified in the GRADE approach for complex interventions.
  • following the online expert panel, a consensus meeting will be hosted with a select group of participants to build consensus for the new GRADE guidance.
  • throughout all the phases of the project, the project team will closely collaborate with the GRADE Working Group to assure the project output follows the rigorous procedures followed by the group

Related publications

Montgomery, P., Movsisyan. A., Grant, S., Macdonald, G. & Rehfuess, E. (2019). Considerations of complexity in rating certainty of evidence in systematic reviews: a primer on using the GRADE approach in global health. BMJ Global Health. 4:e000848. http://dx.doi.org/10.1136/bmjgh-2018-000848 

Grant., S., et al (on behalf of the CONSORT-SPI Group, including Montgomery, P.). (2019). The CONSORT‐SPI 2018 Extension: A New Guideline for Reporting Social and Psychological Intervention Trials. Addiction. Vol. 114(1). p.p.4-8. https://onlinelibrary.wiley.com/doi/full/10.1111/add.14411

Movsisyan A, Dennis, J, Rehfuess E, Grant S, Montgomery P. Rating the quality of a body of evidence on the effectiveness of health and social interventions: A systematic review and mapping of evidence domains. Research Synthesis Methods. 2018 doi: 10.1002/jrsm.1290

Movsisyan A, Melendez-Torres GJ, Montgomery P. Outcomes in systematic reviews of complex interventions never reached "high" GRADE ratings when compared to those of simple interventions. J Clin Epidemiol. 2016;78:22-33. doi: 10.1016/j.jclinepi.2016.03.014

Movsisyan A, Melendez-Torres GJ, Montgomery P. Users identified challenges in applying GRADE to complex interventions and suggested an extension to GRADE. J Clin Epidemiol. 2016;70:191-9 . doi: 10.1016/j.jclinepi.2015.09.010

Movsisyan A, Melendez-Torres GJ, Montgomery P. A harmonised guidance is needed on how to “properly” frame review questions to make the best use of all available evidence in the assessment of effectiveness of complex interventions. J Clin Epidemiol. 2016;77:139-141. doi: 10.1016/j.jclinepi.2016.04.003