How empty are empty reviews?

This study assessed the prevalence of ‘empty reviews’ in the Cochrane Library as a first step towards creating specific guidance for reporting these reviews. 

Principal Investigator: Professor Paul Montgomery

The Cochrane Library  is the world’s largest library of systematic reviews (which synthesise all available evidence about a topic or intervention). Cochrane publishes high-quality reviews and makes them easy to understand and freely available, so that health and social care providers, consumers, researchers and policymakers can make evidence-informed decisions. Cochrane reviews are particularly vital for people who are not researchers, as they are offered with ‘plain-language summaries’ so that even complex studies can be easily understood.

Systematic reviews which find no studies eligible for inclusion are commonly known as ‘empty reviews’. A review might find no relevant studies if it is focused on a highly specific question or uses overly stringent inclusion criteria to find higher-quality evidence. However, a systematic review might also be ‘empty’ because it is looking at an area of study which is very new or has not been sufficiently studied.

Why empty reviews matter

Empty reviews are important, as they tell us who is undertaking a review and thus interested in the topic, can highlight major research gaps and indicate the state of the evidence at a point in time. They can also justify further research and/or funding and even highlight potential harms of an intervention.

Empty reviews can be especially problematic for clinicians and other decision-makers, as it is not clear what to conclude from them. They may offer no conclusions; conclusions based on published evidence that has not been assessed for quality; or conclusions that are not based on evidence. An empty review might simply confuse readers, may be disregarded as irrelevant, or dangerously misinterpreted as evidence that there is, in fact, no evidence that an intervention works.

Findings

This study assessed the prevalence of ‘empty reviews’ in the Cochrane Library as a first step towards creating specific guidance for reporting these reviews. It found that, as of August 2010, nearly 9% of Cochrane reviews were ‘empty reviews’. These were reported in different ways, with study authors as well as Cochrane editors taking different approaches to reporting. The study found that the number of ‘empty reviews’ is increasing, so it is imperative to develop clear and consistent reporting guidelines to ensure that Cochrane reviews remain an important source of knowledge for policy and practice. Read more at http://empty-reviews.org/.

Funded by

UK Cochrane Centre

Related publications

Yaffe, Joanne, Montgomery, Paul, Hopewell, Sally & Shepherd, Lindsay (2012) "Empty Reviews: A Description and Consideration of Cochrane Systematic Reviews with No Included Studies", PLoS ONE  7: 5 e36626. doi: 10.1371/journal.pone.0036626. URL: journals.plos.org/plosone/article.