How to use grey literature

A guide to evaluating and incorporating grey literature as an evidence base for your work

Evaluating grey literature

It is essential to always evaluate and assess the validity of the information before using it, whether in developing management plans, policy, hypothesis generation, literature reviews or meta-analyses, there is an especially important onus on the user to carefully evaluate grey literature.

Since grey literature has usually not been peer-reviewed (to the same extent as journal articles; though there might have been interagency review, which is important to note), the user, in some ways, needs to think like a reviewer and critically assess methodology and inferences, as well as the management and policy implications to ensure that there aren’t obvious biases or agendas. Further, finding and incorporating grey literature requires more time and effort than with traditional peer-reviewed articles1, but the payoff can be substantial.

Hopefully, with the advent of Applied Ecology Resources, finding applied ecology and conservation grey literature will be easier and repeatable.

Fortunately, there are methodologies to guide the evaluation of grey literature. For example, the AACODS checklist2 provides a great framework to guide a user’s assessment of grey literature. AACODS stands for Authority, Accuracy, Coverage, Objectivity, Date and Significance. The checklist lists a series of questions under each of these headings, and requires the user to assess aspects like whether the author is from a reputable organisation, whether the aims and methodology are clearly stated, and whether the date is included.

Examples of the questions a researcher should assess, according to the AACODS checklist include:

  1. Who produced the document? Is it from a reputable organisation?
  2. Does it cite other sources?
  3. When was it produced? How recent is the information?
  4. Does it clearly state transparent methodology?
  5. Does the work seem balanced in its presentation?

At the end of the day, the onus is on the user to assess and evaluate the validity of the grey literature material they wish to use in reviews or policy development. The organisation and the goals of the document should be clearly presented.

For example, a report purporting to dispel the myths of climate change alarmists and subsequent web searches fail to uncover who funds the publishing organisation should cast serious doubt on the validity of the material. A report that clearly states the goal of evaluating the effectiveness of different control measures to reduce invasive plant abundance from a government agency tasked with environmental management should be seen as credible.

To help users perform these checks, all documents archived on AER clearly list the source (who produced it), the date and users can also browse and filter by our member organisations.

Incorporating grey literature as an evidence base

Even when the AACODS checklist supports the use of grey literature articles, this material still should be handled a bit differently than peer-reviewed material. Whether evaluating grey literature for policy development, systematic review or meta-analysis, the following steps should be followed as best practice for incorporating grey literature into your work:

  • The methodology you employ should be clearly described. How grey literature was found (e.g. search terms, databases used, etc.) and the criteria used to appraise validity (e.g. AACODS checklist) need to be recorded.
  • You should include a supplemental file that lists the materials used. This includes not only bibliographic details, but also a DOI or other permanent link, and perhaps a scoring system (e.g. adding up AACODS questions) that quantifies reliability (e.g. a score of ‘5’ as very reliable and ‘1’ as questionable).
  • Make sure the resource is searchable and accessible. If grey literature was found through informal information networks, or the material obtained is not contained in a permanent repository, then the researcher should endeavour to make this material available, by working with the authoring agency or researchers to upload it to a searchable and free-to-access resource, like Applied Ecology Resources. This would generate a DOI or permanent link that then could be included in the bibliographic information.
  • Include sensitivity analysis. Results/interpretation should include sensitivity analysis where inferences are compared to information subsets without grey literature and grey literature only to evaluate how this material alters outcomes and interpretation.
  • Provide a qualitative reflection about the perceived value/concerns from your review of grey literature in the discussion section of your article/report. Much like how it is becoming standard practice for researchers reporting a review or meta-analyses to discuss possible biases that have emerged (e.g. geographical biases), the use of grey literature should similarly be reflected upon. Further, suggestions for improved discoverability or processing of grey literature could be included.

Read the full blog post: “Using grey literature as an evidence base for ecological research and practice” by Professor Marc Cadotte, Chair of Applied Ecology Resources.

References

1Adams, J., F. C. Hillier-Brown, H. J. Moore, A. A. Lake, V. Araujo-Soares, M. White & C. Summerbell (2016) Searching and synthesising ‘grey literature’and ‘grey information’in public health: critical reflections on three case studies. Systematic reviews, 5, 164.

2Tyndall, J. 2010. The AACODS checklist. Flinders University.