Ethics & Engagement across the Wellcome Trust Major Overseas Programmes
Resources list on 'realist' evaluation
Realist evaluation is a valuable method for addressing the complexity of social programmes. It seeks to understand ‘what works, for who, in what circumstances and why’ rather than providing one-off verdicts on programme success or failure (Pawson and Tilley 2007).
Realist evaluation seeks to clarify the ‘programme theories’ behind the way change is expected to happen in an intervention and uses a range of evidence to test whether these theories accurately account for the process of change. It is avowedly multi-method and draws on quantitative, qualitative, comparative and narrative evidence, as well as grey literature and the insights of program staff. In each case it looks to illuminate and sharpen the understanding of how change happened in the programme and why the programme works in some circumstances and not others.
Realist evaluators consider social interventions always to be “complex systems thrust amidst complex systems” (Pawson 2006: 25). It is not programmes that ‘work’, but rather the way people respond to the resources, ideas and practices that programmes introduce (programme ‘mechanisms’ in realist jargon) that creates programme outcomes. In this way, a programme will work differently for different people, and there is no expectation that any particular intervention will be the ‘same’ as it plays out in different contexts.
At the same time, realist evaluation seeks to build a cumulative understanding of how programmes play out differently in different contexts, to produce patterns of outcomes. This picture can then usefully inform policy and subsequent action and programming in any new context. ‘Realist synthesis’ brings the same principles to the systematic review of research and evaluation studies.
A Brief introduction to Realist Evaluation (2 pages) by Dr Gill Westhorp. Introduction to Realist evaluation, what makes it distinctive, and key aspects of the approach.
Realist Impact evaluation: an introduction (12 pages) Dr Gill Westhorpe.
More detailed but accessible introduction to Realist Evaluation highlights distinctive features of the approach, key concepts, when and where the approach is most appropriate, and provides a range of illustrative examples.
Introduction to Realist Evaluation and Realist synthesis - Powerpoint presentation (29 slides) by Ray Pawson, 2006. The first half of the presentation focuses on Realist evaluation, outlines basic principles and provides practical examples. The second half covers the related Realist approach to systematic review, ‘Realist synthesis’ and how to draw out cumulative learning from evaluations.
Realistic Evaluation: an overview (13 pages) conference presentation introduction by Nick Tilley that is more about the rationale of the approach rather than the practicalities. Highlights the roots of the approach in the philosophies of Popper and Campbell and their concerns to see gradual beneficial social change. Gives a critique of the simplistic application of traditional experimental approaches to social programmes and shows how realistic evaluation addresses the complex causality in social processes and programmes.
Please note the four parts to the introduction largely repeat the 'short overview' video, with additional detail, argument and examples. Part 4 also includes some additional material on scientific method.
Realist evaluation short overview (14 minutes). Ray Pawson introduces the basics of Realist evaluation. https://vimeo.com/84215487
Realist evaluation – introduction Part 1 (7 minutes) introduces the key elements of realist evaluation. https://vimeo.com/84216696
Realist evaluation – introduction Part 2 (9 minutes) looks at how realist evaluation helps to address complexity in social programmes. http://vimeo.com/84386879
Realist evaluation – introduction Part 3 (8 minutes) looks at how the method makes use of evidence and the value of a multi-method approach. http://vimeo.com/84227829
Realist evaluation – introduction Part 4 (7.5 minutes) looks at how evaluation can better contribute to cumulative learning and employ ‘scientific’ methods. http://vimeo.com/84228311
Realistic Evaluation (London: Sage,1997). Influential introductory text written by Ray Pawson and Nick Tilley. Gives a critique of existing evaluation practice. Covers the basic principles and approach of realist evaluation, including for realist interviews, and outlines the potential for evaluation to support cumulative learning.
Evidence-Based Policy: a realistic perspective, (London: Sage, Ray Pawson, 2006) Looks at the companion approach ‘Realist synthesis’, which uses realist principles to synthesise evidence to inform policy. Provides a critique of dominant approaches to systematic review and meta-analysis for the way they tend to average out and ultimately obscure the differential effects of context. Includes a range of illustrative examples of how evidence can be gathered and synthesised to make context sensitive recommendations (such as schemes for youth mentoring and crime prevention).
The Science of Evaluation: a realist manifesto, (London: Sage, Ray Pawson 2013).
Makes the case for a ‘science’ of evaluation that can build a coherent body of learning and evidence on social programmes. Summarises key principles of ‘critical realism’ that underpin the method. Includes a critical review of a range of recent published realist evaluations to sharpen the methodological debate and improve practice (also available as a paper – see below ‘A Realist Diagnostic workshop’). Includes a detailed account of an application of the method to inform policy on banning smoking in cars carrying children (also available as a paper – see below Wong et al)
Realist Evaluation (2004) (Chapter - 36 pages) Ray Pawson and Nick Tilley
On-line Community and collections of resources
Email Discussion list on Realist evaluation and Realist synthesis, including archived discussions
On-line Dossier: KIT Royal Tropical Institute Netherlands Dossier on ‘Realist Enquiry’, includes a short introduction and a range of published resources and examples of applying realist evaluation: http://www.kit.nl/-/FAB/51/300.html
The realist hive
Theory-driven inquiry for health systems research
Examples and Case studies
How Do You Modernize a Health Service: a Realist Evaluation of Whole-Scale Transformation in London The Milbank Quarterly, Vol. 87, No. 2, 2009 (pp. 391–416) Paper by Trisha Greenhalgh about applying realist evaluation to understand a range of factors supporting change across four large health care organisations and their delivery of three areas of health services. Also reviews some of the challenges of applying realist methods in this early case study. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2881448/
Policy guidance on threats to legislative interventions in public health: a realist synthesis. (Wong, G., Pawson, R., and Owen, L, in BMC Public Health 2011, 11: 222). This paper explores a detailed ‘case study’ of legislation to ban smoking in cars carrying children, and how the authors drew on a range of different bodies of relevant evidence to assess the likelihood such legislation would be feasible and likely to succeed. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3079647/
‘A Realist Diagnostic Workshop’ paper by Ray Pawson and Ana Manzano-Santaella looking critically at a range of applications of Realist Evaluation to help clarify key elements of the approach (Evaluation 2012, 18: 176): http://evi.sagepub.com/content/18/2/176
These introductory resources have been developed by Robin Vincent as part of supporting the Wellcome Trust linked community of practice on evaluation of public and community engagement.