Ethics & Engagement across the Wellcome Trust Major Overseas Programmes

Engaging with Impact: How do we know if we have made a difference?


Engaging with Impact: How do we know if we have made a difference?

Monitoring and evaluation (M&E) is an area which many want to see strengthened.  When working with a diverse range of objectives, agendas, audiences and mechanisms for community and public engagement it can be a challenge to identify the most fitting approach. 
In this space we bring together reflections on some of the theory and lessons from the practical implementation of M&E in our work as well as useful evaluation resources. 

Members: 27
Latest Activity: May 12, 2016


Engaging with Impact Report

The report of the 'Engaging with Impact' Wellcome Trust workshop held in South Africa in October 2012 summarises discussions on the value and practical challenges of evaluating engagement activities. The meeting brought together engagement practitioners, social science researchers, scientific researchers and development practitioners.There are also brief descriptions of key evaluative/project planning frameworks and data collection tools.


Useful approaches for the evaluation of public engagement

A range of evaluation approaches that are suited to the challenge of evaluating public engagement are being gathered on the 'Resources' pages of the main e-MOPS site here. 'Outcome Mapping' and 'Most Significant Change' are two initial examples but others will be added over time.

‘Journal Club’ discussion papers and summaries

A number of papers relevant to the evaluation of public and community engagement are being discussed in a ‘Journal Club’ held every couple of months by a number of those in the ‘Engaging with Impact’ group. The papers and a summary of the discussion for each one can be found here.

Discussion Forum

Learning from community engagement around malaria elimination in Vanuatu and exploring the use of theory of change approaches for evaluation of complex health interventions

Started by Robin Vincent May 12, 2016. 0 Replies

Hi Everyone,The last two 'Journal club' discussion summaries have been shared on the evaluating with impact group.One discussion focused on a paper that aimed to understand the various factors behind…Continue

integrated indicators for engagement - from education to participation

Started by Robin Vincent Jul 1, 2015. 0 Replies

The latest Journal Club discussion looked at a paper covering the history of the ‘public participation in science' field, with an appendix of suggested indicators to help assess different aspects of…Continue

Revisiting the 'ladder of participation' for understanding engagement

Started by Robin Vincent Nov 10, 2014. 0 Replies

The latest Journal Club discussion looked at understandings of engagement in the light of a critique of Arnstein's influential 'ladder of participation' - which understood citizen involvement as…Continue

Sharing experience and learning on evaluation of engagement

Started by Robin Vincent. Last reply by Robin Vincent Sep 23, 2014. 1 Reply

Hi Everyone, I wanted to encourage everyone to share a little bit about themselves and their evaluation experiences and challenges with the rest of the group. This will make it easier for us to…Continue

Comment Wall

Comment by Sian Aggett on October 24, 2012 at 15:20

Wellcome to the group. Please feel free to start up discussions and share your resources on monitoring and evaluation here.

Comment by Dorcas Kamuya on July 12, 2013 at 11:53

Hi everyone,

Thanks Robin and Sian for putting together a very well written and informative report. I really enjoyed reading it, and particularly because monitoring and evaluation are areas that many of us - community engagement practitioners and researchers - think about, struggle with and would like to learn about and contribute to. Among other very useful sections of the report, I found the juxtaposition of 'stories from the field'  within key areas/themes of the report really very helpful in grounding the content, and the useful links are an incredible addition.

I have a couple of comments, and I will also contribute to Kombe's questions shortly. The idea of framework that can guide conception,  implementation, monitoring and evaluation of engagement project is very useful and reflective. I wondered how, where a CE project is already under implementation without proper prior planning (e.g. it may have started as quick response to a situation in the field which had potential to impede on-going research, and evolved over time to a major activity). In this case, various important steps may have been missed; what would be the starting point in evaluating such a project. The story of change in itself is important, but are there situations where the indicators, frameworks etc are conceptualized later in the project life; rather than at the beginning? Would the previous work then be termed part of formative work? Relatedly, can several frameworks be used on one project, and how does one make the choice, what needs to be considered?

What are some of the possible indicators for assessing impact of community engagement? Given impact is assessed on sustainable changes after a period of time since the project ended, for long-term CE that is integrated as a core activity of the research institution, can there ever be a time to  assess impact?

I am also interested in exploring evaluation as a program activity aimed at informing projects on progress and achievements; and evaluation as a research activity (and can M&E ever be a research activity?)

Thank you,


Comment by Robin Vincent on October 9, 2013 at 11:42

Hi Everyone,

For those of you who are interested in Utlilisation Focused Evaluation - which stresses the importance of engaging with those who are likely to make use of any particular evaluation (so it doesn't remain in a report on a shelf), a good new resource is now available as a pdf to download: http://mande.co.uk/2013/uncategorized/utilization-focused-evaluatio... 

Comment by Robin Vincent on May 15, 2014 at 13:37

The Monitoring and Evaluation News Website has lots of useful resources for the evaluation of complex multi-stakeholder projects. While it's focus is on international development, many of the methods highlighted are relevant to the evaluation of public and community engagement: www.mande.co.uk/

Comment by Robin Vincent on May 23, 2014 at 10:10

Outcome Mapping training in September, Dar Es Salaam

Hi Everyone, for those interesting in the Outcome Mapping method for evaluation of complex multi-stakeholder projects and based in East and Southern Africa, there is a training event coming up in Tanzania (the event is focused on development practitioners, but would be relevant to those involved in research programmes). Below are the details of the announcement from Simon Hearn of the OM learning community cut and pasted from the 'Pelican' evaluation email list:

Dear M&E colleagues, 

The Outcome Mapping Learning Community is hosting their third annual event in September in Dar es Salaam, Tanzania. The OM Lab 2014 is a three-day training and knowledge sharing event to explore the value Outcome Mapping can add to monitoring and evaluation in complex programmes.

It is for development practitioners, managers and funders and monitoring and evaluation professionals. It will offer you opportunities for learning about how OM can help better understand complex, developmental change and will give you advice and support for overcoming the measurement and management challenges you currently face in complex contexts.  

The event will include a range of sessions aimed at different skill levels, so whether you are completely new to OM or an experienced user, you will be able to build your knowledge, hone your skills and share your current challenges and innovations.  

Introductory and intermediate skill-building workshops, case study presentations, panel and plenary sessions involving leaders from the field and emerging practitioners alike will meet a range of capacity needs, as well as bring all participants together to discuss, challenge and plan ways forward on their OM-infused M&E approaches. 

More information and registration details here:http://outcomemapping.ca/resource/files/OM_Lab_2014.pdf  

Sign-up with no commitment before March 31st and receive 15% discount when registration opens in April.



Simon, on behalf of the OMLC Board of Stewards.


Simon Hearn
Research Fellow

Research and Policy in Development
Overseas Development Institute


Comment by Robin Vincent on June 12, 2014 at 16:31

Using stories for monitoring and evaluation - 'Most Significant Change' training in Berlin and London.

Hi Everyone, Berlin and London are not that accessible for many of you, but in case anyone is travelling during these times in July and August I thought it worth highlighting these trainings in MSC - one of the approaches introduced in the evaluation resources pages. This is pasted from the 'pelican' emails list, so apologies to those who already get it:

London and Berlin training on story approach to monitoring and 
evaluation‏ of projects

Do you value useful information on project impacts ? Interested in 
unexpected outcomes?  Do you want to hear how you have made a difference 
to people’s lives in their own words and what is important to them ?  
Need more than reporting on numbers and outputs ?

A 2 day course in the Most Significant Change (MSC) technique in Berlin 
(July 21/22) and London (July 31, Aug 1) will provide you with the 
knowledge and a plan to meet these needs.

MSC is a story based participatory monitoring & evaluation technique 
ideally suited to providing qualitative information on project /program 
impact.  MSC is also an excellent tool for encouraging stakeholder 
participation, surfacing stakeholder values, fostering organisational 
learning and program improvement. It is used widely in international 
development and social change projects.
This is a unique opportunity to have MSC training in Europe, from an 
experienced MSC professional who has delivered training in MSC over 30 

If you are interested in further information contact Theo Nabben at 
LondonMSC@outlook.com or visit http://socialimpactconsulting.com.au

Comment by Robin Vincent on September 4, 2014 at 14:17

Hi Everyone, here is a post from the RAMESES list highlighting a new resource on Realist Impact Evaluation from Gill Westhorpe:

Hi everyone

ODI have asked me to distribute this through evaluation networks.  Please feel free to tweet, forward to other networks, or whatever.


“This introduction has been developed to help evaluators and commissioners of evaluation to decide whether a realist impact evaluation is appropriate and feasible for a particular policy or programme.”

It does rehash introductory material about realist approaches, but it also addresses some more ‘applied’ questions and issues, such as

  • For what purposes is realist impact evaluation most appropriate?
  • In what circumstances is a realist approach more appropriate than other methods?
  • What circumstances make it easier to undertake a realist impact evaluation?
  • Understanding program theory and designing realist evaluations

The paper is available at either of the following:



Happy reading.


Dr Gill Westhorp


Community Matters Pty Ltd

Suite 1, 95 King William Rd

Unley  SA  5063

Ph ++61 8 7325 1801 (Direct)

Ph ++61 8 7325 1800 (Reception)

Mobile 0418 388 224


Comment by Robin Vincent on March 31, 2015 at 9:51

Hi Everyone,

I wanted to flag this upcoming webinar series on digital stories and community based public health put on by the Centre for Digital storytelling (and in case the links below don't work here is the overall url, http://storycenter.org/public-health-webinar-series ).

Please join the Center for Digital Storytelling for a new, five-part webinar seriesDigital Storytelling for Community-Based Public Health. Sign up now


Storytelling and participatory media are powerful, multi-dimensional tools for public health research, strategic communication, community mobilization, and advocacy.


For the past 20 years, the Center for Digital Storytelling has been supporting academics, educators, and advocates in understanding how first person narrative and digital media production can advance a broad range of public health and social justice goals.


Please join us for this five-part webinar series, with two-hour sessions offered every other month, from April through December. The series, which is offered at a discount off the individual webinar price, includes:


Session One: Wed. April 15, 10 am Pacific Time / 1 pm Eastern Time.

Digital Storytelling for Public Health: A Unique Workshop Model


Session Two: Wed. June 10, 10 am Pacific Time / 1 pm Eastern Time.

The Ethics of Digital Storytelling for Public Health


Session Three: Wed. Aug. 12, 10 am Pacific Time / 1 pm Eastern Time.

Digital Storytelling and the Sensitive Story: Potentials, Pitfalls, and Case Studies


Session Four: Wed. Oct. 14, 10 am Pacific Time / 1 pm Eastern Time.

Digital Storytelling and Public Health Research


Session Five: Wed. Dec. 9, 10 am Pacific Time / 1 pm Eastern Time.

Engaging Viewers: A Continuum of Strategies for Distributing Health-Related Stories



Note: This series of webinars is designed to familiarize people with our work and help them understand opportunities for collaboration; it is not a facilitator training program.

Add a Comment

You need to be a member of Engaging with Impact: How do we know if we have made a difference? to add comments!


© 2022   Created by Dina Rippon.   Powered by

Badges  |  Report an Issue  |  Terms of Service