This guide aims to bust the jargon around measuring the changes your work makes
Change is complicated – there is an industry built around how to properly measure it! Fortunately, there are some basic ways to measure change and show that what you do works. This guide aims to de-mystify evaluation and introduce some basic steps to take in order to show the difference you are making, satisfy funders and meet your organisational needs.
This guide is for organisations or projects that do not have the resources to conduct in-depth social value evaluations but who want to embed change measurement into their work to show the difference they make.
FORGET THE JARGON! – the key to measuring change is to simply speak to your service users about the changes you want to help them make in their lives. They will tell you what has changed – you just need to find right way to record this
1 Why measure change?
In the charity sector impact is everything!
Funding: External funders want to know that the money they are investing in your services are making the changes you say they are. It is crucial that you can show the difference it has made to how many people. In short you need to prove it’s a good investment.
Learn and Improve: measuring success can show you what’s working well and what isn’t working – you mind find you aren’t achieving what you set out to do – this gives you an opportunity to change and improve the way you do things. You may also find out that you are achieving some unexpected changes – this may help inform your future delivery
Keeping the board on side: Showing the board you are delivering stuff that is proven to work can influence them to invest and be more supportive
Employee/volunteer satisfaction: being able to communicate clear statistics and figures that evidence success, your staff are more likely to feel proud of their achievements and really understand the difference they are making
Funders want to be confident that their investment is going to make a difference. To inform their investment decision they need to know:
- What you are doing
- Why you are doing it
- Who you are doing it to
- What difference you are making
The two most important elements of measuring change that they need to see are:
- NUMBERS, ie OUTPUTS: the activities you are delivering and the number of people participating.
- CHANGES, ie OUTCOMES: the difference are you making to your service users as result of this activity.
More about outcomes…. put simply an outcome is just another word for change
Outcomes are changes that happen as a result of your work (activity).
To understand outcomes you need to clearly understand how your service users (sometimes referred to as beneficiaries) are affected by your work/projects/activities.
You deliver a support activity > outcome = improved wellbeing
You deliver rehabilitation activity > outcome = increased independence
2 How to measure outcomes (change)
In order to measure outcomes you need to find something that tells you a change has happened – in its simplest form this is done by…..ASKING THE PARTICIPANTS 🙂
The hardest part is deciding what ask to show the change you have made and how you are going to ask it.
Ways to ask
If you identify the difference you intend to make there will be a tool to measure that difference (see links to more information about the different tools available). These include surveys, focus groups, informal and formal observations, 1:1 interviews, activity registers.
Questionnaires: pre and post intervention surveys are great for showing how much things have changed –statistics are powerful – by using a scale you can show how people feel they themselves have changed. Remember to keep your questionnaire simple and focussed make sure every question is going to tell you something about the difference you are making.
Focus groups: These are small groups of service users who take part in a discussion (led by you or someone independent) about what has happened as a result of your work. They are a great way to get in-depth information about what has happened and are often a way of getting good case studies to show case your work.
It is always useful to offer refreshments and travel expenses to encourage people to take part! The best approach is to have some structure to the discussion – go through a list of themes that relate to the changes you aimed to make. This will help you to keep the session focussed. It’s a good idea to record the session so you can evidence the changes properly – just remember to let the participants know they are being recorded
1:1 discussion: Having 1:1 discussions with service users can be useful was to find out personal stories of change – what has happened to people since they started participating. Another great way to get case studies and to know the real impact or your work. Just be aware that sometimes people tend to tell you what you want to hear – they don’t want to offend you by telling you it didn’t work.
Observations: Observations of change noticed by friends and family or other professionals can be recorded and used to show change – these would usually be part of the story and would be more powerful if backed up by change reported by the service user themselves
Activity Registers: These show clear statistics i.e. number of people participating (outputs) – they can be an important tool in evidencing change, for example, 40 people attended exercise classes every week for 3 months – 75% reported an improvement in well-being. The numbers really help to evidence the impact of your work.
If you use a combination of approaches you will have a powerful story to tell that includes statistics, case studies and quotes.
What to ask
You need to ask your service users the right questions. The questions need to tell you if things you set out to change have done. For example, if you deliver a coffee morning that aims to achieve the outcome of ‘Improving emotional wellbeing through the development of peer networks’ you may ask participants to agree or disagree on a scale of 1 – 5 (where 1 = strongly disagree and 5 = strongly agree) their feelings about the following statements:
- I get out of the house regularly
- I have a good network of friends
- I feel confident to meet other people
- I feel connected to my community
The statements are the indicators (the way you are showing the change that is happening). Indicators are what you use to measure whether you have achieved your outcome.
By comparing pre and post survey results you can clearly demonstrate change using statstics, eg 25% of participants felt more connected to the community after participating in the coffee morning for three months. The outcome of this would be reduced social isolation and improved wellbeing.
Our members have developed a number of indicators. These are statements that can be assessed with a scale and relate to the following generic outcome areas for Visionary members:
- Increasing independence
- Improving confidence (see Henshaws example on the right hand column of this page)
- Enhancing wellbeing (see Henshaws example on the right hand column of this page)
- Reducing social isolation
- Increasing knowledge about support available
It is always useful to add an open ended question on a questionnaire – this gives people the opportunity to let people tell you what they think – by asking ‘is there anything else you would like to tell us about the changes you have experienced as a result of this work’ may mean you get to learn more and even find out that you have achieved something you did not expect to. Another great question to ask someone is ‘What would have happened if you hadn’t taken part in our activity?’ – this can often lead to powerful stories about the difference you have made.
Using a basic Survey Monkey package is free – it’s easy to use and also provides an analysis of the data.
Useful information about designing questionnaires can be found in “The 10 steps guide to questionnaire design” on the RCU website and in the “Information collection methods. Choosing tools for assessing impact” guide from the Charities Evaluation Services.
3 Explaining change
The ‘theory of change’ and ‘logic models’ are just ways of explaining change. It’s the story of how what you do creates changes for the people you are working with. These models can be useful to help you focus and think about how to measure outcomes. Simply put this is about:
- What you Invest = inputs (staff time, resources);
- How you do it = outputs (the activities & who is participating)
- The results of you work = outcomes and impact (short and medium term results and longer term impact)
There are 5 key starter questions to ask yourself when you want to explain how change is going to happen
- What’s the problem you’re trying to tackle? – VI people are struggle to leave their homes and are excluded from social activity in their communities.
- What are the changes your organisation aims to bring about? – to increase independence, to improve confidence, to reduce loneliness; to improve wellbeing.
- How does it do this? – social groups, tech support, virtual support groups, delivering orientation skills.
- What are your specific goals? – To enable VI people to gain the necessary skills and confidence to travel to activities; To empower VI people to choose to join new social groups; To equip people with the skills to connect with others online; To develop a strong VI peer support network.
- How will you know if you’ve achieved them? – recording numbers of people attending sessions, recording retention rates asking the people themselves what has changed, asking other stakeholders what has changed – surveys, focus groups, 1:1 discussions, observations.
Use a basic template to explain what changes you expect to make and how you are going to know change is happening
Using the coffee morning example we can complete this template
|Provision of a room
|Weekly coffee morning for visually impaired people (peer support group)||Number of coffee mornings/year
Number of people/week attending
|VI people feel less socially isolated
VI people experience improved mental wellbeing
VI people are more independent
|Number of people reporting having a good network of friends
Number of people reporting that they feel more connected
Number of people reporting that they get out of the house regularly
Further information regarding theory of change and logic models including blank templates can be found here.
4 Recording and analysing data
It’s important to record what you are doing so that you have the evidence to prove what you have achieved.
In the example above you would need to keep a record the sessions held, the number attending and the number of attendances. Ideally these should be input onto a database, this does not have to be an expensive one – a basic Excel spreadsheet will enable you to produce output data at the touch of a button.
A questionnaire or survey will enable you to record and analyse data in the form of numbers (quantitative data). This defines change.
The results of the survey can be input on to a spreadsheet or data base so that you produce statistics for each question. For example, the percentage of participants who feel socially isolated at before the intervention compared with the percentage of participants who feel social isolated after a set period of time. In order to measure change you need to record baseline data then repeat using the same questions in a few weeks or months time. If you use Survey Monkey, responses can be input directly onto a tablet which saves time inputting manually onto a spreadsheet. You may need a member of staff/volunteer to support with completing surveys – this can be don either on the phone or face to face. Survey monkey is also a great quantitative data tool as it will present the results in graphs for you.
Focus group, discussions and observation all provide data that describes the changes that happen (qualitative data). They give the detail and the story behind the changes. Ideally you should transcribe focus groups so that you can analyse the discussions and draw out key themes. Key quotes and stories can be put together in a way that relate to the themes and describes the changes. Case studies can also be used to describe how your work has made a change/changes to someone – one case study may show a number of themes or outcomes. Qualitative data is a powerful and rich way of demonstrating changes – it bring the story to life and promotes and in-depth understanding of what you are doing.
Qualitative data describes whereas quantitative data defines – using both methods in combination is a powerful way to measure and tell your story of change.
Develop a plan for how you’ll review the findings and how you’ll work out what action to take. Think about what you will do with any challenging findings – for example, if some service users tell you a service isn’t actually as good as you think. As much as possible, treat it as an opportunity to learn and improve, so that difficult or negative findings are actually an opportunity, not a problem.
Who is your audience? why do they want the information? how they will use it?
Families/Friends/Carers: may want to know about the type of things you do and some stories the difference you have made to people you have supported – they will want to feel they can trust you to support their person and may be a potential future donor – the information you provide should reflect that.
Professionals: May want to know whether to refer into your organisations, whether to recommend the service to other or whether to collaborate with you. This is a PR exercise – they need to know what you are doing, why you do it and the changes you can evidence. Provide headline statistics and evidence supported by a few good quotes. They may influence future funding opportunities.
Funders: They want to know if your work will help them achieve their aims – are you a good investment for them? Tailor your information to the outcomes they want – focus on their criteria – tell them how you can help them
Staff/Volunteers: They want to know what the whole team is achieving and that their efforts are making a difference. Number of people they have supported, quotes of how good the service is. This can be used as a thank you to staff and to foster team spirit – they have a clear aim and know how they are achieving it.
Board of trustees: They want to know how you are meeting the strategic aims and objectives of the organisation are you making the difference that you say you are for visually impaired people. They need evidence of the difference you are making alongside the investment in each ear of work.