Demystifying Impact Measurement, some notes from the session at #Locality16

Phil Tubla and Elly Townsend ran a session today at the Locality conference around Demystifying Impact, and Impact takes many guises, Environmental, Economical and Social and sometimes all of this in one project – but how do you measure it?

These are some notes and my thoughts from throughout the session:

The session started by asking for introductions from the room, and how groups are currently measuring Impact – and when someone else in the room, who turned out to be a client of ours introduces your app before you get the chance to, it would be rude to not join in the conversation.

Impact can be defined as the change, effects or benefits that result from the services or activity of your organisation, but you need to be aware when measuring and reporting Impact that you are on mission and avoid mission drift – don’t just report for funders, report on the things that matter you.

What are the consequences, intended and unintended of the work you do? It is important to understand the consequences of your work.

Impact measurement helps you to: plan, evaluate, promote and communicate.

Outputs vs Outcomes 

Outputs are immediate and short term – quantitative data, numbers counting “bums on seats”
Outcomes are the changes that occur as the result of the activity – these are the longer term effects of your projects and is more qualitative – it’s harder to turn this into data, into numbers.

There was a lot of feeling that funders look for short term reporting – and that demonstrating impact especially outcomes are difficult on these time frames, and the further you move from your intervention the harder the evidence is to find that the outcome is yours..

The stages of measuring impact are:

  • Inputs
  • Outputs
  • Outcomes
  • Impact

Link the stages together – tell your story!

– Or in our point of view, more importantly tell your client / users story! They are best placed to tell you the impact your project is having for them – and use their voice, that is way more powerful then just your data.

Data

The session then moved no with Gen Maitland Hudson discussing data, data collection, data triangulation – and open data, something we understand from our work earlier this year helping community groups use open data.

We shouldn’t be duplicating measurements when the data already exists – it just needs to be more freely available, and easier for the casual user, or grassroots community activist to understand.

There was then quite a long conversation around data, data collection, surveys and survey collection tools  – but I feel while data and data collection is great for number crunching, and measuring outputs it doesn’t tell the whole story – a survey might capture opinions, but doesn’t capture user experience. It doesn’t tell you the real difference you are making.

 

 

Using Evidence to inform policy and practice: notes from a session at #locality16

Some note from the Panellists ay the Locality Conference in York in 2016.

Richard Harries.  I know how hard it is for community groups to get their voice heard.

Emma Stone:  Every time we fund research it feels like a hatchling – we have no idea which one will survive to make a difference.  We spent 40k on some research on the impact of the social fund, it came at a good time and led to a £250m boost to the fund – which was then scrapped.

At JR Housing – we invested quite a lot in scoping what we could do on dementia by working with people who deal with it or have it.  Academic research was less important than what was being done to address dementia and address loneliness.  Across the country lost of people are having the same conversation.  So the research leads to informal collaborations.

Last example is a long term partnership with Loughborough University around minimum income standards, because it’s only over time that you build up a cumulative effect where people start to trust the relationship and the approach.  How do you keep research grounded in the real world?  Communications is important –  we have had to try and build up new conversations in terms of poverty.  A tool to help people see where thye sat on the scale crashed the BBC website.

We can solve poverty in the UK

We can solve poverty in the UK – click to download report.

Various form of evidence can work – including impact measurement used by the teams at the time.  They are very good at influencing at a local level – but to influence at a national levele you may need more detailed academic research

Work out the value of what you have found out to influence where you want to see change in your world.  (This is just what we work with our clients to do –  to capture the difference they are making and use that to influence the change they want to see)

The key is to be clear on the value. The inference you draw from lighter touch measurement may be different from more in depth research. Be open about what you think it means.

Communicate the finding of your work.

  • Understand what you need to record.
  • Identify the key people to communicate with.
  • Amplify your reach.
  • Invite people to see you at work.
  • Celebrate what you achieve.
  • Share what hasn’t worked so well.

You can also use a wider range of media. Techniques such as story telling can make a huge difference (the stories you can collect through the Impact APP

Click to see the toolkit

Click to see the toolkit

Dan Corry

Evidence needs to be strong. It helps if it’s independent, it also helps if it fits the direction the government is going in.

Mentioned Feinstein about evidence of investment in young people by showing how crucial early years were and amplified surestart.

National Minimum wage and national Living wage policies born out of evidence.  Other work in the 1980’s showed the impact of long term unemployment on people.  Further evidence that short term prison sentences are a waste of time – which included work from Downing St Delivery unit.  This couldn’t change policy because politicians believed it was at odds with what the public want.  Evidence  of the impact of sugar on obesity is slow to take hold because of the power of the sugar lobby.

We are also doing some work on social prescribing to show DR’s how loneliness is effecting health and that introducing people to local community groups will improve help. NSPCC changed it’s tactics by using evidence to influence professionals.

One thing that is crucial is cost effectiveness. Something may work, but if it is to expensive it wont be adopted.

What do we want from evidence

  • Helps us understand problems
  • Causal drivers
  • What are the options to solve problems
  • Robust methods for evaluating
  • We want things that will ‘help ministers achieve their  objectives’

I’ve always wanted to know the ‘right’ answer to the problem we’re tackling. That may not fit with what government or the public wants, but it helps you respond as a civil servant in the most effective way.

What do police makers need:

  • Don’t forget policy takes place in a political environment, shaped as much by that as evidence.
  • Public opinion means we shouldn’t expect an automatic hearing for evidence – so present it well, clear of waffle and jargon and at the right time
  • Ministers are more interested in evidence than you might think.
  • Change may take time so persevere.
  • If you want to influence you need to work through a wide range of challenges, engage with civil servants, build relationships, get a secondment to a department to understand their world.
  • Civil servants keen for new ideas, especially on cheaper ways to do things.
  • We need policy and systems which encourage experimentation and trial and error.

Don’t just engage with civil servants, talk to the ministers or other people advising them.  Civil servants alone may be to late in the policy making process – often decided when politicians are in opposition. So talk to them.   At the moment our present prime minister wasn’t in opposition or able to develop that before becoming PM – chance now to influence what and how.

Also mentioned Ministry of Justice data models which makes it easier for small organisations.

Dan Corry – caught in a trap, between public servants needing simpler information coming back fro the sector and the third sector looking to find fresh ways to measure their impact.

How do charities capture and measure Impact? Notes on the House Lords Charities Committee enquiry.

On Tuesday the House of Lords Charities Select Committee held the first of a set of of evidence collecting sessions for their enquiry into how charities measure impact. There were two sessions.  We watched and made some notes, mainly of the things that struck us as pertinent during the first, which revolved mainly around Impact.

Here they are.

Session one focussed on Impact, the panel was.

There were a few points made around measuring impact, that unsurprisingly we wholeheartedly agree with.

Paul Streets  thought it was unrealistic to expect their to be universal ways to measure impact.

To paraphrase

Every funder will need a different set of metrics which work on a different timescale.  As a funder we shouldn’t be asking for bespoke measurements… we need to find out what charities are measuring first.   If they’re good at it let them do it their way and if they’re not, maybe think twice about funding them.

This make huge sense to us.  Part of what we try and do is encourage charities or social enterprises to be proactive about how they measure and report impact.   We think that getting ahead of your commissioners and funders and being clear about what you do and how you measure that will ultimately helps funders.

What info should donors expect from Charities on the difference / Impact their funding has made?

Paul Streets:

We should start by looking at the organisation – have a conversation with them about how they will report outcomes, Tell us how you are listening to your beneficiaries – and how you’ll measure outcomes. If you have 50 beneficiaries and they are hard to reach that’s o.k just tell us how you’ll listen to them.

You have to measure the soft outcomes, as some projects the first outcome might just be getting someone to catch a bus.”

Dan Corry:

There is a a cost to administrate impact. There is a problem where some smaller charities do not wanted to push up admin costs or  they can’t afford it. What they need is head space to think it through. If you’re are going to collect soft data, that’s ok, but don’t just talk about the success stories.

Gen Maitland Hudson:

You shouldn’t just be collecting data, stories and feedback, you should be acting on it to make change, to make things better, there’s no point collecting it otherwise. Value the information.

Paul Streets;

As a funder we want honesty, if you have under performed tell us, we’ll still want to hear of the impact. As funders we need to allow charities to frame their own impact. Different charities have different Impact. some of it will be qualitative, some quantitative, some financial – but we need to understand this.

Gen Maitland Hudson;

We can help smaller charities measure Impact, we just need to find what tools can be provided – for collection and analysis – and it’s often the analysis that gets left out.”

The analysis side of story capture is what we recognise as the hard part. You can collect stories, but where do you store, them and how to you analyse them?  That’s one thing the impact app does. You collect the stories, it sorts, analyses and begins to to make sense of the content you’re collecting. Your stories becomes searchable, you can begin to spot trends, gaps in service and best practice. You can build case studies and evidence for funder’s old (and new). And we’re not the only ones to recongnise this strength .

The talk then moved on to comparison;

Can you compare the work of charities working in different circumstances?

Dan Corry; 

If you work in the same area – having the same metrics is useful, working together means you can work together, within reason, but you can’t compare different sectors. It’s apples and pears.

There’s been a danger that you try and work out which charity offers the greatest returns on investment and comparing across sectors doesn’t work

Gen Maitland Hudson;

What are you measuring for and what do you want your benchmark to do. Different projects need different benchmarks. It comes back to knowing what you want to use the data for.

We understand that comparing apples and pears isn’t realistic, but we also know that some funders may want to keep track of how they’re money affects people’s lives across a range  providers. The observer account makes it easy for a funder to collect and analyse live stories from a range of organisations they have funded.

The conversation continues;

Paul Streets;

Bench marking can work for funders, instead of for the charities How many people apply to you that don’t get a grant – if that number is high then are we wasting our time and theirs”

Dan Corry:

When you’ve been commissioned you need to remember that you are there to service your beneficiaries, not the funders.

The pay by results model has made charities think about measurement and then you have to think about value for money. PBR can pull charities away from their mission, but they have to know the contract is right. PBR can also make charities skim the easy cases for quick results.

Paul Streets;

Some PBR contracts can work, Meals on wheels, Domiciliary care etc, but contract commissioning can be highly destructive to other charities – as the shift from grants to contracts happen it’s highly unfair to the multiple disadvantaged.”

Cherry picking, or skimming the easy cases, is something some of our clients have recognised across their sectors.  We think that good use of impact measurement can help the best organisations show that they don’t do that.

It’s not simply where you’ve got a client to, but where they started from that important.  The best organisations may do dozens of things to help someone – and by routinely capturing the voice of the people you help we think that builds of the evidence of the myriad small things you do and why they matter

By consistently collecting, and by trusting your clients voice you can show the impact you’re making whether they are the easy to work with, or the hard to reach.

 

 

 

 

 

 

 

 

 

Compliments for the Impact Assessment App from the Royal Society for Public Health

One of our closest clients, Gateway Family Services,  won a couple of big awards last week. Huurah and congratulations.

richard-gatewayusingtheimpactappThey were the Royal Society for Public Health’s Three Year Health and Wellbeing Award and the Public Health Minister’s Award.

One of the things the judges identified was this Impact Assessment app, which we developed with guinea pig help from Gateway, who were among the first users.  They still use it in their daily work.

The judges said this of the way gateway understand what difference their services make..

…the strategy for the compilation of clients’ feedback is accessible and innovative, facilitating the capture of users’ comments and feelings about the service they are receiving.

Below is a video they made of just one workers use of the impact app in one week

https://youtu.be/nRBlv2OM-Kk

and Gateway also  blogged about how they use the app.

We take feedback every session, no matter how short the statement, or how negative the client might be feeling. Often the negative feelings are useful because in future sessions we can look back together and say ‘look how far you’ve come!’

“I find it really useful to be able to show people how the work they’re putting in is having an effect over time.”

As an organisation, the feedback every service gets from every client is extremely useful, as there is such a variety of data. Hearing from clients directly, rather than ticking boxes, means that we are getting a truer picture of the impact we are having. Recording clients in this way means that we often see issues and patterns that might not otherwise have shown up, which helps us to develop our services. It also makes it easier for us to present information about our work, and the impact it has, to others.

The Impact Assessment App allows you to capture what clients say in text (which can they be analysed)  and also video, audio or images.

 

 

Gateway Family Services – capturing impact in public health

Gateway is one of the earliest users of the Impact assessment app.  They have used it to track the progress of more than 8800 clients across 5 projects. Their work is varied.  From health trainers helping people lose weight, stop smoking and get fitter to their Pregnancy Outreach Service, which supports some of the most vulnerable pregnant women in Birmingham.

The app sits very well with Gateway’s approach, which is to recruit staff from the communities they are supporting and build a rapport with the people they help.   The app builds on that trust by providing a simple and friendly way to give their clients a voice.

So feedback can range from a casual “Yeah good ” or “I’m not too bad thanks, have a cold though

through to people sharing intense experiences…

am a drug addict, on a script but still using heroin, i am living in a hostel and am 23 weeks pregnant. i have had two children taken away and know this child will also be taken away, but would like to have some contact with this baby. I have metal health issues and receive the higher level of ESA, I would like support with bettering my situation to enable me to have contact with this child.

or

I disclosed my domestic violence incident and went to hospital for an x-ray on my jaw – the waiting time was too long so I walked out. Since then we split and … I am living in temporary accomodation and feel safe here, I have suffered from depression previously, but feel anxious as my hair is falling out when I brush it. I don’t have much support from family, friends or partner, I am now 21 weeks pregnant. I am grateful that you have come to see me.Thanks for the Tommy’s book to read. You filled out the booklet for me to claim Healthy Start Vouchers.

The focus is always on what the client wants to talk about, creating space for them to share and helping track their journey; their progress in their own words. The culture is to listen and then respond to what comes out.  And people have shared experiences more 25 000 times using the Impact Assessment app.

“It gives us a greater sense of what we are achieving every day.  It gives the managers, the board and out commissioners a much deeper insight into  what the staff and clients are facing and how they are making things better.”  says chief executive Katherine Hewitt.

The Impact Assessment App has helped Gateway:

  • Get all four key services recommissioned in the face of severe budget cuts, work worth more than £1million a year.
  • Develop iterative improvements to services through analysing what the clients say and then share that with partners.
  • Introduce more mobile and digital ways of working, which have replaced a line of filing cabinets several metres long and saved on administration costs.
  • Search the evidence they are collecting quickly and routinely.
  • Take a lead with commissioners on simpler more useful ways to capture and record their impact.
  • Pitch for new work and provide evidence for bid writing to demonstrate the work they do.
  • Run their annual reporting process, with stories told straight from the Impact Assessment App.
  • Tell a constant flow of weekly case studies to show commissioners, colleagues and partners what difference their services make, Stories like Daniel’s and Rumbudzai’s

The Impact App is also been used alongside other systems and is designed to help you cross reference with any system you’re using.

 

 

YMCA Crewe capturing the impact of their work with young people

The Foyer has used outdoor survival to help build confidence and resilience.

The Foyer has used outdoor survival to help build confidence and resilience.

Crewe YMCA is a Foyer that provides a home and support for young people.

Some of their work can be measured in specific qualifications and achievements. Other work is softer; just as important but harder to track.

The Foyer started using the Impact Assessment App to track a healthy conversations project begun with the Foyer Federation.

This not only supports young people with day to day choices they make about their health but also offerers challenges and new experiences. The Impact Assessment  app allowed them to capture what people were experiencing in routine ways but also on special occasions, such as the 24hr survival challenge.

From Project Working to routinely capturing progress.

YMCA_CreweJoel Lewis, from the Foyer, said “It works well for us and now we have moved it from just one project to starting to capture the progress of all the work we do with the people who live here.”

“Not only do we capture their voice, their direct experience, but we use it to make notes from the workers.  By using the hashtag #notes we can separate out what the young people are saying from what the staff are saying and analyse them separately.  It also allows you to work with people to show how they have progressed and lets you track progress through a range of different programmes.”

“The benefit  is being able to report on a almost infinite range of softer outcomes, the challenge is integrating into how we do our work on an hour by hour basis. “