Monthly Archives: November 2016

Demystifying Impact Measurement, some notes from the session at #Locality16

Phil Tubla and Elly Townsend ran a session today at the Locality conference around Demystifying Impact, and Impact takes many guises, Environmental, Economical and Social and sometimes all of this in one project – but how do you measure it?

These are some notes and my thoughts from throughout the session:

The session started by asking for introductions from the room, and how groups are currently measuring Impact – and when someone else in the room, who turned out to be a client of ours introduces your app before you get the chance to, it would be rude to not join in the conversation.

Impact can be defined as the change, effects or benefits that result from the services or activity of your organisation, but you need to be aware when measuring and reporting Impact that you are on mission and avoid mission drift – don’t just report for funders, report on the things that matter you.

What are the consequences, intended and unintended of the work you do? It is important to understand the consequences of your work.

Impact measurement helps you to: plan, evaluate, promote and communicate.

Outputs vs Outcomes 

Outputs are immediate and short term – quantitative data, numbers counting “bums on seats”
Outcomes are the changes that occur as the result of the activity – these are the longer term effects of your projects and is more qualitative – it’s harder to turn this into data, into numbers.

There was a lot of feeling that funders look for short term reporting – and that demonstrating impact especially outcomes are difficult on these time frames, and the further you move from your intervention the harder the evidence is to find that the outcome is yours..

The stages of measuring impact are:

  • Inputs
  • Outputs
  • Outcomes
  • Impact

Link the stages together – tell your story!

– Or in our point of view, more importantly tell your client / users story! They are best placed to tell you the impact your project is having for them – and use their voice, that is way more powerful then just your data.

Data

The session then moved no with Gen Maitland Hudson discussing data, data collection, data triangulation – and open data, something we understand from our work earlier this year helping community groups use open data.

We shouldn’t be duplicating measurements when the data already exists – it just needs to be more freely available, and easier for the casual user, or grassroots community activist to understand.

There was then quite a long conversation around data, data collection, surveys and survey collection tools  – but I feel while data and data collection is great for number crunching, and measuring outputs it doesn’t tell the whole story – a survey might capture opinions, but doesn’t capture user experience. It doesn’t tell you the real difference you are making.

 

 

Using Evidence to inform policy and practice: notes from a session at #locality16

Some note from the Panellists ay the Locality Conference in York in 2016.

Richard Harries.  I know how hard it is for community groups to get their voice heard.

Emma Stone:  Every time we fund research it feels like a hatchling – we have no idea which one will survive to make a difference.  We spent 40k on some research on the impact of the social fund, it came at a good time and led to a £250m boost to the fund – which was then scrapped.

At JR Housing – we invested quite a lot in scoping what we could do on dementia by working with people who deal with it or have it.  Academic research was less important than what was being done to address dementia and address loneliness.  Across the country lost of people are having the same conversation.  So the research leads to informal collaborations.

Last example is a long term partnership with Loughborough University around minimum income standards, because it’s only over time that you build up a cumulative effect where people start to trust the relationship and the approach.  How do you keep research grounded in the real world?  Communications is important –  we have had to try and build up new conversations in terms of poverty.  A tool to help people see where thye sat on the scale crashed the BBC website.

We can solve poverty in the UK

We can solve poverty in the UK – click to download report.

Various form of evidence can work – including impact measurement used by the teams at the time.  They are very good at influencing at a local level – but to influence at a national levele you may need more detailed academic research

Work out the value of what you have found out to influence where you want to see change in your world.  (This is just what we work with our clients to do –  to capture the difference they are making and use that to influence the change they want to see)

The key is to be clear on the value. The inference you draw from lighter touch measurement may be different from more in depth research. Be open about what you think it means.

Communicate the finding of your work.

  • Understand what you need to record.
  • Identify the key people to communicate with.
  • Amplify your reach.
  • Invite people to see you at work.
  • Celebrate what you achieve.
  • Share what hasn’t worked so well.

You can also use a wider range of media. Techniques such as story telling can make a huge difference (the stories you can collect through the Impact APP

Click to see the toolkit

Click to see the toolkit

Dan Corry

Evidence needs to be strong. It helps if it’s independent, it also helps if it fits the direction the government is going in.

Mentioned Feinstein about evidence of investment in young people by showing how crucial early years were and amplified surestart.

National Minimum wage and national Living wage policies born out of evidence.  Other work in the 1980’s showed the impact of long term unemployment on people.  Further evidence that short term prison sentences are a waste of time – which included work from Downing St Delivery unit.  This couldn’t change policy because politicians believed it was at odds with what the public want.  Evidence  of the impact of sugar on obesity is slow to take hold because of the power of the sugar lobby.

We are also doing some work on social prescribing to show DR’s how loneliness is effecting health and that introducing people to local community groups will improve help. NSPCC changed it’s tactics by using evidence to influence professionals.

One thing that is crucial is cost effectiveness. Something may work, but if it is to expensive it wont be adopted.

What do we want from evidence

  • Helps us understand problems
  • Causal drivers
  • What are the options to solve problems
  • Robust methods for evaluating
  • We want things that will ‘help ministers achieve their  objectives’

I’ve always wanted to know the ‘right’ answer to the problem we’re tackling. That may not fit with what government or the public wants, but it helps you respond as a civil servant in the most effective way.

What do police makers need:

  • Don’t forget policy takes place in a political environment, shaped as much by that as evidence.
  • Public opinion means we shouldn’t expect an automatic hearing for evidence – so present it well, clear of waffle and jargon and at the right time
  • Ministers are more interested in evidence than you might think.
  • Change may take time so persevere.
  • If you want to influence you need to work through a wide range of challenges, engage with civil servants, build relationships, get a secondment to a department to understand their world.
  • Civil servants keen for new ideas, especially on cheaper ways to do things.
  • We need policy and systems which encourage experimentation and trial and error.

Don’t just engage with civil servants, talk to the ministers or other people advising them.  Civil servants alone may be to late in the policy making process – often decided when politicians are in opposition. So talk to them.   At the moment our present prime minister wasn’t in opposition or able to develop that before becoming PM – chance now to influence what and how.

Also mentioned Ministry of Justice data models which makes it easier for small organisations.

Dan Corry – caught in a trap, between public servants needing simpler information coming back fro the sector and the third sector looking to find fresh ways to measure their impact.