What stakeholders want from evaluation
Stakeholders are entitled to demand good evidence about whether their involvement in an intervention has been worthwhile. This is particularly true for funders who expect to be provided with information on whether and in what ways the intervention has worked and whether the investment they have put into it has proved value for money.
For many stakeholders, evidence means ‘numbers’ – the total number of participants attending a training programme; the proportion who completed the training; the percentage going on to get a better job. Numbers are equated with ‘quantification’ and quantification, for many reasons, has become synonymous with a particular evaluation approach – the ‘experimental’ method. Ask anyone who is interested in the results of an intervention what kind of evaluation they prefer and it’s likely they’ll ask for an experimental one. And given the choice, they’ll want a particular kind of experimental approach – one that involves randomised control trials (RCTs). RCTs are seen as the ‘gold standard’ of evaluation because they’re good at demonstrating attribution. They ‘prove’ that something causes something else to happen. Most RCTs are carried out inside laboratories or in clinical trials. A pharmaceutical company wants to bring to market a new wonder drug that reduces high blood pressure. It recruits a population of volunteers with similar histories of high blood pressure to test the drug. It measures their individual blood pressure. It randomly assigns the population to either a ‘treatment’ group or a ‘control’ group. The treatment group is given a daily dose of the new drug over a period of a month. The control group is given a placebo over the same period. At the end of the trial, the blood pressure of both groups is again measured and the results show that the average for the treatment group is significantly lower than for the control group. The trail has proved that the new drug reduces high blood pressure.
The problem with experimental evaluation
The problem is that this kind of experimental evaluation method is difficult – some would say impossible – to do in situations involving complex social interventions. This is because a number of factors often conspire to undermine the principle of ‘temporal priority’ on which the validity of RCTs depends. Temporal priority means making sure that the ‘cause’ you are interested in precedes the ‘effect’ you want to prove. But this is not so easy to demonstrate. There could be statistical bias in the treatment and control groups. More volunteers from the control group could withdraw from the treatment group than from the control group. The biggest problem is the effect of intervening variables. In complex social interventions, you are never quite sure whether the effect you are interested in has been caused by something completely external to the intervention.
The evaluation challenge in Carer+
Carer+ is a good example of this kind of complex intervention. The main evaluation objective for Carer+ was to assess whether and in what ways a training programme designed to improve the digital competences of care workers and informal carers ‘caused’ an improvement in the quality of care provided by these care workers and informal carers and, ultimately, an improvement in the quality of life of those they cared for. But this objective posed a number of challenges. There was no way the population of carers could be randomly assigned to a treatment and a control group. The carers who participated in the programme came from very diverse backgrounds. It was not possible to control for the effects of factors outside the training programme that might have an influence on training outcomes – for example the level of access participants had to social media. In short, the stringent conditions necessary for an experimental evaluation could not be applied.
The solution adopted for the Carer+ evaluation was to use a method called ‘Theory of Change’. In essence, Theory of Change substitutes the ‘cause-effect’ evidence chain required in the experimental method with evidence that demonstrates ‘causal pathways’. It uses ’triangulation’ – different kinds of evidence reflecting different stakeholders perspectives and acquired through different data collection methods – to build these causal pathways. Each item of evidence is compared with the other items to arrive at a balanced judgement on the likelihood that an identified effect is likely to be associated with a specific action that the intervention has carried out. Theory of Change requires a framework – an ‘intervention logic’ - to be constructed that shows a clear chain of linked steps between the ‘presenting problem’ an intervention wants to address, and the expected effect the intervention will have on that problem. The framework therefore shows the links between the intervention’s objectives, the activities carried out to achieve these objectives, the outputs that result from these activities, the short-term results (outcomes) associated with using these outputs and the longer term effects (impacts) of the intervention at the broader organisational or societal levels. However, Theory of Change has an important feature that distinguishes it from the conventional ‘logic models’ used in project planning. Whereas logic models are essentially descriptive devices for mapping project components and the relationships between them, Theory of Change models incorporate an explicit explanatory theory of how an intervention is expected to change things. They contain a set of hypotheses or assumptions about what factors cause a particular problem; how the intervention will change these factors and what kinds of changes will come about as a result of the intervention. These hypotheses and assumptions are then tested through the evaluation as the intervention proceeds, and may have to be adjusted – or discarded – if the evaluation evidence does not support them.
The overall Theory of Change for Carer+ is shown in the illustration below.
As the Figure shows, the Carer+ project can be seen as a ‘change journey’ that consists of a sequential progression of ‘step-changes’, each of which has an effect on subsequent steps. The journey begins with an initial ‘theory of change’ - the theory that care givers and care receivers currently lack the ICT knowledge and skills that could improve the quality of care they provide . Following on from this initial theory of change, each step in the change journey then has its own ‘theory of change’. For example, if it is true that care givers and recipients lack ICT knowledge and skills then it follows that a systematic mapping of these skills and knowledge gaps would highlight the improvements in competences that need to be supported (Step 2). The hypothesis for Step 3 is then - ‘Carers' competences can be improved with access to ICT devices and training to use them’, and for Step 4 - ‘If carers competences are improved through access to ICT devices and training, then this will lead to an improvement in the quality of care provided’. Finally, to complete the change journey, the theory of change for Step 5 is ‘An improvement in the quality of care leads to an improvement in the quality of life of care receivers’.
Using this model, the Carer+ evaluation used a range of methods – surveys, focus groups, statistical analysis of training programme participation measures – to collect data on whether and in what ways the assumptions and hypotheses embedded in the model could be substantiated.
Does Theory of Change work?
Did the approach work? Analysis of the evaluation data does seem to suggest clear evidence that Carer+ had a positive impact for both carers and care recipients. For carers, Carer+ contributed to improving communications with care managers, administrators and health and social services providers; increased efficiencies in work organisation and task management; adding value to the quality of care provided and improved social and emotional relationships with clients and their families. For care receivers, Carer+ helped to increase independence, improve some aspects of quality of life and forge a closer bond and relationship with the care provider. It also seems clear that Theory of Change proved a valuable tool in helping the project’s evaluators to extract credible and robust evidence of impact in a complex and difficult terrain.
See the presentation by Dr Cullen on the CARER+ Toolkit for developing the Digital Competences of Carers:
By Dr Joe Cullen, Arcola Research, UK