A study of predicting welfare scroungers and providing targeted support to vulnerable people

“Test predicts which children will grow up to be drain on society – when they are just three years old” screamed the headline in the Telegraph this week. Posting the article on Facebook sparked a flurry of fascinating discussion, ranging from eugenics and normalisation through to private education and Philip Larkin. Some of it rather took me by surprise…

Whereas I had, perhaps naively, assumed that the knowledge of how we could better target resources on preventative interventions to those who most need support, others saw this as a sinister development.

I realised from the outset that the Telegraph had, as is their way, taken a particularly unpleasant slant on the research to blame these people in a barely disguised throwback to the Victorian notion of the ‘undeserving poor’. How else can one explain opinions (rather misleadingly presented as ‘news’) like this:

“A simple test at the age of three can determine whether children will grow up to be a burden on society, needing excessive welfare, ending up in jail or becoming obese.”

Hmmmmm

I had not really considered that this evidence, in the wrong hands (or in the ‘wrong minds’) might provide the justification for writing off children as young as 3 as criminals. Nor had I envisaged how this research might be used to ostracise young children who fail to conform to socially prescribed notions of what is ‘normal’. The authors of the research (and not the Telegraph article) though clearly had, referencing the “…warnings (that) are issued about the myth of early-childhood determinism…”

While, as many pointed out, it is not a new idea to suggest that you can accurately predict future outcomes by looking at the characteristics or test results of young children. The Stanford Marshmallow Experiment (my particular favourite) and a growing body of evidence on Adverse Childhood Experiences (ACEs) support the social determinants model of health inequality.

The social determinants of health are, according to the World Health Organisation the circumstances in which people are born, grow up, live, work and age, and the systems put in place to deal with illness. These circumstances are in turn shaped by a wider set of forces: economics, social policies, and politics.”

 What the WHO doesn’t tell you is how political perspectives can take you from evidence of childhood experience and life outcomes to characterising children as deviant scroungers.

Reporting the same research, the BBC felt it appropriate to point out that “the researchers stress that children’s outcomes are not set at the age of three.”

Now this post was not intended to be about epistemology and the search for knowledge and truth – I’ll leave that to Karl Popper. Nor was it supposed to be about journalistic standards and the way facts are distorted by shoddy hacks and ideologues. It was supposed to be about the way that evidence can enable a more targeted and insight-led deployment of resources. If we are able to identify people that might need support or be at risk in the future then surely investment in preventative interventions is a good idea.

What I have learned is that the values and views of the world which underpin our approach are crucial. If you start with the view that people are inherently good and that with support they can achieve great things the way you might interpret these research findings will completely be completely different to those who see the poor and vulnerable as ‘broken’. Jude Habib has eloquently explained why we need to challenge this ‘deficit language’ and start adopting a strength-based approach which recognises and nurtures people’s talents and capabilities.

Where some see evidence that can inform a sensible way to target support to those who need it, others see it as a reason to write people off. In my work at Social Engine we advocate an evidence-based and insight-led approach but this discussion has reminded me that data are not and cannot be neutral, passive and objective when seen through the eyes of an individual.

 

For the epistemologists among you, the research was published in the journal Nature Human Behaviour and can be found here.

Advertisements

When did it become trendy to start slating randomised controlled trials?

RCTsDismissing Randomised Controlled Trials seems to be an increasingly popular thing to do these days – well, we are living in strange times…Last year Sally Cupitt (Head of NCVO Charities Evaluation Service) asked whether RCTs were the gold standard or just ‘fool’s gold’? A few weeks ago eminent professors Angus Deaton and Nancy Cartwright set out (in a suitably academic manner) their conclusions on the ‘limitations of randomised controlled trials’. Now NfP Synergy’s Joe Saxton has jumped aboard the anti-RCT bandwagon describing them as ‘another false dawn’ and ‘evaluation fantasy’.

Whilst it may make for a good blog post to challenge the growing awareness of and interest in Randomised Controlled Trials, it’s neither helpful nor accurate to dismiss their benefits so readily. Let’s look at some of the criticisms levelled at RCTs:

  • RCTs are ‘fraught with problems and complexity’ and Nesta’s guide to RCTs is 78 pages long so this must be true.

*Sigh*

Some RCTs are complex (though if we are tackling complex challenges in the charity sector –see point 4 below – should we be so scared of complex methods of evaluating impact?). Some too are complicated. But the existence of some poor practice doesn’t justify dismissing an entire method. That’s the sort of insidious thinking that has led some thinkers and politicians to characterise all charities as; poorly run, self-interested, inefficient ‘sock puppets’ of the State. Surely we don’t wish to subscribe to that type of logic?

There is a tendency, as with many specialist techniques to shroud them in complexity and technicality which serves the interests of experts and prevents their wider application. This does not make the method complex or problematic. It merely highlights the need for better understanding, support and application.

  • Nobody mentions double-blinds.

Apart from the fact they do if you are inclined to delve into the academic literature, the real issue here is that it is a red-herring. That’s the sort of sentiment which prevents RCTs from becoming a mainstream evaluation tool. It doesn’t matter if you don’t run a double-blind trial. Let’s be pragmatic about things – there are things which are essential in running trials and things which are nice to haves. In fact, we could talk about experimental methods as a continuum on which RCTs is one approach. It might be the one we aspire to – as reasons of complexity, scale and proportionate costs can make it impractical – but it’s not right for every occasion. Not even the most ardent trial evangelist would consider suggesting that. But there are plenty of instances where they can significantly enhance current practice.

  • It is unethical to withhold an intervention from some (randomly selected) people

This one just makes me laugh and cry in equal measure as it displays an absurd degree of selection bias and wholly misrepresents the very notion of understanding what works. Firstly, if you know something is going to provide a benefit to a particular group then do not (ever!) waste time on a trial. Just do it. Give it to the people as soon as possible.

But if, on the other hand, you think an intervention is going to be effective but you’d like to know for sure if it works a trial could give you confidence in the result. When we test new interventions we don’t know if they are going to work, we are testing them. Sometimes the things we ‘know to be true’ turn out not to be when properly tested. A classic example of this is the treatment of serious head injuries with steroids which had been the standard medical practice for 30 years, until someone tested it through an RCT and immediately found steroids were killing people. And so what was known to be true changed…overnight.

Then there’s the convenient overlooking of the fact that running a pilot is considered perfectly reasonable – indeed the post goes on, without a hint of irony, to talk about an intervention in one South London school…but why was it run in only one school? Was it ethical to deprive students in other schools of this intervention?

Somehow it’s perfectly fine to run a pilot but unethical to run a trial. Hmmmm…

  • Measuring single variables isn’t realistic when charities are often tackling complex challenges

This is more of an argument for RCTs than against them as far as I am concerned. It’s precisely because of the complexity that understanding the impact of single variables is useful. RCTs allow us to separate out the other ‘background noise’ and determine just what difference the intervention makes. Of course that doesn’t mean it’s necessarily sensible to use trials for longitudinal studies where the impact may take place over a generation. RCTs, like every other evaluation method needs to be used appropriately.

What we have found in our work is that small things can make a big difference – changing the text in how we communicate, altering the way information is presented, how we ask someone to do something or the way we design a service to be more customer-centric. All these things can make a significant difference to our impact and I’d suggest it’s our moral duty to put our assumptions to the test.

  • The sample size needs to be big

Yes, it does. So it’s not always going to be appropriate to run a trial in every circumstance – and it’s possible in these instances to use experimental methods without running full-blown trials. It’s also very valuable to recognise the limitations of what we are doing. No one is saying that we must run trials on everything…but we must avoid over-confidence in attributing change to our interventions without considering other external players and environmental factors. To dismiss RCTs simply because they don’t work in every instance is frankly ridiculous.

Contrary to what is suggested there are innumerable instances where RCTs will work, are not overly complex or prohibitively expensive and wholly achievable for a great number of charities. (Indeed where scale or resources are an issue trials might even be a catalyst for collaborating to share learning and increase efficiency).

‘Why have an evaluation standard that is applicable to very few o very few of the interventions that charities make?’

Ummm…because I thought we were in the business of trying to raise standards and quality in the charity sector.

There needs to be an intelligent understanding of what RCTs are, how they work and when it is appropriate to use them. If the starting point of those seeking to support improvement of evaluation and impact assessment in the sector is to rubbish an entire method simply because it’s not a panacea for all the sectors ills, what chance do we have of that?

And if anyone would like some suggestions of practical ways in which RCTs could easily be used by charities feel free to get in touch and I’d be more than happy to oblige.

We shouldn’t put RCTs in the ‘too hard to be bothered with’ box

The excellent article by Sally Cupitt – Head of NCVO Charities Evaluation Services – on Randomised Controlled Trials and their use within the voluntary and community sector provides not just an informed explanation of what they are and how they work, but also a critique of their application. I welcome any contribution that highlights the use of experimental methods – as a paid up member of the experimentalist club – particularly from someone as well informed and experienced as Sally.

RCTs

My experience of Randomised Controlled Trials at Lambeth Council and elsewhere at  is similar to the picture painted in the article – that RCTs are not generally understood, only used very occasionally by not for profit organisations (and indeed by local government and the public sector) though there is growing interest. The list of seven challenges that using RCTs poses to the voluntary sector is certainly very comprehensive and I am not going to argue (much) that they are not valid concerns.

Who can say that scale and timescale, technical skills, ethical issues, generalisation and the need for other evaluation methods aren’t real considerations?

But I look at that list and can’t help thinking that they could be applied to pretty much any decent and reliable evaluation method you might consider using. Of course different methods have their strengths and weaknesses and some will require more technical skill, or pose more challenges of generalisation…but they are all likely to be there with any method we might consider using. That is the nature of evaluation – there is not a single method or approach that will do everything you need all of the time (whatever some of the advocates of these approaches might tell you!).

One thing that Sally and I certainly agree on is that RCTs are not suitable for evaluating every programme or initiative. Not everything can be measured through and RCT and sometimes even if you could, you shouldn’t – the selection of evaluation methods needs to be proportionate to the scale and nature of the programme. In fact that reminds me of discussions Sally and I had about 15 years ago when she was helping me and my team to develop an approach to evaluating influencing policy-making. We realised that we could design a perfect system for evaluating the outcomes we wanted to measure, but only if we spent all our time and all our money on doing it. Evaluation needs to be proportionate. So, like all evaluation methods, we need to use RCTs selectively and appropriately.

I find a lot of people who think that RCTs have to be massively complicated, prohibitively expensive and are only used by moral-lacking purveyors of the ‘dark arts’ of manipulation. And of course there are many (mostly private sector) firms that use trials to sell things they don’t want to unsuspecting people….or something like that. However I think we need to bust a few myths about RCTs here – at least based on my experience.

Whilst I accept that some RCTs are terribly expensive and horribly complicated, they don’t have to be. I know this from having run successful RCTs at a number of local authorities. It comes down to using the method appropriately.

We’ve found it most suitable to test small changes to communications and messaging – to see which variation works best. That is very different to evaluating whole programmes where you have to track people over long periods of time to see how they behave. That way lies complexity and expense. But if we want people to respond in particular ways – whether it’s signing up to take part in an initiative, or to respond to a specific invitation or request in a particular way (behaviour change) then that can work.

I would go so far to say that with ever advancing technology we now have the opportunity to run RCTs at a lower cost and more simply than many other evaluation methods (after a bit of expense on the initial set up).

Scale is an issue – and clearly local authorities have a natural advantage over most charities and community groups in the size of their operation (particularly with universal services). But I see this as an opportunity to encourage collaboration, sharing and ultimately drive up standards across the VCS, by working together to evaluate interventions by using RCTs (that also helps to address the issues around generalisation).

Another concern people often express when I talk to them about using RCTs is that the process is depriving people of something. Of course that is true – but we do that all the time when we pilot new approaches, without batting an eyelid. How is it different? If we knew something would work we would do it and not run pilots or prototype new approaches. We do them because we don’t know but we want to find out – and by using an RCT we can be more confident that the results we observe are down to our actions, not because of any other factors that might make the pilot area or group different to another group.

RCTs do require a degree of technical expertise, that’s true, but it doesn’t mean needing to do a Masters or a PhD in experimental methods (though if you want to go, I can recommend courses that Professor Peter John runs at UCL). There’s a lot you can learn from reading the resources out there, or support that is available. And inevitably, as the use of RCTs grows, so too will the support available to run them – and I’m very happy to share my experience with anyone who’s interested!

Just because they are new and we have to learn how to use them appropriately – much like any other innovation – doesn’t mean we should write them off as being too difficult to bother with. Aiming high and believing things are possible however improbable they might seem is a hallmark of the VCS and one which can be applied to evaluation as much as anything else.

School Days – my day with LiteracyActionNet & Ealing Fields

I spent yesterday discussing schools…nothing particularly unusual there, except that for the most part I wasn’t on Archer Academy business, but talking to others interested in education.

During the day I attended a very interesting seminar organised by Lemos & Crane as part of their LiteracyActionNet project, looking at parental engagement in literacy (and education more generally). Among plenty of interesting presentations and discussion, the thing I found most enlightening was the presentation given by Stephen Hall of the Education Endowment Foundation (EEF) on what impacts on attainment.

The EEF have looked at (and now started to fund) different approaches and interventions and what difference they make to children’s attainment levels. Their findings – which are inevitably changing as they fund and evaluate new projects – make for fascinating reading and challenge some traditional assumptions about the effectiveness of particular methods or interventions. The EEF have sought to quantify the impact in the number of additional months impact the approach has (on average) on children’s educational attainment levels.

Their research suggests that while giving feedback to pupils and to teachers has an average impact of +8 months, ability grouping actually has a detrimental effect on attainment (-1 month). They suggest that school uniform, physical environment, performance related pay and teaching assistants all have (on average) no impact on attainment. But things like peer tutoring, collaborative learning and ‘learning to learn’ strategies all make a significant difference – +5 months or more. Of course, just because school uniform and teaching assistants may not make a difference to attainment levels, they could have benefits on other things…but nonetheless I found the evidence really challenged some of my assumptions about what matters and where we ought to be investing resources.

It’s well worth having a look at the EEF’s evidence.

In the evening, I went to talk to some of the parents behind the proposal to establish the Ealing Fields School in West London. They are, like the Archer Academy’s founders, a group of parents attempting to address a desperate need for secondary school places in their community. The group hope to open in 2015 and are in the process of developing their proposal and building local support for the school. Sitting around the kitchen table brought back many memories of the early days in the development of the Archer Academy and also made me realise quite how far we’ve come.

In many respects the Ealing Fields group are in a better position than we were at the same time in our development. They have solid educational expertise on which to draw and more time to develop their proposal (ours was submitted just two months after deciding we were going to apply). It also appears that the process of applying to set up a free school is getting better with each round – there is now a rolling programme for applications and more support available from the New Schools Network than was the case when we were applying.

What I realised in talking to the group was how important community engagement and communications is to the whole project. Whilst the educational expertise of Archer founders was limited to just a few of the group, we were more experienced in comms and engagement. I think this was crucial in securing community support, developing our vision in a way that reflected the aspirations of our community and building strong foundations that have stood us in good stead throughout the process. Although, of course, the development of a school must be based on a sound vision and educational plan, the importance of local buy-in is without question a ‘deal breaker’. Without it, I cannot believe we would ever have been able to establish the school, or that any other group of parents will be able to.

Having worked in community engagement for many years, I have seen far too much shoddy practice being sold as ‘effective engagement’. Free school groups must ensure that they do not accept just ‘good enough engagement’ but aspire to something much better and much more meaningful. At the Archer Academy we constantly asked ourselves ‘just how good could this school be?’, this same aspirational thinking needs to extend to how we approach community engagement and communication too.

Increasing the evidence base or reducing uncertainty?

Over the years, and in a variety of roles, I have spent a lot of time trying to ‘increase the evidence base’ to inform policy and decision-making. Evidence of what works should guide our plans for the future – this has been the mantra.

Sometimes we’ve had lots of evidence-based policy making – the Social Exclusion Unit’s Policy Action Teams is probably the zenith for evidence based policy development – and at other times we’ve had an almost anti-evidence approach. Talk of ‘conviction politics’ seems more popular these days, less evidence, more how it feels.

The politics is the politics….whatever.

But I’ve been increasingly questioning the sense of ‘increasing evidence’ if we are serious about social innovation.

They say that past behaviour is the best predictor of future behaviour, but is that the case if we are trying to change things?

If we are trying seriously to develop innovative approaches – whether it’s to service delivery, community participation, governance, design or enterprise – we cannot know what the outcome will be. If we know, then surely it can’t be innovation….it’s a dead cert. And that doesn’t sound to me like innovation.

So, with innovation and creativity, there is a risk. We take that risk in trying something new, whether we acknowledge it or not. It may fail. 

Try again, fail again, fail better!

[i just couldn’t resist putting this wonderful Samuel Beckett quote in!]

We can learn from the past, from the evidence base, certainly. But that is useful only as a means of reducing the uncertainty (or the level of risk) in our planned innovation.

 

If we get too hung up on talking about increasing the evidence base, I worry that we will become too confident about the impact we will have when we deliver a particular innovation. The evidence base may be useful – it may tell us how things worked in a different setting with different circumstances. But because life around us changes so rapidly, we cannot be certain that what happened in the past will happen again in the future.

The other thing that makes me nervous talking about increasing the evidence-base, is that it can stifle innovation and (managed) risk taking. If we emphasize having to have an evidence base to prove that we should do something, we may find that the evidence is never there and so the innovation is never tried.

If we talk about reducing uncertainty, by contrast, we are accepting an implicit level of risk (which is there) but that the use of evidence can help us to reduce that risk to a more manageable level. 

That may sound rather subtle but I think in terms of establishing a culture of innovation (thinking in particular about my experience of local government) I think this matters a great deal.

This is not about dismissing the value of evidence….but more about how we frame its use in supporting social innovation.