When did it become trendy to start slating randomised controlled trials?

RCTsDismissing Randomised Controlled Trials seems to be an increasingly popular thing to do these days – well, we are living in strange times…Last year Sally Cupitt (Head of NCVO Charities Evaluation Service) asked whether RCTs were the gold standard or just ‘fool’s gold’? A few weeks ago eminent professors Angus Deaton and Nancy Cartwright set out (in a suitably academic manner) their conclusions on the ‘limitations of randomised controlled trials’. Now NfP Synergy’s Joe Saxton has jumped aboard the anti-RCT bandwagon describing them as ‘another false dawn’ and ‘evaluation fantasy’.

Whilst it may make for a good blog post to challenge the growing awareness of and interest in Randomised Controlled Trials, it’s neither helpful nor accurate to dismiss their benefits so readily. Let’s look at some of the criticisms levelled at RCTs:

  • RCTs are ‘fraught with problems and complexity’ and Nesta’s guide to RCTs is 78 pages long so this must be true.

*Sigh*

Some RCTs are complex (though if we are tackling complex challenges in the charity sector –see point 4 below – should we be so scared of complex methods of evaluating impact?). Some too are complicated. But the existence of some poor practice doesn’t justify dismissing an entire method. That’s the sort of insidious thinking that has led some thinkers and politicians to characterise all charities as; poorly run, self-interested, inefficient ‘sock puppets’ of the State. Surely we don’t wish to subscribe to that type of logic?

There is a tendency, as with many specialist techniques to shroud them in complexity and technicality which serves the interests of experts and prevents their wider application. This does not make the method complex or problematic. It merely highlights the need for better understanding, support and application.

  • Nobody mentions double-blinds.

Apart from the fact they do if you are inclined to delve into the academic literature, the real issue here is that it is a red-herring. That’s the sort of sentiment which prevents RCTs from becoming a mainstream evaluation tool. It doesn’t matter if you don’t run a double-blind trial. Let’s be pragmatic about things – there are things which are essential in running trials and things which are nice to haves. In fact, we could talk about experimental methods as a continuum on which RCTs is one approach. It might be the one we aspire to – as reasons of complexity, scale and proportionate costs can make it impractical – but it’s not right for every occasion. Not even the most ardent trial evangelist would consider suggesting that. But there are plenty of instances where they can significantly enhance current practice.

  • It is unethical to withhold an intervention from some (randomly selected) people

This one just makes me laugh and cry in equal measure as it displays an absurd degree of selection bias and wholly misrepresents the very notion of understanding what works. Firstly, if you know something is going to provide a benefit to a particular group then do not (ever!) waste time on a trial. Just do it. Give it to the people as soon as possible.

But if, on the other hand, you think an intervention is going to be effective but you’d like to know for sure if it works a trial could give you confidence in the result. When we test new interventions we don’t know if they are going to work, we are testing them. Sometimes the things we ‘know to be true’ turn out not to be when properly tested. A classic example of this is the treatment of serious head injuries with steroids which had been the standard medical practice for 30 years, until someone tested it through an RCT and immediately found steroids were killing people. And so what was known to be true changed…overnight.

Then there’s the convenient overlooking of the fact that running a pilot is considered perfectly reasonable – indeed the post goes on, without a hint of irony, to talk about an intervention in one South London school…but why was it run in only one school? Was it ethical to deprive students in other schools of this intervention?

Somehow it’s perfectly fine to run a pilot but unethical to run a trial. Hmmmm…

  • Measuring single variables isn’t realistic when charities are often tackling complex challenges

This is more of an argument for RCTs than against them as far as I am concerned. It’s precisely because of the complexity that understanding the impact of single variables is useful. RCTs allow us to separate out the other ‘background noise’ and determine just what difference the intervention makes. Of course that doesn’t mean it’s necessarily sensible to use trials for longitudinal studies where the impact may take place over a generation. RCTs, like every other evaluation method needs to be used appropriately.

What we have found in our work is that small things can make a big difference – changing the text in how we communicate, altering the way information is presented, how we ask someone to do something or the way we design a service to be more customer-centric. All these things can make a significant difference to our impact and I’d suggest it’s our moral duty to put our assumptions to the test.

  • The sample size needs to be big

Yes, it does. So it’s not always going to be appropriate to run a trial in every circumstance – and it’s possible in these instances to use experimental methods without running full-blown trials. It’s also very valuable to recognise the limitations of what we are doing. No one is saying that we must run trials on everything…but we must avoid over-confidence in attributing change to our interventions without considering other external players and environmental factors. To dismiss RCTs simply because they don’t work in every instance is frankly ridiculous.

Contrary to what is suggested there are innumerable instances where RCTs will work, are not overly complex or prohibitively expensive and wholly achievable for a great number of charities. (Indeed where scale or resources are an issue trials might even be a catalyst for collaborating to share learning and increase efficiency).

‘Why have an evaluation standard that is applicable to very few o very few of the interventions that charities make?’

Ummm…because I thought we were in the business of trying to raise standards and quality in the charity sector.

There needs to be an intelligent understanding of what RCTs are, how they work and when it is appropriate to use them. If the starting point of those seeking to support improvement of evaluation and impact assessment in the sector is to rubbish an entire method simply because it’s not a panacea for all the sectors ills, what chance do we have of that?

And if anyone would like some suggestions of practical ways in which RCTs could easily be used by charities feel free to get in touch and I’d be more than happy to oblige.

Restart Project fixing our broken (relationship with) electronics

Last night I went to a new year party held by the social enterprise the Restart Project and heard about their achievements over the past year and plans for the future.

restart_logo_complete_black1

I’ve previously met Restart’s founders, Janet Gunter and Ugo Vallauri to hear about their work and to explore common interests and opportunities to work together. They are hugely impressive individuals with a clear purpose and a passion that is infectious.

Restart is based on a simple premise – that our relationship with electronics is broken and we need to fix it. Whether it’s the latest must-have iPhone or broken bits of computers that fill our cupboards and drawers, it’s clear that we are, as a society, buying a lot of electronic equipment…without thinking very much about what happens to the ‘old stuff’. Electronic waste is one of the fastest growing waste streams in the UK and it’s costing a huge amount of money to dispose of redundant tech.

What the Restart Project does is encourage and empower people to use their electronics for longer by sharing repair and maintenance skills. They hold events – Restart Parties – where people are invited to bring their electronic equipment to be repaired by a team of enthusiastic volunteer repairers. Those turning up are taught new skills and encouraged to extend the lifespan of electronic devices and divert them from going into waste.

The project clearly offers so benefits on so many levels. It brings people together to interact, meet, talk and do things together, building social capital within communities. It helps people to learn new practical skills that can be used to save money and time. Among younger people ‘getting your hands dirty’ with electronics could easily lead to opening minds – and doors – to career opportunities in computing and coding or engineering; industries that the UK needs to continue to grow the availability of skilled labour in. And of course, Restart delivers on its primary purpose – to prolong the lifespan of electronic equipment, which saves people money and might just save the planet.

Everything about Restart inspires and excites me: the combination of social, economic and environmental benefits is clear and the potential huge.

I can also see the potential for technology to be redistributed to those who may not have access to even older equipment. I can remember back in the 90s taking donations of computers from companies and spending hours on end formatting old PCs and installing basic software on them to then distribute them to homelessness charities and groups.

Even though I do – I must confess – like to upgrade my phone and computer(s) every few years, I always try and find a home for the old kit. The emergence of the resale market for mobile phones over the last few years is an example of how much opportunity there is for us to make better use of old electronics.

IKEA’s head of sustainability recently suggested that we had reached ‘peak stuff’ and that consumers’ appetite for buying home furnishings had reached saturation point. If we want more new stuff, I’d suggest we need to think more about what happens to the old stuff….the Restart Project might just be the answer.

Marketing the charity sector’s spending on marketing

A brief survey of public attitudes towards charity makes for interesting reading. The poll, conducted by the Free Postcode Lottery website, which had a decent sample size of 2,000 people, found that one in four of us think we don’t give enough to charity. 25% isn’t a vast amount, but it would be 15 million people in the UK if the figures were representative (no one is saying they are….) but even if that over-estimated the proportion by 100% it would still mean there were over 7 million people in the UK who might – under the right circumstances give more to charity.

It’s perhaps not surprising to find that over three-quarters of respondents said they were put off from giving money to charity by the marketing practices used by fundraisers. Recent public scandals over fundraising and the very public collapse of Kids Company having received almost £50m of public money have damaged the sector’s reputation. Something I am not convinced that the Etherington Review or NCVO’s and ACEVO’s plans to tackle negative stories will address. This is not just a question of perception – that people see the headlines and need to be reassured that these stories of poor financial management and unethical fundraising are the exception to the rule. People have also experienced first-hand some of the pressurised selling that some charities – and the private companies they contact to fundraise – employ. If we regard this solely as an issue of perception we will never restore confidence and trust in charity.

The third, and final, finding from the survey is that nearly half of respondents say they would give more to charity if they spent less on marketing. Part of me wants to focus on the question’s methodological limitations – we know already of course that three-quarters of people have said they think they give enough to charity already. So it’s difficult to infer too much from this question. Are the half who say they would give more made up of the 25% of respondents who say they don’t give enough – and here is an excuse for them to justify why they don’t? However, even if we assume that half the half (a quarter in total) are inclined to give more anyway. The other 25% are saying that, though they feel they give enough already, they would give more if charities spent less on marketing.

I wonder though how well informed respondents are? Do they know what charities spend on marketing? Which charities? The fact is that the amounts spend vary as wildly as the budgets of charities themselves do – both in actual terms and as a proportion of their budgets. Apparently total charity spending on advertising was £394m in 2013 – a mere 2% of the total of the advertising market worth almost £14bn that year. In the same year the total expenditure of the charity sector was £38bn. So the figures suggest the charity sector as a whole spent around 10% of its money on marketing. That doesn’t sound tremendously excessive.

In that sense I do think it’s a question of addressing perception. I suspect that if people thought charities were spending around 10% of their money on marketing the public might be more sympathetic. Whether or not they would give more to charity is quite another matter though.

Takeaways from the Kids Company Select Committee car-crash

While all and sundry are having their say over the lessons from the demise of Kids Company, anyone who saw the spectacle of former CEO, Camila Batmanghelidjh and Chair Alan Yentob ‘giving evidence’ (I use the term loosely) can’t help to have opinions.

So, at the risk of depressing Nick Temple by adding to the noise, here are my takeaways from the whole affair:

takeawaysignwebsite

  1. Two sides of the same coin

Camila’s charisma, passion and ‘JFDI’ attitude charmed successive Prime Ministers and hypnotised a large band of celebrity supporters. It is precisely the same characteristics which meant strong governance and effective management were neglected at Kids Company. They are, in effect two sides of the same coin and that is just Camila. What attracted so many to the charity also became its Achilles heel. 

  1. Charismatic leaders need good teams around them

Able deputies are essential to fill in the gaps that those with a tendency to focus on the big picture overlook. Detail does matter and whilst there are huge benefits of strategic thinking, you can’t forget the small stuff. You also need people that can challenge leaders when needed and stand up to them if required. A team of acolytes is not healthy for long term sustainability. 

  1. Governance matters

We overlook the importance of strong and effective governance at our peril. We’ve seen this in the private sector and in the charity sector.  The board of RBS was roundly criticised for not having a clue what the company was up to and the risk they were exposed to until it was far too late. Charity governance – like corporate governance and even democratic governance – is patchy. Some are excellent. Some are poor. We need more consistently higher quality governance. Ten years ago (a bygone era) charity governance was considered important and investment made in providing support. That investment – along with much funding for VCS infrastructure – has now gone and that cut comes with a price. 

  1. Kids Company aren’t typical but they aren’t unique

Some have been quick to point out that Kids Company are unusual – a maverick among the hundreds of thousands of charities out there. And indeed they are unusual. But I’d caution those so keen to this sort of thing would never happen elsewhere. Are we really sure that governance is so strong throughout the sector? I’m not saying it’s not, I’m saying I’m not sure. I’m not a betting man, but if pressed I’d probably say there is another Kids Company out there waiting to happen. Suggesting this is a unique, one off never to happen again set of circumstances could backfire terribly. 

  1. A little bit of contrition goes a long way

The self-important posturing and defensiveness of Batmanghelidjh’s and Yentob’s display was unseemly and appeared almost disdainful towards the MPs on the Select Committee. The exasperation of the Committee Chair, Bernard Jenkin was obvious, telling Batmanghelidjh to ‘hurry up’ and ‘stop talking’. Paul Flynn was even more robust, saying ‘we’ve had a lot of psychobabble…can you just answer the question’. Sometimes – and I’d say when appearing before a Select Committee is one of those times – it pays to be sincere, courteous and contrite. You’re not Rupert Murdoch and nor, would I suggest, should you want to be. 

  1. The sector must rebuild trust

Trust in charities has been damaged by a succession of scandals and exposés. I suspect any members of the general public watching these high profile charity sector representatives’ car-crash performance couldn’t help thinking ‘if this is the best you’ve got, what’s the rest of the sector is like?’

It might be unfair and untrue but the impression will have been made in some people’s minds. The die is cast and the sector needs to respond. In my view the response should not be simply to say ‘oh we’re not like that at all’, we need to listen harder take on board the concerns people have and be seen to act. 

  1. Government has been let off the hook

The performance of Kids Company’s senior leaders before MPs has completely deflected attention away from the actions of Prime Ministers (Blair and Brown both courted Camila before Cameron did), their Ministerial colleagues and Whitehall officials. No one should come out of this sorry affair with any credit, but the shocking spectacle of the Select Committee has shifted the focus almost entirely away from the Government. 

  1. Charity is about beneficiaries

The only reason we have charities is to help beneficiaries (normally – but not always – people). We shouldn’t forget that Kids Company did help a large number of extremely vulnerable young people and their families. We should never forget the children who are still in need and aren’t helped by the circus show playing out before our eyes.

Who wants a government funded fundraiser?

The Public Administration and Constitutional Select Committee questioning of Kids Company CEO, Camila Batmanghelidjh and Chair of Trustees, Alan Yentob was excruciating but compelling viewing. There are a large number of issues that were raised that deserve proper scrutiny and discussion – and I am sure they will get that attention. But one thing to have emerged from the questioning shocked and surprised me, even in the context of the circus that has been the Kids Company debacle. It was a brief comment made by Alan Yentob in response to questioning about Kids Company’s relationship with Government.

kids company

He said – as proof of the charity’s good relationship with government – that two civil servants had ‘sat in Kids Company for a year to help the charity find statutory funding’.

I’m intrigued and slightly surprised that the government decided to spend public money by placing its staff in a charity to help them raise money from, errrr, the government.

I would love to know who sanctioned that decision as it strikes me as a wholly inappropriate use of public funds and one which someone in government ought to be held accountable for.

I’m no expert but I also can’t help wondering whether this might breach EU laws on State Aid…but I might be wrong.

Where I am more certain is that a great many charities and community groups would love to have fundraisers offered by the government to come at work for them. Unfortunately they lack the ear of the Prime Minister and Ministers, they don’t have a senior BBC executive as their Chair and are not awash with a host of celebrity supporters.

I would love to know which department these fundraising civil servants came from – was it the Department for Education or the Cabinet Office, or somewhere else?

I’d also like to know who authorised their secondment?

Anyone who knows anything about this, I’d love to hear from you.

How a simple letter got people online – saving Essex Council £20,000 a year

Essex County Council, like most other councils, spend a lot of money processing paper. Some of that goes on administering Blue Badge permits that allow disabled drivers to park closer to where they’re going. With the continued pressure on public service budgets, it’s hardly surprising that they should want to look at whether they can reduce their costs and make the process cheaper and more efficient. An obvious way to do this is to get people to renew their permits (and do an array of other things) online rather than more expensive face-to-face or paper based administration.

So we set out to help them test, through a Randomised Controlled Trial, whether small changes to the Blue Badge renewal process could encourage channel shift – that is moving people from one channel (paper/face-to-face) to another (online).

The results showed it worked – passing standard tests of statistical significance.

Just by making small changes to the renewal letter that goes out to Blue Badge holders we managed to reduce the numbers of people that renewed by post by almost 20% (8.6 percentage points).  And because we used an RCT to test it, we can be confident that the difference can be attributed to the changes we made to the letter.

Blue Badge renewal by type

Blue Badge renewal by type

By simplifying the letter – stripping away anything we didn’t feel was absolutely necessary to include and making it clear to the reader precisely what they had to do – we made it more likely that people used the online renewal process.

Martha Lane-Fox, in her 2010 role as Digital Champion, looked at the costs (and savings) of government conducting business online, rather than face-to-face and paper. She estimated that the public sector would save £12 for every transaction conducted online rather than by post.

For Essex that figure would mean that this small change to the renewal letter will save them around £20,000 a year. Of course set against cuts of £50m this year it’s a drop in the ocean. But if every service or process were able to find similar efficiencies for a fraction of the costs of implementing them it would soon add up.

Simplification wasn’t the only effective way we found of reducing paper renewals. We also found that by telling people that renewing online saved the Council money and helped protect frontline services also encouraged channel shift. This small change in the renewal letter reduced paper renewals by over 18% (7.7 percentage points).

The RCT gave the Council strong evidence of what works, enabling them to adopt the simplified letter as their new standard renewal letter.  Since they did so, only 5 percent of renewals are done by post, with over half of all renewals now being done online (the remainder are done over the phone).

Post-trial renewals by type

Post-trial renewals by type

The trial has helped the Council move toward online renewals for blue badges and we hope it can act as a model for the more general transformation of services across the council.

Small things can make a big difference.

Read the full report on our online Blue Badge renewal trial.