A long time ago, in a galaxy far, far away, well it was about 12 years ago in London, I was at a Christmas party. I was working at the National Foundation for Educational Research (NFER) at the time, a not-for-profit contract research organisation, and the party was for all our clients and contacts, ranging from academics at universities to government departments such as the Department for Education and funding bodies such as the Sutton Trust and the Esme Fairbairn Foundation. With a glass of fizz in one hand and a canape in the other, I got talking to a chap from the Wellcome Trust about randomised controlled trials (RCTs). This is the sort of thing I love to talk about at parties!

With more than 10 years of experience in academic publishing prior to joining NFER, I had done a fair bit of work in medical publishing and was well used to the idea of using RCTs in medicine. They are essential for finding out if new drug treatments, therapies and other interventions truly work or whether they only seem to work. Let me explain.

A randomised controlled trail is three things – it is random, it is controlled and it is a trial. So, if a research team is testing a new drug for a specific disease, following laboratory tests that indicate the drug will be safe for humans to use, they will need a group of patients with the disease to try the drug. This is the trial part. Patients are usually asked to volunteer for drugs trials and in some cases they may be recommended for the trial by their physician, but they must always give their informed consent before embarking on the trial. This is important for two reasons. First, there is at this stage no guarantee that the drug will work and there is no solid information about what the side effects may be. Second, not everyone in the trial will actually be taking the drug – some will be taking a placebo. This is something that looks exactly the same as the real drug, but actually has no medical effect at all on the patient. Patients on the placebo are in the control group – no changes to their health outcomes are expected. Which patients take the drug and which take the placebo is randomly determined – this is the random part of the trial. Patients in the trial do not know whether they are taking the drug or the placebo, so there is a much-reduced chance of participant bias and only the research team and the doctors treating the patients know which patients are taking the drug. In a blind randomised controlled trial, not even the doctors treating the patients know who is taking the drug, to ensure they cannot unconsciously affect patient outcomes. This all makes perfect sense in medicine, as whether drugs work or not can literally be a matter of life and death and those in the trial are already unwell, so there must be many checks and balances to ensure that the findings are accurate.

So, back to the Christmas party. The chap from the Wellcome Trust was keen on RCTs in education. Would the approach work in settings other than medicine? Would it be appropriate? Would it be ethical? What would an RCT in education look like? Over a further glass of fizz and as the other party attendees drifted away from our discussion, we considered an example.

Let’s say you’ve devised a new way of teaching maths to primary school children. If you teach all the children in one year group with this new method, how will you know if the results are better than previous methods without waiting possibly a whole year for end-of-year assessments? And even then you wouldn’t know if that year group would have performed just as well having been taught using the existing methods. You could, instead, teach just one randomly selected group of children in that year group using the new method and then compare their results with those of children who were not in the intervention group. This would be a form of RCT but would be very small and the results would not be likely to be generalisable to the wider population. You could scale this up to be several whole schools whose same year group is involved in the trial. As with all quantitative research, the larger the sample size, the more significant the results will be. But there are many complications. How would you select your sample group for the intervention? How would you ensure that the intervention group was similar enough in terms of ability and motivation to those in the non-intervention group? They might do better or worse just because they are a lower-achieving or higher-achieving group. How would you maintain confidentiality about the new method, such that those in the intervention group don’t tell those in the non-intervention group (or vice versa) that there is an experiment going on and they are the guinea pigs? You would need informed consent from the parents or guardians and just one slip to a child in the non-intervention group might mean that word gets round and that group start to slacken off, aware that they are expected not to do well. And there is a further ethical issue – as a parent giving consent to your child’s involvement in the trial, what of the worry that your child might miss out on the intervention and not be taught using the swish new method that is meant to improve mathematical ability and standards? What if parents agreed on the condition that their child is in the intervention group? That could unbalance the trial. But these issues to do with advantage and disadvantage and the ethical issues around whether it is fair to allow one group of children to have a chance at better success than others, have been well debated and there are solutions. RCTs are not always appropriate, but where they are, they can be very successful.

I left the party having had a great time and discussed the issues with colleagues at NFER the next week. It turned out that there were mixed views on the use of RCTs in education, as you would expect from a mix of researchers with different backgrounds and different epistemological viewpoints. In due course we published a short book on the subject, written by two of our senior statisticians, Dr Dougal Hutchison and Dr Ben Styles, who is now Head of NFER’s Education Trials Unit. If you’d like to read it, it’s available online for free and is very accessible (see reference below) and you can see Ben talking about RCTs in education here.

I hope this has made you think the same way that it made me think. It’s valuable to consider alternative methodologies for research that are used in disciplines other than your own, even if it is just to understand them rather than use them. How could you use RCTs in your research and what challenges might be presented? How would you overcome those challenges?

References and further reading

Hutchison, D. and Styles, B. (2010) A Guide to Running Randomised Controlled Trials for Education Researchers, Slough: NFER, available at https://www.nfer.ac.uk/media/2114/rct01.pdf, accessed 14 April 2020.

NFER (no date) One hundred years of education trials, National Foundation for Educational Research website [online], available at https://www.nfer.ac.uk/news-events/nfer-spotlight/one-hundred-years-of-education-trials/, accessed 14 April 2020.

NFER Education Trials Unit [online], available at https://www.nfer.ac.uk/key-topics-expertise/nfer-education-trials-unit/, accessed 20 June 2020.

NFERTV (2014) Ben Styles talks about randomised controlled trials in education [online], available at https://www.youtube.com/watch?v=wNC55sAo4Cg, accessed 14 April 2020.

Styles, B. (2018) Using Randomised Controlled Trials in Education. Educational Research, 60(3), pp. 371–372.