How two graduate college students uncovered a important error in autism screening tips | Spectrum
Punit Shah
Associate professor, University of Bath
Lucy Waldren
PhD student, University of Bath
When clinicians suspect someone has autism, they often turn to screening tools – standardized questionnaires or checklists that measure autism characteristics. If a high score confirms their suspicion, they refer the person to a specialist for diagnostic evaluation.
These screening tools usually come with guidelines to help doctors use them. But the guidelines also inform researchers who screen potential study participants – a use that ultimately shapes researchers’ definition of autism, says Punit Shah, associate professor of psychology at the University of Bath, England.
In the UK, the National Institute for Health and Care Excellence (NICE), among others, produces a variety of guidelines for screening tools. And earlier this year, two of Shah’s graduate students discovered that NICE guidelines for a particular test called the Autism Quotient-10 (AQ-10) were wrong. They published their results in The Lancet Psychiatry in April.
Last week on June 14th, NICE issued a statement confirming that they would correct the mistake. “We were concerned that the guidelines would not be changed,” said Shah after the announcement. “We were therefore very pleased to see that NICE took the topic seriously and dealt with it in such a timely manner.”
Spectrum spoke to Shah and one of the PhD students, Lucy Waldren, in May about how they discovered the bug, why they are spreading the message, and what needs to happen next.
Spectrum: What are the NICE guidelines and what was the bug you found?
Punit Shah: NICE is a UK-based non-governmental organization. It is an independent, very respected body. The NICE guidelines are often used and adapted by many other countries. So what we do here in the UK and what NICE recommends has far-reaching consequences.
The guidelines recommend that individuals with a score above 6 on the AQ-10 be referred for further assessment and possible clinical diagnosis of autism. This is very different from what was suggested in the original research which included a score of 6.
S: How did you find this bug?
Lucy Waldren: Another Ph.D. Student Rachel Clutterbuck analyzed some data using the AQ-10. The original paper from 2012 defined the AQ-10 from a larger autism spectrum quotient, which is a 50-item measure. Rachel noticed that the original paper and NICE’s had a different cutoff.
We found that really strange. So we went through the rationale they gave for how they got their cutoff score and couldn’t really come up with any real reason why they picked 7 instead of 6.
S: What was your first reaction to this discovery?
LW: I just think surprise. We thought about ourselves again and again and thought: “That can’t be anything. It has been around for almost 10 years and a mistake was made in this guideline. ”It was almost disbelief.
PS: We use the AQ-10 and other measures of autism characteristics in our research. And we think that, by and large, they have been very helpful for autism research. It’s not like we’re trying to attack it or find problems with it. When Rachel encountered this problem, I thought she was making the mistake. I asked her for a screenshot and asked Lucy to dig in it.
S: What did you decide when you were sure of the mistake?
PS: At first, we just kept checking it and double-checking it. We must have checked this a dozen times and thought, “Well, are we safe here?” After being academically convinced that these mistakes can happen, and Lucy did a really good job of questioning all the appendices and the gist of what the NICE guideline group looked at, we decided at this point write about it and send it to a scientific journal ASAP just to try to get people’s attention.
S: Did you contact NICE before you published the paper?
PS: Yes. Our institution contacted NICE to let them know that we found the bug and will write about it. But at least NICE hadn’t answered us at all at this point.
S: What was the reaction to the paper and what has happened since it came out?
LW: I went to NICE’s annual general meeting and asked it as a question that NICE did not want to answer. Since then, they have said they review the guidelines, which is good.
PS: They never contacted us directly. We got quite a bit of media coverage here in the UK and a little bit in the United States. There was a lot of feedback from the autism community on Twitter. As for NICE’s response, however, it was quite disappointing. They said things like, “We don’t really know what happened” or “As part of our fuller review of these guidelines, we will investigate this.”
S: What is the impact of this one-point error over the past 10 years?
PS: While it sounds trivial at first glance, a difference of just one point on a 10-point scale is actually quite large. Since NICE had recommended a higher cutoff, the measurement becomes less sensitive. There will be people who will get a 6, many of whom have autism who may not have been referred for further diagnosis.
LW: We have no way of knowing how many people could be affected. If people miss another exam and diagnosis because of this, they will not have access to the same care and support, including financial assistance. This will affect their mental health due to the lack of support. They also do not have the confirmation of receiving the diagnosis they believe to be correct. So it could have had a really serious impact on people’s lives over the years.
From a research perspective, the measure is used as a cutoff for participants. Data can also be lost if people use the different cutoffs.
PS: Lucy is now looking at this more systematically. Other characteristics of autism spectrum traits have been used quite loudly or in a messy manner in the literature. We found some publications where they used the wrong cutoff to include or exclude people from their studies. I think at some point these researchers will have to reconsider their data or consider trying to replicate its basic effect. Some people have been receptive to this idea and have checked to see if they implemented the wrong cutoff. But in general, people haven’t been so open about mistakes they made.
S: What are you planning next?
LW: We look broadly at the use of autism traits in research and try to figure out what future steps may be needed to improve the practice.
PS: The advantages and disadvantages of these measures have been discussed for a long time. It is only in the last few years that people have started to take a closer look at them. Even if we are to some extent critical of some of these measures, we use them in our research. They have generally been incredibly helpful in advancing autism research. But the next steps are to involve more autistic people and these more rigorous statistical techniques to question and improve them.
