Inexperienced mild for diagnostic autism app raises questions, considerations | Spectrum
Screen Time: An app-based system to aid in diagnosing autism may be available soon.
Courtesy of Cognoa
The U.S. Food and Drug Administration (FDA) has approved an app designed to help doctors diagnose autism in children and marked it safe for use.
However, it is still unclear whether the technology is effective, researchers say, raising concerns that it could actually disrupt the standard diagnostic process and, in some cases, cause delays in starting therapy.
“I think what they did looks beautiful and is an example of what money can buy,” said Catherine Lord, esteemed professor of psychiatry and education at the University of California at Los Angeles and co-founder of Autism Diagnostic Observation Schedule, is considered the “gold standard” for diagnosing autism. “But it can do more harm than good.”
Targeted at primary care physicians and pediatricians, the app was developed by Cognoa, a California-based company founded by autism researcher Dennis Wall. It’s part of a machine learning-based system called Canvas Dx that uses algorithms to evaluate videos and other information about a child that has been uploaded by their parents and a doctor. The company plans to release the app and the rest of the Canvas Dx system in the US by the end of 2021, says Sharief Taraman, chief medical officer of Cognoa.
“We believe Canvas Dx will help primary care practitioners diagnose autism fairly in young children, regardless of gender, race, ethnicity, socioeconomic status or geography, and hope that the use of Canvas Dx will reduce the time to diagnosis which may allow initiation of previous interventions, ”Taraman wrote in an email to Spectrum.
In 2018, Cognoa caused confusion among some autism researchers by announcing that the app was classified as an FDA Class II diagnostic medical device long before the agency made the classification earlier this month. On June 2, the FDA authorized Cognoa to market the app as a tool for diagnosing autism in children aged 18 months to 5 years whose clinicians or caregivers are monitoring possible autism characteristics.
The company received approval via the FDA’s “de novo premarket review path”, which was introduced in 2018 for novel medical devices with low to medium risk.
“I think it is important for Cognoa marketing purposes not to confuse de novo classification with product approval [from the FDA]”Says Matthew Goodwin, director of the Computational Behavioral Science Lab at Northeastern University in Boston, Massachusetts.
Test results:
To use Canvas Dx, a parent or caregiver uploads two videos of their child and fills out a questionnaire in the app. A doctor then meets with the family and fills out a second questionnaire in the app. An algorithm analyzes the data and provides a conclusive “yes” or “no” as to whether the child has autism or suggests that the child be screened for other conditions. The process takes a few weeks, says Taraman.
As part of its safety review, the FDA reviewed data from the Cognoa clinical study, which compared the app’s assessment of autism in 425 children, including 153 girls, to that of a panel of three experts. None of the children had previously been formally screened for autism, but their carers or pediatricians had raised concerns about their development.
According to the doctors’ assessment, 122 of the children are autistic, 40 are neurotypical and 263 have a neurodevelopmental disorder other than autism.
The app provided conclusive results for 135 children or 32 percent of the group. Of the 290 children who received an inconclusive result, according to the panel of experts, 264 or 91 percent have at least one other neurodevelopmental disorder.
The app correctly identified autism in 63 of the 122 autistic children, and 1 autistic child received a false negative result. For the remaining 58 autistic children, the app suggested that they need further evaluation.
Of the 71 non-autistic children who received a result, 15 were incorrectly identified as being autistic. All 15 had a different state of development; 5 was diagnosed as autistic by one of the three experts.
Cognoa plans to release the results before the app launches, Taraman says.
The technology enables pediatricians to diagnose autism in a subset of children who would otherwise have to wait for a specialist, he says.
“Pediatricians are more than capable of diagnosing a good proportion of children with autism,” he says. “You just need the right tools.”
But Goodwin has a number of concerns: the app has missed autism in many cases, which could delay care for children with autism if their parents do not seek further evaluation; much of Cognoa’s research on the app was not published in autism journals, meaning the peer reviewers had no experience of diagnosing autism; and in his view, the company was not clear about the severity of the study participants’ characteristics, making it difficult to gauge how effective the app is at distinguishing autism from other conditions such as attention deficit hyperactivity disorder.
“I haven’t seen any evidence-based demonstration to suggest that Cognoa is better than the gold standard, cheaper than the gold standard, more accurate than the gold standard, or more scalable than the gold standard,” says Goodwin.
Uncertain future:
The approval raises questions about where technologies like Cognoa’s should fit into the diagnostic process, says Stephen Sheinkopf, associate professor of psychiatry and human behavior at Brown University in Providence, Rhode Island.
“Given the performance previously published by Cognoa, this appears to be a potential second level screening tool. There are other examples of such tools in the literature and if used properly they could help improve access to care, ”he says. “But I would be wary of using this tool as a formal diagnostic test right now.”
While the Cognoa app could theoretically help reduce waiting times for diagnosis, it cannot solve the underlying problems with access to care, Sheinkopf adds.
“Scientifically rigorous and advanced machine learning approaches are of great value,” he says. “These are going to be very important tools for us, but we also need to think about the basics, and the basics are that we need well-trained people who are properly paid to look after the families we see in the clinic.”
Cognoa has not yet set a price for the system, but is working to make sure it is covered by health insurance, Taraman says.
Much more data needs to be gathered on its effectiveness before families should pay the costs, Goodwin said. “I don’t want to say that this is a bad approach that shouldn’t be taken and shut everything down. But it is far too early to commercialize for profit and incriminate families for lack of evidence. “
Quote this article: https://doi.org/10.53053/XHTV4627
