Detecting Autism Through Machine Learning On Home Videos Of Children

Autism Spectrum Disorder is one of the fastest growing developmental delays and has risen by 700% since 1996, now impacting 1 in 40 children in the United States. This causes a significant financial burden on families, with individual costs going up to $60,000 per year to support someone diagnosed with autism.

The current method of diagnosis for autism is based on observation and parental interview and involves several in-person appointments with trained clinicians who measure between 20-100 behaviors. These appointments can last several hours and therefore, with the growing demand for diagnostic resources for autism, have resulted in long waitlists. This causes a significant delay in diagnosis for several children, and, in some cases, the children are not diagnosed until after the age at which interventions can be most effective.

Through machine learning and large scale data mining, the Wall Lab, with Principal Investigator Dr. Dennis Wall, has built a mobile system that quantifies and tracks the severity of autism spectrum disorder. This can be a scalable, low-cost, and effective solution to break the diagnosis bottleneck and have a global impact across cultures and different development delays.

Standard diagnostic instruments for autism include the Autism Diagnostic Observation Schedule (ADOS) and the Autism Diagnostic Interview-Revised (ADI-R). In previous work, Wall’s team collected clinical scoresheets from ADOS and ADI-R and by performing feature reduction, determined the minimal number of features required in order to reach an accurate diagnosis. This work has resulted in 8 unique machine learning models, several of which have been independently validated by the Wall Team using external data as well as by other researchers using distinct cohorts of children with and without autism, showing balanced measures of accuracy consistently above 80% and often above 90%. In aggregate these models use 30 features that the Wall group hypothesized could be measured, or “tagged,” accurately by untrained individuals in a short home video under 3 minutes in length and often as short as 60 seconds, depicting the child at risk in his or her natural setting.

The video raters, chosen deliberately to be non-expert, namely without explicit training in the field and not certified for clinical practice in developmental delay detection, were given minimal video feature analysis training by the Wall Lab’s clinical managers and the task of accessing the videos through a secure portal and rating the 30 features on 162 videos of American children. The features tagged by these nonexpert video analysts included social interaction, interaction with objects, and speech-language traits. The videos were collected through online crowdsourcing or through public data sources such as YouTube. The independent tagging measurements generated by nonexpert raters on the 30 features were delivered as input to 8 validated classifiers in order to prospectively validate them on this new feature set from video data. The features being input to these classifiers were different from the features they were trained on for various reasons.

The Wall team tested and determined that three video raters, blind to the diagnosis and working independently on each video, would be sufficient to reach majority rules consensus on the diagnosis for the child in question. With this minimally viable “crowd” of nonexpert raters, the Wall team ran a prospective home video analysis on 116 children with autism ages 2-8 and 46 neurotypical children ages 2-6. The median time to complete the 30 feature tagging task was 4 minutes, resulting, in that short period of time, in 3 unique and independent feature vectors enriched for the child’s natural behavior for use in model computation.

Although several models performed well, a sparse 5-feature Logistic Regression classifier yielded the highest accuracy of 92% across all ages tested, importantly including children at and under 2 years of age. The Wall team then used a separate and independent prospective dataset of 66 videos (33 ASD and 33 non-ASD) with the same 30 features tagged on them by 3 new nonexpert raters’ measurements to validate the outcome, achieving slightly lower but still comparable accuracy at 89%.

Image republished from PLOS Medicine from

This mobile machine learning process for diagnosis can accelerate the detection and enable triage to ameliorate the diagnostic bottleneck but can also instantiate an “action-to-data” feedback loop by enabling point-of-care diagnostics at scale while building feature labels on images that can be used later for deep learning model development. Such labeled image data could have even greater value for detection and tracking of autism over time. The work also suggests that non-experts working virtually — a crowd workforce — may enable rapid and economical feature analysis without compromising accuracy. Compliance, structure, and volume are essential for scaling video analysis of the kind described here to the global population in need of initial detection and in need of being quantitatively tracked during therapy. The Wall Lab project Guess What is a mobile video game that encourages and records social interactions between child and caregiver enriched for the features needed by our classifiers. As a record-while-you-play style of game for both Android and iOS, it has the potential to facilitate video collection at high volume with little to no technical barriers. For developing countries, this screening system may be particularly useful.

To this end, the Wall lab has recently completed the first phases of a follow-on project, recently published in JMIR, which shows the ability for the mobile video and machine learning processes to work on children at risk for a host of developmental delays, well under 4 years of age and native to Bangladesh, a developing country with few pediatric health resources. This expansion and adaptation of the prior work show that such an approach is culturally scalable, can help with the diagnosis of other developmental delays, and provide a vehicle for mapping the prevalence of developmental disorders across the country.

These findings are described in the article entitled Mobile detection of autism through machine learning on home video: A development and prospective validation study, recently published in the journal PLOS Medicine.

About The Author

Qandeel Tariq

Qandeel Tariq is a research data scientist at Stanford University School of Medicine. She holds a Master’s degree from Stanford University in Learning, Design, and Technology and has published more than 5 papers in the domain of Artificial Intelligence and autism. She is currently working on the video phenotyping project where she is developing machine learning methods to improve the screening process for autism while testing scalable methods for reliable annotation using crowdsourcing and automatic extraction using computer vision.

Dennis P. Wall

Dennis P. Wall, Ph.D., received his Ph.D. in Integrative Biology from the University of California, Berkeley and a National Science Foundation postdoctoral fellowship in Computational Genetics at Stanford University before joining the faculty at Harvard Medical School. He is now Associate Professor of Pediatrics, Psychiatry and Biomedical Data Sciences at Stanford Medical School. He leads a lab in Pediatric Innovation focused on developing methods in biomedical informatics to disentangle complex conditions that originate in childhood and perpetuate through the life course, including autism and related developmental delays. Dr. Wall has innovated, adapted and deployed bioinformatic strategies to enable precise and personalized interpretation of high resolution genetic, microbiological and phenotypic data. Dr. Wall has pioneered the use of machine learning and artificial intelligence for fast, quantitative and mobile detection of neurodevelopmental disorders in children, as well as the use of machine learning systems on wearable devices, such as Google Glass, for real-time “exclinical" therapy. These same precision health approaches enable quantitative tracking of progress during treatment throughout an individual’s life enabling big data generation of a type and scale never before possible and have defined a new paradigm for behavioral detection and therapy. Dr. Wall has founded several companies, including, serves as advisor on several others in working on genomic and precision health, and has won several awards including a spot in the top ten of the World’s top 30 autism researchers, the Vice Chancellor's Award for Research, three awards for excellence in teaching, the Harvard Medical School Leadership award, and the Slifka/Ritvo Clinical Innovation in Autism Research Award for outstanding advancements in clinical translation.

Speak Your Mind!


Two Cases Of Unusual Migraine Auras

Neurological symptoms during sex need urgent tests to look for a serious cause such as a bleed on the brain.  “Thunderclap” headaches, a sudden severe headache like a blow to the head, raise alarm bells and can happen at the time of orgasm. People who experience this headache need to go to the emergency department for […]

The Search For Novel Inexpensive And Efficient Non-Noble Catalysts

In the past few decades, the rapid consumption of non-renewable fossil fuels has caused significant environmental and social problems on a global scale. Renewable energy such as wind power, solar energy, and tides have been greatly developed and widely applied, and other efficient energy sources with attractive performance, low cost, and environmental benefits are in […]

Strategies For Encouraging Solar Technology Adoption Through Public Engagement And Visual Exposure

Globally, electricity systems are undergoing a significant transformation to green energy in response to technology change and climate policy. Yet, in some countries with high renewable energy potential, the transition is hardly evident. In Canada, for example, installed solar capacity is approximately 1.0% of total capacity in 2014 (National Energy Board, 2016, p. 81). This […]

Diverse Microbial Communities Aid In Pollution Control For Coastal Environments

High-latitude coastal regions are increasingly suffering the effects of human pressure, either from local activities or as a result of global-scale effects. Demographic expansion, resource overexploitation, habitat degradation, pollutant release on land and at sea, and climate change are rapidly affecting these environments with unpredictable effects. Oil exploration and exploitation in Arctic and Subantarctic regions, […]

Nitric Oxide And Hydrogen Sulfide Cross-Talk For Blood Vessel Wellness

The importance of the gaseous transmitter nitric oxide (NO) was definitely recognized by the 1998 Nobel Prize for Medicine/Physiology. Depending on its concentration, NO controls vessel tone and blood fluidity, oxidative stress, inflammation and body defenses, and growth or death of different types of cells, including cancer cells. Due to its wide range of properties, […]

Changes To Barrier Island Systems During The Holocene

Barrier islands systems, briefly described as long and relatively narrow sandy islands parallel to the shore and separated from it by a backbarrier lagoon, represent almost 10% of all continental shorelines. Most of the US eastern coastline features such systems, developing long strings of barrier islands and spits, such as the Outer Banks (North Carolina) […]

Potential Therapeutic Relevance For TIM-3 In Breast Cancer Patients

Since the first FDA approval of an immune checkpoint blockade (ICB) therapy in 2011, immunotherapy has increasingly become a standard-of-care treatment for cancer. ICB can work exceedingly well in a subset of patients, but disparities in response rates exist, particularly between tumor types (1). The other major factor predicting response to ICB is the extent […]