#Product Trends
An Augmented Reality App to Help Children with Autism
Computer software company PTC in partnership with the Boston Children’s Hospital has developed a prototype of an augmented reality (AR) application to assist learning and communication for children with autism.
The results of the clinical tests show a tremendous improvement for these children in terms of attention to the activity, speaking in three-word phrases, participating in pretend play, or following single-step directives.
During the LiveWorx20 digital transformation conference organized exceptionally on a virtual basis this year due to Covid-19, Dr Howard Shane, Director of the Center for Communication Enhancement at the Boston Children’s Hospital, explained that children with Autism spectrum disorder (ASD) have little or no functional speech but a strong interest in electronic screen media:
“If you look at their development, you’ll find that they had very limited early speech sounds: These were children who were pretty quiet and that’s often one of the first indicators that something is going wrong with their development. These children also have a really strong predilection for what is visual. The internet is a great attraction to them. They love computer graphics, animations, video clips, etc. Along with that, they have extraordinary difficulty comprehending spoken language. Much of their learning takes place by watching. They have a strong visual process for learning compared to auditory.”
He added:
“The visual channel is a way to rebuild their spoken language capabilities and their abilities to understand. So the augmented reality technology presents us with a really fascinating opportunity.”
A Multisensory App
In 2018, a team of PTC employees began doing volunteer work to understand autism and the requirements to create an effective AR solution. They came up with a prototype of an AR application that would provide a multisensory experience to the user.
They devised a virtual replica of a traditional children’s toy: a farmhouse with different characters such as a boy, a horse, a barn, a chicken, an egg, etc. This toy allows you to take animals, move them in and out of the farmhouse, and so on. From a tablet, the user can overlay the AR technology on top of it.
Once it’s locked onto the farmhouse, the AR animation reveals some verbs, nouns and prepositions associated with visual cues that the child can see. The child can create a three-word sentence using the icons, then the AR animation shows the result of what they have created. For example, if the result is “the boy riding the horse,” a synthetic voice gives, at the same time, an auditory cue: “The boy rides the horse.” It turns into something that children with autism can really understand.
Christina Yu, Speech Language Pathologist at the Boston’s Children Hospital, explained:
“If we give this toy to a child with autism without any visual cues or instruction, he may not necessarily interact with it appropriately. He may just pick up the various characters and not put the chicken next to the eggs or make the boy ride the horse. But with these visual supports using AR, children are able to process the visual information and interact with the toy appropriately.”
Don Busiek, Senior Vice President, Strategic Alliances at PTC, said:
“PTC has been building up the prototype, the Children’s Hospital’s team has been working on some clinical trials and we have been updating the app based on the clinical evaluation feedback. Now there’s a transition from the volunteer team to full-time employees working on building up the next-generation prototype.”
The AR app could be applied to a variety of objects.
Clinical Tests
The Boston Children’s Hospital’s team led by Dr Shane and Christina Yu did a series of clinical tests. One of them was made with Jessica, a six-year-old girl with Autism spectrum disorder. The team presented the toy to her without the AR app and with the AR app.
Without the AR app, she attended to the activity for a very limited amount of time. She was able to label the nouns such as boy, cow and pig but she was not able to combine words into full phrases. So she wasn’t able to say the boy rides the horse. In terms of pretend play, she did one single step: She put the boy in the barn but she wasn’t able to do any of the other pretend play steps that a typical six-year-old would do. Finally, she was unable to follow a single-step directive such as “Can you put the chicken next to the nest?”
When they presented the AR application, she attended significantly longer to the activity. She didn’t get up once and she was able to label the nouns. She selected each of the icons and she repeated after the voice output. She was able to speak in two to three-word phrases, which she previously could not do, then she sequenced three icons together to say “the pig is in the barn.” She was able to repeat this three-word sentence aloud.
Christina Yu said:
“This is really significant because we have been working on two to three-word phrases for over a year now in speech therapy, and it just really hasn’t stuck with her. Her parents have been working tremendously hard on it at home too. So this app provides a medium for her to stay interested and attended. And we know that when a child is attending to an activity and is more interested, they process information and learn the information more quickly.”
Regarding pretend play, it doesn’t come naturally to kids with autism, Christina Yu said. And here with the augmented reality, Jessica was able to participate in numerous pretend play opportunities. Finally, in terms of single-step directives, usually, if the doctor would say “Can you put the boy next to the horse?” Jessica would not be able to do that. But with the visual mode using AR, she completed all of the single-step directives.
Dr Shane said:
“Given our history of working with technology and children with a whole range of developmental disabilities, the AR app has been key in helping them improve their capabilities. We’re beginning to see that this is a technology that can be a critical differentiator.”
According to the Centers for Disease Control and Prevention (CDC), 1 in 88 children has some form of ASD in the United States—a 78 percent increase in comparison to a decade ago.
