A girl woke up in the middle of the night around 3:00AM, as she did most nights. As she rubbed her eyes and mustered the strength to get out of bed and head to the bathroom, she noticed something eerie. Her dog was sitting up at the end of the bed staring into the long, dark, hallway. She peered down the hallway, focusing on the same point as the dog, fearing that she may find an intruder. To her relief, she spotted her cat, head down, tail in the air, ready to pounce.
This story may sound mundane, but some people with autism may not recognize the dog’s locked eyes as a cue to look down the hallway. Most people respond to eye gaze cues every day without thinking about it. They may thank a waiter for a drink before it’s on the table, feel uncomfortable as someone watches over their shoulder as they converse, or silently share a knowing glance with a coworker in a meeting. In these situations, no one directly says, “someone just asked that question two minutes ago,” in the middle of a meeting; they communicate it with their eyes.
Teaching someone to understand eye gaze cues can be challenging. But Suzanne Scherf, a psychologist and neuroscientist at The Pennsylvania State University, found a way to make learning a mundane task fun with video games. She designed a detective game with a cast of cute critters that teaches children with autism how to read eye gaze cues and recognize facial expressions (1-2).
How do we normally interpret facial expressions?
An ongoing debate amongst face recognition experts is whether the brain develops specialized computations for recognizing faces and dedicates tissues specifically for this purpose. There were some cool studies done where typically developing adults were trained to recognize novel objects created with 3D graphics to understand how the brain normally processes faces. Researchers opted not to use faces because people bring their whole history and experiences that might confound the research. The researchers created new objects that required the same type of processing demands as faces. They determined that the brain uses the same neural systems to process these objects as it does to process faces. These objects were individual little characters called greebles. People could learn and remember names for the individual characters. They understood that this greeble is Joe and not Bob, Mary, Sam, or anyone else they were introduced to. We thought, what if we tried to train kids on the autism spectrum to recognize greebles? Maybe this would tell us if their brains can still do this process, but it just struggles to do this process with faces.
How did you use greebles to understand how kids with autism process faces?
First, we determined that kids on the spectrum could learn to recognize these greeble characters. Then we wondered why they struggle to recognize faces. What is it about faces? We didn’t want to bring kids into the lab because this would be a huge burden for the families, so we created a home-based game intervention.
In the game, there was a greeble jewel thief who traveled the world stealing jewels (3). Ultimately, he stole a huge diamond, and the kids played as detectives tasked with catching him. We showed them pictures of the greeble, and they had to sort which greeble was which to find the jewel thief. It involved storytelling, simple pictures, and videos. It wasn’t anything elaborate. All the kids did was look at pictures and videos on the screen with a name and say if it was or was not the right match. They looked through 17,000 files in two and a half months. It was really an incredibly boring, mundane task, but the kids loved the story. They asked to do more. The sad part is that we found that it did nothing to improve how the kids recognized faces.
How did you modify the game to better address challenges with facial recognition?
When I became a new faculty member at The Pennsylvania State University, I wanted to do something that was really going to make a difference for kids with autism. I went back to the core phenotype of autism to try to understand what behavior that is part of the diagnosis snowballs and contributes to all kinds of social communicative problems. If we could support learning these skills early on, we may ultimately help kids in a lot of different ways, even ways we did not expect.
I focused on eye gaze. Recognizing eye gaze cues is a subtle, but complicated skill that is highly preserved across mammalian species. For example, dogs can follow human gazes, but it doesn’t necessarily mean that they understand why someone is looking at something else in the room or what they intend to communicate. The psychological connection part is not there. We designed a serious video game to help kids with autism make that psychological connection (1). We listened to what the families told us about the greeble game. They loved the narrative, so we wanted to keep the detective theme. This time, we made it about a pet detective.
How does the pet detective game work?
Kids must identify the pet that they’re looking for, and then navigate their way through this whole game to find their individual pet. In the meantime, they perform small side tasks through a whole series of interactions with other pet avatars.
We very carefully and subtly trained the participants to interpret eye gazes to navigate the game. We never told the kids to look at the face of the avatar or pay attention to the eyes. There was nothing explicit about any of it. We present problems to solve, and to solve them, the player must pay attention to nonverbal behavior. Initially it was just nonverbal greetings and avatars turning their heads in a certain direction. As the player becomes more successful and moves through the game, it gets increasingly difficult. We had distractor items such as sparkling items that make the player look away from where the avatars eyes are looking. We made the items closer together so that the player had to determine exactly where the avatar looked rather than just the general region. We included multiple avatars, and the player had to find the object that held the gaze of both avatars. Not all of the avatar gestures point out objects. Sometimes the character moves because they are thinking or are about to tell the player something. In real life, we use movement for these reasons too. I wanted them to differentiate between these different types of social-communicative gestures.
Will you walk me through a scene in the game?
In one of my favorite scenes, the player enters a school and is directed to their locker. The lock on their locker is covered in gum, and a note instructs the player to go to the chemistry lab and talk to a teacher. The player goes to the chemistry lab and finds the teacher with all of her flasks in front of her. She greets them and then proceeds to use nonverbal cues such as pointing to a flask to tell the player how to develop a solution to remove the gum from their locker. For example, there may be only two objects on the table. The teacher will point her body, gaze, and point her finger to one of them, indicating that the player should click on the potion. When they click the potion, it flies up and pours a solution into a flask, and then a new object shows up on the table. Now there are three objects on the table and eventually four or five. The player goes through about twenty trials until the they’ve made a magic potion to pour onto their locker. They then find clues inside their locker about what to do next.
How has the pet detective game changed since your first iteration?
We’re on the second iteration now. In this version, the player is in a town filled with people busy at work doing different jobs. There is a cat in the town that helps people complete their jobs. For example, there is a farmer who grows vegetables for the town, but when birds come and eat his seeds, the cat scares away the birds. The cat disappears, and the townspeople are distraught. They hire the player to find the cat and help them do their jobs while the cat is away. The player helps different characters. For example, they help someone collect seashells on the beach so that she can make and sell jewelry. While the player helps the townspeople, they get hints from the cat saying that he is traveling. We wanted to appeal not only to younger kids, but also kids between ten and eighteen years old. We didn’t want it to be corny, so we made a strong narrative to keep the players motivated to find out what happens next.
How do you determine if the game helps the participants?
We recently presented some of our results at a conference. We measured the effectiveness of the game in a few different ways. The novel way is using eye-tracking metrics. We test how much kids have learned from the game by showing them photographs and videos of real humans. We also test their response to eye gaze cues using real people, including researchers and their parents. We focus on trying to understand where people looked and how that changed over time in response to stimuli that required the player to interpret an eye shift. We examine if they look at the person’s face or eyes at all to try to understand when we ask them to tell us what the person is looking at. Then we see if they land on the object the person is looking at, such as a coffee cup or a book. It’s complicated though. The eye tracking data alone don’t tell a story. We must consider the behavioral performance as well. In other words, when they get something right, what do they do with their eyes versus when they get something wrong.
Has your game improved the patients’ response to eye gaze cues and facial recognition?
We compared the results of our behavioral test between children with autism who played the game at home and another group who did not. The group who played the game showed an average increase of seven or eight percent improvement in performance while the control group showed no improvement over time. When we measured the behavior of kids who had a minimum dose of about ten hours of game play before and after playing the game, we saw a 12% increase in performance.
What do you hope the impact of your games will be?
We are not forcing folks on the spectrum to look at faces. We're not trying to encourage them to act in a particular communicative style. We're simply providing learning opportunities for them to understand how others use their eyes in a nonverbal way to provide real information about objects in the world. They're missing out on so much information if they can't understand that social communicative style. The goal here is to help them learn how to read that information so that they can decide what they want to do with it and how they want to act on it themselves.
References
- Scherf, K.S. et al. Improving sensitivity to eye gaze cues in autism using serious game technology: study protocol for a phase I randomised controlled trial. BMJ Open 8, e023682 (2018).
- White, E.M. et al. Designing Serious Game Interventions for Individuals with Autism. J Autism Dev Disord 45, 3820-3831 (2015).
- Scherf, K.S. et al. Atypical development of face and greeble recognition in autism. The Journal of Child Psychology and Psychiatry 49, 838-847 (2008).
