INTRODUCTION
Virtual environments (VEs) have been increasingly used in education, skill training, gaming, and the study of human behavior [1,2]. They offer novel options in the neuropsychological assessment and rehabilitation of many cognitive processes, including attention deficit hyperactivity disorder (ADHD) [1], anxiety disorder [3], autism spectrum disorder (ASD) [4], and obsessive compulsive disorder [5]. One of key advantages to applying VEs in neuropsychological cognitive processing research is their ability to provide natural and realistic testing environments [3,6,7] that have ecological validity in which participants can interact dynamically [8,9]. Along with the ecological validity, the VEs also allow researchers to control whole situations and environments, which could be beneficial in scientific analyses of human behavior [7]. Other VE advantages are cost reductions and higher participant motivations compared with classical methods [7,10,11].
One well-known example of VE applications for the neuropsychological assessment is a virtual classroom that simulated a real classroom for school-aged children in order to evaluate attention performance [1,12]. School-aged children spend a lot of time in classrooms, where they must be attentive to classroom tasks. Virtual classrooms are similar to real classrooms and immersive for school-aged children to help them learn to pay attention in their real classrooms [13]. Virtual classrooms have high ecological validity, and student participants enjoyed training more in the VE than they did in a clinician’s office or hospital, which are tedious and unfamiliar settings that evoked a negative reaction in children [14].
In a previous study, Rizzo et al. [3] compared the pattern of attention between an ADHD group and a healthy control group using an AX version of continuous performance tasks (AX-CPTs) in a virtual classroom, which they found to be a reliable method for assessing attention in school-aged children. Pollak et al. [14] compared the effects of methylphenidate (MPH), a medication used to treat ADHD, in a virtual classroom task and a traditional task and concluded that the virtual class room task was the more sensitive measure for MPH effects. They also measured participants’ feelings during the assessment tasks using a self-reported scale and found that participants experienced the content in the virtual classroom to be more immersive and entertaining than the computerized traditional neuropsychological assessment. Díaz-Orueta et al. analyzed correlations between a virtual classroom task and the Conners’ Continuous Performance Test [15]. They showed convergent validity between the virtual classroom and traditional measures [16]. Along with those studies, the use of a virtual classroom has extended beyond ADHD research to study of other neuropsychological disorders including social anxiety disorder (SAD), and higher functioning autism spectrum disorder (HFASD) [3,17].
Previous virtual classrooms had good ecological validity, but they had limited components for social attention (i.e., joint attention: interaction between a virtual teacher and the participating student). We believe that incorporating joint attention (JA) into the VE will improve the ecological validity of virtual classrooms. JA, a social interaction concept, has been defined as the ability to coordinate with others and share attention to a common object [18]. JA is a method by which individuals generate shared focus on an object/person. Several studies have shown that JA plays a key role in the development of language and learning [19-21], and some argue that a lack of this skill leads to difficulty in social interactions [22]. JA is used as a key concept in various social interaction scenarios [23,24] and is vital to social competence at all ages. Adolescents and adults who cannot follow, initiate, or join in rapid changes in shared attention during social interactions can be impaired in their capacity for relatedness and relationships [25-27]. Therefore, our use of JA in this study will enhance the ecological validity of social interactions in virtual classrooms.
The objective in the current study is the development of a joint attention virtual reality classroom (JA-VR classroom) to evaluate attention in social situations between participants and a virtual avatar teacher. For this objective, we included an traditional AX-CPT, a VR control condition, and two levels of JA conditions. We tested two hypotheses. The first hypothesis is that JA affects the attention of participants during a neuropsychological assessment in the JA-VR classroom. The second hypothesis is that the dependent measures of the new JA-VR classroom program will have correlations with those of previous measures, including a VR classroom without a JA component.
METHODS
Pilot study
Before starting our main study, we developed an initial version of the JA-VR classroom and tested the level of task difficulty with 25 pilot participants (24.0% female, n=6; age: M=24.9, SD=3.21). We used a within-subjects design, counterbalanced for order across participants. During the pilot study, each participant was seated and asked to complete AX-CPTs in three conditions [28]. In the first condition, the virtual classroom contained a virtual teacher who did not interact with participants (VR control condition). In the second condition, the virtual teacher only pointed to the left or right screen, and participants were asked to follow his directions (low responding JA; low RJA condition). In the third condition, the virtual teacher said “look at me,” “look at this” as he pointed to each screen, and participants were asked to follow his direction (high RJA condition). As previous virtual classroom studies [29-31], participants performed one button AX-CPTs. Participants only response to the valid stimuli ‘X’ after ‘A’. In each condition, 120 stimuli were presented, 32 target stimuli and 88 non-target stimuli. All stimuli were presented for 200 ms, and intertrial intervals were varied from 1,500 ms to 2,500 ms.
We conducted ANOVAs to compare the dependent measures in each condition. We found no significant difference in accuracy, p>0.84, commission errors, p>0.39, omission errors, p>0.84, or response time, p>0.37. Participants had accuracies greater than 99% in all conditions (VR control: 99.2%; low RJA: 99.1%; high RJA: 99.0%). This pilot study suggested an important implication. The current version of the CPT had ceiling effects in measuring the attention abilities of healthy participants. Previous studies have also suggested the ceiling effects [29-31], therefore, we decided to revise the task. We revised three things from previous virtual classroom. First, we increased the level of complexity of the CPTs in all conditions in order to reduce the ceiling effect (details below). Second, we measured head telemetries with time stamps, which enabled us to analyze the head movements of participants during whole sequences. Third, we included an additional control trial to measure CPT performance characteristics without any VE or virtual avatar components.
Participants
This protocol was approved by the University Institutional Review Board prior to recruitment (IRB number: HYI-14-100-1). Participants were university students recruited through advertisements on campus. The 33 participants (39.40% female, n=13) provided consent to participate (age: M=23.14, SD=4.6) for the main study. No participants reported being currently prescribed any psychiatric medication, and none of the participants had major psychiatric problems, as assessed using the Symptom-Checklist-90-R [32].
Hardware and software
We created the VR paradigm for this study using a 3D development platform (Vizard 5.1; WorldViz, Santa Barbara, CA, USA). Our VR classroom was implemented with a stereo headmounted display system (HMD; Oculus lift DK 2, Oculus VR, Irvine, CA, USA) with a 100° field of view (screen resolution was 960×1,080 per eye). During the experiment, we used adjustable headbands to fit the HMD on each participant’s head. A built-in three-degrees-of-freedom sensor tracked the participants’ head movements. The system was controlled by a desktop workstation running Windows 7 (Microsoft) equipped with a high-end graphics card (nVidia). A standard keyboard was used to record participants’ responses, and a standard speaker was used for auditory stimulation.
Modification of CPTs
To increase the difficulty of the attention task, we used a two-button AX-CPT [33,34]. Participants were needed to press the right arrow key only target stimuli that presented stimuli ‘X’ after seeing an ‘A.’ In response to non-target stimuli, participants were required to press the left arrow key. In addition to the changes of response buttons, we also changed the number of target stimuli and the stimuli length. All stimuli were presented for only 100 ms, and there were only 24 target stimuli. As in the pilot study, inter trial intervals were varied from 1,500 ms to 2,500 ms.
Task conditions
We added a new condition to the three that we used in the pilot study. In the new condition, participants were instructed to complete the two button AX-CPT on a white background without any virtual classroom or virtual teacher components. The task was thus the same as a traditional AX-CPT except it was conducted through an HMD. Therefore, participants had an equipment control (equipment control condition: same with a traditional AX-CPT except an equipment) to compare with the other three conditions.
In the present study, we designed a VE that integrated a virtual avatar and HMD along with the human participants’ JA in a virtual classroom. In all three VR conditions (VR control condition, low RJA condition, high RJA condition), participants were presented with the image of the upper body and face of a male avatar (Figure 1) and asked to complete CPTs that appeared to the right and left screens of the avatar. In other words, participants saw a male virtual teacher standing behind a teaching desk in the middle of a white board, with projection screens at the teacher’s left and right sides. The sitting students were instructed to fixate first on the virtual teacher, and then their view changed according to their head movements.
As in the pilot study, the three VR conditions differed in how the participants were directed to view the screens to the right and left of the teacher avatar (Figure 1). In the first VR condition (VR control condition), participants were instructed to complete the AX-CPT without any interaction with the teacher avatar. Participants simply conducted half of the trials on the left and the other half on the right screen, according to the program’s guidelines, which counterbalanced the order of screens across participants. In the second VR condition, the virtual avatar teacher chose the left or right screen and directed participants to look at that screen by pointing (low RJA condition). The screens changed 12 times during this condition, and other task variables remained the same as in the other conditions. In the third VR condition, the teacher avatar both pointed and said, “look at me” when he wanted the user to attend to him and “look at this” when he changed screens (high RJA condition). Again, the screens changed 12 times, and all other details were the same as in the other conditions.
Dependent measures
The dependent measures in this study were accuracy, commission errors, omission errors, response time, response time variability, and total head movements. Accuracy was the number of correct responses for target and non-target stimuli, commission errors were the number of incorrect responses toward stimuli, and omission errors were the number of missing responses toward stimuli. Because the sum of correct trials (accuracy), commission and omission errors is the total number of trials, we analyzed only commission and omission errors. To index participants’ head movements, we analyzed their head-telemetries. The human head has three rotation axes (yaw, pitch, and roll). To quantify those movements, we sampled current frame values every 1/60 second and subtracted them from previous frame values. The sum of those differences was defined as the total head movement (THM) in degrees (°). The formulation for THM is as follows:
Participants needed to turn from screen to screen 12 times in the low and high RJA conditions, but they needed only a single head turn in the VR control condition, while the task in the equipment control condition required no head turns. To eliminate this variability, we marked all time-stamps in a log file and excluded the task demand head movements data from the THM. We called that THM without task demand head turns.
Procedures
Upon arrival at the research site, the participating adults were informed of the nature of the research, and they provided signed consent, following the university-approved IRB protocol. They were then asked to complete a brief questionnaire to gather data on their age, sex, and current psychiatric symptoms. Participants were then seated on a chair, and the experimenter helped them put on the HMD. They adjusted their headband and completed a practice session of examples of all conditions so participants were familiar with each condition. The setup and practice session required 10–15 min per participant.
Following the practice session, participants were presented with the four conditions. The order of conditions was counterbalanced across participants to control for systematic order effects. To prevent potential motion sicknesses, participants rested 5 min after each condition, and seated on a chair in the whole tasks. After completion of all conditions, participants were debriefed about the purpose of the study. The experiment lasted for -60 minutes.
RESULTS
We conducted ANOVAs to compare the dependent measures across conditions (Table 1). We found a statistically significant difference in commission errors, F (3, 96)=3.097, p=0.030, η2=0.088. Post-hoc analyses suggested a significant difference between the equipment control condition and the high RJA condition, t (32)=2.652, p=0.012, and between the VR control condition and high RJA condition, t (32)=2.197, p=0.035 (Figure 2). Response time also showed a statistically significant difference F (3, 96)=3.634, p=0.016, η2=0.102. Post-hoc analyses suggested significant differences between the equipment control condition and VR control condition, t (32)=2.903, p=0.007, the equipment control condition and low RJA condition, t (32)=2.113, p=0.043, and the equipment control condition and high RJA condition, t (32)=2.446, p=0.020. We found no significant differences in omission errors, p=0.46, or response time variability, p=0.17, across conditions.
Because the equipment control condition was same with the traditional AX-CPT, we first conducted correlation analyses between equipment control and other VR conditions. Result suggested that there was significant correlations in commission errors (all ps<0.001), response time (all ps<0.001), and response time variability (all ps<0.001). In omission errors, we found significant positive correlations of equipment control condition with VR control and high RJA conditions (all ps<0.001), but low RJA condition was not significantly correlated with equipment control condition (p=0.825). Our correlation analysis also found significant consistency in the patterns of individual differences across the VR conditions (VR control condition, low RJA condition, and high RJA condition) with regard to commission errors (all ps<0.004), response time (all ps<0.001), and response time variability (all ps<0.001). In omission errors, we found significantly positive correlations between VR control condition and high RJA condition (p=0.001), but the low RJA condition was not significantly correlated with other conditions (all ps>0.291).
We conducted an ANOVA to compare the THM in each condition (Table 2). We excluded one of the 33 participants because of a data saving error in the head-telemetry system, so the final THM sample included 32 participants. We found a statistically significant difference in THM, F (3, 93)=260.884, p=0.001, η2=0.882 (Figure 3). Post-hoc analyses suggested strong significant differences in all conditions (all ps<0.001) except between the low RJA condition and high RJA condition, p=0.448. To exclude the head turns related to each task condition, we also compared the THM without task demand head turns and found a statistically significant difference across conditions, F (3, 93)=5.526, p=0.002, η2=0.151. Post-hoc analyses suggested strong significant differences between the equipment control condition and low RJA condition, t (31)=-2.287, p=0.007, the equipment control condition and high RJA condition, t (31)=-2.621, p=0.013, the VR control condition and low RJA condition, t (31)=-2.901, p=0.007, and the VR control condition and high RJA condition, t (31)=-2.522, p=0.017 (Figure 3). We found no significant differences between the equipment control condition and VR control condition, p=0.824, or between the low RJA condition and high RJA condition, p=0.656.
DISCUSSION
We developed a JA-VR classroom and conducted initial experiments with healthy adults. We had two initial hypotheses: that JA affected the attention of participants in the JA-VR classroom, and that the parameters of the JA-VR classroom are correlated with those of previous assessments. The first hypothesis was confirmed. The results of the main study suggest that the higher JA condition did better promote attention processing than the lower JA conditions in terms of commission errors and response time. The second hypothesis was also confirmed. The dependent measures of commission errors, response time, and response time variability had high correlations across conditions, and omission errors also had correlations with some conditions.
In this study, we added an important core concept to virtual classroom and social interaction research. Previous studies have suggested that virtual classrooms had a unique ecological value for school-aged children [1,13,35], and that virtual classroom distractors negatively affected to attentional processing [3,16]. Following those studies, we proposed a JA-VR classroom in which participants could share joint attention with a virtual teacher and showed that the virtual social interaction promoted the attentional processes in healthy adult participants. Merging our results with those of previous studies, we found that virtual classrooms have ecological validity [3,13,14,16], provide a way to suppress attention using non-social distractors [3,16], and offer a way to promote attentional processing using JA with a virtual avatar teacher.
We found both commonalities and differences between the classical VR classroom and our new JA-VR classroom. As we hypothesized, the classical VR classroom parameters were highly correlated with those of both the JA-VR classroom and the equipment control condition, reflecting the effects of the CPT task performed in each condition. In previous studies, the results from a virtual classroom attention task were highly correlated with those from traditional tools [16,36], which accords with our current result. Another common point between the VR classroom and the new JA-VR classroom is controllability. One important advantage of VR is the ability to easily control environments, sounds, and tasks [7,10]. In addition to those advantages, our JA-VR classroom includes the possibility of sharing joint attention with a virtual teacher avatar, allowing researchers to control the level of social interaction within an environment.
One difference between the classical VR and new JA-VR classrooms is the effects of head movements. We found significant differences in head movements across conditions, which we attribute to a JA effect. In a JA situation, participants often need to look back and forth between the target and the interaction partner. Classical VR classrooms do not require that back and forth movement. This difference generated more total head movements in the JA-VR classroom, which could be correlated with social interaction traits. In previous studies, head movements were considered an indicator of participants’ distraction levels [3,16,36] or psycho-pathological problems [1,36-38]. However, head movements could instead be considered social interaction markers, as in our JA-VR classroom. Another difference between the classical VR and new JA-VR classrooms is attention processing. As described above, participants had fewer commission errors in the JA-VR conditions than in the VR control condition, which could indicate that JA promotes attentional processing. By merging our response time results and those of previous studies between a VR classroom and a computer-monitor-based measure [3], we found that the VR environment and the JA required in the JA-VR classroom have separate effects.
The current JA-VR classroom could be extended to clinical populations with ASD and ADHD. JA is a core element of social attention, and a previous study has suggested that JA impairments are linked with the core symptoms of ASD [17]. For school-aged ASD children, the current JA-VR classroom might be an important neuropsychological assessment that includes interactions with a teacher. The JA-VR classroom could also be useful for children with ADHD. Previous studies have suggested that a virtual classroom was a valid tool for school-aged children with ADHD [3,39]. Our JA-VR classroom extends the usefulness of that tool to social interactions among children with ADHD.
Our study had some limitations that should be addressed in subsequent studies. First, the analysis was restricted because of the small number and limited measures of participants. For example, gender and intelligence quotient effects should be evaluated in subsequent studies. Second, experiments using interactions with a female avatar partner would be beneficial. We only had a male teacher avatar, and that limited our subsequent analyses on the sexes of the interaction partner and participants. Third, we need to mention the ceiling effect for omission errors. Although we increased the difficulty of the CPT in the main study, the omission errors still showed a ceiling effect not only in the ANOVA analysis, but also in the bivariate correlation analyses. Therefore, we might need to consider other high level attention measures in our follow-up studies. Fourth, we assessed head telemetry, but humans can move their eyes without head movements. Therefore, eye and head movements need to be examined together. Fifth, the participants in the present study were all university students. The reaction to a virtual classroom might differ according to age group. Therefore, younger and older groups should be included in future studies. Finally, we need to investigate patients with pathologies such as ADHD and ASD. It would be beneficial to verify an effect of JA in each pathology.
Conclusions
In this study, we suggested a new JA-VR classroom and measured participants’ attentional processing. We found that JA promoted attentional processing among participants, and we showed convergent validity between our JA-VR classroom and previous measures. We hope these results contribute to new methods, new questions, and renewed enthusiasm for virtual classroom research. This work is an important foundation for the study of social interactions in school life, and we hope it will ultimately help people with and without handicaps in social attention.
