Visuo-Haptic-Based Multimodal Feedback Virtual Reality Solution to Improve Anxiety Symptoms: A Proof-of-Concept Study

Article information

Psychiatry Investig. 2019;16(2):167-171
Publication date (electronic) : 2019 February 21
doi :
1Department of Psychiatry, College of Medicine, Korea University, Seoul, Republic of Korea
2Department of Psychology, University of British Columbia, Vancouver, Canada
3Department of Computer Science and Engineering, Digital Experience Laboratory, Korea University, Seoul, Republic of Korea
4Department of Computer Science and Engineering, Pohang University of Science and Technology, Pohang, Republic of Korea
5Department of Psychology, Chung-Ang University, Seoul, Republic of Korea
6Department of Human Factors Engineering, Ulsan National Institute of Science and Technology, Ulsan, Republic of Korea
7Department of Software, Sungkyunkwan University, Suwon, Republic of Korea
8Department of Film & Multimedia, Korea National University of Arts, Seoul, Republic of Korea
Correspondence: Chul-Hyun Cho, MD, PhD Department of Psychiatry, Korea University Hospital, Korea University College of Medicine, 73 Inchon-ro, Seongbuk-gu, Seoul 02841, Republic of Korea Tel: +82-2-920-6272, Fax: +82-2-927-2836, E-mail:
Received 2018 August 1; Revised 2018 November 7; Accepted 2018 December 25.


With proper guidance, virtual reality (VR) can provide psychiatric therapeutic strategies within a simulated environment. The visuo-haptic-based multimodal feedback VR solution has been developed to improve anxiety symptoms through immersive experience and feedback. A proof-of-concept study was performed to investigate this VR solution. Nine subjects recently diagnosed with panic disorder were recruited, and seven of them eventually completed the trial. Two VR sessions were provided to each subject. Depression, anxiety, and VR sickness were evaluated before and after each session. Although there was no significant effect of the VR sessions on psychiatric symptoms, we could observe a trend of improvement in depression, anxiety, and VR sickness. The VR solution was effective in relieving subjective anxiety, especially in panic disorder without comorbidity. VR sickness decreased over time. This study is a new proof-of-concept trial to evaluate the therapeutic effect of VR solutions on anxiety symptoms using visuo-haptic-based multimodal feedback simultaneously.


Virtual reality (VR) can offer an interactive and immersive experience within an environment simulated by a computer, and create a computer-generated virtual situation which can be used for psychological treatment and psychoeducation [1,2]. The sense of presence or “being there” in VR is important for the stimulation of multiple senses and the active exploration of the virtual environment, and it can be facilitated through the use of technology such as head-mounted displays, synthesized sounds, and vibrotactile platforms [3]. A participant can be engaged with the VR environment in a more naturalistic and intuitive way [4,5].

Anxiety disorder is one of the common mental health problems [6]. Cognitive behavioral therapy (CBT), and specifically exposure therapy, is an effective treatment of anxiety disorders [7-9]. VR-based techniques can be applied to exposure therapy, as the sense of presence experienced in VR provides the opportunity to immerse the patient in the feared environment, tailored to match specific aspects of his or her fear structures, thereby activating those structures and allowing them to be modified [10]. Exposure therapy typically involves a patient repeatedly confronting the feared stimulus in a graded way, either in imagination or in vivo [11]. Although in vivo therapy can be expensive and impractical to conduct, VR allows the possibility of constructing inexpensive approaches and exposures that might be difficult to perform in vivo [10].

VR technology could thus offer an alternative way of performing exposure treatment [11]. CBT using VR exposure therapy has been shown to be effective in the treatment of anxiety disorders [12]. It allows the patient to be exposed to anxiety-generating stimuli and confront a feared experience in a safe and controlled way [13-15]. A situation simulated by VR, physically impossible to recreate in real life, can be therapeutically helpful, provided it is administered with proper guidance. It can be repeated many times, is available without time limits, and allows powerful adjustments of the treatment input. Various studies have applied VR in various ways: Not all have succeeded, but the consensus about the usefulness of VR-based therapy is increasing [16]. The feedback of physiological changes due to anxiety and psychological interventions through VR could constitute an immersive therapeutic method. VR could also be used to make relaxation easier in stressed or anxious patients, by visually presenting key relaxing images [17,18].

VR systems can merge visual, auditory and tactile perceptions of both virtual and real objects. The experiential effects can be maximized by simultaneously applying various types of visual, auditory, and tactile perceptions. In addition, using these techniques, it is possible to increase the possibility of achieving therapeutic effect through interactions within the VR environment, by measuring the user’s physiological changes and giving appropriate feedback in real time. This can be called a ‘visuo-haptic based multimodal feedback VR solution’ [19]. The aim of this pilot study was to develop a visuohaptic-based multimodal feedback VR solution and investigate, in a proof-of-concept study, the effectiveness of the system, especially on anxiety symptoms.



Subjects were recruited from the Psychiatry Clinic of the Korea University Hospital according to the following inclusion criteria: 1) recent diagnosis (within 6 months) of panic disorder according to the Mini-International Neuropsychiatric Interview [20]; 2) older than 19 years of age; 3) no evidence of organic mental disorders, intellectual disability, schizophrenia spectrum disorders, or substance use disorders; 4) no seizure or vestibular abnormalities. Among the nine subjects selected for the study, two dropped out due to personal reasons, and seven completed the study. The subjects signed a written informed consent. The study was approved by the Institutional Review Board of Korea University Hospital (2017 AN0126).

Study protocol of a visuo-haptic-based multimodal feedback VR solution

The visuo-haptic-based multimodal feedback VR solution system [19] consists of a computer to drive VR contents, an Oculus CV1 head-mounted display (Facebook Inc., Menlo Park, CA, USA), Leap Motion hand/finger tracking (Leap Motion Inc., San Francisco, CA, USA), a PSL-iECG2 heart rate monitor (Physiolab Co., Busan, South Korea), and a Falcon device for haptic feedback (Novint Technologies Inc., Albuquerque, NM, USA). The content was developed using Unity 3D (Unity Technologies, San Francisco, CA, USA).

The protocol was based on two VR sessions, one session per week. This VR solution consists of 1) an introductory part, to select one out of two scenarios: one shows a virtual party, while the other is set in an elevator; 2) a scenario part to mildly increase anxiety; 3) an intervention part with real-time heartbeat visuo-haptic feedback, with a breathing guide provided by narration, where the subjects can visually see their own heart beating and feel it through the haptic machine which synchronizes in real time with the subjects’ heart rate ( [20]. Subjects were instructed to experience both VR scenarios.

Psychiatric assessment

The following psychometric questionnaires were administered at baseline and immediately after each VR session: Montgomery–Åsberg Depression Rating Scale, Korean version [21], for measuring depressive mood; Hamilton Anxiety Rating Scale [22], Beck Anxiety Inventory (BAI) [23], and State-Trait Anxiety Inventory [24] for measuring objective or subjective anxiety; Simulator Sickness Questionnaire (SSQ) [25] for measuring the subjective discomfort caused by the VR experience. In addition, the subjects stated whether they had prior experience in VR.

Physiological measurement

Real-time heart rate was measured by a sensor in both forearms during the whole VR experience. Unfortunately, we found out that the heart rate log data of four subjects were not recorded properly, due to system errors.

Statistical analysis

The Wilcoxon signed-rank test, a non-parametric statistical test, was used to investigate the effect of VR sessions, by comparing psychiatric assessment scores between, before and after the VR sessions. Using repeated measures analysis of variance (ANOVA) on psychiatric assessments after VR sessions, the effect of the VR solutions was analyzed across groups and sessions. For the statistical analysis the subjects were classified by gender, comorbid psychiatric disease, and previous VR experience. All statistical tests were performed using SPSS, version 16 (SPSS Inc., Chicago, IL, USA).


The basic information and the raw data of several assessments for each subject are presented in Supplementary Table 1 (in the online-only Data Supplement). The mean age of the seven participants was 23.3 years [standard deviation (SD)=4.5]. Two of the seven participants were women. Although there was no significant effect of the VR sessions on the assessment scores, we could detect a trend of improvement in K-MADRS, HAM-A, and SSQ scores when compared before and after the VR sessions (Table 1). Comparing the groups with and without comorbid psychiatric disorders [n=5 and n=2, respectively, Supplementary Table 2 (in the online-only Data Supplement)], there was a significant decrease in the BAI score between sessions, and there was a significant interaction effect between the session and group factors. The comparison of the groups with and without previous VR experience [n=3 and n=4, respectively, Supplementary Table 2 (in the online-only Data Supplement)], showed a significant reduction of the SSQ score between sessions in both groups.

Results of comparison of psychiatric scales across haptic-based multimodal feedback VR sessions

Due to the inadequacy of the heart rate recorded data, it was difficult to find meaningful systematic results. Instead, a sample graph of measured heart rate changes in one patient during the VR session is shown in Figure 1.

Figure 1.

A sample graph of measured heart rate changes of one subject during a VR session. A: HR of one subject during a VR treatment session: the first 20 seconds represent the baseline HR before the start of the VR session (baseline HR), “Elevator” represents the experience of the VR elevator scenario, and MF represents the breathing guide with the visuo-haptic-based multimodal feedback system. There is a clear jump between baseline HR and Elevator. The heart rate decreases between Elevator and MF, and the heart rate variability is wider at MF. B: The dot represents the mean value of the heart rate, and the arrow represents its SD. Point 1 represents the mean and SD of heart rate at baseline, point 2 represents the same data at Elevator, and point 3 at MF. VR: virtual reality, HR: heart rate, MF: multimodal feedback, SD: standard deviation.


From this study, we were able to observe a tendency towards improvement of scales measuring depression (K-MADRS) and anxiety (HAM-A) as an effect of VR sessions. Although there were limitations such as the small sample size and the insufficient number of VR sessions, the present study is meaningful in showing that an immersive experience using a visuohaptic based multimodal feedback VR solution could have a therapeutic effect in controlling psychological discomfort. The subjective anxiety decreased after the VR session for panic patients both with and without comorbid psychiatric disorders, but in particular the group without comorbidities showed a more significant decrease than the comorbid group. This result suggests that the developed VR solution may be effective to relieve subjective anxiety, especially in the absence of other mood disorders. This is due to the fact that our VR solution was developed specifically for anxiety symptoms. Therefore, it can be assumed to be more effective in patients who had anxiety symptoms alone, rather than in those who had comorbid psychiatric disorders, in agreement with a previous study that reported insufficient treatment response in the comorbid groups [26].

VR sickness is the most common complaint related to VR use, and there have been various efforts to improve it [27]. Sometimes, such discomfort may be fatal to the therapeutic program, especially for people experiencing anxiety symptoms. In this study, we found that the subjective discomfort due to VR was decreased over time as the VR session progressed, according to the somatic symptoms questionnaire (SSQ) results. VR sickness can be somewhat problematic at first, but it could be gradually adapted to as the VR experience progresses [28,29]. VR sickness, of course, will vary depending on the each person’s susceptibility [30], but VR sessions are expected to become more tolerable and effective over time as VR sessions are repeated.

This study is a new trial to evaluate, as a proof-of-concept, the effectiveness of VR therapy by using physiological information and multimodal feedback simultaneously. Until now, most research related to the physiological information gained from VR was aimed only towards data acquisition and analysis. However, in this study, physiological (heart rate) information for simultaneous feedback was used as a part of the VR treatment system, through a visuo-haptic-based implementation.

There are several studies on VR treatment for mental health: For example, treatments for post-traumatic stress disorder (PTSD) and specific phobias are well-known. It is however much more challenging to develop a VR system for panic symptom than for PTSD and specific phobias. When an effective VR treatment system for anxiety disorder is developed, it will be effectively and powerfully used in many patients. Overall, almost all participants were satisfied with the VR solution, according to their subjective reports, and they reported willingness to use the VR solution in the future. Particularly, they mentioned that it was meaningful to experience visuo-haptically the decrease in their own heart rate obtained by following the breathing guide. Moreover, all participants were clinical patients, and the analysis of the therapeutic effects of VR sessions on these patients is the main strength of this study.

There are a few limitations to this study. Firstly, the sample size was too small. The results must be assessed while considering that it is a proof-of-concept study. The effectiveness of the VR sessions could become statistically reliable with a larger sample size. Secondly, the number of sessions was relatively small compared to traditional CBT treatment, which is designed to be carried out in 12–18 sessions of approximately 50–60 minutes [31]. This proof-of-concept study included only two sessions, which might not be enough to evaluate the effectiveness of the VR therapeutic intervention. Lastly, there were technical issues on the hardware system, so that the heart rate information was not fully recorded and thus could not be statistically analyzed.

It is necessary to devise a therapeutic approach fully exploiting those unique features of VR, which cannot be reproduced by any other approach. Far from being simply an application of new technologies, VR will be able to fill areas that are difficult or impossible to implement with other therapeutic approaches. In this regard, the visuo-haptic based multimodal feedback VR solution implemented and applied in this study will be relevant to future psychiatric applications. In the future, the development of a more complete and technically sound psychiatric VR solution, and related clinical research with a sufficient number of subjects and VR sessions, may suggest the possibility of new immersive therapeutic options.

Supplementary Materials

The online-only Data Supplement is available with this article at


This research was supported by the Brain Research Program through the National Research Foundation of Korea (NRF), funded by the Ministry of Science, ICT & Future Planning (NRF-2016M3C7A1914448) and ICT & Future Planning for convergent research in Development program for convergence R&D over Science and Technology Liberal Arts (NRF2017M3C1B6070978).


1. Gould NF, Holmes MK, Fantie BD, Luckenbaugh DA, Pine DS, Gould TD, et al. Performance on a virtual reality spatial memory navigation task in depressed patients. Am J Psychiatry 2007;164:516–519.
2. Parsons TD, Rizzo AA. Affective outcomes of virtual reality exposure therapy for anxiety and specific phobias: A meta-analysis. J Behav Ther Exp Psychiatry 2008;39:250–261.
3. Rothbaum BO, Hodges L, Kooper R. Virtual reality exposure therapy. J Psychother Pract Res 1997;6:219–226.
4. Brooks FP. What’s real about virtual reality? IEEE Computer Graph Appl 1999;19:16–27.
5. Anderson PL, Rothbaum BO, Hodges L. Virtual reality: Using the virtual world to improve quality of life in the real world. Bull Menninger Clin 2001;65:78–91.
6. Ormel J, VonKorff M, Ustun TB, Pini S, Korten A, Oldehinkel T. Common mental disorders and disability across cultures: results from the WHO Collaborative Study on Psychological Problems in General Health Care. JAMA 1994;272:1741–1748.
7. Butler AC, Chapman JE, Forman EM, Beck AT. The empirical status of cognitive-behavioral therapy: a review of meta-analyses. Clin Psychol Rev 2006;26:17–31.
8. Heimberg RG, Dodge CS, Hope DA, Kennedy CR, Zollo LJ, Becker RE. Cognitive behavioral group treatment for social phobia: Comparison with a credible placebo control. Cognit Ther Res 1990;14:1–23.
9. Foa EB, Kozak MJ. Emotional processing of fear: exposure to corrective information. Psychol Bull 1986;99:20–35.
10. Maples-Keller JL, Bunnell BE, Kim SJ, Rothbaum BO. The Use of Virtual Reality Technology in the Treatment of Anxiety and Other Psychiatric Disorders. Harv Rev Psychiatry 2017;25:103–113.
11. Gerardi M, Cukor J, Difede J, Rizzo A, Rothbaum BO. Virtual reality exposure therapy for post-traumatic stress disorder and other anxiety disorders. Curr Psychiatry Rep 2010;12:298–305.
12. Opriş D, Pintea S, García‐Palacios A, Botella C, Szamosközi Ş, David D. Virtual reality exposure therapy in anxiety disorders: a quantitative meta‐analysis. Depress Anxiety 2012;29:85–93.
13. Riva G. Virtual environments in clinical psychology. Psychother Theor Res Pract Train 2003;40:68–76.
14. Glantz K, Durlach NI, Barnett RC, Aviles WA. Virtual reality (VR) for psychotherapy: From the physical to the social environment. Psychother Theor Res Pract Train 1996;33:464–473.
15. Wiederhold BK, Wiederhold MD. Virtual Reality Therapy for Anxiety Disorders: Advances in Evaluation and Treatment Washington: American Psychological Association; 2005.
16. Wiederhold BK, Wiederhold MD. Virtual reality for posttraumatic stress disorder and stress inoculation training. J Cyber Ther Rehabil 2008;1:23–35.
17. Villani D, Riva F, Riva G. New technologies for relaxation: The role of presence. Int J Stress Manag 2007;14:260.
18. Freeman J, Lessiter J, Keogh E, Bond F, Chapman K. Relaxation Island: virtual, and really relaxing. Proceedings of 7th international workshop on presence 2004. 1467–72.
19. Seol E, Min S, Seo S, Jung S, Lee Y, Lee J, et al. Drop the beat: virtual reality based mindfulness and cognitive behavioral therapy for panic disorder---a pilot study. Proceedings of the 23rd ACM Symposium on Virtual Reality Software and Technology. ACM 2017. 57
20. Hergueta T, Baker R, Dunbar GC. The Mini-International Neuropsychiatric Interview (MINI): the development and validation of a structured diagnostic psychiatric interview for DSM-IV and ICD-10. J Clin Psychiatry 1998;59(Suppl 20):22–33.
21. Ahn YM, Lee KY, Yi JS, Kang MH, Kim DH, Kim JL, et al. A validation study of the Korean-version of the Montgomery-Asberg depression rating scale. J Korean Neuropsychiatr Assoc 2005;44:466–476.
22. Hamilton M. The assessment of anxiety states by rating. Br J Med Psychol 1959;32:50–55.
23. Beck AT, Epstein N, Brown G, Steer RA. An inventory for measuring clinical anxiety: psychometric properties. J Consult Clin Psychol 1988;56:893–897.
24. Spielberger CD, Gorsuch RL, Lushene RE. Manual for the State-Trait Anxiety Inventory Palo Alto: Consulting Psychologists Press; 1970.
25. Kennedy RS, Lane NE, Berbaum KS, Lilienthal MG. Simulator sickness questionnaire: An enhanced method for quantifying simulator sickness. Int J Aviat Psychol 1993;3:203–220.
26. van Balkom AJ, van Boeijen CA, Boeke AJ, van Oppen P, Kempe PT, van Dyck R. Comorbid depression, but not comorbid anxiety disorders, predicts poor outcome in anxiety disorders. Depress Anxiety 2008;25:408–415.
27. LaViola Jr JJ. A discussion of cybersickness in virtual environments. ACM SIGCHI Bulletin 2000;32:47–56.
28. Wada Y, Nishiike S, Kitahara T, Yamanaka T, Imai T, Ito T, et al. Effects of repeated snowboard exercise in virtual reality with time lags of visual scene behind body rotation on head stability and subjective slalom run performance in healthy young subjects. Acta Otolaryngol 2016;136:1121–1124.
29. Kennedy RS, Stanney KM, Dunlap WP. Duration and exposure to virtual environments: sickness curves during and across sessions. Presence 2000;9:463–472.
30. Tanaka N, Takagi H. Virtual reality environment design of managing both presence and virtual reality sickness. J Physiol Anthropol Appl Human Sc 2004;23:313–317.
31. Barlow DH, Farchione TJ, Sauer-Zavala S, Latin HM, Ellard KK, Bullis JR, et al. Unified Protocol for Transdiagnostic Treatment of Emotional Disorders: Therapist Guide Oxford: Oxford University Press; 2017.

Article information Continued

Figure 1.

A sample graph of measured heart rate changes of one subject during a VR session. A: HR of one subject during a VR treatment session: the first 20 seconds represent the baseline HR before the start of the VR session (baseline HR), “Elevator” represents the experience of the VR elevator scenario, and MF represents the breathing guide with the visuo-haptic-based multimodal feedback system. There is a clear jump between baseline HR and Elevator. The heart rate decreases between Elevator and MF, and the heart rate variability is wider at MF. B: The dot represents the mean value of the heart rate, and the arrow represents its SD. Point 1 represents the mean and SD of heart rate at baseline, point 2 represents the same data at Elevator, and point 3 at MF. VR: virtual reality, HR: heart rate, MF: multimodal feedback, SD: standard deviation.

Table 1.

Results of comparison of psychiatric scales across haptic-based multimodal feedback VR sessions

Z -1.357 -1.363 -0.943 -0.210 -0.368 -1.439
P value 0.175 0.173 0.345 0.833 0.713 0.150

K-MADRS: Montgomery-Åsberg Depression Rating Scale, Korean version, HAM-A: Hamilton Anxiety Rating scale, BAI: Beck Anxiety Inventory, STAI: State-Trait Anxiety Inventory, SSQ: Simulator sickness questionnaire