Diagnostic Usefulness of an Ultra-Brief Screener to Identify Risk of Online Gaming Disorder for Children and Adolescents
Article information
Abstract
Objective
This study examined the diagnostic validity of a three-item ultra-brief screening tool for online gaming disorder in line with the gaming disorder criteria in the International Classification of Diseases 11th Revision.
Methods
The Three-item Gaming disorder Test-Online-Centered (TIGTOC) was composed of three items using a four-point Likert scale selected from the Internet Game Use-Elicited Symptom Screen (IGUESS). Among a cohort of 2319 young-adolescent Internet users, the baseline data of 228 healthy controls and 45 Internet-gaming-disorder cases were analyzed. Receiver operation characteristic (ROC) analysis was performed using mental health specialists’ diagnoses as the gold standard.
Results
The ROC curve analysis showed an area under the curve of 86%. Using a cut-off score of 4 from a full range of 0–9, the sensitivity, specificity, and Cronbach’s α were 72%, 90%, and 0.811, respectively. TIGTOC scores was positively associated with time spent on online gaming, depressive symptoms, attention-deficit/hyperactivity disorder symptoms, and addictive Internet use.
Conclusion
The TIGTOC appears to be a brief, valid, and reliable screening tool for online gaming disorder within the community or in primary care settings.
INTRODUCTION
Addressing online gaming and the mental-health consequences among children and adolescents is particularly important, since most children and adolescents play online games [1-3] and, considering the normal trajectory of neural development, adolescence marks a period of heightened limbic-system maturation. Thus, adolescents are more likely to engage in addictive substances and behaviors than adults [4]. Further, game makers strive to reinforce addictive features in their products [5,6]. Addiction to online gaming makes it difficult for individuals in this age group to accomplish psychosocial developmental tasks typically learned during this period; this may have long-term consequences on their social lives [7]. Therefore, asking children and adolescents about their use of online games and gaming-elicited problems is crucial to provide preventive interventions in a timely manner [8,9].
The World Health Organization (WHO)’s new International Classification of Diseases 11th Revision (ICD-11) now includes gaming disorders in the addictive disorders section [10]. The diagnostic criteria for gaming disorder includes three symptoms, namely, impaired control over gaming, increasing priority given to gaming over other activities (to the extent that gaming takes precedence over other interests and daily activities), and continuation or escalation of gaming despite the occurrence of negative consequences. The subtypes of gaming disorder include offline-predominant and online-predominant; the latter corresponds to Internet gaming disorder (IGD) as defined by the Diagnostic and Statistical Manual of Mental Disorders, 5th Edition (DSM-5), which pre-dates the ICD-11. In the DSM-5, IGD is defined as the “persistent and recurrent use of the Internet to engage in games, often with other players, leading to clinically significant impairment or distress [11].” Among the nine diagnostic criteria of IGD, three items are similar to the ICD-11 gaming disorder diagnostic criteria mentioned above.
As the three criteria are included in both the ICD-11 and in the DSM-5, a screening tool utilizing them would reflect the concepts of both online-predominant gaming disorder (ICD-11) and IGD (DSM-5). This would be useful for the quick assessment of children’s and adolescents’ health risk behaviors in community or primary-care settings. However, if self-reported data is used to set a gold standard and the validation of the screening tool is assessed based on that gold standard, the acceptability might be relatively low, no matter how fast or reliable the tool [12,13].
We aimed to investigate the diagnostic validity and reliability of online gaming screening tools based on three items using a mental health specialists’ diagnoses as the gold standard.
METHODS
Study population
Baseline data from the Internet user Cohort for Unbiased Recognition of gaming disorder in Early Adolescence (iCURE) study were used in this study. The iCURE study was a prospective study on 2319 children and adolescents attending one of 21 elementary or middle schools. Participants were enrolled from 2015 to 2016. At the time of enrollment, informed assent and written informed consent were obtained from all individual participants and their parent/guardian, respectively. Details of the iCURE study have been described previously [14].
In this study, we analyzed the data of 273 young adolescents, including 1) 228 who were not diagnosed with IGD from a sub-cohort of 232 representing the iCURE cohort (this representative sub-cohort was constructed by randomly sampling 10% of the whole iCURE cohort, and 2) all 45 from the whole cohort who were diagnosed with IGD [15,16] to ensure a sufficient number of patients to evaluate the diagnostic usefulness of the new screening tool (these 45 cases included 4 subjects diagnosed with IGD who belonged to the representative subcohort). All subjects were tested with an IGD screening tool that used nine items based on the DSM-5 criteria, which includes the three items corresponding to the ICD-11 gaming disorder diagnostic criteria. Additionally, regardless of the result, all subjects had a one-on-one clinical interview with a mental health specialist to confirm the IGD diagnosis.
The present study complies with the tenets put forth in the Declaration of Helsinki and was approved by the Institutional Review Board of the Catholic University of Korea (MC14ONMI0085).
Measurements
The Three-Item Gaming disorder Test–Online-Centered (TIGTOC)
The TIGTOC is an online gaming disorder screening tool that utilized three items and a four-point Likert scale (Table 1). The three items present the symptoms mentioned as diagnostic criteria of gaming disorder in the ICD-11; they were selected from the Internet Game Use–Elicited Symptom Screen (IGUESS), for which reliability and diagnostic validity had been previously established [12]. To evaluate the diagnostic validity of the TIGTOC, we analyzed subjects’ responses to those three items collected in the iCURE study.
Three items comprising the Three-Item Gaming disorder Test-Online-Centered (TIGTOC), an online gaming disorder screening tool with 4-point Likert scale
The TIGTOC is a primary testing tool designed to identify individuals at risk for online gaming disorder in community and primary-care settings. It is self-administered form and comprises three items, which respondents are asked to rate on a four-point scale (0, not at all; 1, occasionally; 2, frequently; 3, always) for their own experiences of last 12 months.
Two extra reversed items had been inserted when the nine-item screening test, the IGUESS, was used to detect insufficient-effort responses. Cases in which an IGD diagnosis had been made but there was a score of zero on all nine items (regardless of reversed items) were defined as intentional negative respondents. As false negatives can be caused by either problems with the screening tool or respondent-related factors, we attempted to determine the sensitivity of the TIGTOC by performing the analyses on both the ideal group (which excluded intentional negative respondents) and the real-world group (which included intentional negative respondents).
Diagnosis of online gaming disorder
As a gold standard to evaluate the criterion-related validity of the TIGTOC, we used the diagnosis of IGD, confirmed by a mental-health specialist and based on the DSM-5 in the iCURE study. All of the subjects underwent a screening test, followed by a one-on-one diagnostic interview 1 week later; interviews were conducted by psychiatrists or clinical psychologists with >5 years of clinical experience and master’s degrees. In addition, IGD was diagnosed using a semi-structured diagnostic face-to-face interview schedule, the Diagnostic interview for Internet Addiction (DIA) [17], which was developed based on the IGD diagnostic criteria proposed in the DSM-5. Inter-rater concordance for mental health status was reported as low as 0.20–0.39 [18-20]. In this study, the inter-interviewer reliability of DIA was fair, with Cohen’s kappa=0.38 [21]. In order to improve the agreement between the interviewers, on the day of the diagnosis interview, a case conference was attended by all the interviewers, and the diagnosis were confirmed after sufficient discussion.
The parents of the children and adolescents who were diagnosed with IGD were contacted to provide an opportunity to receive professional help, either through introductions to the mental-health department at an affiliated hospital or through the supply of information related to pertinent community resources.
We next attempted to evaluate the convergent validity of the TIGTOC. For this purpose, self-reported data pertaining to Internet addiction, depressive symptoms, attention-deficit/hyperactivity disorder (ADHD) symptoms, and time spent on Internet gaming were included in the analysis. For time spent on Internet gaming, we surveyed both the frequency (number of days) and duration (time per day) the subjects spent engaged in Internet game play over the course of the week, as well as over the weekend. The average daily time spent on Internet gaming was computed using their responses, and then the subjects were divided into those who spent 2 hours or more on Internet gaming and those who spent less than 2 hours on Internet gaming.
Internet addiction was measured using the Internet Addiction Test (IAT)-a tool specifically developed to screen for Internet addiction [22]. Each item on the 20-item test is rated on a five-point Likert scale (1: not at all; 5: always). A total score of 40–69 indicates problematic use, and a score of 70 or higher indicates Internet addiction requiring professional help. In the present study, a score of 70 or higher was defined as positive on the IAT. The IAT was used to examine convergent validity of the TIGTOC. As games account for one of the main contents of the Internet use, the correlation between Internet addiction and IGD has been reported to be very high; the tool used in such a report was the IAT, which is widely used around the world [22].
Depressive symptoms were surveyed using the Child Depression Inventory (CDI), a depression screening tool widely used for children and adolescents [23]. This tool comprises 27 items, and, for each item, respondents are asked to choose one of three response options that most closely resembles their lives within the past 2 weeks. Each response is rated from 0 to 2. Scores of 22–25, 26–28, and 29 or higher indicate mild, moderate, and severe depression, respectively. In the present study, a score of 22 or higher was defined as positive on the depression screening test.
ADHD symptoms were measured based on self-reported responses to the observer version of the Korean ADHD Rating Scale (K-ARS). Responses were made by the guardian of the registered child or adolescent. The K-ARS comprises 18 items whereby each item is scored from 0 (never or rarely) to 3 (very often). A score of 19 or higher was defined as positive on the ADHD screening test [24].
Statistical analyses
Cronbach’s α was computed to assess the reliability of the TIGTOC. To assess the criterion-related validity of the TIGTOC, the area under the curve (AUC) was computed with receiver operating characteristic (ROC) analysis using diagnoses of online gaming disorder as the gold standard. To set the optimal cut-off score for the TIGTOC, the sensitivity, specificity, positive and negative predictive values, positive and negative likelihood ratios, and diagnostic accuracy were computed. As mentioned above, analyses were performed both on the total study population, on the so-called real-world group, and on the ideal group in which the six intentional negative respondents were excluded. This approach was adopted to examine differences in the diagnostic validity of the TIGTOC before and after reflecting real-world conditions in which false negatives due to respondent-related factors were present. Associations between major variables related to online gaming disorder and the screening test results were analyzed via chisquare test, t-test or correlation analysis to assess the convergent validity of the TIGTOC.
RESULTS
Table 2 shows the distribution of the general characteristics of the 273 subjects included in this study. About 54.9% of the participants were male, and 88.6% were middle-school students. The mean age was 12.6±1.1 years. A total of 33.2% of the participants spent ≥2 hours/day on Internet gaming, and 8.8% were positive on the IAT; 8.1% and 13.8% of the participants were positive for depressive and ADHD symptoms, respectively. The median and mean scores of the TIGTOC were 1 (range 0–9) and 1.8 (SD=2.1), respectively, and 16.5% of the participants were diagnosed with IGD after a one-on-one diagnostic interview.
General characteristics of sample (N=273)
In the IGD group, the proportions of middle-school students (97.8% vs. 86.8%), Internet gamers playing 2 hours or more per day (54.8% vs. 28.4%), Internet addicts (42.2% vs. 2.2%), those with depression (22.2% vs. 5.3%), and those with ADHD (28.9% vs. 10.7%) were significantly higher than in the non-IGD group.
Figures 1 and 2 show the ROC curves of the TIGTOC. The AUC for the ideal group without the six intentional negative respondents was 86.2%, while the AUC for the total study population, including the intentional negative respondents, was 77.6%. Cronbach’s α for the three items of the TIGTOC was 0.811 for the ideal group and 0.814 for the total study population.
ROC curve of the Three-item Gaming disorder Test-Online-Centered (TIGTOC) with a goldstandard of diagnosed Internet gaming disorder of 273 subjects. Area under the curve (AUC) was 77.6%. ROC: receiver-operating characheristic.
ROC curve of the Three-item Gaming disorder Test-Online-Centered (TGITOC) with a goldstandard of diagnosed Internet gaming disorder of 267 subject (intentional negative response cases excluded). Area under the curve (AUC) was 86.2%. ROC: receiveroperating characheristic.
Table 3 shows the diagnostic performance index in relation to the cut-off point of the TIGTOC. An analysis excluding the intentional false negative responses showed a diagnostic accuracy of 85.7% at a cut-off point of 4. At this cut-off point, sensitivity was 71.8% and specificity was 90.4%, with a positive likelihood ratio (LR+) of 7.5 and a negative likelihood ratio (LR-) of 0.3. Under the same conditions, the positive predictive value (PPV) was 56.0% and the negative predictive value (NPV) was 94.9%.
Diagnostic performance of the Three-Item Gaming disorder Test-Online-Centered (TIGTOC)
A similar pattern was observed in the analysis that included the intentional false negative responses; diagnostic accuracy was 85.7% at a cut-off point of 4. At this cut-off point, sensitivity was 62.2% and specificity was 90.4%, with an LR+ of 6.5 and an LR- of 0.4. In this case, the PPV was 56.0% and the NPV was 92.4%, which is similar to the results of the analysis without the intentional false negative responses.
Table 4 shows the results of screening test by general characteristics using the TIGTOC based on a cut-off point of 4. Weekly time spent on online gaming was significantly greater in the TIGTOC positive group (21.8±17.8 hours) than in the negative group (10.1±11.5 hours) (p<0.001). The IAT score (positive: 10.1±11.5 vs. negative: 38.7±14.0), depressive symptoms (positive: 15.5±11.0 vs. negative: 8.5±7.3, p<0.001), and ADHD symptoms (positive: 7.8±6.5 vs. negative: 5.6±5.6, p=0.014) were also significantly higher in the TIGTOC positive group than in the negative group. As a result of an additional correlation analysis, the correlation coefficient between the total score of the TIGTOC and the total score of IAT, CDI, and K-ARS was 0.659 (p<0.001), 0.407 (p<0.001), and 0.027 (p<0.001) respectively.
Results of screening test by general characteristics using the Three-Item Gaming disorder Test-Online-Centered (TIGTOC) (N=273)
DISCUSSION
In modern society, online gaming is a part of many people’s daily lives. Thus, it is important to distinguish between non-problematic and problematic gaming. The identification of gaming disorders requires the use of clinically defined screening tools to target individuals who may require professional help [9].
When discussing the characteristics associated with the problem of online gaming, one of the frequently mentioned variables is excessive use of games, that is, the amount of time spent on online gaming. Prior to this study, we had analyzed whether determination of the average gaming time as more than 2 hours a day or more than 3 hours a day would be useful as a simple tool to screen for online gaming disorder risk. However, although not presented in this study, we found that the sensitivity and specificity of time spent on gaming were not enough to function as a screening tool for assessing risk of online gaming disorder. Therefore, we established the TIGTOC to provide researchers and clinicians an instrument that could quickly and accurately identify individuals at risk for online gaming disorder.
The greatest strength of the current study is that we evaluated the criterion-related validity of the TIGTOC using methodology that satisfies the Quality Assessment of Diagnostic Accuracy Studies tool [25]—the international quality standard for diagnostic studies. To do so, a blinded mental-health specialist conducted one-on-one diagnostic interviews with all subjects to determine an IGD diagnosis based on DSM-5 criteria; this diagnosis was used as the external gold standard. In this study, an empirical cut-off score of 4 on the TIGTOC was sufficient to distinguish between non-problematic and problematic online game use. Moreover, we found that the TIGTOC score was positively associated with weekly time spent playing online games, Internet addiction measures obtained via the IAT, ADHD symptoms, and depressive symptoms; the TIGTOC was a reliable tool with a high Cronbach’s alpha. However, despite the strength that the goldstandard of IGD diagnosis was determined through such a rigorous process, since the three items constituting the TIGTOC were also important criteria in the goldstandard, there is a limitation that the relation of the screening test and the goldstandard wouldn’t be completely free from the error of circular reasoning [26].
An additional strength of the TIGTOC with a four-point Likert scale is that it is an ultra-brief instrument that is contextually in line with the diagnostic criteria for online-dominant gaming disorder as suggested by the WHO in the latest ICD revision. Although debates regarding the diagnostic criteria for IGD have been ongoing [12,27-30], recent empirical and clinical studies generally report the above three symptoms as the main indicators of gaming disorder. Moreover, this brief instrument poses significant benefits when considering the busy nature of primary care and/or community settings, as well as the fact that a substantial number of variables are already included and measured in child/adolescent behavior or mental-health surveys.
In screening tests, false positive and false negative results are inevitable. They are not solely due to low test accuracy, but also occur when respondents make intentionally false answers in an effort to obtain desired test results. In the real world, the size and frequency of such respondent-related factors are unknown. However, in the present study, we tried to detect this aspect by performing a sensitivity analysis; we found that six subjects marked 0 for all nine items on the IGUESS but were diagnosed with IGD after the one-on-one diagnostic interview. When the diagnostic performance excluding their data was compared to the diagnostic performance including their data, we found that all features were similar, with the exception of sensitivity: the sensitivity was 72% in the data excluding the six intentional negative respondents and 62% in the data including them. Since the difference in sensitivity was not induced by the screening tool itself, it does not undermine the usefulness of this instrument. Nevertheless, screening service providers should be aware of intentional false negative responses and should have countermeasures in place.
Unlike previous studies, the present study evaluated the criterion-related validity as well as the convergent validity of the online gaming disorder screening tool. The usefulness of this tool can be highlighted in that it specifically targets early adolescence, a time when exposure to online games begins to increase.
Overall, the findings of the current study support the concept of gaming disorder in the ICD-11 [10] and preceding research on IGD of the DSM-5. The present study provides a valid, reliable, and ultra-brief online gaming screening tool for researchers and practitioners in this field [1]. Further studies should investigate whether the TIGTOC would be useful across different cultures and ages.
Acknowledgements
This research was supported by a grant of the Korea Mental Health R&D Project, funded by the Ministry of Health & Welfare, Republic of Korea (grant number: HL19C0048).
Notes
The authors have no potential conflicts of interest to disclose.
Author Contributions
Conceptualization: Sun-Jin Jo, Hyeon Woo Yim. Data curation: Sun-Jin Jo, Hae Kook Lee, Hyunsuk Jeong, Hyeon Woo Yim. Formal analysis: Sun-Jin Jo. Funding acquisition: Hyeon Woo Yim. Investigation: Sun-Jin Jo, Hyunsuk Jeong, Seung-Yup Lee, Hye Jung Son. Methodology: Sun-Jin Jo, Hyeon Woo Yim. Project administration: Hye Jung Son. Resources: Hyeon Woo Yim, Yong-Sil Kweon. Software: Hye Jung Son. Supervision: Yong-Sil Kweon, Hae Kook Lee. Validation: Hyunsuk Jeong. Visualization: Sun-Jin Jo, Hyeon Woo Yim. Writing—original draft: Sun-Jin Jo. Writing—review & editing: all of the authors.