Reliability and Validity of the Korean Version of Disruptive Behavior Disorders Rating Scale, DSM-5 Version-Parent Form
Article information
Abstract
Objective
Disruptive behavior disorder (DBD) adversely impacts children and adolescents. However, a comprehensive and cost-effective scale to assess DBD is lacking in Korea. Therefore, this study translated the Disruptive Behavior Disorders Rating Scale (DBDRS) into Korean and analyzed its psychometric properties.
Methods
Parents and primary caregivers of non-clinical (n=429) and clinical (n=28) children and adolescents aged 6–15 years were included in the analysis. Confirmatory factor analysis was conducted; further, concurrent validity and internal consistency were investigated using correlation analysis and Cronbach’s alpha, respectively. Furthermore, discriminative capacity was estimated using receiver operating characteristic curve analysis.
Results
The four-factor model of K-DBDRS showed good model fit indices and factor loadings, which supported the construct validity of the scale. Strong correlations between K-DBDRS and related measurements were observed, and a robust level of Cronbach’s alpha was confirmed (0.891–0.933). The discriminative capacity of the scale was good, based on the area under the curve values (0.933–0.953).
Conclusion
This study indicated that the K-DBDRS is an appropriate screening tool for Korean children and adolescents. Thus, this scale can be applied in clinical and community settings to identify children and adolescents with disruptive behavior disorders.
INTRODUCTION
Disruptive behavior disorder (DBD) is characterized by social disruptions, which encompasses attention-deficit/hyperactivity disorder (ADHD), oppositional defiant disorder (ODD), and conduct disorder (CD). DBD is common in Korean children and adolescents; moreover, reports suggest that the prevalence of ADHD in Korean elementary school students ranged from 5.9%–6.5%, and the rate of sub-threshold symptom level was 9.0% [1,2]. In addition, the prevalence of childhood ADHD, retrospectively reported by college students, was 7.3% [3]. These rates indicate that at least one student in a class may have ADHD. Further, the prevalence of ODD in Korean elementary school students was 4.9%, and the rate of those with subclinical symptoms was 7.7% [1]. The prevalence of CD and the rate for sub-threshold levels of CD were 0.4% and 0.8%, respectively [1]. The rates of ADHD and ODD were similar, whereas CD was less prevalent than ADHD and ODD in the community. However, a study showed that more than half of the adolescents who were juvenile offenders in custody met the diagnostic criteria for CD [4].
Children and adolescents with DBD during the developmental course could experience long-term negative outcomes, including poor intellectual performance [5], familial and interpersonal conflicts [6,7], criminal activities [8-10], and poor health conditions or early death [11,12]. DBD symptoms diagnosed in childhood tend to persist until early adulthood, thus increasing the probability of comorbidities with other psychiatric disorders [13-15]. In addition, DBD leads to increased social costs and loss of human resources. In the United States, the loss of labor productivity related to ADHD was estimated at $67 billion–$116 billion for one year [5], and the cost of providing mental health services or treatment for CD exceeded $7,000 per child [16]. Thus, previous results highlight the need for screening DBD symptoms in the early developmental stages to prevent adverse effects of the disorders.
To identify DBD, prior studies utilized distinct measurements to assess ADHD, ODD, or CD symptoms. Valid and widely used scales in Korea include the ADHD Rating Scale (ARS) [17], ADHD Diagnostic System [18], and Computerized Neurocognitive Function Tests [19]. Regarding ODD and CD, the reliabilities, validities, and diagnostic accuracy of the Oppositional Defiant Disorder Rating Scale [20] and Conduct Disorder Rating Scale [21], which were developed in Canada, have been verified. However, these scales have not yet been translated and validated in the Korean context. Moreover, there is a dearth of other assessments for ODD or CD. Previous studies found that ADHD, ODD, and CD were often diagnosed as comorbid conditions, and that these cases indicated more severe symptoms, poorer functional abilities, and more difficulties in school [22,23]. Therefore, a comprehensive assessment for DBD is required to discover symptoms that cannot be detected by a single scale. The integrated measurements widely used to assess DBD among Korean children and adolescents are the Child Behavior Checklist (CBCL) [24] and Behavioral Assessment System for Children [25]. Their psychometric properties have been substantiated in previous Korean studies [26,27]. Nevertheless, they have limitations in their application in clinical and community settings owing to their expensive and time-consuming nature.
The Disruptive Behavior Disorders Rating Scale (DBDRS) [28] was developed to facilitate the screening of DBD symptoms and diagnostic decision-making. At first, the DBDRS was developed using the Diagnostic and Statistical Manual of Mental Disorders (DSM)-III-R diagnostic criteria, but it has been updated following the publication of DSM-IV-TR and DSM5. The DBDRS includes items measuring ADHD, ODD, and CD symptoms. Pelham et al. [28] suggested that the proportion of boys who meet the diagnostic cutoffs for ADHD, ODD, or CD using the DBDRS was similar to the prevalence rate reported in prior research. Further, prior reports confirmed that the DBDRS has an appropriate level of reliability and validity [28,29]. In addition, a previous study demonstrated that the DBDRS was as valid as a clinical interview [30]. Particularly, there was an 87.5% degree of agreement between the DBDRS as reported by a parent and the Diagnostic Interview Schedule for Children-IV in classifying children as ADHD or non-ADHD, which is considered relatively high. These prior studies implied that the DBDRS can overcome the constraints of the aforementioned measurements, specifically because it is an encompassing scale for DBD and less time-consuming. Utilizing the DBDRS as a screening tool in community and clinical settings can help identify unrecognized symptoms and promote the formulation of early intervention plans for children and adolescents with DBD symptoms.
Although some researchers translated and used DBDRS to measure DBD symptoms in Korean children [31], no study has confirmed the validity of a Korean version of the DBDRS. Therefore, we investigated the reliability and validity of the Korean version of DBDRS (K-DBDRS) to measure DBD among children and adolescents. While both parents and teachers can respond to the DBDRS, we only recruited parents and primary caregivers for the following reasons. First, teachers can evaluate social, behavioral, and academic problems because they are exposed to students in the classroom; however, the majority of Korean students attend online classes because of the coronavirus disease (COVID-19). Even though some students attend in-person school, teachers are unable to observe children’s and adolescents’ behaviors, given that activities and interactions are restricted to avoid contagion. Moreover, teachers may face difficulties in assessing all students, given that approximately 22 students are allocated per teacher in Korean schools [32]. Considering the availability of the DBDRS, especially for large-scale screening, collecting data from parents is more efficient.
The objectives of this study are as follows: 1) to examine the reliability and validity of the K-DBDRS in children and adolescents; 2) to compare the non-clinical and clinical groups diagnosed with ADHD, while investigating sex and age differences in the non-clinical group based on the results of prior research [33,34]; 3) to investigate the discriminative capacity and diagnostic accuracy of the K-DBDRS by comparing it to other ADHD measurements, and confirming the cutoff points to screen children and adolescents with a clinical level of symptoms; and 4) to explore diagnostic predictabilities for each item of the K-DBDRS.
METHODS
Ethics
This study was approved by the Institutional Review Board of the National Center for Mental Health (approval number: 116271-2021-05). Written informed consent to participate in the research was obtained from all participants. All methods used in this study were conducted in accordance with the relevant guidelines and regulations.
Participants
Non-clinical group
Data were collected from 510 parents and primary caregivers of children and adolescents aged 6–15 years who were not diagnosed with ADHD, ODD, or CD. We recruited non-clinical participants from online communities (e.g., Daum and NAVER cafes), which are easily accessible in various regions of South Korea. Eligibility was based on (a) no current diagnosis of mental disorders including DBD and (b) non-participation in special education. Furthermore, duplicate or inconsistent responses were excluded. Consequently, we used 429 participants’ data for analysis.
Clinical group
The clinical group was recruited from a child and adolescent outpatient clinic at a mental health center in Seoul, Korea. Eligibility was based on: (a) diagnosis of one or more DBD and (b) no indication of an intellectual disorder, autism, a psychotic disorder, epilepsy, or gross brain damage. Similar to the non-clinical group, parents or primary caregivers of children and adolescents aged 6–15 years responded to the survey. We collected and used data from 28 parents and primary caregivers of children and adolescents with ADHD (n=21), ADHD with ODD (n=2), or ADHD with a mood disorder (n=5).
Table 1 shows the socio-demographic characteristics of participants in this study. There were 230 girls (53.6%) and 199 boys (46.4%) in the non-clinical group, and 5 girls (17.9%) and 23 boys (82.1%) in the clinical group. The age was stratified into the following ranges: 6–8 years (39.6%, non-clinical group; 42.9%, clinical group), 9–11 years (37.3%, non-clinical group; 35.7%, clinical group), and 12–15 years (23.1%, nonclinical group; 21.4%, clinical group). A majority of the respondents were mothers (93.5%, non-clinical group; 85.7%, clinical group). Regarding the education level of parents and primary caregivers, the majority were university/college graduates in both groups (80.9%, non-clinical group; 60.7%, clinical group). A chi-squared test of independence was conducted, and all socio-demographic characteristics except for age showed a significant difference (see Table 1).
Participants’ socio-demographic characteristics
Procedure
We created an online survey, which was only administered to parents or primary caregivers who voluntarily accessed the link and consented to participate in the study. The description detailed the outline and purpose of the study, as well as the research process. Moreover, participant eligibility was displayed on the online community to recruit non-clinical participants, whereas a flyer was displayed on a bulletin board to recruit clinical participants. We explained our study procedures to participants, after they expressed their interest in participation. Next, we sent them a link to the online survey. All participants provided written informed consent before responding to the questionnaire. After completion of the survey, participants were rewarded with a mobile gift card of 5,000 KRW for the non-clinical group and 15,000 KRW for the clinical group. Data collection lasted over 4 months, from April–July 2021.
Translation
The English DBDRS was translated into Korean by two researchers with a master’s degree in psychology and one with a master’s degree in occupational therapy. Next, a bilingual professional back-translated the original translation. Experts including a psychology professor, psychiatrist, and researchers thoroughly reviewed and revised all of the translated versions to reach a consensus. Finally, three psychiatrists who did not participate in the translation reviewed the K-DBDRS to determine whether the contents were relevant to the Korean cultural context.
Measures
Background information
To identify the socio-demographic characteristics of all participants, information from parents or primary caregivers (i.e., relationship with children and adolescents, sex, age, education level, and household income) and information from the target children and adolescents (i.e., sex, age, psychiatric history, and present psychiatric diagnosis) were investigated.
Disruptive Behavior Disorders Rating Scale
The DBDRS is a scale comprising 45 items extracted from the diagnostic criteria of DBD [28]. It has four subscales: ADHD inattention symptoms (ADHD-IA), ADHD hyperactivity/impulsivity symptoms (ADHD-HI), ODD symptoms, and CD symptoms. Each item is rated on a 4-point Likert scale ranging from 0 (not at all) to 3 (very much). A rating of 0 or 1 indicates the absence of symptoms, whereas that of 2 or 3 indicates the presence of symptoms. As reported by Pelham et al. [28], the DBDRS is a reliable scale in terms of internal consistency, with 0.95 for ADHD-IA, 0.95 for ADHD-HI, and 0.96 for ODD. Moreover, the parent-reported DBDRS has good internal consistency, ranging from 0.94–0.97 [30]. In this study, we received the DSM-5 version of the DBDRS from the original author and used 41 items, excluding 4 items from the previous version of the DSM. The subscale scores of the DBDRS can be computed in two ways: averaging the items of each factor or counting symptoms rated 2 or 3. In this study, the subscale scores were calculated by averaging the items of each factor. This calculation was used to increase the utility of the DBDRS as a screening tool. As a result, the range of each subscale score was from 0 to 3.
ADHD Rating Scale
The ARS was used to test the concurrent validity of the K-DBDRS [17]. The ARS was developed to measure ADHD symptoms in school-aged children. Moreover, it was standardized in Korea for elementary school students with and without ADHD [35]. The scale comprises two subscales (i.e., ADHD inattention and ADHD hyperactivity/impulsivity), each 9 items, for a total of 18 items. The items are rated on a 4-point Likert scale ranging from 0 (never or rarely) to 3 (very often) based on the frequency of symptoms. Two subscale scores were derived from summing the scores of each of the nine items (subscale scores ranged from 0–27). Previous studies with Korean parents confirmed the reliability and validity of the ARS [36]. The Cronbach’s alpha for the ARS subscales in this study were 0.93 and 0.91 for ADHD-IA and ADHD-HI, respectively.
Child Behavior Checklist for ages 6–18
The CBCL was used to determine the concurrent validity of the K-DBDRS. The CBCL is used to assess various emotional and behavioral problems among children and adolescents [24]. We used DSM-oriented scales comprising classified items depending on the diagnostic criteria of DSM. Among them, DSM ADHD, DSM ODD, and DSM CD were included in this study. Each behavior problem is rated on a 3-point Likert scale, ranging from 0 (not at all) to 2 (often or a lot). The Korean standardization study reported good internal consistency and 1-week test-retest reliability of three DSM-oriented scales; however, these measures were lower than those reported in the normative group in the United States [37]. In this study, the Cronbach’s alpha for the three scales were 0.88 for ADHD, 0.82 for ODD, and 0.86 for CD.
Statistical analyses
Statistical analyses were conducted using IBM Statistical Package for the Social Sciences (SPSS; IBM Co., Armonk, NY, USA) version 22.0 and Mplus 8.5 (Muthén & Muthén, Los Angeles, CA, USA). We examined descriptive statistics to understand the nature of the data, which revealed that the DBDRS scores did not follow a normal distribution. In addition, the items of the DBDRS are rated on a four-point Likert scale, which is considered to have ordinal characteristics. We analyzed the data based on these conditions. First, we performed confirmatory factor analysis (CFA). Pelham et al. [28] yielded three factors from the DBDRS, including inattention, impulsivity/hyperactivity, and oppositional/defiant by principal components analysis. However, these factors comprised 26 of the total 45 items, and the items to measure conduct problems were not included. Moreover, we considered that the threefactor structure derived from the DSM-III-R was unsuitable for the DSM-5 version of the DBDRS, given that the description of the items had changed and the three-factor model of the latest version of the DBDRS had not been tested. Therefore, we conducted CFA based on the diagnostic criteria of DSM-5. To analyze the ordinal data, we used weighted least squared means and variance adjusted (WLSMV) estimation. It has been suggested that WLSMV can more precisely estimate the magnitude of factor loadings with ordinal data, compared with maximum likelihood, regardless of the sample size [38]. The statistical measures used to evaluate the goodness of fit of the CFA model included the comparative fit index (CFI), Tucker–Lewis index (TLI), and the root mean square error of approximation (RMSEA). CFI and TLI values greater than 0.90 [39] and RMSEA values of less than 0.05 [40] indicate a good model fit.
The Mann–Whitney U tests were used to compare the nonclinical and clinical groups according to their levels of DBD symptoms on the DBDRS, ARS, and CBCL. Moreover, sex differences in the non-clinical group were examined on the DBDRS subscales. The Kruskal–Wallis test was conducted for all DBDRS subscales to investigate the differences in the ages of the non-clinical group. Spearman’s rank-order correlations were calculated between the DBDRS, ARS, and CBCL to determine the concurrent validity. Cronbach’s alpha for each subscale was yielded, and a value greater than 0.60 was considered to indicate good reliability [41].
Receiver operating characteristic (ROC) curve analysis was conducted to provide a cut-off point and test the discriminative capacity and diagnostic accuracy for the K-DBDRS. The T score of the CBCL DSM-oriented scales for diagnosing the clinical level of ADHD, ODD, and CD is 70T and above in Koreans [37]. With this reference, the diagnosis presence or absence cases were classified to conduct ROC curve analysis. Sensitivity and specificity were computed, and the cut-off points were estimated using Youden’s index for each subscale of the DBDRS [42]. In addition, we investigated the area under the curve (AUC), which enables the examination of the discriminative capacity and diagnostic accuracy of the scale. An AUC value of 0.70 or above indicates appropriate discrimination and accuracy [43,44]. Furthermore, we compared the ADHD subscales of the DBDRS with those of the ARS and CBCL and confirmed which scale had the highest discriminative capacity. The cases were categorized according to whether the children and adolescents received an ADHD diagnosis from their psychiatrists.
Finally, we tested the diagnostic predictive power of each item of the DBDRS. Predictive power refers to the probability of agreement between the diagnosis and the result of the test. The positive predictive power (PPP) or negative predictive power (NPP) can be computed according to the test outcomes. The PPP refers to the proportion of true-positive cases among all positive cases, while the NPP is the rate of true-negative cases among all negative cases [45].
RESULTS
Confirmatory factor analysis
CFA was conducted. As shown in Table 2, the fit indices of the four-factor model were good (χ2=1226.116, df=773, p<0.001, CFI=0.982, TLI=0.981, standardized root mean residual [SRMR]=0.089, RMSEA=0.036). All factor loadings were significant and higher than 0.70. Factor loadings of the items were 0.755–0.928 and 0.703–0.888 in ADHD-IA and ADHDHI, respectively. Factor loadings of ODD ranged from 0.822–0.933, and those of CD ranged from 0.780–0.968 (Figure 1).
Goodness-of-fit indices of the four-factor model
Four-factor confirmatory factor analysis model of the Disruptive Behavior Disorders Rating Scale. ADHD-IA, ADHD inattention symptoms; ADHD-HI, ADHD hyperactivity/impulsivity symptoms; ODD, oppositional defiant disorder; CD, conduct disorder.
Comparison
To investigate the differences between the non-clinical and clinical groups, we performed the Mann–Whitney U tests. As shown in Table 3, there were significant differences between the two groups in the DBDRS subscales: ADHD-IA (Mann–Whitney U=1,435.50, Z=-6.848, p<0.001), ADHD-HI (Mann–Whitney U=1,589.00, Z=-6.664, p<0.001, ODD (Mann–Whitney U=2,093.50, Z=-5.942, p<0.001), and CD (Mann–Whitney U=3,473.50, Z=-4.398, p<0.001).
Comparisons between mean scores of non-clinical and clinical samples for DBDRS and other related measurements
Based on the chi-squared test results for socio-demographic characteristics, we conducted additional analysis to compare the mean scores after controlling for confounders (i.e., sex, respondent’s relationship to children and adolescents, education level, and household income). Analysis of covariance using the rank transformation was performed. Overall subscale scores of the DBDRS were significantly different after controlling for these confounders (F=40.451, p<0.001 for ADHD-IA; F=36.679, p<0.001 for ADHD-HI; F=29.817, p<0.001 for ODD; F=12.711, p<0.001 for CD).
Table 4 indicates the sex differences in the non-clinical group. On the ADHD-IA (Mann–Whitney U=18,013.00, Z=-3.87, p<0.001), ADHD-HI (Mann–Whitney U=19,045.00, Z=-3.08, p<0.01), ODD (Mann–Whitney U=19,576.00, Z=-2.67, p<0.01), and CD (Mann–Whitney U=20,592.50, Z=-2.15, p<0.05), significant differences were found between boys’ and girls’ scores.
Comparing mean scores by sexes in non-clinical samples for the DBDRS
The Kruskal–Wallis test showed no significant differences among the age groups (Table 5), except for ADHD-HI (Kruskal–Wallis=12.37, df=2, p<0.01). Post hoc tests revealed that the 12–15 years age group was significantly different from the 6–8 years (Mann–Whitney U=6,320.00, Z=-3.50, p<0.001) and the 9–11 years age group (Mann–Whitney U=6,447.00, Z=-2.60, p<0.01).
Comparing mean scores by age groups in non-clinical samples for DBDRS
Concurrent validity
The DBDRS ADHD-IA was significantly correlated with the ARS ADHD-IA (rs=0.85, p<0.01) and CBCL ADHD (rs=0.78, p<0.01). Moreover, the DBDRS ADHD-HI indicated a significant correlation with the ARS ADHD-HI (rs=0.84, p<0.01) and CBCL ADHD (rs=0.73, p<0.01). In addition, the DBDRS ODD and CD subscales had significant relationships with the CBCL ODD (rs=0.71, p<0.01) and CD (rs=0.55, p<0.01), respectively. Table 6 shows the associations between the DBDRS and other measurements.
Correlations between the DBDRS and other related measurement
Reliability
The internal consistency of the DBDRS was examined based on Cronbach’s alpha for each subscale. Cronbach’s alpha was 0.924 for ADHD-IA, 0.891 for ADHD-HI, 0.906 for ODD, and 0.933 for CD, thus indicating excellent consistency for all subscales.
Cut-off point
ROC curves using the DBDRS for the total sample are presented in Figure 2. All ROC curves lie in the upper left part of the reference line, and all of the values of AUC ranged from 0.933–0.953 (Table 7). These results indicate that the DBDRS had a good discriminative capacity for ADHD, ODD, and CD diagnoses. The cut-off point for ADHD-IA was 1.82, with a sensitivity of 94% and specificity of 88% (Youden’s Index=0.82), whereas that for ADHD-HI was 1.85, with a sensitivity of 94% and specificity of 92% (Youden’s Index=0.85). Moreover, the cut-off point for ODD was 1.80, with a sensitivity of 100% and specificity of 80% (Youden’s Index=0.80), whereas that for CD was 1.83, with a sensitivity of 93% and specificity of 89% (Youden’s Index=0.83).
ROC curves for DBDRS’s ADHD, ODD, and CD subscales. ROC, receiver operating characteristic; DBDRS, Disruptive Behavior Disorders Rating Scale; ADHD, attention-deficit/hyperactivity disorder; ODD, oppositional defiant disorder; CD, conduct disorder.
AUC, SE, 95% confidence interval, and p-value for DBDRS
In addition, we yielded and compared the AUC values of all ADHD measurements. The DBDRS ADHD-IA was 0.880, DBDRS ADHD-HI was 0.868, ARS ADHD-IA was 0.865, ARS ADHD-HI was 0.853, and CBCL ADHD was 0.863 (Figure 3 and Table 8). All of these measures demonstrated a discriminative power for diagnosing ADHD, and among the scales, the DBDRS ADHD-IA and ADHD-HI showed the highest value of AUC.
ROC curves for ADHD measurements. DBDRS, Disruptive Behavior Disorders Rating Scale; ADHD-IA, ADHD inattention symptoms; ADHD-HI, ADHD hyperactivity/impulsivity symptoms; ARS, ADHD Rating Scale; CBCL, Child Behavior Checklist; ADHD, attention-deficit/hyperactivity disorder; ROC, receiver operating characteristic.
AUC, SE, 95% confidence interval, and p-value for ADHD measurements
Positive and negative predictive power
The conditional probability statistics for the items of the DBDRS were analyzed to investigate the diagnostic predictive power of each DBD symptom. Table 9 presents the results. The presence of the symptom was determined using the ratings “pretty much” or “very much.” Moreover, the diagnoses of ADHD, ODD, and CD were determined by the number of symptoms suggested in DSM-5.
Conditional probabilities of ADHD, ODD, and CD (N=457)
The PPP rates of ADHD-IA symptoms in predicting ADHD predominantly inattentive type diagnosis ranged from 0.37–0.61. Specifically, the symptom with the highest PPP was “has difficulty sustaining attention,” whereas the lowest PPP was “easily distracted.” Regarding the ADHD-HI, the PPP rates in predicting ADHD for predominantly hyperactive/impulsive type diagnosis varied from 0.18–0.47. “Acts as if driven by a motor” had the highest PPP, whereas “talks excessively” had the lowest PPP. Overall, PPP rates of ADHD-HI symptoms were lower than those of ADHD-IA symptoms. In ODD symptoms, the PPP rates for predicting ODD diagnosis ranged from 0.54–0.89. The symptom with the highest PPP was “spiteful or vindictive,” whereas that with the lowest PPP was “blames others for one’s mistakes.” The PPP rates of CD symptoms predicting CD diagnosis varied from 0.40–1.00, and it was higher than the PPP ranges of ADHD and ODD symptoms. The majority of PPPs for CD symptoms was close to 1.00, except for “lies to obtain goods and favors” and “truant from school.” Finally, NPP ranged from 0.97–1.00 for all symptoms of DBDRS, suggesting high rates.
DISCUSSION
This study investigated the psychometric properties of the DBDRS for use with Korean parents and primary caregivers of children and adolescents aged 6–15. The findings demonstrate that the K-DBDRS can suitably measure disruptive behavior symptoms in both clinical and non-clinical participants. This study had four purposes: 1) to test the reliability and validity of the DBDRS in the Korean context; 2) to compare the K-DBDRS scores in the clinical and non-clinical groups, including by sex and age; 3) to examine the cut-off point and the discriminative capacity of the K-DBDRS; and 4) to estimate the diagnostic predictive power of the K-DBDRS items. The results of the study are summarized as follows.
First, we evaluated the four-factor model in compliance with the disorder categories and diagnostic criteria of DSM-5. The results of the CFA confirmed a robust level of model fit indices and a good level of factor loadings, which demonstrates appropriate construct validity of the K-DBDRS. In a previous study, CFA was conducted for ADHD and community teenagers using the DSM-IV version of the DBDRS [46]. Specifically, the researchers analyzed a three-factor model composed of ADHD-IA, ADHD-HI, and ODD, and used items identical to those used in our study. The results indicated that model fit indices were acceptable, and factor loadings were within an appropriate range, similar to ours. Moreover, the three-factor model was best fitted to the data compared to alternative models, which included the DBD single-factor model and two-factor model consisting of ADHD and ODD. The factor analysis outcomes for the ADHD-IA, ADHD-HI, and ODD items concurred with our results, except for the CD items. In addition, the internal consistencies of the DBDRS subscales were in excellent ranges, and the correlations between the DBDRS, ARS, and CBCL were significant, which supports the concurrent validity of the K-DBDRS.
Second, significant differences were found between the clinical and non-clinical groups on the DBDRS subscale scores. The mean scores overall were higher in the clinical group than in the non-clinical group. In other words, participants with ADHD showed a higher symptom frequency. The ARS and CBCL subscales measuring similar constructs to the DBDRS also presented significant differences between the two groups and higher mean scores in the clinical group. Moreover, as expected, sex differences were observed in all DBDRS subscale scores. These findings were consistent with previously published studies, where boys reported a higher number of, or more severe ADHD symptoms than girls [47,48]. In the Korean school-based population aged 6–16 years, boys showed more symptoms of ADHD-IA and HI than girls [49]. By contrast, prior studies report inconclusive findings on sex differences between ODD and CD symptoms [50-52]. The result of Park et al. [49] indicated an absence of sex differences in CD scores, which contrasts with the results of our study. These results can be attributed to methodological discrepancies, such as sampling. In addition, we stratified age into three groups to investigate the differences in the DBDRS subscale scores and found a significant difference only in ADHD-HI scores. This was also seen in the results of prior studies [33,48,53]. Specifically, the number or severity of symptoms was higher among younger children, similar to prior studies.
These comparison outcomes should be interpreted with a caveat. Some researchers suggested that the distribution of data could be biased, featuring very high or low scores, if the items were constructed based on the diagnostic criteria of DSM [26]. The DBDRS also consisted of items derived from the diagnostic criteria of DSM-5, and the distributions of all subscales were positively skewed. In other words, the majority of children and adolescents were rated with a score of nearly 0. Consequently, we analyzed the data with nonparametric methods, since the normality of distribution and homogeneity of variance was violated. Thus, we recommend that readers construe our results with caution.
Third, based on the results of ROC curve analyses, all AUC values of the DBDRS subscales far exceeded 0.70, which indicated that DBDRS has a good level of discriminative capacity and diagnostic accuracy to determine DBD symptoms. The optimal cut-off point in this study was 1.82 for ADHD-IA, 1.85 for ADHD-HI, 1.80 for ODD, and 1.83 for CD. DBDRS can accurately identify 90% of cases with a clinical level of DBD symptoms, based on the cut-off points. Furthermore, ROC curve analysis results for the DBDRS, ARS, and CBCL subscales assessing ADHD showed appropriate AUC values on all scales, and among them, the DBDRS ADHD-IA and HI had the highest AUC values. These results imply that the DBDRS subscales possess the discriminatory ability and diagnostic utility as a screening tool for DBD among Korean children and adolescents.
Finally, we investigated the diagnostic utility of each symptom of DBD by estimating conditional probability statistics for the DBDRS items. The symptoms with the highest PPP had simultaneously high NPP, implying that these symptoms are useful as inclusion and exclusion criteria for DBD when parents or caregivers act as informants. For the ADHD symptoms, our results indicated that “has difficulty sustaining attention” and “acts as if driven by a motor” were highest among the ADHD-IA and ADHD-HI symptoms, respectively; however, this result was inconsistent with previous results. Pelham et al. [28] investigated PPPs and NPPs for the DSM-III-R version of the DBDRS rated by teachers, finding that “often engages in dangerous activities without considering consequences” had the highest PPP. However, this finding is not directly comparable to our results, because not only were the analyses conducted without distinguishing the IA and HI symptoms, but also the items were slightly different from those of the DSM-5 version. On the other hand, some researchers reported the highest PPP as “forgetful in daily activities” for IA and “runs or climbs excessively” for HI based on the DSM-IV-TR diagnostic criteria [54]. Other researchers revealed the highest PPP was “has difficulty organizing tasks and activities” for IA and “acts as if driven by a motor” for HI, based on the DSM-IV diagnostic criteria [55]. Regarding ODD, the symptom with the highest PPP was “spiteful or vindictive” in this study, which was relatively consistent with prior results [28,55]. Moreover, the findings of Owens and Hoza [54] were consistent with ours, where “spiteful or vindictive” had the highest PPP and “blames others” had the lowest PPP. The CD symptoms with the highest PPPs in this study, including “setting fire with the intention of causing serious harm,” “deliberately destroyed property,” and “stays out at night,” were similar to those of Frick et al. [55].
This study has several limitations. First, the number of clinical participants was small, and all the primary diagnoses were ADHD. Thus, it was impossible to compare the DBDRS scores by sex and age in clinical samples and analyze the ROC curve for ODD- or CD-diagnosed cases. In a previous study, diverse sex differences were observed for externalizing behavior, attention problems, and aggressive behavior, depending on the presence or kind of diagnoses such as ADHD, ODD, or co-existing ADHD and ODD [56]. Thus, future studies should recruit a larger sample of children and adolescents with DBD diagnoses. By comparing the DBDRS scores in clinical samples, future studies can provide a better understanding of the nature of DBD symptoms in Koreans. In addition, there is limited knowledge about whether the clinical participants in our study received medication or psychotherapy. Given that treatment status can affect symptom severity, future studies should collect information about specific treatment conditions such as type and duration. Further, analyzing ROC curves for those with ODD or CD can strengthen the diagnostic utility of the DBDRS. Second, as previously stated, we gathered data from parents and primary caregivers given that many students were attending online classes due to COVID-19. However, for diagnosing DBD, children’s and adolescents’ behavior should be assessed in various settings by multiple informants. Future research on testing the psychometric properties of the DBDRS rated by teachers or any other informants, as well as examining the agreement among informants on the scale is required. Third, internal consistency was the only reliability measure in this study. Future research should conduct additional reliability analyses such as test-retest reliability, which examines the consistency of scores by repeating the same assessment with the same participants at certain time intervals. Finally, response bias may exist given that some parents and primary caregivers might have provided socially desirable responses to the questions about their children’s behavioral problems. To reduce response bias, future research could include measurements that assess social desirability (e.g., the social desirability scale).
Despite the limitations discussed, our findings have clinical implications. The results of our study showed that the K-DBDRS is a brief but comprehensive and relatively accurate scale for detecting DBD symptoms. As a preemptive screening tool for children and adolescents suspected of having DBD, the K-DBDRS is practical and beneficial in community settings. Furthermore, the K-DBDRS can be used in clinical settings by psychiatrists, psychologists, or other mental health professionals to identify the number of symptoms and their frequency. Moreover, it is a less time-consuming tool, thus allowing clinicians to spend less time identifying symptoms using the K-DBDRS and more time acquiring additional information such as severity and settings in which the symptoms present.
In conclusion, this study is the first attempt to investigate the utility of the K-DBDRS. Moreover, the results show an appropriate level of reliability and validity of the K-DBDRS. We particularly targeted children and adolescents aged 6–15 years, which is the most prevailing period for DBD symptoms, and collected data from their parents or primary caregivers who spend a majority of time with them. Thus, our findings demonstrate the suitability of the K-DBDRS as a screening tool for Korean children and adolescents, and can promote the formulation of early interventions for those at risk.
Notes
Availability of Data and Material
The datasets generated or analyzed during the study are not publicly available because the participants did not consent to share their raw data, but are available from the corresponding author on reasonable request.
Conflicts of Interest
Subin Park, a contributing editor of the Psychiatry Investigation, was not involved in the editorial evaluation or decision to publish this article. All remaining authors have declared no conflicts of interest.
Author Contributions
Conceptualization: Vin Ryu, Subin Park. Data curation: Eun Sol Lee. Formal analysis: Eun Sol Lee, Hyeree Han. Funding acquisition: Subin Park. Investigation: Eun Sol Lee, Jungwon Choi, Yunhye Oh, Jin Woong Yoon, Hyeon Hong, Hye Jung Son, Ji Hyun Lee. Methodology: Eun Sol Lee, Hyeree Han, Ji Hyun Lee. Project administration: Vin Ryu, Subin Park. Supervision: Vin Ryu, Subin Park. Visualization: Eun Sol Lee. Writing—original draft: Eun Sol Lee, Vin Ryu, Hyeon Hong, Hye Jung Son, Subin Park. Writing—review & editing: Eun Sol Lee, Vin Ryu, Subin Park.
Funding Statement
This work was supported by a grant from the Korea Mental Health R&D Project, funded by the Ministry of Health & Welfare, Republic of Korea (grant number HL19C0048).
Acknowledgements
The authors would like to thank William E. Pelham and his colleagues for approving the translation and validation of the Korean version of the DBDRS. In addition, we are very grateful to all participants for contributing their time and efforts to this study.