• Home
• E-Submission
• Sitemap
Psychiatry Investig Search

CLOSE

 Psychiatry Investig > Volume 19(11); 2022 > Article
Lee, Ryu, Choi, Oh, Yoon, Han, Hong, Son, Lee, and Park: Reliability and Validity of the Korean Version of Disruptive Behavior Disorders Rating Scale, DSM-5 Version-Parent Form

Objective

Disruptive behavior disorder (DBD) adversely impacts children and adolescents. However, a comprehensive and cost-effective scale to assess DBD is lacking in Korea. Therefore, this study translated the Disruptive Behavior Disorders Rating Scale (DBDRS) into Korean and analyzed its psychometric properties.

Methods

Parents and primary caregivers of non-clinical (n=429) and clinical (n=28) children and adolescents aged 6-15 years were included in the analysis. Confirmatory factor analysis was conducted; further, concurrent validity and internal consistency were investigated using correlation analysis and Cronbach’s alpha, respectively. Furthermore, discriminative capacity was estimated using receiver operating characteristic curve analysis.

Results

The four-factor model of K-DBDRS showed good model fit indices and factor loadings, which supported the construct validity of the scale. Strong correlations between K-DBDRS and related measurements were observed, and a robust level of Cronbach’s alpha was confirmed (0.891-0.933). The discriminative capacity of the scale was good, based on the area under the curve values (0.933-0.953).

Conclusion

This study indicated that the K-DBDRS is an appropriate screening tool for Korean children and adolescents. Thus, this scale can be applied in clinical and community settings to identify children and adolescents with disruptive behavior disorders.

INTRODUCTION

Disruptive behavior disorder (DBD) is characterized by social disruptions, which encompasses attention-deficit/hyperactivity disorder (ADHD), oppositional defiant disorder (ODD), and conduct disorder (CD). DBD is common in Korean children and adolescents; moreover, reports suggest that the prevalence of ADHD in Korean elementary school students ranged from 5.9%-6.5%, and the rate of sub-threshold symptom level was 9.0% [1,2]. In addition, the prevalence of childhood ADHD, retrospectively reported by college students, was 7.3% [3]. These rates indicate that at least one student in a class may have ADHD. Further, the prevalence of ODD in Korean elementary school students was 4.9%, and the rate of those with subclinical symptoms was 7.7% [1]. The prevalence of CD and the rate for sub-threshold levels of CD were 0.4% and 0.8%, respectively [1]. The rates of ADHD and ODD were similar, whereas CD was less prevalent than ADHD and ODD in the community. However, a study showed that more than half of the adolescents who were juvenile offenders in custody met the diagnostic criteria for CD [4].
Children and adolescents with DBD during the developmental course could experience long-term negative outcomes, including poor intellectual performance [5], familial and interpersonal conflicts [6,7], criminal activities [8-10], and poor health conditions or early death [11,12]. DBD symptoms diagnosed in childhood tend to persist until early adulthood, thus increasing the probability of comorbidities with other psychiatric disorders [13-15]. In addition, DBD leads to increased social costs and loss of human resources. In the United States, the loss of labor productivity related to ADHD was estimated at $67 billion-$116 billion for one year [5], and the cost of providing mental health services or treatment for CD exceeded \$7,000 per child [16]. Thus, previous results highlight the need for screening DBD symptoms in the early developmental stages to prevent adverse effects of the disorders.
To identify DBD, prior studies utilized distinct measurements to assess ADHD, ODD, or CD symptoms. Valid and widely used scales in Korea include the ADHD Rating Scale (ARS) [17], ADHD Diagnostic System [18], and Computerized Neurocognitive Function Tests [19]. Regarding ODD and CD, the reliabilities, validities, and diagnostic accuracy of the Oppositional Defiant Disorder Rating Scale [20] and Conduct Disorder Rating Scale [21], which were developed in Canada, have been verified. However, these scales have not yet been translated and validated in the Korean context. Moreover, there is a dearth of other assessments for ODD or CD. Previous studies found that ADHD, ODD, and CD were often diagnosed as comorbid conditions, and that these cases indicated more severe symptoms, poorer functional abilities, and more difficulties in school [22,23]. Therefore, a comprehensive assessment for DBD is required to discover symptoms that cannot be detected by a single scale. The integrated measurements widely used to assess DBD among Korean children and adolescents are the Child Behavior Checklist (CBCL) [24] and Behavioral Assessment System for Children [25]. Their psychometric properties have been substantiated in previous Korean studies [26,27]. Nevertheless, they have limitations in their application in clinical and community settings owing to their expensive and time-consuming nature.
The Disruptive Behavior Disorders Rating Scale (DBDRS) [28] was developed to facilitate the screening of DBD symptoms and diagnostic decision-making. At first, the DBDRS was developed using the Diagnostic and Statistical Manual of Mental Disorders (DSM)-III-R diagnostic criteria, but it has been updated following the publication of DSM-IV-TR and DSM5. The DBDRS includes items measuring ADHD, ODD, and CD symptoms. Pelham et al. [28] suggested that the proportion of boys who meet the diagnostic cutoffs for ADHD, ODD, or CD using the DBDRS was similar to the prevalence rate reported in prior research. Further, prior reports confirmed that the DBDRS has an appropriate level of reliability and validity [28,29]. In addition, a previous study demonstrated that the DBDRS was as valid as a clinical interview [30]. Particularly, there was an 87.5% degree of agreement between the DBDRS as reported by a parent and the Diagnostic Interview Schedule for Children-IV in classifying children as ADHD or non-ADHD, which is considered relatively high. These prior studies implied that the DBDRS can overcome the constraints of the aforementioned measurements, specifically because it is an encompassing scale for DBD and less time-consuming. Utilizing the DBDRS as a screening tool in community and clinical settings can help identify unrecognized symptoms and promote the formulation of early intervention plans for children and adolescents with DBD symptoms.
Although some researchers translated and used DBDRS to measure DBD symptoms in Korean children [31], no study has confirmed the validity of a Korean version of the DBDRS. Therefore, we investigated the reliability and validity of the Korean version of DBDRS (K-DBDRS) to measure DBD among children and adolescents. While both parents and teachers can respond to the DBDRS, we only recruited parents and primary caregivers for the following reasons. First, teachers can evaluate social, behavioral, and academic problems because they are exposed to students in the classroom; however, the majority of Korean students attend online classes because of the coronavirus disease (COVID-19). Even though some students attend in-person school, teachers are unable to observe children’s and adolescents’ behaviors, given that activities and interactions are restricted to avoid contagion. Moreover, teachers may face difficulties in assessing all students, given that approximately 22 students are allocated per teacher in Korean schools [32]. Considering the availability of the DBDRS, especially for large-scale screening, collecting data from parents is more efficient.
The objectives of this study are as follows: 1) to examine the reliability and validity of the K-DBDRS in children and adolescents; 2) to compare the non-clinical and clinical groups diagnosed with ADHD, while investigating sex and age differences in the non-clinical group based on the results of prior research [33,34]; 3) to investigate the discriminative capacity and diagnostic accuracy of the K-DBDRS by comparing it to other ADHD measurements, and confirming the cutoff points to screen children and adolescents with a clinical level of symptoms; and 4) to explore diagnostic predictabilities for each item of the K-DBDRS.

Ethics

This study was approved by the Institutional Review Board of the National Center for Mental Health (approval number: 116271-2021-05). Written informed consent to participate in the research was obtained from all participants. All methods used in this study were conducted in accordance with the relevant guidelines and regulations.

Participants

Non-clinical group

Data were collected from 510 parents and primary caregivers of children and adolescents aged 6-15 years who were not diagnosed with ADHD, ODD, or CD. We recruited non-clinical participants from online communities (e.g., Daum and NAVER cafes), which are easily accessible in various regions of South Korea. Eligibility was based on (a) no current diagnosis of mental disorders including DBD and (b) non-participation in special education. Furthermore, duplicate or inconsistent responses were excluded. Consequently, we used 429 participants’ data for analysis.

Clinical group

The clinical group was recruited from a child and adolescent outpatient clinic at a mental health center in Seoul, Korea. Eligibility was based on: (a) diagnosis of one or more DBD and (b) no indication of an intellectual disorder, autism, a psychotic disorder, epilepsy, or gross brain damage. Similar to the non-clinical group, parents or primary caregivers of children and adolescents aged 6-15 years responded to the survey. We collected and used data from 28 parents and primary caregivers of children and adolescents with ADHD (n=21), ADHD with ODD (n=2), or ADHD with a mood disorder (n=5).
Table 1 shows the socio-demographic characteristics of participants in this study. There were 230 girls (53.6%) and 199 boys (46.4%) in the non-clinical group, and 5 girls (17.9%) and 23 boys (82.1%) in the clinical group. The age was stratified into the following ranges: 6-8 years (39.6%, non-clinical group; 42.9%, clinical group), 9-11 years (37.3%, non-clinical group; 35.7%, clinical group), and 12-15 years (23.1%, nonclinical group; 21.4%, clinical group). A majority of the respondents were mothers (93.5%, non-clinical group; 85.7%, clinical group). Regarding the education level of parents and primary caregivers, the majority were university/college graduates in both groups (80.9%, non-clinical group; 60.7%, clinical group). A chi-squared test of independence was conducted, and all socio-demographic characteristics except for age showed a significant difference (see Table 1).

Procedure

We created an online survey, which was only administered to parents or primary caregivers who voluntarily accessed the link and consented to participate in the study. The description detailed the outline and purpose of the study, as well as the research process. Moreover, participant eligibility was displayed on the online community to recruit non-clinical participants, whereas a flyer was displayed on a bulletin board to recruit clinical participants. We explained our study procedures to participants, after they expressed their interest in participation. Next, we sent them a link to the online survey. All participants provided written informed consent before responding to the questionnaire. After completion of the survey, participants were rewarded with a mobile gift card of 5,000 KRW for the non-clinical group and 15,000 KRW for the clinical group. Data collection lasted over 4 months, from April-July 2021.

Translation

The English DBDRS was translated into Korean by two researchers with a master’s degree in psychology and one with a master’s degree in occupational therapy. Next, a bilingual professional back-translated the original translation. Experts including a psychology professor, psychiatrist, and researchers thoroughly reviewed and revised all of the translated versions to reach a consensus. Finally, three psychiatrists who did not participate in the translation reviewed the K-DBDRS to determine whether the contents were relevant to the Korean cultural context.

Measures

Background information

To identify the socio-demographic characteristics of all participants, information from parents or primary caregivers (i.e., relationship with children and adolescents, sex, age, education level, and household income) and information from the target children and adolescents (i.e., sex, age, psychiatric history, and present psychiatric diagnosis) were investigated.

Disruptive Behavior Disorders Rating Scale

The DBDRS is a scale comprising 45 items extracted from the diagnostic criteria of DBD [28]. It has four subscales: ADHD inattention symptoms (ADHD-IA), ADHD hyperactivity/impulsivity symptoms (ADHD-HI), ODD symptoms, and CD symptoms. Each item is rated on a 4-point Likert scale ranging from 0 (not at all) to 3 (very much). A rating of 0 or 1 indicates the absence of symptoms, whereas that of 2 or 3 indicates the presence of symptoms. As reported by Pelham et al. [28], the DBDRS is a reliable scale in terms of internal consistency, with 0.95 for ADHD-IA, 0.95 for ADHD-HI, and 0.96 for ODD. Moreover, the parent-reported DBDRS has good internal consistency, ranging from 0.94-0.97 [30]. In this study, we received the DSM-5 version of the DBDRS from the original author and used 41 items, excluding 4 items from the previous version of the DSM. The subscale scores of the DBDRS can be computed in two ways: averaging the items of each factor or counting symptoms rated 2 or 3. In this study, the subscale scores were calculated by averaging the items of each factor. This calculation was used to increase the utility of the DBDRS as a screening tool. As a result, the range of each subscale score was from 0 to 3.

The ARS was used to test the concurrent validity of the K-DBDRS [17]. The ARS was developed to measure ADHD symptoms in school-aged children. Moreover, it was standardized in Korea for elementary school students with and without ADHD [35]. The scale comprises two subscales (i.e., ADHD inattention and ADHD hyperactivity/impulsivity), each 9 items, for a total of 18 items. The items are rated on a 4-point Likert scale ranging from 0 (never or rarely) to 3 (very often) based on the frequency of symptoms. Two subscale scores were derived from summing the scores of each of the nine items (subscale scores ranged from 0-27). Previous studies with Korean parents confirmed the reliability and validity of the ARS [36]. The Cronbach’s alpha for the ARS subscales in this study were 0.93 and 0.91 for ADHD-IA and ADHD-HI, respectively.

Child Behavior Checklist for ages 6-18

The CBCL was used to determine the concurrent validity of the K-DBDRS. The CBCL is used to assess various emotional and behavioral problems among children and adolescents [24]. We used DSM-oriented scales comprising classified items depending on the diagnostic criteria of DSM. Among them, DSM ADHD, DSM ODD, and DSM CD were included in this study. Each behavior problem is rated on a 3-point Likert scale, ranging from 0 (not at all) to 2 (often or a lot). The Korean standardization study reported good internal consistency and 1-week test-retest reliability of three DSM-oriented scales; however, these measures were lower than those reported in the normative group in the United States [37]. In this study, the Cronbach’s alpha for the three scales were 0.88 for ADHD, 0.82 for ODD, and 0.86 for CD.

Statistical analyses

Statistical analyses were conducted using IBM Statistical Package for the Social Sciences (SPSS; IBM Co., Armonk, NY, USA) version 22.0 and Mplus 8.5 (Muthén & Muthén, Los Angeles, CA, USA). We examined descriptive statistics to understand the nature of the data, which revealed that the DBDRS scores did not follow a normal distribution. In addition, the items of the DBDRS are rated on a four-point Likert scale, which is considered to have ordinal characteristics. We analyzed the data based on these conditions. First, we performed confirmatory factor analysis (CFA). Pelham et al. [28] yielded three factors from the DBDRS, including inattention, impulsivity/hyperactivity, and oppositional/defiant by principal components analysis. However, these factors comprised 26 of the total 45 items, and the items to measure conduct problems were not included. Moreover, we considered that the threefactor structure derived from the DSM-III-R was unsuitable for the DSM-5 version of the DBDRS, given that the description of the items had changed and the three-factor model of the latest version of the DBDRS had not been tested. Therefore, we conducted CFA based on the diagnostic criteria of DSM-5. To analyze the ordinal data, we used weighted least squared means and variance adjusted (WLSMV) estimation. It has been suggested that WLSMV can more precisely estimate the magnitude of factor loadings with ordinal data, compared with maximum likelihood, regardless of the sample size [38]. The statistical measures used to evaluate the goodness of fit of the CFA model included the comparative fit index (CFI), Tucker-Lewis index (TLI), and the root mean square error of approximation (RMSEA). CFI and TLI values greater than 0.90 [39] and RMSEA values of less than 0.05 [40] indicate a good model fit.
The Mann-Whitney U tests were used to compare the nonclinical and clinical groups according to their levels of DBD symptoms on the DBDRS, ARS, and CBCL. Moreover, sex differences in the non-clinical group were examined on the DBDRS subscales. The Kruskal-Wallis test was conducted for all DBDRS subscales to investigate the differences in the ages of the non-clinical group. Spearman’s rank-order correlations were calculated between the DBDRS, ARS, and CBCL to determine the concurrent validity. Cronbach’s alpha for each subscale was yielded, and a value greater than 0.60 was considered to indicate good reliability [41].
Receiver operating characteristic (ROC) curve analysis was conducted to provide a cut-off point and test the discriminative capacity and diagnostic accuracy for the K-DBDRS. The T score of the CBCL DSM-oriented scales for diagnosing the clinical level of ADHD, ODD, and CD is 70T and above in Koreans [37]. With this reference, the diagnosis presence or absence cases were classified to conduct ROC curve analysis. Sensitivity and specificity were computed, and the cut-off points were estimated using Youden’s index for each subscale of the DBDRS [42]. In addition, we investigated the area under the curve (AUC), which enables the examination of the discriminative capacity and diagnostic accuracy of the scale. An AUC value of 0.70 or above indicates appropriate discrimination and accuracy [43,44]. Furthermore, we compared the ADHD subscales of the DBDRS with those of the ARS and CBCL and confirmed which scale had the highest discriminative capacity. The cases were categorized according to whether the children and adolescents received an ADHD diagnosis from their psychiatrists.
Finally, we tested the diagnostic predictive power of each item of the DBDRS. Predictive power refers to the probability of agreement between the diagnosis and the result of the test. The positive predictive power (PPP) or negative predictive power (NPP) can be computed according to the test outcomes. The PPP refers to the proportion of true-positive cases among all positive cases, while the NPP is the rate of true-negative cases among all negative cases [45].

Confirmatory factor analysis

CFA was conducted. As shown in Table 2, the fit indices of the four-factor model were good (χ2=1226.116, df=773, p<0.001, CFI=0.982, TLI=0.981, standardized root mean residual [SRMR]=0.089, RMSEA=0.036). All factor loadings were significant and higher than 0.70. Factor loadings of the items were 0.755-0.928 and 0.703-0.888 in ADHD-IA and ADHDHI, respectively. Factor loadings of ODD ranged from 0.822-0.933, and those of CD ranged from 0.780-0.968 (Figure 1).

Comparison

To investigate the differences between the non-clinical and clinical groups, we performed the Mann-Whitney U tests. As shown in Table 3, there were significant differences between the two groups in the DBDRS subscales: ADHD-IA (Mann-Whitney U=1,435.50, Z=-6.848, p<0.001), ADHD-HI (Mann-Whitney U=1,589.00, Z=-6.664, p<0.001, ODD (Mann-Whitney U=2,093.50, Z=-5.942, p<0.001), and CD (Mann-Whitney U=3,473.50, Z=-4.398, p<0.001).
Based on the chi-squared test results for socio-demographic characteristics, we conducted additional analysis to compare the mean scores after controlling for confounders (i.e., sex, respondent’s relationship to children and adolescents, education level, and household income). Analysis of covariance using the rank transformation was performed. Overall subscale scores of the DBDRS were significantly different after controlling for these confounders (F=40.451, p<0.001 for ADHD-IA; F=36.679, p<0.001 for ADHD-HI; F=29.817, p<0.001 for ODD; F=12.711, p<0.001 for CD).
Table 4 indicates the sex differences in the non-clinical group. On the ADHD-IA (Mann-Whitney U=18,013.00, Z=-3.87, p<0.001), ADHD-HI (Mann-Whitney U=19,045.00, Z=-3.08, p<0.01), ODD (Mann-Whitney U=19,576.00, Z=-2.67, p<0.01), and CD (Mann-Whitney U=20,592.50, Z=-2.15, p<0.05), significant differences were found between boys’ and girls’ scores.
The Kruskal-Wallis test showed no significant differences among the age groups (Table 5), except for ADHD-HI (Kruskal-Wallis=12.37, df=2, p<0.01). Post hoc tests revealed that the 12-15 years age group was significantly different from the 6-8 years (Mann-Whitney U=6,320.00, Z=-3.50, p<0.001) and the 9-11 years age group (Mann-Whitney U=6,447.00, Z=-2.60, p<0.01).

Concurrent validity

The DBDRS ADHD-IA was significantly correlated with the ARS ADHD-IA (rs=0.85, p<0.01) and CBCL ADHD (rs=0.78, p<0.01). Moreover, the DBDRS ADHD-HI indicated a significant correlation with the ARS ADHD-HI (rs=0.84, p<0.01) and CBCL ADHD (rs=0.73, p<0.01). In addition, the DBDRS ODD and CD subscales had significant relationships with the CBCL ODD (rs=0.71, p<0.01) and CD (rs=0.55, p<0.01), respectively. Table 6 shows the associations between the DBDRS and other measurements.

Reliability

The internal consistency of the DBDRS was examined based on Cronbach’s alpha for each subscale. Cronbach’s alpha was 0.924 for ADHD-IA, 0.891 for ADHD-HI, 0.906 for ODD, and 0.933 for CD, thus indicating excellent consistency for all subscales.

Cut-off point

ROC curves using the DBDRS for the total sample are presented in Figure 2. All ROC curves lie in the upper left part of the reference line, and all of the values of AUC ranged from 0.933-0.953 (Table 7). These results indicate that the DBDRS had a good discriminative capacity for ADHD, ODD, and CD diagnoses. The cut-off point for ADHD-IA was 1.82, with a sensitivity of 94% and specificity of 88% (Youden’s Index=0.82), whereas that for ADHD-HI was 1.85, with a sensitivity of 94% and specificity of 92% (Youden’s Index=0.85). Moreover, the cut-off point for ODD was 1.80, with a sensitivity of 100% and specificity of 80% (Youden’s Index=0.80), whereas that for CD was 1.83, with a sensitivity of 93% and specificity of 89% (Youden’s Index=0.83).
In addition, we yielded and compared the AUC values of all ADHD measurements. The DBDRS ADHD-IA was 0.880, DBDRS ADHD-HI was 0.868, ARS ADHD-IA was 0.865, ARS ADHD-HI was 0.853, and CBCL ADHD was 0.863 (Figure 3 and Table 8). All of these measures demonstrated a discriminative power for diagnosing ADHD, and among the scales, the DBDRS ADHD-IA and ADHD-HI showed the highest value of AUC.

Positive and negative predictive power

The conditional probability statistics for the items of the DBDRS were analyzed to investigate the diagnostic predictive power of each DBD symptom. Table 9 presents the results. The presence of the symptom was determined using the ratings “pretty much” or “very much.” Moreover, the diagnoses of ADHD, ODD, and CD were determined by the number of symptoms suggested in DSM-5.
The PPP rates of ADHD-IA symptoms in predicting ADHD predominantly inattentive type diagnosis ranged from 0.37-0.61. Specifically, the symptom with the highest PPP was “has difficulty sustaining attention,” whereas the lowest PPP was “easily distracted.” Regarding the ADHD-HI, the PPP rates in predicting ADHD for predominantly hyperactive/impulsive type diagnosis varied from 0.18-0.47. “Acts as if driven by a motor” had the highest PPP, whereas “talks excessively” had the lowest PPP. Overall, PPP rates of ADHD-HI symptoms were lower than those of ADHD-IA symptoms. In ODD symptoms, the PPP rates for predicting ODD diagnosis ranged from 0.54-0.89. The symptom with the highest PPP was “spiteful or vindictive,” whereas that with the lowest PPP was “blames others for one’s mistakes.” The PPP rates of CD symptoms predicting CD diagnosis varied from 0.40-1.00, and it was higher than the PPP ranges of ADHD and ODD symptoms. The majority of PPPs for CD symptoms was close to 1.00, except for “lies to obtain goods and favors” and “truant from school.” Finally, NPP ranged from 0.97-1.00 for all symptoms of DBDRS, suggesting high rates.

DISCUSSION

This study investigated the psychometric properties of the DBDRS for use with Korean parents and primary caregivers of children and adolescents aged 6-15. The findings demonstrate that the K-DBDRS can suitably measure disruptive behavior symptoms in both clinical and non-clinical participants. This study had four purposes: 1) to test the reliability and validity of the DBDRS in the Korean context; 2) to compare the K-DBDRS scores in the clinical and non-clinical groups, including by sex and age; 3) to examine the cut-off point and the discriminative capacity of the K-DBDRS; and 4) to estimate the diagnostic predictive power of the K-DBDRS items. The results of the study are summarized as follows.
First, we evaluated the four-factor model in compliance with the disorder categories and diagnostic criteria of DSM-5. The results of the CFA confirmed a robust level of model fit indices and a good level of factor loadings, which demonstrates appropriate construct validity of the K-DBDRS. In a previous study, CFA was conducted for ADHD and community teenagers using the DSM-IV version of the DBDRS [46]. Specifically, the researchers analyzed a three-factor model composed of ADHD-IA, ADHD-HI, and ODD, and used items identical to those used in our study. The results indicated that model fit indices were acceptable, and factor loadings were within an appropriate range, similar to ours. Moreover, the three-factor model was best fitted to the data compared to alternative models, which included the DBD single-factor model and two-factor model consisting of ADHD and ODD. The factor analysis outcomes for the ADHD-IA, ADHD-HI, and ODD items concurred with our results, except for the CD items. In addition, the internal consistencies of the DBDRS subscales were in excellent ranges, and the correlations between the DBDRS, ARS, and CBCL were significant, which supports the concurrent validity of the K-DBDRS.
Second, significant differences were found between the clinical and non-clinical groups on the DBDRS subscale scores. The mean scores overall were higher in the clinical group than in the non-clinical group. In other words, participants with ADHD showed a higher symptom frequency. The ARS and CBCL subscales measuring similar constructs to the DBDRS also presented significant differences between the two groups and higher mean scores in the clinical group. Moreover, as expected, sex differences were observed in all DBDRS subscale scores. These findings were consistent with previously published studies, where boys reported a higher number of, or more severe ADHD symptoms than girls [47,48]. In the Korean school-based population aged 6-16 years, boys showed more symptoms of ADHD-IA and HI than girls [49]. By contrast, prior studies report inconclusive findings on sex differences between ODD and CD symptoms [50-52]. The result of Park et al. [49] indicated an absence of sex differences in CD scores, which contrasts with the results of our study. These results can be attributed to methodological discrepancies, such as sampling. In addition, we stratified age into three groups to investigate the differences in the DBDRS subscale scores and found a significant difference only in ADHD-HI scores. This was also seen in the results of prior studies [33,48,53]. Specifically, the number or severity of symptoms was higher among younger children, similar to prior studies.
These comparison outcomes should be interpreted with a caveat. Some researchers suggested that the distribution of data could be biased, featuring very high or low scores, if the items were constructed based on the diagnostic criteria of DSM [26]. The DBDRS also consisted of items derived from the diagnostic criteria of DSM-5, and the distributions of all subscales were positively skewed. In other words, the majority of children and adolescents were rated with a score of nearly 0. Consequently, we analyzed the data with nonparametric methods, since the normality of distribution and homogeneity of variance was violated. Thus, we recommend that readers construe our results with caution.
Third, based on the results of ROC curve analyses, all AUC values of the DBDRS subscales far exceeded 0.70, which indicated that DBDRS has a good level of discriminative capacity and diagnostic accuracy to determine DBD symptoms. The optimal cut-off point in this study was 1.82 for ADHD-IA, 1.85 for ADHD-HI, 1.80 for ODD, and 1.83 for CD. DBDRS can accurately identify 90% of cases with a clinical level of DBD symptoms, based on the cut-off points. Furthermore, ROC curve analysis results for the DBDRS, ARS, and CBCL subscales assessing ADHD showed appropriate AUC values on all scales, and among them, the DBDRS ADHD-IA and HI had the highest AUC values. These results imply that the DBDRS subscales possess the discriminatory ability and diagnostic utility as a screening tool for DBD among Korean children and adolescents.
Finally, we investigated the diagnostic utility of each symptom of DBD by estimating conditional probability statistics for the DBDRS items. The symptoms with the highest PPP had simultaneously high NPP, implying that these symptoms are useful as inclusion and exclusion criteria for DBD when parents or caregivers act as informants. For the ADHD symptoms, our results indicated that “has difficulty sustaining attention” and “acts as if driven by a motor” were highest among the ADHD-IA and ADHD-HI symptoms, respectively; however, this result was inconsistent with previous results. Pelham et al. [28] investigated PPPs and NPPs for the DSM-III-R version of the DBDRS rated by teachers, finding that “often engages in dangerous activities without considering consequences” had the highest PPP. However, this finding is not directly comparable to our results, because not only were the analyses conducted without distinguishing the IA and HI symptoms, but also the items were slightly different from those of the DSM-5 version. On the other hand, some researchers reported the highest PPP as “forgetful in daily activities” for IA and “runs or climbs excessively” for HI based on the DSM-IV-TR diagnostic criteria [54]. Other researchers revealed the highest PPP was “has difficulty organizing tasks and activities” for IA and “acts as if driven by a motor” for HI, based on the DSM-IV diagnostic criteria [55]. Regarding ODD, the symptom with the highest PPP was “spiteful or vindictive” in this study, which was relatively consistent with prior results [28,55]. Moreover, the findings of Owens and Hoza [54] were consistent with ours, where “spiteful or vindictive” had the highest PPP and “blames others” had the lowest PPP. The CD symptoms with the highest PPPs in this study, including “setting fire with the intention of causing serious harm,” “deliberately destroyed property,” and “stays out at night,” were similar to those of Frick et al. [55].
This study has several limitations. First, the number of clinical participants was small, and all the primary diagnoses were ADHD. Thus, it was impossible to compare the DBDRS scores by sex and age in clinical samples and analyze the ROC curve for ODD- or CD-diagnosed cases. In a previous study, diverse sex differences were observed for externalizing behavior, attention problems, and aggressive behavior, depending on the presence or kind of diagnoses such as ADHD, ODD, or co-existing ADHD and ODD [56]. Thus, future studies should recruit a larger sample of children and adolescents with DBD diagnoses. By comparing the DBDRS scores in clinical samples, future studies can provide a better understanding of the nature of DBD symptoms in Koreans. In addition, there is limited knowledge about whether the clinical participants in our study received medication or psychotherapy. Given that treatment status can affect symptom severity, future studies should collect information about specific treatment conditions such as type and duration. Further, analyzing ROC curves for those with ODD or CD can strengthen the diagnostic utility of the DBDRS. Second, as previously stated, we gathered data from parents and primary caregivers given that many students were attending online classes due to COVID-19. However, for diagnosing DBD, children’s and adolescents’ behavior should be assessed in various settings by multiple informants. Future research on testing the psychometric properties of the DBDRS rated by teachers or any other informants, as well as examining the agreement among informants on the scale is required. Third, internal consistency was the only reliability measure in this study. Future research should conduct additional reliability analyses such as test-retest reliability, which examines the consistency of scores by repeating the same assessment with the same participants at certain time intervals. Finally, response bias may exist given that some parents and primary caregivers might have provided socially desirable responses to the questions about their children’s behavioral problems. To reduce response bias, future research could include measurements that assess social desirability (e.g., the social desirability scale).
Despite the limitations discussed, our findings have clinical implications. The results of our study showed that the K-DBDRS is a brief but comprehensive and relatively accurate scale for detecting DBD symptoms. As a preemptive screening tool for children and adolescents suspected of having DBD, the K-DBDRS is practical and beneficial in community settings. Furthermore, the K-DBDRS can be used in clinical settings by psychiatrists, psychologists, or other mental health professionals to identify the number of symptoms and their frequency. Moreover, it is a less time-consuming tool, thus allowing clinicians to spend less time identifying symptoms using the K-DBDRS and more time acquiring additional information such as severity and settings in which the symptoms present.
In conclusion, this study is the first attempt to investigate the utility of the K-DBDRS. Moreover, the results show an appropriate level of reliability and validity of the K-DBDRS. We particularly targeted children and adolescents aged 6-15 years, which is the most prevailing period for DBD symptoms, and collected data from their parents or primary caregivers who spend a majority of time with them. Thus, our findings demonstrate the suitability of the K-DBDRS as a screening tool for Korean children and adolescents, and can promote the formulation of early interventions for those at risk.

Notes

Availability of Data and Material

The datasets generated or analyzed during the study are not publicly available because the participants did not consent to share their raw data, but are available from the corresponding author on reasonable request.

Conflicts of Interest

Subin Park, a contributing editor of the Psychiatry Investigation, was not involved in the editorial evaluation or decision to publish this article. All remaining authors have declared no conflicts of interest.

Author Contributions

Conceptualization: Vin Ryu, Subin Park. Data curation: Eun Sol Lee. Formal analysis: Eun Sol Lee, Hyeree Han. Funding acquisition: Subin Park. Investigation: Eun Sol Lee, Jungwon Choi, Yunhye Oh, Jin Woong Yoon, Hyeon Hong, Hye Jung Son, Ji Hyun Lee. Methodology: Eun Sol Lee, Hyeree Han, Ji Hyun Lee. Project administration: Vin Ryu, Subin Park. Supervision: Vin Ryu, Subin Park. Visualization: Eun Sol Lee. Writing—original draft: Eun Sol Lee, Vin Ryu, Hyeon Hong, Hye Jung Son, Subin Park. Writing—review & editing: Eun Sol Lee, Vin Ryu, Subin Park.

Funding Statement

This work was supported by a grant from the Korea Mental Health R&D Project, funded by the Ministry of Health & Welfare, Republic of Korea (grant number HL19C0048).

ACKNOWLEDGEMENTS

The authors would like to thank William E. Pelham and his colleagues for approving the translation and validation of the Korean version of the DBDRS. In addition, we are very grateful to all participants for contributing their time and efforts to this study.
Figure 1.
Four-factor confirmatory factor analysis model of the Disruptive Behavior Disorders Rating Scale. ADHD-IA, ADHD inattention symptoms; ADHD-HI, ADHD hyperactivity/impulsivity symptoms; ODD, oppositional defiant disorder; CD, conduct disorder.
Figure 2.
ROC curves for DBDRS’s ADHD, ODD, and CD subscales. ROC, receiver operating characteristic; DBDRS, Disruptive Behavior Disorders Rating Scale; ADHD, attention-deficit/hyperactivity disorder; ODD, oppositional defiant disorder; CD, conduct disorder.
Table 1.
Participants’ socio-demographic characteristics
Non-clinical Clinical Total χ2
Total 429 (93.9) 28 (6.1) 457 (100)
Sex 13.453***
Male 199 (46.4) 23 (82.1) 222 (48.6)
Female 230 (53.6) 5 (17.9) 235 (51.4)
Age (yr) 0.118
6-8 170 (39.6) 12 (42.9) 182 (39.8)
9-11 160 (37.3) 10 (35.7) 170 (37.2)
12-15 99 (23.1) 6 (21.4) 105 (23.0)
Parents & primary caregivers
Relationship 31.671***
Mother 401 (93.5) 24 (85.7) 425 (93.0)
Father 20 (4.7) 2 (7.1) 22 (4.8)
Grandmother 4 (0.9) - 4 (0.9)
Siblings & relations 4 (0.9) - 4 (0.9)
Facility caregiver - 2 (7.1) 2 (0.4)
Education 11.005**
≤High school 47 (11.0) 9 (32.1) 56 (12.3)
University/college 347 (80.9) 17 (60.7) 364 (79.6)
≥Graduate school 35 (8.2) 2 (7.1) 37 (8.1)
Income level (KRW) 31.224***
0-1.9 million 9 (2.1) 6 (21.4) 15 (3.3)
2-3.9 million 106 (24.7) 6 (21.4) 112 (24.5)
4-5.9 million 180 (42.0) 8 (28.6) 188 (41.1)
≥6 million 134 (31.2) 8 (28.6) 142 (31.1)

Values are presented as number (%).

** p<0.01;

*** p<0.001

Table 2.
Goodness-of-fit indices of the four-factor model
Model χ2 df CFI TLI SRMR RMSEA (90% CI)
Four-factor 1,226.116*** 773 0.982 0.981 0.089 0.036 (0.032-0.040)

*** p<0.001.

CFI, comparative fit index; TLI, Tucker-Lewis index; SRMR, standardized root mean residual; RMSEA, root mean square error of approximation; CI, confidence interval

Table 3.
Comparisons between mean scores of non-clinical and clinical samples for DBDRS and other related measurements
Non-clinical (N=429)
Clinical (N=28)
Mann-Whitney U value
M SD Mrank M SD Mrank
DBDRS
ADHD-IA 0.39 0.46 218.35 1.37 0.68 392.23 1,435.50***
ADHD-HI 0.30 0.40 218.70 1.10 0.56 386.75 1,589.00***
ODD 0.31 0.43 219.88 0.92 0.57 368.73 2,093.50***
CD 0.08 0.24 223.10 0.15 0.16 319.45 3,473.50***
ARS
ADHD-IA 3.76 4.30 218.77 12.11 5.64 385.77 1,616.50***
ADHD-HI 2.79 3.89 219.10 9.82 5.03 380.64 1,760.00***
CBCL
ADHD 53.90 6.45 218.83 68.50 12.85 384.88 1,641.50***
ODD 54.10 6.91 220.84 66.75 14.21 354.02 2,505.50***
CD 53.77 6.58 220.72 62.07 8.25 355.86 2,454.00***

*** p<0.001.

DBDRS, Disruptive Behavior Disorders Rating Scale; ADHD, attention-deficit/hyperactivity disorder; ADHD-IA, ADHD inattention symptoms; ADHD-HI, ADHD hyperactivity/impulsivity symptoms; ODD, oppositional defiant disorder; CD, conduct disorder; ARS, ADHD Rating Scale; CBCL, Child Behavior Checklist; M, mean; SD, standard deviation; Mrank, the mean rank used to analyze the differences between groups in the Mann-Whitney U test

Table 4.
Comparing mean scores by sexes in non-clinical samples for the DBDRS
Girl (N=230)
Boy (N=199)
Mann-Whitney U value
M SD Mrank M SD Mrank
ADHD-IA 0.32 0.42 193.82 0.47 0.49 239.48 18,013.00***
ADHD-HI 0.26 0.38 198.30 0.35 0.42 234.30 19,045.00**
ODD 0.27 0.38 200.61 0.37 0.48 231.63 19,576.00**
CD 0.08 0.27 205.03 0.07 0.19 226.52 20,592.50*

* p<0.05;

** p<0.01;

*** p<0.001.

DBDRS, Disruptive Behavior Disorders Rating Scale; ADHD-IA, ADHD inattention symptoms; ADHD-HI, ADHD hyperactivity/impulsivity symptoms; ODD, oppositional defiant disorder; CD, conduct disorder; M, mean; SD, standard deviation; Mrank, the mean rank used to analyze the differences between groups in the Mann-Whitney U test

Table 5.
Comparing mean scores by age groups in non-clinical samples for DBDRS
6-8 yr (N=170)
9-11 yr (N=160)
12-15 yr (N=99)
Kruskal-Wallis
M SD Mrank M SD Mrank M SD Mrank
ADHD-IA 0.40 0.49 218.82 0.39 0.48 212.02 0.35 0.37 213.25 0.283
ADHD-HI 0.36 0.45 232.06 0.32 0.41 219.17 0.17 0.22 178.96 12.37**
ODD 0.28 0.40 207.74 0.36 0.50 221.24 0.29 0.32 217.39 1.10
CD 0.10 0.28 217.24 0.08 0.25 221.87 0.03 0.07 200.05 2.86

** p<0.01.

DBDRS, Disruptive Behavior Disorders Rating Scale; ADHD-IA, ADHD inattention symptoms; ADHD-HI, ADHD hyperactivity/impulsivity symptoms; ODD, oppositional defiant disorder; CD, conduct disorder; M, mean; SD, standard deviation; Mrank, the mean rank used to analyze the differences between groups in the Kruskal-Wallis test

Table 6.
Correlations between the DBDRS and other related measurement
1 2 3 4 5 6 7 8 9
3 DBDRS ODD 0.72** 0.75** 1
4 DBDRS CD 0.49** 0.54** 0.56** 1
5 ARS ADHD-IA 0.85** 0.73** 0.71** 0.48** 1
6 ARS ADHD-HI 0.72** 0.84** 0.69** 0.52** 0.81** 1
7 CBCL ADHD 0.78** 0.73** 0.69** 0.48** 0.79** 0.76** 1
8 CBCL ODD 0.59** 0.64** 0.71** 0.44** 0.65** 0.67** 0.69** 1
9 CBCL CD 0.58** 0.60** 0.65** 0.55** 0.62** 0.61** 0.65** 0.70** 1

** p<0.01.

DBDRS, Disruptive Behavior Disorders Rating Scale; ADHD-IA, ADHD inattention symptoms; ADHD-HI, ADHD hyperactivity/impulsivity symptoms; ODD, oppositional defiant disorder; CD, conduct disorder; ARS, ADHD Rating Scale; CBCL, Child Behavior Checklist

Table 7.
AUC, SE, 95% confidence interval, and p-value for DBDRS
AUC SE 95% confidence interval
p-value
Lower Upper
DBDRS ADHD-IA 0.950 0.025 0.901 1.000 <0.001
DBDRS ADHD-HI 0.945 0.035 0.877 1.000 <0.001
ODD
DBDRS ODD 0.953 0.014 0.926 0.098 <0.001
CD
DBDRS CD 0.933 0.032 0.871 0.996 <0.001

Classified a case with symptoms based upon CBCL DSM subscales T score ≥70. DBDRS, Disruptive Behavior Disorders Rating Scale; ADHD, attention-deficit/hyperactivity disorder; ADHD-IA, ADHD inattention symptoms; ADHD-HI, ADHD hyperactivity/impulsivity symptoms; ODD, oppositional defiant disorder; CD, conduct disorder; AUC, area under the curve; SE, standard error

Table 8.
AUC, SE, 95% confidence interval, and p-value for ADHD measurements
AUC SE 95% confidence interval
p-value
Lower Upper
DBDRS ADHD-IA 0.880 0.034 0.813 0.948 <0.001
DBDRS ADHD-HI 0.868 0.038 0.794 0.942 <0.001
ARS ADHD-IA 0.865 0.042 0.784 0.947 <0.001
ARS ADHD-HI 0.853 0.042 0.771 0.936 <0.001
CBCL ADHD 0.863 0.039 0.786 0.941 <0.001

Classified a case with symptoms based upon Diagnostic and Statistical Manual of Mental Disorders-5 recommended cutoffs for diagnosis as ADHD. AUC, area under the curve; SE, standard error; ADHD, attention-deficit/hyperactivity disorder; DBDRS, Disruptive Behavior Disorders Rating Scale; ADHD-IA, ADHD inattention symptoms; ADHD-HI, ADHD hyperactivity/impulsivity symptoms; ARS, ADHD Rating Scale; CBCL, Child Behavior Checklist

Table 9.
Conditional probabilities of ADHD, ODD, and CD (N=457)
No. Symptom Base ratea SENb SPEc PPPd NPPe
9 Easily distracted 0.09 0.84 0.94 0.37 0.99
18 Does not seem to listen 0.06 0.58 0.96 0.38 0.98
23 Fails to give close attention to details 0.09 0.95 0.95 0.46 1.00
27 Does not follow instructions 0.07 1.00 0.97 0.56 1.00
29 Has difficulty sustaining attention 0.07 1.00 0.97 0.61 1.00
34 Loses things 0.07 0.63 0.95 0.38 0.98
37 Avoids engaging in tasks requiring sustained mental effort 0.07 0.89 0.97 0.55 1.00
42 Has difficulty organizing tasks and activities 0.05 0.63 0.97 0.50 0.98
44 Forgetful in daily activities 0.05 0.53 0.97 0.45 0.98
1 Interrupts others 0.07 0.73 0.94 0.24 0.99
7 Talks excessively 0.11 0.82 0.91 0.18 1.00
12 Fidgets or squirms 0.09 0.82 0.93 0.23 1.00
19 Blurts out answers 0.08 0.64 0.94 0.20 0.99
22 Has difficulty playing quietly 0.05 0.55 0.96 0.26 0.99
25 Leaves seat 0.05 0.82 0.97 0.43 1.00
30 Has difficulty awaiting turn 0.04 0.64 0.97 0.37 0.99
33 Acts as if “driven by a motor” 0.04 0.82 0.98 0.47 1.00
35 Runs about or climbs excessively 0.03 0.55 0.98 0.40 0.99
ODD
3 Argues with adults 0.06 0.81 0.98 0.65 0.99
13 Spiteful or vindictive 0.02 0.38 1.00 0.89 0.97
15 Blames others for one’s mistakes 0.08 0.95 0.96 0.54 1.00
17 Actively defies to comply adults’ requests or rules 0.05 0.62 0.98 0.57 0.98
24 Angry and resentful 0.07 0.90 0.97 0.63 1.00
26 Touchy or easily annoyed 0.06 0.76 0.97 0.59 0.99
28 Loses temper 0.04 0.62 0.99 0.72 0.98
39 Deliberately annoys people 0.02 0.29 0.99 0.67 0.97
CD
2 Ran away from home overnight 0.02 0.67 1.00 1.00 0.99
4 Lies to obtain goods and favors 0.04 0.67 0.97 0.40 0.99
6 Physically cruel to people 0.02 0.50 1.00 0.75 0.99
8 Has stolen items without confronting a victim 0.01 0.42 1.00 0.83 0.98
11 Truant from school 0.01 0.17 1.00 0.50 0.98
16 Deliberately destroyed property 0.01 0.42 1.00 1.00 0.98
20 Initiates physical fights 0.01 0.33 1.00 0.67 0.98
31 Forced someone into sexual activity 0.01 0.33 1.00 1.00 0.98
32 Bullies, threatens, or intimidates 0.01 0.42 1.00 0.83 0.98
36 Physically cruel to animals 0.01 0.33 1.00 1.00 0.98
38 Stays out at night 0.01 0.42 1.00 1.00 0.98
40 Has stolen while confronting a victim 0.00 0.17 1.00 1.00 0.98
41 Setting fire with the intention of causing serious harm 0.02 0.67 1.00 1.00 0.99
43 Has broken into house, building, or car 0.01 0.50 1.00 1.00 0.99
45 Has used a weapon 0.01 0.42 1.00 0.83 0.98

Disruptive Behavior Disorders Rating Scale items are briefly described.

a base rate=N who show disorder/total N;

b SEN=N with disorder who show symptom/N with disorder;

c SPE=N without disorder who do not show symptom/N without the disorder;

d PPP=N with disorder who show symptom/N with symptom;

e NPP=N without disorder who do not show symptom/N without symptom.

ADHD, attention-deficit/hyperactivity disorder; ODD, oppositional defiant disorder; CD, conduct disorder; ADHD-IA, ADHD inattention symptoms; ADHD-HI, ADHD hyperactivity/impulsivity symptoms; SEN, sensitivity; SPE, specificity; PPP, positive predictive power; NPP, negative predictive power

REFERENCES

1. Park S, Kim BN, Cho SC, Kim JW, Shin MS, Yoo HJ. Prevalence, correlates, and comorbidities of DSM-IV psychiatric disorders in children in Seoul, Korea. Asia Pac J Public Health 2015;27:NP1942-NP1951.

2. Yang SJ, Cheong S, Hong SD. Prevalence and correlates of attention deficit hyperactivity disorder: school-based mental health services in Seoul. J Korean Neuropsychiatr Assoc 2006;45:69-76.

3. Kim SJ, Ha EH. The prevalence and maladjustments according to persistence & decline of childhood ADHD and late-onset of adulthood ADHD in college students: executive functioning deficits, psychopathology and interpersonal problems. Kor J Clin Psychol 2020;39:134-147.

4. Choi BS, Kim JI, Kim BN, Kim B. Comorbidities and correlates of conduct disorder among male juvenile detainees in South Korea. Child Adolesc Psychiatry Ment Health 2017;11:44

5. Biederman J, Faraone SV. The effects of attention-deficit/hyperactivity disorder on employment and household income. MedGenMed 2006;8:12

6. Burke JD, Rowe R, Boylan K. Functional outcomes of child and adolescent oppositional defiant disorder symptoms in young adult men. J Child Psychol Psychiatry 2014;55:264-272.

7. Greene RW, Biederman J, Zerwas S, Monuteaux MC, Goring JC, Faraone SV. Psychiatric comorbidity, family dysfunction, and social impairment in referred youth with oppositional defiant disorder. Am J Psychiatry 2002;159:1214-1224.

8. Hodgins S, Cree A, Alderton J, Mak T. From conduct disorder to severe mental illness: associations with aggressive behaviour, crime and victimization. Psychol Med 2008;38:975-987.

9. Mannuzza S, Klein RG, Konig PH, Giampino TL. Hyperactive boys almost grown up. IV. Criminality and its relationship to psychiatric status. Arch Gen Psychiatry 1989;46:1073-1079.

10. Mordre M, Groholt B, Kjelsberg E, Sandstad B, Myhre AM. The impact of ADHD and conduct disorder in childhood on adult delinquency: a 30 years follow-up study using official crime records. BMC Psychiatry 2011;11:57

11. Barbaresi WJ, Colligan RC, Weaver AL, Voigt RG, Killian JM, Katusic SK. Mortality, ADHD, and psychosocial adversity in adults with childhood ADHD: a prospective study. Pediatrics 2013;131:637-644.

12. Pedersen W, Mastekaasa A. Conduct disorder symptoms and subsequent pregnancy, child-birth and abortion: a population-based longitudinal study of adolescents. J Adolesc 2011;34:1025-1033.

13. Barkley RA, Fischer M, Smallish L, Fletcher K. The persistence of attention-deficit/hyperactivity disorder into young adulthood as a function of reporting source and definition of disorder. J Abnorm Psychol 2002;111:279-289.

14. Lavigne JV, Cicchetti C, Gibbons RD, Binns HJ, Larsen L, DeVito C. Oppositional defiant disorder with onset in preschool years: longitudinal stability and pathways to other disorders. J Am Acad Child Adolesc Psychiatry 2001;40:1393-1400.

15. Offord DR, Boyle MH, Racine YA, Fleming JE, Cadman DT, Blum HM, et al. Outcome, prognosis, and risk in a longitudinal follow-up study. J Am Acad Child Adolesc Psychiatry 1992;31:916-923.

16. Foster EM, Jones DE. The high costs of aggression: public expenditures resulting from conduct disorder. Am J Public Health 2005;95:1767-1772.

17. DuPaul GJ. Parent and teacher ratings of ADHD symptoms: psychometric properties in a community-based sample. J Clin Child Adolesc Psychol 1991;20:245-253.

18. Cho SZ, Chun SY, Hong KE, Shin MS. A study of the development and standardization of ADHD diagnostic system. J Korean Acad Child Adolesc Psychiatry 2000;11:91-99.

19. Ha KS, Kwon JS, Lyoo IK. Development and standardization of the computerized attention assessment for Korean adults. J Korean Neuropsychiatr Assoc 2002;41:335-346.

20. Hommersen P, Murray C, Ohan JL, Johnston C. Oppositional defiant disorder rating scale: preliminary evidence of reliability and validity. J Emot Behav Disord 2006;14:118-125.

21. Waschbusch DA, Elgar FJ. Development and validation of the conduct disorder rating scale. Assessment 2007;14:65-74.

22. Connor DF, Doerfler LA. ADHD with comorbid oppositional defiant disorder or conduct disorder: discrete or nondistinct disruptive behavior disorders? J Atten Disord 2008;12:126-134.

23. Nordström T, Ebeling H, Hurtig T, Rodriguez A, Savolainen J, Moilanen I, et al. Comorbidity of disruptive behavioral disorders and attentiondeficit hyperactivity disorder--indicator of severity in problematic behavior? Nord J Psychiatry 2013;67:240-248.

24. Achenbach TM, Rescorla LA. Manual for the ASEBA school-age forms & profiles: an integrated system of multi-informant assessment. Burlington: ASEBA; 2001.

25. Reynolds CR, Kamphaus RW. Behavior assessment system for children: manual. Circle Pines: American Guidance Service; 1998.

26. Baek SE, Park SY, Ahn CM. The behavior assessment system for children-2, parent rating scales-adolescent form (K.BASC-2, PRS-A): a construct validation study. Korean J Educ Psychol 2017;31:639-661.

27. Lee SJ, Shin MS, Kim BN, Yune HS, Shin YJ, Kim YA, et al. Clinical utility of the Korean version of CBCL6-18 in the diagnosis of attentiondeficit hyperactivity disorder. Kor J Clin Psychol 2015;34:829-850.

28. Pelham WE Jr, Gnagy EM, Greenslade KE, Milich R. Teacher ratings of DSM-III-R symptoms for the disruptive behavior disorders. J Am Acad Child Adolesc Psychiatry 1992;31:210-218.

29. Van Eck K, Finney SJ, Evans SW. Parent report of ADHD symptoms of early adolescents: a confirmatory factor analysis of the disruptive behavior disorders scale. Educ Psychol Meas 2010;70:1042-1059.

30. Wright KD, Waschbusch DA, Frankland BW. Combining data from parent ratings and parent interview when assessing ADHD. J Psychopathol Behav Assess 2007;29:141-148.

31. Lee Y, Kim BN, Park MH, Park S. Familial, cognitive, and behavioral characteristics of adolescents with depression. J Korean Acad Child Adolesc Psychiatry 2017;28:168-173.

32. Korean Statistical Information Service. The number of students per class. Available at: https://kosis.kr/statHtml/statHtml.do?orgId=101&tblId=DT_1YL15001&conn_path=I2. Accessed September 20, 2021.

33. Loeber R, Green SM, Lahey BB, Christ MAG, Frick PJ. Developmental sequences in the age of onset of disruptive child behaviors. J Child Fam Stud 1992;1:21-41.

34. Rucklidge JJ. Gender differences in ADHD: implications for psychosocial treatments. Expert Rev Neurother 2008;8:643-655.

35. Kim YS, So YK, Noh JS, Choi NK, Kim SJ, Koh YJ. Normative data on the Korean ADHD Rating Scales (K-ARS) for parents and teacher. J Korean Neuropsychiatr Assoc 2003;42:352-359.

36. So YK, Noh JS, Kim YS, Ko SG, Koh YJ. The reliability and validity of Korean parent and teacher ADHD rating scale. J Korean Neuropsychiatr Assoc 2002;41:283-289.

37. Oh KJ, Kim YA. Manual for the Korean version of ASEBA school-age forms & profiles. Seoul: Huno Inc; 2011.

38. Beauducel A, Herzberg PY. On the performance of maximum likelihood versus means and variance adjusted weighted least squares estimation in CFA. Struct Equ Model 2006;13:186-203.

39. Bryant FB, Yarnold PR. Principal-components analysis and exploratory and confirmatory factor analysis. In: Grimm LG, Yarnold PR, editor. Reading and understanding multivariate statistics. Washington, DC: American Psychological Association, 1995, p. 99-136.

40. Byrne BM. A primer of LISREL: basic applications and programming for confirmatory factor analytic models. New York: Springer Science & Business Media; 2012.

41. Nunnally JC, Bernstein IH. Psychometric theory. 3rd ed. New York: McGraw-Hill; 1994.

42. Fluss R, Faraggi D, Reiser B. Estimation of the Youden index and its associated cutoff point. Biom J 2005;47:458-472.

43. Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 1982;143:29-36.

44. Muller MP, Tomlinson G, Marrie TJ, Tang P, McGeer A, Low DE, et al. Can routine laboratory tests discriminate between severe acute respiratory syndrome and other causes of community-acquired pneumonia? Clin Infect Dis 2005;40:1079-1086.

45. Baldessarini RJ, Finklestein S, Arana GW. The predictive power of diagnostic tests and the effect of prevalence of illness. Arch Gen Psychiatry 1983;40:569-573.

46. Molina BS, Smith BH, Pelham WE. Factor structure and criterion validity of secondary school teacher ratings of ADHD and ODD. J Abnorm Child Psychol 2001;29:71-82.

47. Gershon J. A meta-analytic review of gender differences in ADHD. J Atten Disord 2002;5:143-154.

48. Levy F, Hay DA, Bennett KS, McStephen M. Gender differences in ADHD subtype comorbidity. J Am Acad Child Adolesc Psychiatry 2005;44:368-376.

49. Park JH, Bang YR, Kim CK. Sex and age differences in psychiatric disorders among children and adolescents: high-risk students study. Psychiatry Investig 2014;11:251-257.

50. Lehto-Salo P, Närhi V, Ahonen T, Marttunen M. Psychiatric comorbidity more common among adolescent females with CD/ODD than among males. Nord J Psychiatry 2009;63:308-315.

51. Loeber R, Burke JD, Lahey BB, Winters A, Zera M. Oppositional defiant and conduct disorder: a review of the past 10 years, part I. J Am Acad Child Adolesc Psychiatry 2000;39:1468-1484.

52. Munkvold LH, Lundervold AJ, Manger T. Oppositional defiant disorder-gender differences in co-occurring symptoms of mental health problems in a general population of children. J Abnorm Child Psychol 2011;39:577-587.

53. Nolan EE, Gadow KD, Sprafkin J. Teacher reports of DSM-IV ADHD, ODD, and CD symptoms in schoolchildren. J Am Acad Child Adolesc Psychiatry 2001;40:241-249.

54. Owens J, Hoza B. Diagnostic utility of DSM-IV-TR symptoms in the prediction of DSM-IV-TR ADHD subtypes and ODD. J Atten Disord 2003;7:11-27. discussion 29.

55. Frick PJ, Lahey BB, Applegate B, Kerdyck L, Ollendick T, Hynd GW, et al. DSM-IV field trials for the disruptive behavior disorders: symptom utility estimates. J Am Acad Child Adolesc Psychiatry 1994;33:529-539.

56. Carlson CL, Tamm L, Gaub M. Gender differences in children with ADHD, ODD, and co-occurring ADHD/ODD identified in a school population. J Am Acad Child Adolesc Psychiatry 1997;36:1706-1714.

Editorial Office
#522, 27, Seochojungang-ro 24-gil, Seocho-gu, Seoul 06601, Korea
Tel: +82-2-717-0892    E-mail: psychiatryinvest@gmail.com