Validation of the Korean Version of the Delirium Diagnostic Tool-Provisional (K-DDT-Pro)

Article information

Psychiatry Investig. 2022;19(9):748-753

Publication date (electronic) : 2022 September 22

doi : https://doi.org/10.30773/pi.2022.0151

Kyeong Mee Kim¹, Man-shik Shim², Dahyun Yi³, So Yeon Jeon^,1,4, Jeong Lan Kim^,1,4

¹Department of Psychiatry, Chungnam National University Hospital, Daejeon, Republic of Korea

²Department of Cardiovascular & Thoracic Surgery, Chungnam National University Hospital, Daejeon, Republic of Korea

³Institute of Human Behavioral Medicine, Seoul National University Medical Research Center, Seoul, Republic of Korea

⁴Department of Psychiatry, Chungnam National University College of Medicine, Daejeon, Republic of Korea

Correspondence: So Yeon Jeon, MD, PhD Department of Psychiatry, Chungnam National University Hospital, Chungnam National University College of Medicine, 282 Munhwa-ro, Jung-gu, Daejeon 35015, Republic of Korea Tel: +82-42-280-7281, Fax: +82-42-280-8423, E-mail: iris870422@gmail.com

Correspondence: Jeong Lan Kim, MD, PhD Department of Psychiatry, Chungnam National University Hospital, Chungnam National University College of Medicine, 282 Munhwa-ro, Jung-gu, Daejeon 35015, Republic of Korea Tel: +82-42-280-7281, Fax: +82-42-280-8423, E-mail: kimjl@cnu.ac.kr

Received 2022 June 5; Revised 2022 July 14; Accepted 2022 July 19.

Abstract

INTRODUCTION

Delirium is a disturbance of consciousness, attention, and baseline cognitive function according to the Diagnostic and Statistical Manual of Mental Disorders, 5th Edition (DSM-5) [1]. Delirium increases morbidity, the hospitalization time, and mortality [2-5]. Delirium is common (7%–47%) in elderly inpatients [6-8]. As populations age, delirium will become more prominent. It is important to prevent delirium or at least to detect delirium early; this allows rapid medical intervention. However, in clinical studies, about two-thirds of delirium cases were not diagnosed; in emergency settings, 57%–83% of delirium cases were missed, because it is difficult for medical staff lacking expertise with psychiatric interviews and differential diagnosis to identify and evaluate the characteristic neuropsychiatric symptoms of delirium and to distinguish them from those of other diseases [9-11]. There is a need for a simple screening tool that can be used by medical staff who are not specialists in psychiatry or neurology. However, the validity of a delirium evaluation must be verified, as must the capacity of the tool to distinguish delirium from dementia. Also, the various aspects of delirium must be revealed, and, finally, the tool must be practical [12].

The Confusion Assessment Method algorithm (CAM-A) is one of the most commonly used tools that enables nonpsychiatric clinicians to detect delirium quickly and easily. The CAM-A evaluates the presence of four important symptoms of delirium: acute onset and fluctuating course, inattention, disorganized thinking, and altered level of consciousness. The CAM-A can be completed in less than 5 minutes. Because of its simplicity and convenience, CAM-A is widely used by nonexpert clinicians for delirium detection. However, some training is required for optimal use of the CAM. The CAM allows stupor or coma to be diagnosed as delirium and is a dichotomous tool not allowing it to be a continuous measure even with adaptations [13-16]. The 4A’s Test (4AT), another brief assessment tools for delirium and cognitive impairment, developed in the UK. 4AT screens patients with four items that assess the alertness, orientation, attention, and history of a patient’s acute change or fluctuation course in cognition or other mental function. It is short (<2 minutes), simple screening tool for delirium and shows good diagnostic test accuracy for delirium. In the recent meta-analysis study, the pooled sensitivity was 88% and the pooled specificity was 88%. It does not require special training and has the advantage of being easy to implement. However, the test results are affected by whether dementia is accompanied [17]. In addition to CAM-A and 4AT, many tools to detect delirium are being developed and verified.

The Delirium Diagnostic Tool-Provisional (DDT-Pro) developed by Kean et al. [18] is an excellent screen allowing a provisional diagnosis of delirium; the tool is delirium-specific, sensitive, and reliable. The DDT-Pro evaluates the three core symptom domains of delirium: cognition (vigilance in DDTPro), higher-level thinking (comprehension in DDT-Pro), and circadian rhythm (sleep-wake cycle in DDT-Pro). The presence of these core domain symptoms could accurately reflect delirium while other brief tools do not evaluate all the three core symptom domains. In comparison to the CAM-A, because of its highly structured, quantitative items, the DDT-Pro would be more useful to provisionally diagnose delirium. The previous validation study of the DDT-Pro in general hospital patients shows a significant advance over the commonly used CAM. Although the English and Spanish versions of DDT-Pro have been validated with excellent results, validation in other languages employed by specific patient populations is also required [16,19,20]. A Korean version would be useful as Korean hospitals lack such tools. We translated the DDT-Pro from English to Korean (Korean version of the DDT-Pro, K-DDT-Pro). We compared the performance of the K-DDT-Pro to that of the Korean version of the Delirium Rating Scale-Revised-98 (DRS-R98-K), which is validated and is commonly used in Korea [21,22]. We explored the inter-rater reliability of the K-DDT-Pro between an expert and a non-expert; the reference diagnosis was that of a psychiatrist using DSM-5 criteria.

METHODS

Inter-rater reliability and validation of delirium assessment

Prior to the full-scale evaluation, all raters met to discuss the assessment method. One experienced psychiatrist evaluated all patients using DSM-5 delirium criteria. Then, one psychiatric trainee; a psychiatry resident who had graduated from medical school (who used form A) and one research nurse (who used form B) independently performed delirium assessments on the same subjects using the K-DDT-Pro. The trainee was the psychiatric expert, and the nurse was the non-expert. During the evaluation, the trainee also evaluated patients using the DRS-R98-K with information from their family members and caregivers. The patient assessments of the three raters were completely independent (Figure 1). All evaluations were conducted between 14:00 and 18:00 to minimalize any possible bias arising from fluctuations in each patient’s condition.

Figure 1.

The process of the assessment. DSM-5, Diagnostic and Statistical Manual of Mental Disorders, 5th Edition; K-DDTPro, Korean version of the Delirium Diagnostic Tool-Provisional; DRS-R98-K, Korean version of the Delirium Rating Scale-Revised-98.

RESULTS

Baseline characteristics

The patient flow diagram is shown in Figure 2. We interviewed 95 patients who met the inclusion criteria, but 53 also met the exclusion criteria. The ‘communication problem’ of the exclusion criteria is when the patient has severe hearing loss or can neither talk nor motion a response. Therefore, we enrolled 42 patients. The sociodemographic characteristics of the subjects are summarized in Table 1. Using the DSM-5 criteria, 25 were diagnosed with delirium by the psychiatrist. Eight had previously been diagnosed with dementia by psychiatrists or neurologists. The independent t test showed that patients with delirium were older than others. However, no other parameter differed significantly between the two groups.

Figure 2.

Flow diagram of the study participants. ^*when the patient has severe hearing loss or can neither talk nor motion a response. DSM-5, Diagnostic and Statistical Manual of Mental Disorders, 5th Edition.

Table 1.

Baseline characteristics of participants

Internal and inter-rater reliability of the Korean version of the Delirium Diagnostic Tool-Provisional

The Cronbach’s α coefficients for the psychiatric trainee and the nurse were 0.839 and 0.822, respectively. The internal reliability of the K-DDT-Pro was very good. The Cohen weighted κ coefficients for the K-DDT-Pro items were (item 1) 0.555±0.102 (p<0.001), (item 2) 0.776±0.062 (p<0.001), (item 3) 0.647±0.081 (p<0.001); and that for the total score 0.718±0.048 (p<0.001) (Table 2). The inter-rater agreement was moderate to substantial.

Table 2.

Inter-rater reliability of K-DDT-Pro (psychiatric trainee and nurse)

Validity of the Korean version of the Delirium Diagnostic Tool-Provisional and selection of the cutoff score for delirium

The K-DDT-Pro and the DRS-R98-K total and severity scale scores were inversely related as revealed by the Pearson rho values (Table 3). The K-DDT-Pro assessments of the psychiatric trainee correlated with the DRS-R98-K severity score (-0.850 [p<0.001]) and with the DRS-R98-K total score (-0.857 [p<0.001]). The figures for the nurse were -0.821 (p<0.001) and -0.815 (p<0.001), respectively. Thus, the correlations were very high regardless of the rater’s psychiatric expertise.

Table 3.

Correlations among K-DDT-Pro and DRS-R98-K

The areas under the ROC curves (AUCs) for K-DDT-Pro diagnostic accuracy using the DSM-5 criteria as the reference were 97.4% (95% confidence interval [CI]=93.3%–100%) for the psychiatric trainee and 89.4% (95% CI=80%–98.6%) for the nurse (Figure 3). The sensitivity and specificity of the K-DDT-Pro diagnoses compared to the of DSM-5 diagnoses as revealed by ROC analysis are listed in Table 4. The ascending sensitivity and descending specificity gradients were determined for various cutoff values. K-DDT-Pro scores ≤6 and ≤7 both seemed to balance sensitivity and specificity. Scores ≤7 balanced sensitivity and specificity only in the assessments of the psychiatric trainee. Therefore, the optimal cutoff may be ≤6 or ≤7.

Figure 3.

AUC for the K-DDT-Pro for delirium diagnosis (DSM-5 independent diagnosis). K-DDT-Pro, Korean version of the Delirium Diagnostic Tool-Provisional; AUC, area under the receiver operating characteristic curve; CI, confidence interval.

Table 4.

Sensitivity and specificity analysis using patient operating characteristic for the K-DDT-Pro for delirium diagnosis in accordance with DSM-5 (total N=42)

DISCUSSION

We prepared a K-DDT-Pro and evaluated its reliability and validity in terms of diagnosing delirium. In terms of reliability, all three items evidenced high internal consistency, with Cronbach’s α coefficient of 0.839 (the trainee) and 0.822 (the nurse). In terms of the Cohen weighted κ coefficients for each item, the tool evidenced moderate (0.555±0.102, item 1) to substantial (0.776±0.062, item 2) inter-rater reliability between the psychiatric expert and the non-expert. Note that the inter-rater reliability of item 1 was lower than that of others. For comparison, when the item 1 scores of the two raters were averaged, that of the psychiatric trainee (2.4286) was higher than that of the nurse (2.2143). One possible reason is that form A (used by the trainee) could be more intuitively understood by the elderly patients than form B (used by the nurse). Also, the possibility that the evaluation methods differed (even slightly) between the raters cannot be excluded.

The nurse ratings in this study were a little less strong than the trainee. In our opinions, because the research nurse can’t observe the patient’s conditions bedside unlike clinical nurses (representing non-experts in previous studies), it must have been more difficult for research nurse to acquire information about patients than clinical nurses or the trainee. Of course, it can be explained with absolute results of the present study, but further research on clinical nurses (instead of the research nurse) may be needed to explain that the DDT-Pro can be used as useful in non-experts (e.g., nurses, rehabilitation staff) as physicians.

A significant inverse Pearson correlation was apparent between the K-DDT-Pro and DRS-R98-K scores. As the DRS-R98-K is well-validated, this constitutes indirect evidence that the K-DDT-Pro will also usefully screen for delirium. We then confirmed the validity of the K-DDT-Pro.

To ensure that the tool would be clinically useful, the optimal cutoff for delirium diagnosis was obtained via ROC analyses; this was ≤6 or ≤7, as for the English and Spanish versions [16,18,19]. Considering subsyndromal delirium (SSD), the scores of 6 and 7 indicate SSD and full delirium is denoted by the lower score range.19 Compared to the original validation studies of DDT-Pro (sensitivity 100% and specificity 94% in Kean et al. [18], 2010; sensitivity 90% and specificity 87% in Franco et al.,16 2020), the K-DDT-Pro exhibited similar sensitivity and specificity values.

Our work had several limitations. First, the sample size was too small (n=42) to compare the outcomes of patients with and without dementia. Given the small sample size, we sought to render the samples homogeneous, but this was difficult. In the present study, a research nurse was the non-psychiatric expert. Clinical nurses who observe the patient’s conditions bedside should be evaluated as raters.

In conclusion, the K-DDT-Pro exhibited high validity and reliability in terms of delirium screening regardless of the rater’s psychiatric expertise. Thus, the K-DDT-Pro will find clinical applications in Korean hospitals. However, additional studies with larger sample sizes are required. Also, clinical nurses who observe patients’ conditions bedside should be evaluated as raters.

We hope that all medical staff, regardless of their psychiatric expertise, will use the brief K-DDT-Pro to improve the prognosis of elderly inpatients with delirium. Early detection of delirium will trigger rapid medical intervention.

Article information Continued