This study had two stages: initial screening with self-report questionnaires, including the CAPE-15, and semi-structured interviews to investigate the instrument’s diagnostic validity. The semi-structured interviews were conducted by phone with subjects who gave informed consent by trained mental health professionals (clinical psychologists, nurses, and social workers). In the second evaluation, the subjects were assessed with the Comprehensive Assessment of At-Risk Mental States (CAARMS) [13] to investigate whether they truly had UHR symptoms. When investigators strongly suspected that subjects did not meet the CAARMS UHR criteria, they made final diagnoses of no UHR of psychosis. Subjects whose diagnoses were not confirmed by phone interviews were then interviewed face-to-face by experienced psychiatrists. Final diagnoses were reviewed at a research team meeting with experienced psychiatrists.

All study subjects were college students aged between 18 and 30 years. In the classroom setting, professors and research associates provided an explanation of the purpose and procedure of this study to the prospective student participants (n=3,000), 2,010 (67.0%) of whom voluntarily completed the anonymous self-screening questions. Of these, 261 students older than 30 years of age were excluded and 1,749 students were the final subjects for the initial analysis. Of the initial study population, 1,225 (70.0%) consented to potentially being contacted a second time and thus provided their phone number. In line with the procedure of our previous validation studies [7], individuals with CAPE-15 scores ≥4 (n=224, 18.3%) and with depressive symptoms regardless of CAPE-15 score (n=76, 6.2%) were selected for semi-structured interviews. Of the subjects for whom the next investigation was indicated (n=300), 191 (63.7%) participated in a semi-structured interview. From these procedure, 20 participants were invited for face-to-face interviews, and 14 subjects underwent the face-to-face interviews. Seven more college students who visited an early intervention center for the evaluation of psychosis-like symptoms during the study period were also included in the final study population for diagnosis validation after completing the same face-to-face interview.

The Institutional Review Board of Chonnam National University Hospital approved this study (CNUH-2018-228), and all participants provided informed consent. The initial prescreening was performed in September and October 2018 and the diagnostic interviews were conducted between October 2018 and March 2018.

The initial screening questionnaires included questions regarding demographic characteristics, in addition to the Patient Health Questionnaire (PHQ-9), mKPQ-16, and CAPE-15. The PQ-16 is a self-report screening questionnaire that assesses the presence of attenuated psychotic symptoms using dichotomous questions (true/false). The level of distress associated with each symptom is rated on a four-point scale ranging from “no distress” to “much distress.” The modified Korean version of the PQ-16 includes three additional item drawn from the ESI, in addition to the 16 items of the original instrument [7,8].

The original CAPE is a self-report screening questionnaire that comprised 42 items [14]. The CAPE-15 is a shorter subscale that has been widely used as a screening tool to identify people at UHR in a clinical setting [15-17]. It assesses the presence of psychotic-like experiences with a three-factor structure: persecutory ideation (five items), bizarre experiences (seven items), and perceptual abnormalities (three items). It measures both the frequency of and distress associated with these experiences. Each item uses a 4-point Likert scale from 0, ‘never,’ through ‘sometimes’ and ‘often,’ to 3, ‘nearly always.’ For each endorsed item, distress was rated on a 4-point Likert scale from 0, ‘not distressed,’ through ‘a bit distressed’ and ‘quite distressed,’ to 3, ‘very distressed,’ which produced a total score of 0-45. Higher scores indicated a higher frequency of psychotic experiences and an increased level of distress due to these experiences. Three authors (SWK, YSK, and JSY) translated the CAPE-15 into Korean after obtaining permission from its original authors. Bilingual translators majoring in medicine back-translated the CAPE-15 into English, after which the instrument was finally edited. No significant discrepancies between the translated and original instruments were identified.

The CAARMS was used as the gold standard for UHR diagnosis [13]. The Korean version of the CAARMS, which was developed through a standard translation/back-translation procedure (followed by a final revision based on feedback from the original authors) and applied in various studies validating screening measures for UHR populations, was also used in this study [6,18]. The original CAARMS consists of seven subscales, but symptomatic criteria for psychosis risk are based exclusively on positive symptom items (disordered thought contents, perceptual abnormalities, and disorganized speech). In this study, four items from the positive symptom subscale were administered to subjects for the diagnosis of UHR of psychosis.

The Korean version of the nine-item Patient Health Questionnaire (PHQ-9), validated by Han et al. [19,20], was administered to measure the level of depressive symptoms.

Using data from the screening samples, the degree of correlation between the CAPE-15 and mKPQ-16 scores was assessed using Spearman’s correlation coefficient to examine convergent validity. Associations among these scales and PHQ-9 were investigated with Spearman’s correlation. The internal consistency was estimated with Cronbach’s alpha. Using data from the subjects who underwent structured evaluations, we conducted receiver operating characteristic (ROC) analyses for the diagnosis of UHR by the CAARMS interview. We determined the optimal cutoff values for the CAPE-15 frequency and distress scores based on the sensitivity, specificity, PPV, and negative predictive value (NPV). The accuracy of a test depends on how well it separates the group being tested into those with and without the disease in question, and is measured by the area under the curve (AUC). The validity of the mKPQ-16 item total score, which we previously proved to be the most valid measure, was also calculated for the study subjects to compare with the CAPE-15. The test-retest reliability of the CAPE-15 was estimated using the intraclass correlation coefficient (ICC) in the subjects who went through face-toface interviews. The data were analyzed using SPSS for Windows (ver. 21.0; IBM Corp., Armonk, NY, USA), and p-values< 0.05 were deemed statistically significant.

The mean age of the subjects who participated in the first screening (n=1,749) was 21.8±1.9 years. Female participants were more numerous than male participants (64.3%). For the screening dataset, Table 1 shows the Spearman’s coefficients of correlation between the mKPQ-16 and CAPE. Scores for the CAPE-15 frequency and distress scores were significantly associated with the mKPQ-16 item total and distress scores (all p-values<0.001). The PHQ-9 scores were significantly associated with the CAPE and mKPQ-16 scores. The correlation between frequency and distress scores of the CAPE-15 was strong (r=0.878, p<0.001).

The mean age of the subjects who underwent further investigations (n=198) was also 20.8±2.1 years, and females (74.3%) predominated. Among the subjects who were interviewed, 12 met the CAARMS criteria for UHR of psychosis. Two subjects met subthreshold frequency criteria and 10 subjects met subthreshold intensity criteria. To predict CAARMS-based diagnoses of UHR of psychosis, ROC curves were plotted (Figure 1). The AUCs were all significant for all measures, which were put into the model (all p-values<0.001). The AUC of the CAPE-15 distress score was the greatest (0.936) among the three measures.

Table 2 summarizes the diagnostic validity of the CAPE-15 based on the AUCs and optimal cutoff points for sensitivity and specificity. For the CAPE-15 distress score, a cutoff score of 6 resulted in the best balance of sensitivity (91.7%) and specificity (85.2%), with a favorable PPV of 32.4%. For the CAPE-15 frequency score, a cutoff score of 6 resulted in the greatest sensitivity (100%), despite a relatively low specificity (67.6%) and a PPV of 16.9%. A cutoff item total mKPQ-16 score of 7 showed favorable sensitivity (83.3%) and specificity (79.1%), and a PPV of 20.8%.

Cronbach’s alpha values for the CAPE-15 frequency and distress items were 0.867 and 0.835, respectively. The test-retest reliability was evaluated using data from subjects who underwent face-to-face evaluations, and the ICC for the CAPE-15 frequency and distress scores was 0.659 (p=0.010) and 0.859 (p<0.001), respectively.