Shim and Jeong: Predicting Suicidal Ideation in College Students with Mental Health Screening Questionnaires

Abstract

Objective

The present study aimed to identify risk factors for future SI and to predict individual-level risk for future or persistent SI among college students.

Methods

Mental health check-up data collected over 3 years were retrospectively analyzed. Students were categorized as suicidal ideators and non-ideators at baseline. Logistic regression analyses were performed separately for each group, and the predicted probability for each student was calculated.

Results

Students likely to exhibit future SI had higher levels of mental health problems, including depression and anxiety, and significant risk factors for future SI included depression, current SI, social phobia, alcohol problems, being female, low self-esteem, and number of close relationships and concerns. Logistic regression models that included current suicide ideators revealed acceptable area under the curve (AUC) values (0.7–0.8) in both the receiver operating characteristic (ROC) and precision recall (PR) curves for predicting future SI. Predictive models with current suicide non-ideators revealed an acceptable level of AUCs only for ROC curves.

Conclusion

Several factors such as low self-esteem and a focus on short-term rather than long-term outcomes may enhance the prediction of future SI. Because a certain range of SI clearly necessitates clinical attention, further studies differentiating significant from other types of SI are necessary.

INTRODUCTION

College entrance can be particularly stressful for students who are psychologically vulnerable and have poor support because this period involves a transition from adolescence to “emerging adulthood.” [1] College students are at high risk for suicidal ideation (SI), planning, and attempts [2]. Approximately one of four to six college students have experienced some form of SI during college, and about 40% and 20% of students with SI report suicide plans and attempts, respectively [3]. As for the college students in Korea, one-month prevalence of suicidal ideation, plan, and attempt was 6.7%, 0.5%, and 0.1%, respectively [4].
Current SI and a history of suicide attempts are generally considered to be important predictors of later suicidal behavior [5]. A recent study of patients with depression found that the severity of past SI is the most important factor differentiating suicide attempters from non-attempters, even after controlling for other significant variables [6]. The authors suggested that severe SI should not be overlooked because it might be a major independent predictor of suicidal behaviors. Suicidal thoughts and behaviors are markers of extreme psychological distress [7] and are linked with long-term adverse outcomes, such as depression, substance dependence, and unemployment [8]. For example, 15-year-olds with SI are more likely to exhibit psychopathology, suicidal behavior, and compromised functioning at age 30 [9]; thus, there is a need for early identification of and continued intervention with those with SI.
Many studies have attempted to identify the risk factors of SI among college students. Generally, this issue is thought to be associated with depressive symptoms [7,10], hopelessness, stressful life events, lack of social belongingness [11], smoking, alcohol and drug use [12], and a poor family environment [13]. According to Kraemer et al. [14], a risk factor is a special type of correlate that temporally precedes the outcome of interest and divides individuals into high- and low-risk groups [15]. Thus, strictly speaking, the aforementioned factors are correlates or concomitants rather than risk factors or longitudinal predictors because they are derived from cross-sectional rather than longitudinal studies.
Thus, the present study aimed to identify the risk factors for future SI in college students. Because current SI may be the most important contributor to future SI, students were categorized as suicide ideators and non-ideators at baseline to estimate the risk factors, predict the individual-level risk for future or persistent SI, and test the accuracy of a predictive model.

Methods

Procedures

The present study retrospectively analyzed mental health check-up data collected between April 2014 and March 2017. All students at our institution are required to participate in web-based mental health screening once per year as part of a routine medical check-up. The first-year screening was completed by 4,872 students, and 3,362 of these students completed the check-up twice or more over 3 years. Although some students (n=391) participated in the check-up more than three times, only up to two sets of follow-up data per student were used for statistical analysis. The follow-up data with the longest intervals were selected from each of two timepoints (i.e., before, within the median follow-up interval of 13 months, and after the 13 months); these timepoints are referred to as the 1st-half and 2nd-half follow-up periods, respectively. In total, 1,057 and 698 students were followed in only the 1st-half or 2nd-half periods, respectively, and 1,607 students were followed in both periods.
The study protocol was approved by the KAIST Institutional Review Board (IRB No. KH2017-56). Before answering the questionnaire, all students provided informed consent stating their mental health check-up data could be used for research purposes after data anonymization.

Measures

Sociodemographic variables

Students were categorized as never smokers, ex-smokers, or current smokers based on whether they smoked more than five packs during their lifetimes. Sleep patterns were classified as follows based on sleep and wake-up routines during the semester: regular for both, regular for either of the two, and irregular for both. The total number of concerns was calculated based on answers to the question “Please check the box next to each problem about which you have been worried recently (you can check more than one),” and number of close relationships was assessed with the question “With how many people in school do you have close relationships?”

Scales

Several scales were administered at baseline (over the first year): the CAGE quiz, which screens for problematic alcohol use [16]; the Smartphone Addiction Scale [17]; the Pittsburgh Sleep Quality Index [18]; the Patient Health Questionnaire-9 [19]; the Generalized Anxiety Disorder-7 (GAD-7) scale [20]; the Liebowitz Social Anxiety Scale [21]; the Verbal Abuse Questionnaire [22]; the Impact of the Event Scale-Revised version [23]; the Adult ADHD Self-Report Scale version 1.1 Screener [24]; the Beck Scale for Suicide Ideation [25]; and the KAIST Scale for Suicide Ideation (KSI).
The 19 self-rated items on the PSQI were combined to form scores for seven components: subjective sleep quality, sleep latency, sleep duration, habitual sleep efficiency, sleep disturbances, use of sleeping medication, and daytime dysfunction. The LSAS scores were divided into two subscale scores (fear and avoidance), the IES-R was scored as three subdomains (intrusion, avoidance, and hyperarousal), and the VAQ scores were calculated as total scores for parental, superior, and peer verbal abuse. The ASRS Screener score reflected the number of checkmarks that corresponded to a certain frequency range (from ‘sometimes’ or ‘often’ to ‘very often’), as described by Kessler et al. [24], and the BSI score was the sum of the scores on the first 19 items. The KSI score was the sum of the scores on the first 14 items, each of which was rated from 0 to 4; the KSI was developed by our research team to assess various levels of SI over the past 2 weeks or the last year on a scale ranging from mild (“I would rather fall asleep and not wake up”) to severe (“I will carry out my thoughts of wanting to take my own life”) (Supplementary Table 1 in the online-only Data Supplement). The KSI score for the past 2 weeks was significantly correlated with the BSI score in the present study (Kendall’s τ=0.35, p<0.001).
Beginning in April 2015, two additional measures, the Rosenberg’s Self-Esteem Scale [26] and the Resilience Appraisals Scale [27], were added to the survey, and the five questionnaires used at baseline (SAS, VAQ, IES-R, ASRS Screener, and BSI) were no longer used for the mental health survey. Korean versions of nine self-report scales were validated: SAS, PSQI, PHQ-9, GAD-7, LSAS, VAQ, IES-R, BSI, and RSES.

Analyses

Statistical analyses were performed with R [28]. Initially, the presence of SI was determined by the KSI score for the last 2 weeks because these data were available for a period spanning more than 3 years; a KSI score of 0 indicated no SI whereas a KSI score >0 indicated SI. To identify risk factors for future or persistent SI, logistic regression analyses were independently performed for suicide ideators and non-ideators at baseline. The presence or absence of future SI, based on follow-up KSI score, was entered as the dependent variable, and risk factors were identified with a backward conditional stepwise elimination strategy after all continuous variables were transformed into z-scores. If a student had a KSI score >0 at least once during the follow-up survey, they were regarded as having future SI.
Using the final variables, we assessed whether the relationships between the continuous predictors and the logit were linear using the Box-Tidwell test [29]. If the relationships were not linear and the sample size was not large enough, a polynomial model was applied [30]. The Hosmer-Lemeshow goodness statistic was calculated to assess how well the chosen model fit the data, and multicollinearity was tested using tolerance and the variance inflation factor [31]. The Nagelkerke pseudo-R2 value was used as a measure of the total effect size of the final model.
The predicted probability for future SI was calculated from the final model using the leave-one-out sample method (i.e., all the remaining data without oneself) to verify predictability with greater accuracy. Using the calculated predicted probability, receiver operating characteristic (ROC) curves and precision recall (PR) curves were generated, and area-underthe-curve (AUC) values were calculated to evaluate prediction accuracy using the ‘PRROC’ package [32,33]. Because ROC curves may present an overly optimistic view of a classifier’s performance when there is a large skew in the class distribution [34], PR curves were also generated.
The optimal threshold that provided the maximum correct prediction was found based on the Youden Index, which is the difference between the sensitivity and the false positive rate and equals the vertical distance between the ROC curve and a line of equality or a chance line [35]. Maximizing the Youden index allows for the identification of an optimal cut-off point on the ROC curve that is independent of the prevalence. The definitions for these terms were used as following Table 1.
The presence of SI at the 2nd-half follow up among students for whom both sets of data were available was predicted based on the data from the 1st-half follow up and the five scales that were administered only at baseline. After applying the same procedure for this modeling, we expected that the combined model using the baseline and 1st-half follow-up data would be superior to the original model using only baseline data because the former included more predictors and the prediction interval was relatively shorter.

Results

Table 2 presents the baseline sociodemographic characteristics and scores on the clinical scales of suicide ideators and non-ideators separately as well as the comparison results based on future SI. Suicide non-ideators who showed future SI were more likely to be female (χ2=19.21, p<0.001), have irregular sleep patterns (χ2=9.97, p=0.01), and have more concerns and alcohol problems; more concerns and alcohol problems were also evident in suicidal ideators who showed persistent SI. Students who showed future SI already exhibited more clinical problems at baseline compared to those who did not, irrespective of baseline SI. Students who did not have future SI (at the 2nd half) had significantly higher scores on the RAS and RSES compared to those who did, regardless of SI at the 1st half (Supplementary Table 2 in the online-only Data Supplement).
In the logistic regression model, the risk factors for future SI in non-ideators were being female, long night-time sleep duration, higher levels of depressive symptoms, verbal abuse from superiors (e.g., academic supervisors or seniors), SI according to the BSI, and attention deficit hyperactivity disorder (ADHD) symptoms (Table 3). The total effect size of the predictive models using the above six risk factors with a constant was 0.11 (Nagelkerke R2), and the strongest three predictors were depressive symptoms, SI, and being female. Risk factors for persistent SI in suicide ideators were higher levels of alcohol problems, depressive symptoms, avoidance due to social phobia, and SI measured by the BSI. Low sleep efficiency, as measured by the PSQI, was a protective factor against future SI in suicide ideators. The total effect size was 0.26, and the strongest three predictors were avoidance due to social phobia, SI, and alcohol problems.
In the logistic regression analysis predicting the presence of future SI in non-ideators at the 1st-half follow up, the significant predictors were being a current smoker (vs. ex-smoker), number of close relationships and concerns, poor sleep quality, use of sleeping medication, higher levels of depressive symptoms, verbal abuse from peers, and SI measured by the BSI (Table 4). Low sleep efficiency and high self-esteem were protective factors against SI. The total effect size was 0.21, and the strongest three predictors were low self-esteem and number of close relationships and concerns. Risk factors for persistent SI in suicide ideators at the 1st-half follow up were higher levels of depressive symptoms, fear due to social phobia, and resilience appraisal. High self-esteem was a protective factor against persistent SI. The total effect size of this model was 0.33, and the strongest three predictors were depressive symptoms, low self-esteem, and fear due to social phobia.
Based on the Hosmer-Lemeshow goodness-of-fit statistic, the above four models fitted the data well [baseline suicide non-ideators: χ2(8)=12.87, p=0.12; baseline suicide ideators, χ2(8)=13.92, p=0.08; 1st-half follow up suicide non-ideators: χ2(8)=9.62, p=0.29; and 1st-half follow up suicide ideators: χ2(8)=11.06, p=0.20]. Tests for multicollinearity did not reveal an alarming level of problems, as, with the exception of CAGE (VIF=3.39) and CAGE squared (VIF=3.40) in the baseline suicide ideators model, all the VIF values for each predictor in the four models were <2.50. The average VIF values were 1.13 (baseline suicide non-ideators), 1.95 (baseline suicide ideators), 1.20 (1st-half follow up suicide non-ideators), and 1.53 (1st-half follow up suicide ideators) for each model. Additionally, there were no noticeable issues with multicollinearity (Supplementary Table 3 in the online-only Data Supplement).
The ROC and PR curves based on the abovementioned four prediction models are shown in Figure 1. Two models predicting SI at the 2nd-half follow up showed larger AUCs in the ROC curves relative to the other two prediction models. However, the two prediction models that used suicide ideators performed better than the other two models that used suicide non-ideators on the PR curves due to the higher prevalence of future SI in the groups with suicide ideators.
Table 5 presents the evaluation metrics of the four predictive models. The predictive model using suicide non-ideators at the 1st-half follow up had the highest sensitivity but the lowest precision. The two prediction models using suicide ideators showed relatively higher precision and F1 scores compared to the other two prediction models using suicide nonideators. The prediction model using suicide ideators at the 1st-half follow up had the best performance based on F1 score. Detailed confusion matrices of the four groups are presented in Supplementary Table 4 (in the online-only Data Supplement).

Discussion

The present study found that students who would have SI in the future already had higher levels of mental health problems, such as depression and anxiety, at baseline than did students who did not. Additionally, the significant risk factors for future SI were depression, current SI, social phobia, alcohol problems, being female, low self-esteem, and number of close relationships and concerns. Logistic regression models with current suicide ideators showed an acceptable level of AUCs (0.7–0.8) in the ROC and PR curves for the prediction of future SI. However, predictive models using current suicide non-ideators showed an acceptable level of AUCs only in the ROC curves, whereas the AUCs of the PR curves were below 0.3 due to the low prevalence of future SI among current suicide non-ideators.
Students with higher levels of depression and hopelessness, alcohol-related problems, and poor social support are more likely to exhibit SI [7,36]. Female college students have a higher risk of SI [13] and are more prone to report their suicidal thoughts to others compared to males [37]. In the present study, students who reported a larger number of close relationships were more likely to have future SI. However, reporting the larger number of close friends may not necessarily mean that these students acquire more solid social networks and are surrounded by close enough relationships whom to report their most personal thoughts, and thus feel less socially isolated. In this regard, the present findings on close relationships should be interpreted with caution.
Walsh and colleagues [38] demonstrated that future suicide attempts can be accurately predicted (AUC: 0.84, precision: 0.79, recall: 0.95) by applying machine learning to electronic medical records. Another study reported that future suicidal behavior can be predicted with 43% sensitivity and 90–95% specificity [39]. Additionally, two prediction models for use with suicide attempters showed reasonable sensitivity (64%) and specificity (76%), with an AUC of 70% [40] and an overall accuracy greater than 80% [41]. However, when interpreting a prediction model’s performance on binary classification, the proportion of two outcomes must be kept in mind. In the above study by Walsh and colleagues [38], suicide attempters (i.e., cases) were more than 1.5 times as many as controls, which may have affected the relatively high AUC of the PR curves. In the present study, only about 9% of suicide nonideators showed future SI, whereas about 47% of suicide ideators showed persistent SI in the future. Because the prediction models using non-ideators showed noticeably low precision related to a high false positive rate, it is possible that the model based on current suicide ideators was more applicable than the model based on non-ideators.
One study investigating the link between childhood maltreatment and adult suicidality found that childhood emotional, but not physical or sexual, maltreatment is uniquely associated with SI in college students [42]. Furthermore, this relationship was partially mediated by the cognitive styles and hopelessness levels of students. A study assessing the association between SI and family background in college students showed that suicidal ideators have poor family structures and relationships and parents with unstable work habits and improper parenting styles [13]. Thus, based on the stress-diathesis model of suicidal behavior [43], it can be postulated that recent interpersonal stress, such as verbal abuse from peers and superiors, can lead to future SI in students with a history of childhood maltreatment, which may have resulted in vulnerable personality traits, including low self-esteem. Similarly, interpersonal stress can lead to feeling like a burden to others and to experiencing a lack of belonging, which, according to the interpersonal theory of suicide, are two main interpersonal cognitive-affective states contributing to SI [44]. Thus, further evaluations of the childhood maltreatment experienced by current suicidal ideators will be helpful for improving the prediction of future SI.
SI, as measured by the BSI, was a significant risk factor for future SI even among non-ideators. Compared to the KSI, the BSI was more sensitive for capturing subtle SI. This indicates that the use of fine-grained measures that clearly differentiate between more and less severe SI will be optimal for predictive models. In addition to SI, depressive symptoms also emerged as a strong and consistent risk factor in the present study. However, according to previous studies, students with subclinical depression are also liable to have SI, and this was confirmed across different samples [45]. Therefore, a variety of risk factors should be assessed concomitantly and may be considered potential therapeutic targets during mental health screening.
Suicidal thoughts and behaviors are also linked to reduced rates of professional help-seeking for mental health problems [46]. Studies conducted in various countries have found the usage rates of mental health services among adolescents and young adults with SI and related behaviors to be at or below 50% [47-49]. Furthermore, adolescents with more severe SI have lower rates of help-seeking behaviors than those with less severe ideation [46]. Young people with more severe SI are also more likely to be hopeless, pessimistic, and have maladaptive adaptive coping strategies, which, in part, can lead to such ‘help-negation’ [50]. Therefore, we need focused efforts to connect students with severe and persistent SI with appropriate services and to encourage them to remain in treatment.
Interestingly, students with longer sleep durations, higher sleep efficiency, and more positive self-appraisals of resilience were more likely to have future SI in the present study. A growing body of evidence supports the idea that sleep problems, including insomnia, are risk factors for SI, related behaviors, and death [51]. However, a recent study using large survey data from Korean adults found that both short-sleeping and longsleeping individuals were more likely to have SI than people who sleep 7 hours a day [52]. Higher levels of resilience should also lead to less suicidality, because resilience leads to a quick recovery and psychosocial growth in the face of adverse life situations [53]. However, a study of patients with depression and anxiety showed that resilience is not independently associated with moderate-severe SI [54]. The effects of the interaction between resilience, on the one hand, and depression and anxiety, on the other, were significant in that study, which implies that resilience primarily moderates the risks posed by depression and anxiety symptoms for SI. In the present study, approximately 70% of the variance in the regression coefficients for both the RAS and RSES scores in the logistic modeling was associated with the smallest eigenvalue, which indicates that some level of dependency between the two variables. Considering the relatively low levels of significance for sleep duration, sleep efficiency, and resilience revealed by the predictive models, the limited range of the self-reported PSQI to evaluate sleep problems, and the potential interaction effect between significant variables, it may be more appropriate to postpone reaching a definitive interpretation of the above findings.
Another limitation of this study should also be discussed. The expression of SI is traditionally considered a risk factor for suicide, but controversy remains regarding the association between SI and suicide. Although Posner and colleagues [55] consider SI to be one of the most salient risk factors for suicide, Large and Nielssen [56] assert that the association between these two variables is unclear and that the presence of SI in the absence of a suicide attempt has not been firmly established as a risk factor for suicide. However, recent systematic meta-analyses found weak associations between these variables in psychiatric inpatients [57,58] and showed that the association between SI and action is stronger in the short-term than in the long-term [59,60]. Taken together, these data suggest that a certain range of SI clearly requires clinical attention; thus, further research aimed at differentiating significant SI from other kinds of SI will be necessary.
To the best of our knowledge, this study is the first to attempt to predict future SI based on self-reported data from college students. The present study found that predictive modeling may require a variety of factors in addition to current SI and depressive symptoms and that short-term outcomes enhance predictive performance better than long-term outcomes. College may constitute a venue in which young people at risk for suicide can be identified. The present findings show that students who will exhibit future or persistent SI can be identified using a screening survey. This modeling approach can serve as an early warning system to help clinicians identify high-risk students for further screening, which can improve the costeffectiveness of the current screening system. Additionally, the accurate identification of college students at risk for suicide and the referral of these individuals to effective treatment services are mandatory for suicide prevention purposes.

Supplementary Materials

The online-only Data Supplement is available with this article at https://doi.org/10.30773/pi.2018.08.21.3

ACKNOWLEDGEMENTS

This research was supported by a grant of the Brain Research Program through the National Research Foundation of Korea (NRF), funded by the Ministry of Science, ICT & Future Planning (NRF-2016M3C7A1914448). We thank Min Young You for her assistance on data acquisition. We also thank Alicia S. Kang for her advice on English expression.

Figure 1.
Receiver operator characteristic curves (ROC) and precision-recall (PR) curves for classifying future suicidal ideation in four groups. Each area under the curve (AUC) value is presented in the graph. SI- vs. SI+ mean suicidal nonideators vs. suicidal ideators, respectively. FPR: false positive rate.
pi-2018-08-21-3f1.tif
Table 1.
Definitions of common evaluation metrics
Sensitivity (=recall) TP/(TP+FP)
FPR FP/(FP+TN)
Precision TP/(TP+FP)
NPV TN/(TN+FN)
F1 2*precision*recall/(precision+recall)

FN: false negative, FP: false positive, FPR: false positive rate, NPV: negative predictive value, TN: true negative, TP: true positive

Table 2.
Comparison of baseline characteristics according to the presence of future suicidal ideation in suicide ideators and non-ideators at baseline
Future suicidal ideation Baseline suicide non-ideators (N=2,938)
Baseline suicide ideators (N=424)
Absence (N=2,633) Presence (N=305) χ2, t, or U p Absence (N=214) Presence (N=210) χ2, t, or U p
Male (%) 2,178 (82.7) 221 (72.5) 19.21 <0.001 167 (78) 159 (75.7) 0.32 0.57
Smoking (%)
 Never smoker 2,248 (85.4) 261 (85.6) 183 (85.5) 175 (83.3)
 Ex-smoker 179 (6.5) 18 (5.9) 0.18 0.92 11 (5.1) 17 (8.1)
 Current smoker 215 (8.2) 26 (8.5) 20 (9.3) 18 (8.6)
Sleep pattern (bedtime and wake-up time) (%)
 Regular for both 2,216 (84.2) 236 (77.4) 167 (78) 155 (73.8)
 Regular for either one 191 (7.3) 28 (9.2) 9.97 0.01 20 (9.3) 21 (10.0) 1.24 0.54
 Irregular for both 226 (8.6) 41 (13.4) 27 (12.6) 34 (16.2)
Age (yrs) 22.7±3.7 22.5±3.7 0.71 0.48 22.4±3.4 22.7±3.5 -0.65 0.52
Close relationships 10.0 (14.0) 10.0 (15.0) 3.8E+5 0.35 10.0 (15.0) 10.0 (8.3) 2.0E+4 0.48
Concerns 1.0 (2.0) 1.0 (2.0) 3.3E+5 <0.001 2.0 (2.0) 2.0 (3.0) 2.0E+4 0.03
CAGE 0.0 (0.0) 0.0 (1.0) 3.7E+5 <0.005 0.0 (0.0) 0.0 (1.0) 1.9E+4 <0.001
SAS 63.0 (38.0) 70.0 (40.0) 3.3E+5 <0.001 73.0 (34.3) 74.0 (31.3) 2.1E+4 0.32
PSQI 4.0 (3.0) 5.0 (3.0) 3.1E+5 <0.001 6.0 (4.0) 6.0 (4.0) 2.0E+4 0.06
PHQ-9 1.0 (3.0) 3.0 (5.0) 2.7E+5 <0.001 5.0 (5.0) 7.0 (6.0) 1.6E+4 <0.001
GAD-7 0.0 (1.0) 1.0 (3.0) 2.9E+5 <0.001 2.0 (4.0) 4.0 (6.0) 1.7E+4 <0.001
LSAS 9.0 (25.0) 18.0 (28.5) 3.1E+5 <0.001 25.0 (29.0) 35.5 (32.5) 1.5E+4 <0.001
VAQ (parents) 0.0 (3.0) 1.0 (6.0) 3.4E+5 <0.001 3.0 (8.0) 2.5 (10.3) 2.2E+4 0.84
VAQ (superiors) 0.0 (1.0) 1.0 (4.0) 3.3E+5 <0.001 1.0 (5.3) 1.0 (8.0) 2.1E+4 0.26
VAQ (peers) 0.0 (2.0) 1.0 (6.5) 3.3E+5 <0.001 3.0 (9.0) 4.0 (16.0) 2.1E+4 0.18
IES-R 0.0 (5.0) 4.0 (12.5) 2.9E+5 <0.001 9.0 (16.3) 15.0 (20.3) 1.7E+4 <0.001
BSI 2.0 (1.0) 3.0 (4.0) 3.1E+5 <0.001 6.0 (5.0) 9.0 (8.0) 1.4E+4 <0.001
ASRS Screener 0.0 (1.0) 1.0 (2.0) 3.0E+5 <0.001 2.0 (2.0) 2.0 (3.0) 1.9E+4 <0.005

Data was presented as number (%), mean±SD, or median (IQR: interquartile range), and statistical analysis was performed using Pearson’s χ2 test, independent t-test, or Mann-Whitney test, respectively. As for self-report scale, global score was provided. Some data were not available for ‘close relationships’ (N=26), ‘PSQI’ (N=44), and ‘VAQ (peers)’ (N=1). ASRS Screener: Adult ADHD Self-Report Scale-v1.1 Screener, BSI: Beck scale for Suicide Ideation, GAD-7: 7-item Generalized Anxiety Disorder Scale, IES-R: Impact of Event Scale-Revised, LSAS: Liebowitz Social Anxiety Scale, PHQ-9: Patient Health Questionnaire-9, PSQI: Pittsburgh Sleep Quality Index, SAS: Smartphone Addiction scale, VAQ: Verbal Abuse Questionnaire

Table 3.
Logistic regression model with final selected predictors for the presence of future suicidal ideation in suicide ideators and nonideators at baseline
Predictor B Wald χ2 p Odds ratio (95% CI)
Baseline suicide non-ideators (N=2,938): χ2(6)=168.56, Nagelkerke R2=0.11 Gender (male) -0.47 10.57 0.001 0.62 (0.47–0.83)
PSQI comp.3 (sleep duration) -0.13 4.08 0.04 0.88 (0.77–1.00)
PHQ-9 0.47 37.98 <0.001 1.61 (1.38–1.87)
VAQ (superiors) 0.15 7.31 0.007 1.17 (1.04–1.30)
BSI 0.43 35.65 <0.001 1.54 (1.34–1.77)
ASRS screener 0.18 8.02 0.005 1.20 (1.06–1.37)
Constant -1.78 199.73 <0.001 0.17
Baseline suicide ideators (N=424): χ2(6)=90.24, Nagelkerke R2=0.26 CAGE 0.65 11.68 <0.001 1.92 (1.32–2.80)
PSQI comp.4 (sleep efficiency) -0.17 4.02 0.05 0.85 (0.72-1.00)
PHQ-9 0.23 5.30 0.02 1.26 (1.03–1.53)
LSAS (avoidance subscale) 0.46 18.61 <0.001 1.58 (1.28–1.94)
BSI 0.35 16.31 <0.001 1.42 (1.20–1.68)
CAGE squared* -0.15 6.36 0.01 0.86 (0.76–0.97)
Constant -0.89 26.04 <0.001 0.41

* as the interaction between CAGE score and ln (CAGE score +1) was significant [B=-0.60, Wald χ2=8.72, p<0.005, odds ratio=0.55 (0.37–0.82)] in the Box-Tidwell test, we added squared value of CAGE score to the model.

ASRS Screener: Adult ADHD Self-Report Scale-v1.1 Screener, BSI: Beck scale for Suicide Ideation, LSAS: Liebowitz Social Anxiety Scale, PHQ-9: Patient Health Questionnaire-9, PSQI comp.: Pittsburgh Sleep Quality Index component, VAQ: Verbal Abuse Questionnaire

Table 4.
Logistic regression model with final selected predictors for the presence of suicidal ideation at the 2nd-half follow up in suicide ideators and non-ideators at the 1st-half follow up
Predictor B Wald χ2 p Odds ratio (95% CI)
Suicide non-ideators at the 1st-half follow up (N=1,316): χ2(11)=118.91, Nagelkerke R2=0.21 Smoking (never smoker) 0.75 2.69 0.10 2.12 (0.86–5.20)
Smoking (ex-smoker vs. current smoker) 2.02 5.39 0.02 7.58 (1.37–41.87)
Close relationships 0.28 9.57 <0.005 1.32 (1.11–1.58)
Concerns 0.40 9.60 <0.005 1.49 (1.16–1.92)
PSQI comp.1 (sleep quality) 0.26 3.89 0.05 1.30 (1.00–1.69)
PSQI comp.4 (sleep efficiency) -0.37 5.80 0.02 0.69 (0.51–0.93)
PSQI comp.6 (sleeping medication) 0.23 6.74 0.01 1.26 (1.06–1.51)
PHQ-9 0.38 6.59 0.01 1.46 (1.09–1.95)
VAQ (peers) 0.18 3.96 0.05 1.20 (1.00–1.43)
BSI 0.32 7.89 <0.005 1.37 (1.10–1.71)
RSES -0.49 11.66 <0.001 0.61 (0.46–0.81)
Constant -3.07 101.72 <0.001 0.05
Suicide ideators at the 1st-half follow up (N=162): χ2(4)=45.44, Nagelkerke R2=0.33 PHQ-9 0.62 10.53 <0.005 1.85 (1.28–2.68)
LSAS (fear subscale) 0.45 6.06 0.01 1.56 (1.10–2.23)
RSES -0.86 8.50 <0.005 0.42 (0.24–0.75)
RAS 0.60 4.46 0.03 1.82 (1.04–3.19)
Constant -1.71 26.17 <0.001 0.18

LSAS: Liebowitz Social Anxiety Scale, PHQ-9: Patient Health Questionnaire-9, PSQI comp.: Pittsburgh Sleep Quality Index component, RAS: Resilience Appraisals Scale, RSES: Rosenberg’s Self-Esteem Scale, BSI: Beck scale for Suicide Ideation, VAQ: Verbal Abuse Questionnaire

Table 5.
Evaluation metrics of predictive models for four groups. Optimal threshold point was based on Youden index that maximizes the value of ‘sensitivity-false positive rate’
Baseline
1st-half follow up
Suicide nonideators (N=2,938) Suicide ideators (N=424) Suicide nonideators (N=1,316) Suicide ideators (N=162)
Cutoff point 0.11 0.59 0.05 0.46
Youden index 0.32 0.36 0.45 0.41
FPR 0.26 0.16 0.38 0.23
Sensitivity (=recall) 0.58 0.52 0.82 0.64
Precision 0.21 0.76 0.15 0.66
NPV 0.94 0.64 0.98 0.75
F1 0.30 0.62 0.25 0.65

FPR: false positive rate, NPV: negative predictive value

REFERENCES

1. Arnett JJ. Emerging adulthood. A theory of development from the late teens through the twenties. Am Psychol 2000;55:469–480.
crossref pmid
2. Eskin M, Sun JM, Abuidhail J, Yoshimasu K, Kujan O, Janghorbani M, et al. Suicidal behavior and psychological distress in university students: a 12-nation study. Arch Suicide Res 2016;20:369–388.
crossref pmid
3. Mortier P, Cuijpers P, Kiekens G, Auerbach RP, Demyttenaere K, Green JG, et al. The prevalence of suicidal thoughts and behaviours among college students: a meta-analysis. Psychol Med 2018;48:554–565.
crossref pmid
4. Roh MS, Jeon HJ, Lee HW, Lee HJ, Han SK, Hahm BJ. Suicide-related behaviors among the college students. J Korean Neuropsychiatr Assoc 2007;46:35–40.

5. Mann JJ. A current perspective of suicide and attempted suicide. Ann Intern Med 2002;136:302–311.
crossref pmid
6. Park EH, Hong N, Jon DI, Hong HJ, Jung MH. Past suicidal ideation as an independent risk factor for suicide behaviours in patients with depression. Int J Psychiatry Clin Pract 2017;21:24–28.
crossref pmid
7. Garlow SJ, Rosenberg J, Moore JD, Haas AP, Koestner B, Hendin H, et al. Depression, desperation, and suicidal ideation in college students: results from the American Foundation for Suicide Prevention College Screening Project at Emory University. Depress Anxiety 2008;25:482–488.
crossref pmid
8. Goldman-Mellor SJ, Caspi A, Harrington H, Hogan S, Nada-Raja S, Poulton R, et al. Suicide attempt in young people: a signal for long-term health care and social needs. JAMA Psychiatry 2014;71:119–127.
crossref pmid pmc
9. Reinherz HZ, Tanner JL, Berger SR, Beardslee WR, Fitzmaurice GM. Adolescent suicidal ideation as predictive of psychopathology, suicidal behavior, and compromised functioning at age 30. Am J Psychiatry 2006;163:1226–1232.
crossref pmid
10. Konick LC, Gutierrez PM. Testing a model of suicide ideation in college students. Suicide Life Threat Behav 2005;35:181–192.
crossref pmid
11. Van Orden KA, Witte TK, James LM, Castro Y, Gordon KH, Braithwaite SR, et al. Suicidal ideation in college students varies across semesters: the mediating role of belongingness. Suicide Life Threat Behav 2008;38:427–435.
crossref pmid
12. Toprak S, Cetin I, Guven T, Can G, Demircan C. Self-harm, suicidal ideation and suicide attempts among college students. Psychiatry Res 2011;187:140–144.
crossref pmid
13. Zhai H, Bai B, Chen L, Han D, Wang L, Qiao Z, et al. Correlation between family environment and suicidal ideation in university students in China. Int J Environ Res Public Health 2015;12:1412–1424.
crossref pmid pmc
14. Kraemer HC, Kazdin AE, Offord DR, Kessler RC, Jensen PS, Kupfer DJ. Coming to terms with the terms of risk. Arch Gen Psychiatry 1997;54:337–343.
crossref pmid
15. Franklin JC, Ribeiro JD, Fox KR, Bentley KH, Kleiman EM, Huang X, et al. Risk factors for suicidal thoughts and behaviors: a meta-analysis of 50 years of research. Psychol Bull 2017;143:187–232.
crossref pmid
16. Ewing JA. Detecting alcoholism. The CAGE questionnaire. JAMA 1984;252:1905–1907.
crossref pmid
17. Kwon M, Kim DJ, Cho H, Yang S. The smartphone addiction scale: development and validation of a short version for adolescents. PLoS One 2013;8:e83558.
crossref pmid pmc
18. Sohn SI, Kim DH, Lee MY, Cho YW. The reliability and validity of the Korean version of the Pittsburgh Sleep Quality Index. Sleep Breath 2012;16:803–812.
crossref pmid
19. An JY, Seo ER, Lim KH, Shin JH, Kim JB. Standardization of the Korean version of Screening Tool for Depression (Patient Health Questionnaire-9, PHQ-9). J Korean Soc Biol Ther Psychiatry 2013;19:47–56.

20. Seo JG, Park SP. Validation of the Generalized Anxiety Disorder-7 (GAD-7) and GAD-2 in patients with migraine. J Headache Pain 2015;16:97
crossref pmid pmc
21. Yu ES, Ahn CI, Park KH. Factor structure and diagnostic efficiency of A Korean version of the Liebowitz Social Anxiety Scale. Kor J Clin Psychol 2007;26:251–270.
crossref
22. Jeong B, Lee SW, Lee JS, Yoo JH, Kim KW, Cho S, et al. The psychometric properties of the Korean version of the verbal abuse questionnaire in university students. Psychiatry Investig 2015;12:190–196.
crossref pmid pmc
23. Lim HK, Woo JM, Kim TS, Kim TH, Choi KS, Chung SK, et al. Reliability and validity of the Korean version of the Impact of Event Scale-Revised. Compr Psychiatry 2009;50:385–390.
crossref pmid
24. Kessler RC, Adler L, Ames M, Demler O, Faraone S, Hiripi E, et al. The World Health Organization Adult ADHD Self-Report Scale (ASRS): a short screening scale for use in the general population. Psychol Med 2005;35:245–256.
crossref pmid
25. Shin MS, Park KB, Oh KJ, Kim ZS. A study of suicidal ideation among high school students: the structural relation among depression, hopelessness, and suicidal ideation. Kor J Clin Psychol 1990;9:1–19.

26. Lee JY, Nam SK, Lee MK, Lee JH, Lee SM. Rosenberg’ self-esteem scale: analysis of item-level validity. Korean J Couns Psychother 2009;21:173–189.

27. Johnson J, Gooding PA, Wood AM, Tarrier N. Resilience as positive coping appraisals: testing the schematic appraisals model of suicide (SAMS). Behav Res Ther 2010;48:179–186.
crossref pmid
28. R Core Team. A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing; 2017.

29. Wuensch KL. Independent Samples T Tests versus Binary Logistic Regression. In: Karl Wuensch’s Statistics Lessons. Available at: http://core.ecu.edu/psyc/wuenschk/StatsLessons.htm. Accessed September 11, 2017.

30. Wuensch KL. Binary Logistic Regression with SPSS. In: Karl Wuensch’s Statistics Lessons. Available at: http://core.ecu.edu/psyc/wuenschk/StatsLessons.htm. Accessed February 17, 2017.

31. Field A. Discovering Statistics Using SPSS. London, UK: Sage; 2009.

32. Grau J, Keilwagen J. Package ‘PRROC’. April 21, 2017. Available at: https://CRAN.R-project.org/package=PRROC. Accessed October 19, 2017.

33. Grau J, Grosse I, Keilwagen J. PRROC: computing and visualizing precision-recall and receiver operating characteristic curves in R. Bioinformatics 2015;31:2595–2597.
crossref pmid pmc
34. Davis J, Goadrich M. The Relationship between Precision-Recall and ROC Curves. In: Cohen WW, editor. Proceedings of the 23rd International Conference on Machine Learning. Pittsburgh, Pennsylvania, USA: ACM, 2006, p. 233–240.

35. Kumar R, Indrayan A. Receiver operating characteristic (ROC) curve for medical researchers. Indian Pediatr 2011;48:277–287.
crossref pmid
36. Arria AM, O’Grady KE, Caldeira KM, Vincent KB, Wilcox HC, Wish ED. Suicide ideation among college students: a multivariate analysis. Arch Suicide Res 2009;13:230–246.
crossref pmid pmc
37. Isometsa ET, Heikkinen ME, Marttunen MJ, Henriksson MM, Aro HM, Lonnqvist JK. The last appointment before suicide: is suicide intent communicated? Am J Psychiatry 1995;152:919–922.
crossref pmid
38. Walsh CG, Ribeiro JD, Franklin JC. Predicting risk of suicide attempts over time through machine learning. Clin Psychol Sci 2017;5:457–469.
crossref
39. Barak-Corren Y, Castro VM, Javitt S, Hoffnagle AG, Dai Y, Perlis RH, et al. Predicting suicidal behavior from longitudinal electronic health records. Am J Psychiatry 2017;174:154–162.
crossref pmid
40. Hack L, Jovanovic T, Carter S, Ressler K, Smith A. Suicide prediction using machine learning techniques in screening and clinician-derived data. Biol Psychiatry 2017;81:S361
crossref
41. Oh J, Yun K, Hwang JH, Chae JH. Classification of suicide attempts through a machine learning algorithm based on multiple systemic psychiatric scales. Front Psychiatry 2017;8:192
crossref pmid pmc
42. Gibb BE, Alloy LB, Abramson LY, Rose DT, Whitehouse WG, Hogan ME. Childhood maltreatment and college students’ current suicidal ideation: a test of the hopelessness theory. Suicide Life Threat Behav 2001;31:405–415.
crossref pmid
43. Mann JJ, Waternaux C, Haas GL, Malone KM. Toward a clinical model of suicidal behavior in psychiatric patients. Am J Psychiatry 1999;156:181–189.
crossref pmid
44. Joiner TE. Why People Die by Suicide. Cambridge, MA: Harvard University Press; 2005.

45. Cukrowicz KC, Schlegel EF, Smith PN, Jacobs MP, Van Orden KA, Paukert AL, et al. Suicide ideation among college students evidencing subclinical depression. J Am Coll Health 2011;59:575–581.
crossref pmid pmc
46. Hom MA, Stanley IH, Joiner TE. Evaluating factors and interventions that influence help-seeking and mental health service utilization among suicidal individuals: a review of the literature. Clin Psychol Rev 2015;40:28–39.
crossref pmid
47. Biddle L, Gunnell D, Sharp D, Donovan JL. Factors influencing help seeking in mentally distressed young adults: a cross-sectional survey. Br J Gen Pract 2004;54:248–253.
pmid pmc
48. Borges G, Benjet C, Medina-Mora ME, Orozco R, Familiar I, Nock MK, et al. Service use among Mexico City adolescents with suicidality. J Affect Disord 2010;120:32–39.
crossref pmid pmc
49. Cheung AH, Dewa CS. Mental health service use among adolescents and young adults with major depressive disorder and suicidality. Can J Psychiatry 2007;52:228–232.
crossref pmid
50. Rudd MD, Joiner TE Jr, Rajab MH. Help negation after acute suicidal crisis. J Consult Clin Psychol 1995;63:499–503.
crossref
51. McCall WV, Black CG. The link between suicide and insomnia: theoretical mechanisms. Curr Psychiatry Rep 2013;15:389
crossref pmid pmc
52. Kim JH, Park EC, Cho WH, Park CY, Choi WJ, Chang HS. Association between total sleep duration and suicidal ideation among the Korean general adult population. Sleep 2013;36:1563–1572.
crossref pmid pmc
53. Sánchez-Teruel D, Robles-Bello MA. Protective factors promoting resilience to suicide in young people and adolescents. Papeles del Psicólogo 2014;35:181–192.

54. Min JA, Lee CU, Chae JH. Resilience moderates the risk of depression and anxiety symptoms on suicidal ideation in patients with depression and/or anxiety disorders. Compr Psychiatry 2015;56:103–111.
crossref pmid
55. Posner K, Brown GK, Stanley B, Brent DA, Yershova KV, Oquendo MA, et al. The Columbia-Suicide Severity Rating Scale: initial validity and internal consistency findings from three multisite studies with adolescents and adults. Am J Psychiatry 2011;168:1266–1277.
crossref pmid pmc
56. Large MM, Nielssen O. Suicidal ideation and later suicide. Am J Psychiatry 2012;169:662author reply 663.
crossref
57. Large M, Smith G, Sharma S, Nielssen O, Singh SP. Systematic review and meta-analysis of the clinical factors associated with the suicide of psychiatric in-patients. Acta Psychiatr Scand 2011;124:18–29.
crossref pmid
58. Large M, Sharma S, Cannon E, Ryan C, Nielssen O. Risk factors for suicide within a year of discharge from psychiatric hospital: a systematic meta-analysis. Aust N Z J Psychiatry 2011;45:619–628.
crossref pmid
59. Pokorny AD. Prediction of suicide in psychiatric patients: report of a prospective study. Arch Gen Psychiatry 1983;40:249–257.
crossref pmid
60. Goldstein RB, Black DW, Nasrallah A, Winokur G. The prediction of suicide: sensitivity, specificity, and predictive value of a multivariate model applied to suicide among 1906 patients with affective disorders. Arch Gen Psychiatry 1991;48:418–422.
crossref pmid