Predicting Suicidal Ideation in College Students with Mental Health Screening Questionnaires
Article information
Abstract
Objective
The present study aimed to identify risk factors for future SI and to predict individual-level risk for future or persistent SI among college students.
Methods
Mental health check-up data collected over 3 years were retrospectively analyzed. Students were categorized as suicidal ideators and non-ideators at baseline. Logistic regression analyses were performed separately for each group, and the predicted probability for each student was calculated.
Results
Students likely to exhibit future SI had higher levels of mental health problems, including depression and anxiety, and significant risk factors for future SI included depression, current SI, social phobia, alcohol problems, being female, low self-esteem, and number of close relationships and concerns. Logistic regression models that included current suicide ideators revealed acceptable area under the curve (AUC) values (0.7–0.8) in both the receiver operating characteristic (ROC) and precision recall (PR) curves for predicting future SI. Predictive models with current suicide non-ideators revealed an acceptable level of AUCs only for ROC curves.
Conclusion
Several factors such as low self-esteem and a focus on short-term rather than long-term outcomes may enhance the prediction of future SI. Because a certain range of SI clearly necessitates clinical attention, further studies differentiating significant from other types of SI are necessary.
INTRODUCTION
College entrance can be particularly stressful for students who are psychologically vulnerable and have poor support because this period involves a transition from adolescence to “emerging adulthood.” [1] College students are at high risk for suicidal ideation (SI), planning, and attempts [2]. Approximately one of four to six college students have experienced some form of SI during college, and about 40% and 20% of students with SI report suicide plans and attempts, respectively [3]. As for the college students in Korea, one-month prevalence of suicidal ideation, plan, and attempt was 6.7%, 0.5%, and 0.1%, respectively [4].
Current SI and a history of suicide attempts are generally considered to be important predictors of later suicidal behavior [5]. A recent study of patients with depression found that the severity of past SI is the most important factor differentiating suicide attempters from non-attempters, even after controlling for other significant variables [6]. The authors suggested that severe SI should not be overlooked because it might be a major independent predictor of suicidal behaviors. Suicidal thoughts and behaviors are markers of extreme psychological distress [7] and are linked with long-term adverse outcomes, such as depression, substance dependence, and unemployment [8]. For example, 15-year-olds with SI are more likely to exhibit psychopathology, suicidal behavior, and compromised functioning at age 30 [9]; thus, there is a need for early identification of and continued intervention with those with SI.
Many studies have attempted to identify the risk factors of SI among college students. Generally, this issue is thought to be associated with depressive symptoms [7,10], hopelessness, stressful life events, lack of social belongingness [11], smoking, alcohol and drug use [12], and a poor family environment [13]. According to Kraemer et al. [14], a risk factor is a special type of correlate that temporally precedes the outcome of interest and divides individuals into high- and low-risk groups [15]. Thus, strictly speaking, the aforementioned factors are correlates or concomitants rather than risk factors or longitudinal predictors because they are derived from cross-sectional rather than longitudinal studies.
Thus, the present study aimed to identify the risk factors for future SI in college students. Because current SI may be the most important contributor to future SI, students were categorized as suicide ideators and non-ideators at baseline to estimate the risk factors, predict the individual-level risk for future or persistent SI, and test the accuracy of a predictive model.
Methods
Procedures
The present study retrospectively analyzed mental health check-up data collected between April 2014 and March 2017. All students at our institution are required to participate in web-based mental health screening once per year as part of a routine medical check-up. The first-year screening was completed by 4,872 students, and 3,362 of these students completed the check-up twice or more over 3 years. Although some students (n=391) participated in the check-up more than three times, only up to two sets of follow-up data per student were used for statistical analysis. The follow-up data with the longest intervals were selected from each of two timepoints (i.e., before, within the median follow-up interval of 13 months, and after the 13 months); these timepoints are referred to as the 1st-half and 2nd-half follow-up periods, respectively. In total, 1,057 and 698 students were followed in only the 1st-half or 2nd-half periods, respectively, and 1,607 students were followed in both periods.
The study protocol was approved by the KAIST Institutional Review Board (IRB No. KH2017-56). Before answering the questionnaire, all students provided informed consent stating their mental health check-up data could be used for research purposes after data anonymization.
Measures
Sociodemographic variables
Students were categorized as never smokers, ex-smokers, or current smokers based on whether they smoked more than five packs during their lifetimes. Sleep patterns were classified as follows based on sleep and wake-up routines during the semester: regular for both, regular for either of the two, and irregular for both. The total number of concerns was calculated based on answers to the question “Please check the box next to each problem about which you have been worried recently (you can check more than one),” and number of close relationships was assessed with the question “With how many people in school do you have close relationships?”
Scales
Several scales were administered at baseline (over the first year): the CAGE quiz, which screens for problematic alcohol use [16]; the Smartphone Addiction Scale [17]; the Pittsburgh Sleep Quality Index [18]; the Patient Health Questionnaire-9 [19]; the Generalized Anxiety Disorder-7 (GAD-7) scale [20]; the Liebowitz Social Anxiety Scale [21]; the Verbal Abuse Questionnaire [22]; the Impact of the Event Scale-Revised version [23]; the Adult ADHD Self-Report Scale version 1.1 Screener [24]; the Beck Scale for Suicide Ideation [25]; and the KAIST Scale for Suicide Ideation (KSI).
The 19 self-rated items on the PSQI were combined to form scores for seven components: subjective sleep quality, sleep latency, sleep duration, habitual sleep efficiency, sleep disturbances, use of sleeping medication, and daytime dysfunction. The LSAS scores were divided into two subscale scores (fear and avoidance), the IES-R was scored as three subdomains (intrusion, avoidance, and hyperarousal), and the VAQ scores were calculated as total scores for parental, superior, and peer verbal abuse. The ASRS Screener score reflected the number of checkmarks that corresponded to a certain frequency range (from ‘sometimes’ or ‘often’ to ‘very often’), as described by Kessler et al. [24], and the BSI score was the sum of the scores on the first 19 items. The KSI score was the sum of the scores on the first 14 items, each of which was rated from 0 to 4; the KSI was developed by our research team to assess various levels of SI over the past 2 weeks or the last year on a scale ranging from mild (“I would rather fall asleep and not wake up”) to severe (“I will carry out my thoughts of wanting to take my own life”) (Supplementary Table 1 in the online-only Data Supplement). The KSI score for the past 2 weeks was significantly correlated with the BSI score in the present study (Kendall’s τ=0.35, p<0.001).
Beginning in April 2015, two additional measures, the Rosenberg’s Self-Esteem Scale [26] and the Resilience Appraisals Scale [27], were added to the survey, and the five questionnaires used at baseline (SAS, VAQ, IES-R, ASRS Screener, and BSI) were no longer used for the mental health survey. Korean versions of nine self-report scales were validated: SAS, PSQI, PHQ-9, GAD-7, LSAS, VAQ, IES-R, BSI, and RSES.
Analyses
Statistical analyses were performed with R [28]. Initially, the presence of SI was determined by the KSI score for the last 2 weeks because these data were available for a period spanning more than 3 years; a KSI score of 0 indicated no SI whereas a KSI score >0 indicated SI. To identify risk factors for future or persistent SI, logistic regression analyses were independently performed for suicide ideators and non-ideators at baseline. The presence or absence of future SI, based on follow-up KSI score, was entered as the dependent variable, and risk factors were identified with a backward conditional stepwise elimination strategy after all continuous variables were transformed into z-scores. If a student had a KSI score >0 at least once during the follow-up survey, they were regarded as having future SI.
Using the final variables, we assessed whether the relationships between the continuous predictors and the logit were linear using the Box-Tidwell test [29]. If the relationships were not linear and the sample size was not large enough, a polynomial model was applied [30]. The Hosmer-Lemeshow goodness statistic was calculated to assess how well the chosen model fit the data, and multicollinearity was tested using tolerance and the variance inflation factor [31]. The Nagelkerke pseudo-R2 value was used as a measure of the total effect size of the final model.
The predicted probability for future SI was calculated from the final model using the leave-one-out sample method (i.e., all the remaining data without oneself) to verify predictability with greater accuracy. Using the calculated predicted probability, receiver operating characteristic (ROC) curves and precision recall (PR) curves were generated, and area-underthe-curve (AUC) values were calculated to evaluate prediction accuracy using the ‘PRROC’ package [32,33]. Because ROC curves may present an overly optimistic view of a classifier’s performance when there is a large skew in the class distribution [34], PR curves were also generated.
The optimal threshold that provided the maximum correct prediction was found based on the Youden Index, which is the difference between the sensitivity and the false positive rate and equals the vertical distance between the ROC curve and a line of equality or a chance line [35]. Maximizing the Youden index allows for the identification of an optimal cut-off point on the ROC curve that is independent of the prevalence. The definitions for these terms were used as following Table 1.
Definitions of common evaluation metrics
The presence of SI at the 2nd-half follow up among students for whom both sets of data were available was predicted based on the data from the 1st-half follow up and the five scales that were administered only at baseline. After applying the same procedure for this modeling, we expected that the combined model using the baseline and 1st-half follow-up data would be superior to the original model using only baseline data because the former included more predictors and the prediction interval was relatively shorter.
Results
Table 2 presents the baseline sociodemographic characteristics and scores on the clinical scales of suicide ideators and non-ideators separately as well as the comparison results based on future SI. Suicide non-ideators who showed future SI were more likely to be female (χ2=19.21, p<0.001), have irregular sleep patterns (χ2=9.97, p=0.01), and have more concerns and alcohol problems; more concerns and alcohol problems were also evident in suicidal ideators who showed persistent SI. Students who showed future SI already exhibited more clinical problems at baseline compared to those who did not, irrespective of baseline SI. Students who did not have future SI (at the 2nd half) had significantly higher scores on the RAS and RSES compared to those who did, regardless of SI at the 1st half (Supplementary Table 2 in the online-only Data Supplement).
Comparison of baseline characteristics according to the presence of future suicidal ideation in suicide ideators and non-ideators at baseline
In the logistic regression model, the risk factors for future SI in non-ideators were being female, long night-time sleep duration, higher levels of depressive symptoms, verbal abuse from superiors (e.g., academic supervisors or seniors), SI according to the BSI, and attention deficit hyperactivity disorder (ADHD) symptoms (Table 3). The total effect size of the predictive models using the above six risk factors with a constant was 0.11 (Nagelkerke R2), and the strongest three predictors were depressive symptoms, SI, and being female. Risk factors for persistent SI in suicide ideators were higher levels of alcohol problems, depressive symptoms, avoidance due to social phobia, and SI measured by the BSI. Low sleep efficiency, as measured by the PSQI, was a protective factor against future SI in suicide ideators. The total effect size was 0.26, and the strongest three predictors were avoidance due to social phobia, SI, and alcohol problems.
Logistic regression model with final selected predictors for the presence of future suicidal ideation in suicide ideators and nonideators at baseline
In the logistic regression analysis predicting the presence of future SI in non-ideators at the 1st-half follow up, the significant predictors were being a current smoker (vs. ex-smoker), number of close relationships and concerns, poor sleep quality, use of sleeping medication, higher levels of depressive symptoms, verbal abuse from peers, and SI measured by the BSI (Table 4). Low sleep efficiency and high self-esteem were protective factors against SI. The total effect size was 0.21, and the strongest three predictors were low self-esteem and number of close relationships and concerns. Risk factors for persistent SI in suicide ideators at the 1st-half follow up were higher levels of depressive symptoms, fear due to social phobia, and resilience appraisal. High self-esteem was a protective factor against persistent SI. The total effect size of this model was 0.33, and the strongest three predictors were depressive symptoms, low self-esteem, and fear due to social phobia.
Logistic regression model with final selected predictors for the presence of suicidal ideation at the 2nd-half follow up in suicide ideators and non-ideators at the 1st-half follow up
Based on the Hosmer-Lemeshow goodness-of-fit statistic, the above four models fitted the data well [baseline suicide non-ideators: χ2(8)=12.87, p=0.12; baseline suicide ideators, χ2(8)=13.92, p=0.08; 1st-half follow up suicide non-ideators: χ2(8)=9.62, p=0.29; and 1st-half follow up suicide ideators: χ2(8)=11.06, p=0.20]. Tests for multicollinearity did not reveal an alarming level of problems, as, with the exception of CAGE (VIF=3.39) and CAGE squared (VIF=3.40) in the baseline suicide ideators model, all the VIF values for each predictor in the four models were <2.50. The average VIF values were 1.13 (baseline suicide non-ideators), 1.95 (baseline suicide ideators), 1.20 (1st-half follow up suicide non-ideators), and 1.53 (1st-half follow up suicide ideators) for each model. Additionally, there were no noticeable issues with multicollinearity (Supplementary Table 3 in the online-only Data Supplement).
The ROC and PR curves based on the abovementioned four prediction models are shown in Figure 1. Two models predicting SI at the 2nd-half follow up showed larger AUCs in the ROC curves relative to the other two prediction models. However, the two prediction models that used suicide ideators performed better than the other two models that used suicide non-ideators on the PR curves due to the higher prevalence of future SI in the groups with suicide ideators.
Receiver operator characteristic curves (ROC) and precision-recall (PR) curves for classifying future suicidal ideation in four groups. Each area under the curve (AUC) value is presented in the graph. SI- vs. SI+ mean suicidal nonideators vs. suicidal ideators, respectively. FPR: false positive rate.
Table 5 presents the evaluation metrics of the four predictive models. The predictive model using suicide non-ideators at the 1st-half follow up had the highest sensitivity but the lowest precision. The two prediction models using suicide ideators showed relatively higher precision and F1 scores compared to the other two prediction models using suicide nonideators. The prediction model using suicide ideators at the 1st-half follow up had the best performance based on F1 score. Detailed confusion matrices of the four groups are presented in Supplementary Table 4 (in the online-only Data Supplement).
Evaluation metrics of predictive models for four groups. Optimal threshold point was based on Youden index that maximizes the value of ‘sensitivity-false positive rate’
Discussion
The present study found that students who would have SI in the future already had higher levels of mental health problems, such as depression and anxiety, at baseline than did students who did not. Additionally, the significant risk factors for future SI were depression, current SI, social phobia, alcohol problems, being female, low self-esteem, and number of close relationships and concerns. Logistic regression models with current suicide ideators showed an acceptable level of AUCs (0.7–0.8) in the ROC and PR curves for the prediction of future SI. However, predictive models using current suicide non-ideators showed an acceptable level of AUCs only in the ROC curves, whereas the AUCs of the PR curves were below 0.3 due to the low prevalence of future SI among current suicide non-ideators.
Students with higher levels of depression and hopelessness, alcohol-related problems, and poor social support are more likely to exhibit SI [7,36]. Female college students have a higher risk of SI [13] and are more prone to report their suicidal thoughts to others compared to males [37]. In the present study, students who reported a larger number of close relationships were more likely to have future SI. However, reporting the larger number of close friends may not necessarily mean that these students acquire more solid social networks and are surrounded by close enough relationships whom to report their most personal thoughts, and thus feel less socially isolated. In this regard, the present findings on close relationships should be interpreted with caution.
Walsh and colleagues [38] demonstrated that future suicide attempts can be accurately predicted (AUC: 0.84, precision: 0.79, recall: 0.95) by applying machine learning to electronic medical records. Another study reported that future suicidal behavior can be predicted with 43% sensitivity and 90–95% specificity [39]. Additionally, two prediction models for use with suicide attempters showed reasonable sensitivity (64%) and specificity (76%), with an AUC of 70% [40] and an overall accuracy greater than 80% [41]. However, when interpreting a prediction model’s performance on binary classification, the proportion of two outcomes must be kept in mind. In the above study by Walsh and colleagues [38], suicide attempters (i.e., cases) were more than 1.5 times as many as controls, which may have affected the relatively high AUC of the PR curves. In the present study, only about 9% of suicide nonideators showed future SI, whereas about 47% of suicide ideators showed persistent SI in the future. Because the prediction models using non-ideators showed noticeably low precision related to a high false positive rate, it is possible that the model based on current suicide ideators was more applicable than the model based on non-ideators.
One study investigating the link between childhood maltreatment and adult suicidality found that childhood emotional, but not physical or sexual, maltreatment is uniquely associated with SI in college students [42]. Furthermore, this relationship was partially mediated by the cognitive styles and hopelessness levels of students. A study assessing the association between SI and family background in college students showed that suicidal ideators have poor family structures and relationships and parents with unstable work habits and improper parenting styles [13]. Thus, based on the stress-diathesis model of suicidal behavior [43], it can be postulated that recent interpersonal stress, such as verbal abuse from peers and superiors, can lead to future SI in students with a history of childhood maltreatment, which may have resulted in vulnerable personality traits, including low self-esteem. Similarly, interpersonal stress can lead to feeling like a burden to others and to experiencing a lack of belonging, which, according to the interpersonal theory of suicide, are two main interpersonal cognitive-affective states contributing to SI [44]. Thus, further evaluations of the childhood maltreatment experienced by current suicidal ideators will be helpful for improving the prediction of future SI.
SI, as measured by the BSI, was a significant risk factor for future SI even among non-ideators. Compared to the KSI, the BSI was more sensitive for capturing subtle SI. This indicates that the use of fine-grained measures that clearly differentiate between more and less severe SI will be optimal for predictive models. In addition to SI, depressive symptoms also emerged as a strong and consistent risk factor in the present study. However, according to previous studies, students with subclinical depression are also liable to have SI, and this was confirmed across different samples [45]. Therefore, a variety of risk factors should be assessed concomitantly and may be considered potential therapeutic targets during mental health screening.
Suicidal thoughts and behaviors are also linked to reduced rates of professional help-seeking for mental health problems [46]. Studies conducted in various countries have found the usage rates of mental health services among adolescents and young adults with SI and related behaviors to be at or below 50% [47-49]. Furthermore, adolescents with more severe SI have lower rates of help-seeking behaviors than those with less severe ideation [46]. Young people with more severe SI are also more likely to be hopeless, pessimistic, and have maladaptive adaptive coping strategies, which, in part, can lead to such ‘help-negation’ [50]. Therefore, we need focused efforts to connect students with severe and persistent SI with appropriate services and to encourage them to remain in treatment.
Interestingly, students with longer sleep durations, higher sleep efficiency, and more positive self-appraisals of resilience were more likely to have future SI in the present study. A growing body of evidence supports the idea that sleep problems, including insomnia, are risk factors for SI, related behaviors, and death [51]. However, a recent study using large survey data from Korean adults found that both short-sleeping and longsleeping individuals were more likely to have SI than people who sleep 7 hours a day [52]. Higher levels of resilience should also lead to less suicidality, because resilience leads to a quick recovery and psychosocial growth in the face of adverse life situations [53]. However, a study of patients with depression and anxiety showed that resilience is not independently associated with moderate-severe SI [54]. The effects of the interaction between resilience, on the one hand, and depression and anxiety, on the other, were significant in that study, which implies that resilience primarily moderates the risks posed by depression and anxiety symptoms for SI. In the present study, approximately 70% of the variance in the regression coefficients for both the RAS and RSES scores in the logistic modeling was associated with the smallest eigenvalue, which indicates that some level of dependency between the two variables. Considering the relatively low levels of significance for sleep duration, sleep efficiency, and resilience revealed by the predictive models, the limited range of the self-reported PSQI to evaluate sleep problems, and the potential interaction effect between significant variables, it may be more appropriate to postpone reaching a definitive interpretation of the above findings.
Another limitation of this study should also be discussed. The expression of SI is traditionally considered a risk factor for suicide, but controversy remains regarding the association between SI and suicide. Although Posner and colleagues [55] consider SI to be one of the most salient risk factors for suicide, Large and Nielssen [56] assert that the association between these two variables is unclear and that the presence of SI in the absence of a suicide attempt has not been firmly established as a risk factor for suicide. However, recent systematic meta-analyses found weak associations between these variables in psychiatric inpatients [57,58] and showed that the association between SI and action is stronger in the short-term than in the long-term [59,60]. Taken together, these data suggest that a certain range of SI clearly requires clinical attention; thus, further research aimed at differentiating significant SI from other kinds of SI will be necessary.
To the best of our knowledge, this study is the first to attempt to predict future SI based on self-reported data from college students. The present study found that predictive modeling may require a variety of factors in addition to current SI and depressive symptoms and that short-term outcomes enhance predictive performance better than long-term outcomes. College may constitute a venue in which young people at risk for suicide can be identified. The present findings show that students who will exhibit future or persistent SI can be identified using a screening survey. This modeling approach can serve as an early warning system to help clinicians identify high-risk students for further screening, which can improve the costeffectiveness of the current screening system. Additionally, the accurate identification of college students at risk for suicide and the referral of these individuals to effective treatment services are mandatory for suicide prevention purposes.
Supplementary Materials
The online-only Data Supplement is available with this article at https://doi.org/10.30773/pi.2018.08.21.3
Acknowledgements
This research was supported by a grant of the Brain Research Program through the National Research Foundation of Korea (NRF), funded by the Ministry of Science, ICT & Future Planning (NRF-2016M3C7A1914448). We thank Min Young You for her assistance on data acquisition. We also thank Alicia S. Kang for her advice on English expression.