Standardization of the Discrimination and Stigma Scale-Korean Version (DISC 12-K) in Patients with Depressive Disorders

Gyu-On Kim; Tae-Young Yoo; Nam-Jun Kim; Hee-Joon Lee; Min Jhon; Ju-Wan Kim; Hee-Ju Kang; Sung-Wan Kim; Jae-Min Kim

doi:10.30773/pi.2020.0111

Psychiatry Investig > Volume 17(7); 2020 > Article

Kim, Yoo, Kim, Lee, Jhon, Kim, Kang, Kim, and Kim: Standardization of the Discrimination and Stigma Scale-Korean Version (DISC 12-K) in Patients with Depressive Disorders

Original Article

Psychiatry Investigation 2020;17(7):654-661.

Published online: June 24, 2020

DOI: https://doi.org/10.30773/pi.2020.0111

Standardization of the Discrimination and Stigma Scale-Korean Version (DISC 12-K) in Patients with Depressive Disorders

Department of Psychiatry, Chonnam National University Medical School, Gwangju, Republic of Korea

Correspondence: Jae-Mim Kim, MD, PhD Department of Psychiatry, Chonnam National University Medical School, 160 Baekseo-ro, 12 Dong-gu, Gwangju 61469, Republic of Korea Tel: +82-62-220-6143, Fax: +82-62-225-2351, E-mail: jmkim@chonnam.ac.kr

Received March 23, 2020 Revised April 1, 2020 Accepted April 2, 2020

This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/4.0/) which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

Objective

The Discrimination and Stigma Scale (DISC 12), which assesses behavioral and experienced stigma, has not been translated into Korean. We developed and standardized the Korean version of the DISC 12 (DISC 12-K) in patients with depressive disorders.

Methods

The study included 230 patients with depressive disorders who were assessed on the four subscales of the DISC 12-K: Unfair Treatment, Stopping Self, Overcoming Stigma, and Positive Treatment. Additionally, stigma was assessed using the Internalized Stigma of Mental Illness scale, depressive symptoms using the Hamilton Depression Rating Scale and Beck Depression Inventory, level of functioning using the Social and Occupational Functioning Assessment Scale, self-esteem using the Rosenberg Self-Esteem Scale, and quality of life was assessed using the EuroQol-5D. The reliability of DISC 12 was assessed by internal consistency using Cronbach’s alpha coefficient and estimating the intercorrelation of items and corrected item-total correlations; interrater reliability and test-retest reliability were assessed using intraclass correlation coefficients at the item and subscale levels; and the concurrent validity of the DISC 12-K relative to the other assessment scales was assessed using Spearman’s correlation coefficient.

Results

All of the DISC 12-K subscales had high reliability. The validity was good for the Unfair Treatment and Stopping Self subscales, but only fair for the Overcoming Stigma and Positive Treatment subscales.

Conclusion

The Unfair Treatment and Stopping Self subscales of the new DISC 12-K are reliable and valid measures of stigma in patients with depressive disorders. Future studies are needed to test the validity of this scale in other mental disorders.

Keywords: Discrimination and Stigma Scale (DISC 12), Scale standardization, Reliability, Validity, Depression

INTRODUCTION

Stigma is classically defined as “pertaining to the shame that a person may feel when he or she fails to meet other people’s standards, and to the fear of being discredited—which causes the individual not to reveal his or her shortcomings [1].” Social psychology characterizes stigma as the disapproval of a person based on physical or behavioral characteristics that distinguish them from others. Stigma is frequently attached to individuals with mental disorders and is associated with negative consequences including discrimination and rejection [2]. Traditionally, stigma research has focused on schizophrenia, one of the most stigmatized mental disorders [3]; however, depression has garnered increasing research attention [4-7]. Depression is a common mental disorder and one of the leading causes of disability worldwide [8]. Stigma related to depression is associated with various disadvantages in social participation and vocational integration [7].

Stigma is most often measured in terms of public attitudes toward specific scenarios rather than through the experiences of individuals with mental health problems [9]. The Perceived Devaluation and Discrimination Scale [10] and the Self-stigma of Mental Illness Scale are examples of tests that measure public attitudes [11]. The primary limitation of this approach, i.e., the absence of direct assessment of the stigmatized person’s experience, is that stigma may be overestimated, particularly in terms of problems related to knowledge (ignorance) and to attitudes (prejudice) [2]. Instruments that measure experienced stigma in people with mental health problems include the Consumer Experiences of Stigma Questionnaire [12] and the Rejection Experience Scale [13]. However, most measures of experienced stigma have not been psychometrically validated [14].

The Discrimination and Stigma Scale (DISC 12) was developed to fill this research gap and focuses on the responses and experiences of stigmatized persons [15]. The psychometric properties of the DISC 12, including reliability, validity, acceptability, and feasibility, have been demonstrated [14]. Several studies have used the DISC 12 to assess the overall patterns of experienced discrimination and anticipation [16,17]. The DISC 12 has been translated into several languages and has been used in studies of schizophrenia in 27 countries [15] and of depression studies in 30 countries [7].

No Korean version of the DISC 12 has been available; therefore, we developed and standardized a Korean version of the test (DISC 12-K) for use in Korean patients with depressive disorders.

METHODS

Study protocol

This study is a sub-study of the MAKE Biomarker discovery for Enhancing anTidepressant Treatment Effect and Response (MAKE BETTER) trial, which was undertaken to develop a treatment-response prediction index of biomarkers for patients with depressive disorders. Details of the study have been published as a design paper [18] and registered at cris.nih.go.kr (identifier: KCT0001332). Briefly, the naturalistic 2-year prospective study was designed to identify biomarkers for predicting treatment response in real-world settings. Therefore, participants were enrolled regardless of depression subtype or physical comorbidity. Treatment interventions were performed naturalistically using the type, dose, and regimen of antidepressant and other medications preferred by the patients and prescribed by the clinicians; however, the protocol was guided by pre-planned assessments at given time points. The DISC 12-K was administered 1 year after the initiation of antidepressant therapy to assess the level of stigma experienced in the prior 12 months. Demographic and clinical characteristics were obtained at the same time as assessment tests were administered. The DISC 12-K was re-administered 4 weeks later to assess test-rest reliability. All data were obtained using a structured clinical report form (CRF) completed by clinical research coordinators who were blinded to the treatments. The clinicians were trained by the research psychiatrists on the use of the CRF and data collection methods. The study was approved by the Chonnam National University Hospital Institutional Review Board (CNUH 2013-163).

Participants

Participants in the MAKE BETTER study who attended the outpatient psychiatric department of Chonnam National University Hospital and had been followed for 1 year were consecutively enrolled in our study between November 2013 and August 2017. The research psychiatrists used the MiniInternational Neuropsychiatric Interview (MINI) [19], a structured diagnostic psychiatric interview based on the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition (DSM-IV) criteria, to diagnose depressive disorders [20]. Inclusion criteria were 1) diagnosed with major depressive disorder (MDD), dysthymic disorder, or depressive disorder not otherwise specified (NOS); 2) aged 18-70 years; 3) able to complete questionnaires, understand the objective of the study, and sign the informed consent form. Exclusion criteria were 1) unstable or uncontrolled medical condition; 2) unable to complete the psychiatric assessment or comply with the medication regimen due to a severe physical illness; 3) current or lifetime DSM-IV diagnosis of bipolar disorder, schizophrenia, schizoaffective disorder, schizophreniform disorder, psychotic disorder NOS, or other psychotic disorder; 4) history of organic psychosis, epilepsy, or seizure disorder; 5) hospitalization for any psychiatric diagnosis apart from depressive disorder (e.g., alcohol/drug dependence); and 6) electroconvulsive therapy received for the current depressive episode. All participants reviewed the consent form and provided written informed consent.

Original and Korean versions of the DISC 12

The DISC 12 is a structured interview for assessing discrimination experienced by individuals with a mental disorder. The DISC 12 contains 32 questions concerning aspects of everyday life including work, marriage, parenting, housing, and leisure and religious activities divided into four subscales: 1) Unfair Treatment (items 1-21) measures experienced discrimination (e.g., “Have you been treated unfairly in making or keeping friends?”); 2) Stopping Self (items 22-25) measures anticipated discrimination (e.g., “Have you stopped yourself from applying for work?”); 3) Overcoming Stigma (items 26 and 27) assesses the coping strategies patients use to overcome discrimination (e.g., “Have you been able to use your personal skills or abilities in coping with stigma and discrimination?”); 4) and Positive Treatment (items 28-32) assesses positive treatment received as a result of a mental health problem (e.g., “Have you been treated more positively by your family?”). Responses were rated on a four-point Likert scale (0=not at all, 1=a little, 2=moderately, and 3=a lot) with a “not applicable” response option for items that the patients judged as not relevant to their situation. Scores on the Positive Treatment subscale were reverse coded so that a high score indicated a lack of positive treatment or occasions when the person received help or support due to their mental health problem that was not available to others. The total score for each subscale was generated by counting the number of items on which the score was 1, 2, or 3 (disadvantage scores), with higher scores indicating greater stigma.

Formal permission to translate the DISC-12 into Korean was obtained from its developer (Prof. G. Thornicroft). The standard translation procedure was used, which included forward and backward translation as well as pilot tests for acceptability and feasibility. The final DISC 12-K is available in the online supplement. The DISC 12-K is administered and scored in the same way as the original DISC 12. We used the DISC 12-K to assess experienced stigma during the 12 months after the initiation of antidepressant treatment.

Patient characteristics

The sociodemographic characteristics obtained included age, sex, years of formal education, marital status (currently married or not), cohabitation status (living alone or not), religion (religious affiliation or no religious preference), occupation (currently employed or not), and annual income (above or below $2,000 USD). The clinical characteristics included duration of illnesses, number of previous depressive episodes, history of psychiatric hospitalization, and awareness of illness (agree or disagree with diagnosis).

Assessment scales

Internalized Stigma of Mental Illness (ISMI) scale

The ISMI scale is a 29-item self-report measure designed to assesses mental health service users’ subjective experience of internalized stigma [21]. The scale is composed of five subscales: alienation, stereotype endorsement, perceived discrimination, social withdrawal, and stigma resistance, with higher scores indicating greater stigma. The reliability and validity of the ISMI scale were established in a sample of 127 mental health outpatients [21], and the scale has been formally standardized in Korean [22].

Other psychometric assessment scales

Several psychometric scales were used to assess participants’ symptoms and level of functioning. Depressive symptoms were assessed objectively using the Hamilton Depression Rating Scale (HAMD) [23] and subjectively using the Beck Depression Inventory (BDI) [24]. Level of functioning was assessed using the Social and Occupational Functioning Assessment Scale (SOFAS) [20], self-esteem using the Rosenberg Self-Esteem Scale (RSES) [25], and quality of life using the EuroQol-5D (EQ-5D) [26]. Higher scores on the HAMD, BDI, RSES, and EQ-5D and lower scores on the SOFAS indicated more severe symptoms. All of the assessment scales have been formally standardized in Korean [27-31].

Statistical analysis

Descriptive data are expressed as mean (standard deviation, SD), minimum-maximum values, or as number (percentage), as appropriate. The proportion of “Agree” responses for each DISC 12-K item is expressed as the combined total of the “disadvantage” response categories (a little, moderate, and a lot). Reliability and validity of each DISC 12-K subscale were assessed. Reliability was assessed as internal consistency using Cronbach’s alpha coefficient and by estimating the intercorrelation of items and corrected item-total correlations. The interrater reliability and test-retest reliability were assessed using intraclass correlation coefficients at the item and subscale levels. The concurrent validity of the DISC 12-K with the other assessment scales was assessed using Spearman’s correlation coefficient. All statistical analyses were performed using SPSS software version 21.0 (IBM Corp., Armonk, NY, USA).

RESULTS

Patient characteristics

The study included 230 patients with depressive disorders. Patient sociodemographic and clinical characteristics are shown in Table 1. The proportion of responses for each DISC 12-K item is shown in Table 2. Disadvantage was frequently reported (>20%) on items 1 (making or keeping friends), 3 (dating or intimate relationships), and 7 (family) of the Unfair Treatment subscale; on items 24 (close personal relationship) and 25 (concealed or hidden mental health problem) of the Stopping Self subscale; and on all five items of the Positive Treatment subscale. In contrast, the frequency of reported disadvantage was low (5-6%) for the two items on the Overcoming Stigma subscale. Moreover, on each of the DISC 12-K items, many participants (25.7-59.6%) selected the “not applicable” option. Scores for all of the psychometric scales are shown in Table 3. The mean scores on the HAMD (5.9), BDI (8.0), and SOFAS (80.3) tests indicated that most of the participants were stable.

Reliability analyses

The findings of the reliability analyses of the DISC 12-K subscales are shown in Table 4. Cronbach’s alpha coefficients were high for all of the DISC 12-K subscales (0.919-0.987). The item intercorrelations were high for the Unfair Treatment, Stopping Self, and Overcoming Stigma subscales and slightly low for the Positive Treatment subscale. The corrected itemtotal correlations were high for all subscales. The clinical research coordinators assessed the interrater reliability of the DISC 12-K in a subsample (n=30) of patients. The intraclass correlation coefficients for interrater agreement were high at the individual item and subscale levels. The DISC 12-K was readministered to 200 patients 1 month after they completed the initial test to evaluate test-retest validity. The intraclass correlation coefficients for test-retest stability were high at the item and subscale levels.

Validity analyses

The correlations among DISC 12-K subscale scores and between the DISC 12-K total and other assessment scale scores are shown in Table 5. The only significant correlations were between the Unfair Treatment and Stopping Self subscale scores. Comparison of the DISC 12-K and ISMI scales revealed that the Unfair Treatment and Stopping Self subscale scores were significantly correlated with all of the ISMI subscale scores. Furthermore, a significant inverse relationship was found between the Overcoming Stigma subscale and the ISMI Stigma Resistance subscale. The Positive Treatment subscale was not correlated with any of the ISMI subscales. The DISC 12-K Unfair Treatment and Stopping Self subscale scores were significantly correlated with the HAMD, BDI, SOFAS, EuroQol-5D, and RSES scores; thus, higher stigma scores were associated with more severe pathology. Scores on the Overcoming Stigma and Positive Treatment subscales were significantly correlated with RSES scores, but not with the HAMD, BDI, SOFAS, or EuroQol-5D scores.

DISCUSSION

Our standardization study of the DISC 12-K in Korean patients with depressive disorders found that the four DISC 12-K subscales had high reliability. The validity of the Unfair Treatment and Stopping Self subscales was good; however, the Overcoming Stigma and Positive Treatment subscales had only fair validity.

The DISC 12 is widely used to assess experienced stigma in patients with mental disorders [9]. Although the DISC 12 has been translated into several languages and is frequently used in international research [7,15], only a few validation studies have been conducted [14,32]. Thus, our study is one of the few studies to validate the DISC 12. Cronbach’s alpha coefficients and inter-item correlations were high for all of the subscale scores, indicating that the questionnaire construction was consistent across the subscale items. The interrater reliability was good, suggesting that the DISC 12-K items could be rated uniformly and consistently by assessors. The test-retest reliability was also good, suggesting that the DISC 12-K reflected particular individual characteristics consistently over time.

The scores on the Unfair Treatment and Stopping Self subscales of the DISC 12-K were highly correlated, indicating that the subscale items were based on similar constructs and measured related aspects of stigma. Moreover, the Unfair Treatment and Stopping Self subscales were significantly correlated with all of the ISMI subscale scores and the HAMD, BDI, SOFAS, RSES, and EuroQol-5D scores. These findings indicate that the experienced stigma measured by two DISC 12-K subscales reflects depressive symptoms per se as well as social functioning, self-esteem, and quality of life, which are multi-dimensional aspects of depression. Together, our findings suggest that the Unfair Treatment and Stopping Self subscales of the DISC 12-K are valid instruments for measuring experienced stigma in patients with depressive disorder.

Scores on the Overcoming Stigma and Positive Treatment subscales had good reliability but only fair validity. These subscale scores were not correlated with the Unfair Treatment and Stopping Self subscales scores or with any of the ISMI subscale scores. Although the Overcoming Stigma and Positive Treatment subscale scores were correlated with the RSES score, they were not significantly associated with the HAMD, BDI, SOFAS, or EureQol-5D scores. These findings suggest that the Overcoming Stigma and Positive Treatment subscales are not valid measures of experienced stigma in patients with depressive disorders. Previous studies have reported similar results; only the Unfair Treatment and Stopping Self subscales have been used in international research [7], and a previous DISC 12 validation study found that the Unfair Treatment subscale was the only valid measure of experienced discrimination [14]. Taken together, our findings and those of previous studies support the use of the Unfair Treatment and Stopping Self subscales as valid measures of stigma.

However, it should be noted that the proportion of “disadvantage” responses to individual items in the Overcoming Stigma and Positive Treatment subscales was low (5 items had scores >20%) (Table 3) compared with responses reported previously in patients with depression (10 items scored >20%) [7]. This difference may be explained by the fact that we administered the DISC 12-K when the participants were in the continuation or maintenance phase of antidepressant therapy (i.e., 1 year after the initiation of treatment), so most were stable. However, data concerning the depression severity and antidepressant treatment status of the patients in the previous study was not available. Nevertheless, despite differences in study design and the proportion of “disadvantage” responses, our findings and those of the previous study indicate that higher levels of experienced stigma were associated with poor social functioning.

The strengths of our study include the sample size (n=230), which was larger than those of previous DISC 12 validation studies (n=86-89) [14,32], and the structured research protocol and use of well-established, standardized scales. The limitations of our study include the fact that participants were recruited from a single center, which may limit the generalizability of our findings, although a single-center study has potential strengths in terms of consistency in evaluation and treatment. Furthermore, participants included patients with dysthymic disorder and depressive disorder NOS as well as MDD, whereas previous studies focused on patients with MDD [7,32]. However, our findings were similar to those previously reported, indicating that the DISC 12 is a useful assessment tool for a broad range of depressive disorders.

In summary, the new DISC 12-K is a reliable and valid instrument for assessing stigma in patients with depressive disorders. In particular, the Unfair Treatment and Stopping Self subscales have good reliability and validity. Recently, stigma has been recognized as target for depression treatment, and the effect of stigma reduction on social rehabilitation has received considerable research attention [33]. The DISC 12-K offers a standardized and efficient approach to assessing stigma and helping patients reach treatment goals. We recommend that this instrument be administered as a complementary tool to existing assessment scales, as it may provide unique information that could be critical for comprehensive assessment in depressive patients. Future studies are needed to assess the validity of the DISC 12-K in other mental disorders. In addition, consequences of stigma should also be investigated since stigma may have negative impacts in patients with mental disorders [34].

ACKNOWLEDGEMENTS

The study was funded by a grant of National Research Foundation of Korea Grant (NRF-2019M3C7A1031345) to Professor Jae-Min Kim.

The authors have no potential conflicts of interest to disclose.

Author contribution

Conceptualization: Kyu-On Kim, Jae-Min Kim. Data curation: Kyu-On Kim, Tae-Young Yoo, Nam-Jun Kim. Formal analysis: Kyu-On Kim, Hee-Joon Lee, Min Jhon, Jae-Min Kim. Funding acquisition: Sung-Wan Kim, Jae-Min Kim. Investigation: Kyu-On Kim, Ju-Wan Kim, Hee-Ju Kang, JaeMin Kim. Methodology: Jae-Min Kim. Project administration: Kyu-On Kim, Hee-Ju Kang, Jae-Min Kim. Resources: Hee-Joon Lee, Min Jhon, Ju-Wan Kim. Software: Ju-Wan Kim. Supervision: Jae-Min Kim. Validation: SungWan Kim. Visualization: Kyu-On Kim. Writing—original draft: Kyu-On Kim. Writing—review & editing: Jae-Min Kim.

Table 1.

Patient sociodemographic and clinical characteristics

Socio-demographic characteristics
Age, mean (SD) years	55.3 (12.3)
Gender, N (%) female	160 (69.6)
Education, mean (SD) years	9.6 (4.3)
Marital status, N (%) unmarried	52 (22.6)
Living alone, N (%)	19 (8.3)
Religious observance, N (%)	133 (57.8)
Unemployed status, N (%)	132 (57.4)
Monthly income, N (%) <2,000 USD	108 (47.0)
Clinical characteristics
Duration of illness, mean (SD) months	29.8 (48.4)
Number of depressive episodes, mean (SD)	1.2 (1.5)
History of psychiatric hospitalization, N (%)	20 (8.7)
Agree with diagnosis, N (%)	207 (90.0)

Table 2.

Proportion of responses for each item on the Discrimination and Stigma Scale-Koran version (DISC 12-K)

Item		Disadvantage	No difference	Not applicable
Unfair treatment
	7. Family	94 (40.4)	64 (27.8)	73 (31.7)
	1. Making or keeping friends	60 (26.1)	84 (36.5)	86 (37.4)
	3. Dating or intimate relationships	47 (20.4)	84 (36.5)	99 (43.0)
	21. Avoided or shunned by other people	43 (18.7)	90 (39.1)	97 (42.2)
	9. Keeping a job	38 (16.5)	86 (37.4)	106 (46.5)
	16. Mental health staff	38 (16.5)	94 (40.9)	98 (42.6)
	2. Neighborhood	33 (14.3)	96 (41.7)	101 (43.9)
	13. Social life	30 (13.0)	96 (41.7)	104 (45.2)
	4. Housing	24 (10.4)	101 (43.9)	105 (45.7)
	8. Finding a job	24 (10.4)	93 (40.4)	113 (49.1)
	5. Education	21 (9.1)	92 (40.0)	117 (50.9)
	18. Personal safety and security	15 (6.5)	108 (47.0)	107 (46.5)
	20. Role as a parent	15 (6.5)	108 (47.0)	107 (46.5)
	6. Marriage or divorce	14 (6.1)	97 (42.2)	119 (51.7)
	12. Religious practices	14 (6.1)	104 (45.2)	112 (48.7)
	11. Welfare benefits or disability pensions	13 (5.7)	106 (46.1)	111 (48.3)
	10. Public transport	12 (5.2)	106 (46.1)	112 (48.7)
	15. Physical health	9 (3.9)	114 (49.6)	107 (46.5)
	14. Police	7 (3.0)	108 (47.0)	115 (50.0)
	17. Personal privacy	7 (3.0)	111 (48.3)	112 (48.7)
	19. Starting a family or having children	7 (3.0)	107 (46.5)	116 (50.4)
Stopping self
	24. Close personal relationship	56 (24.3)	88 (38.3)	86 (37.4)
	25. Concealed or hidden mental health problem	53 (23.0)	83 (36.1)	94 (40.9)
	22. Applying for a job	41 (17.8)	85 (37.0)	104 (45.2)
	23. Applying of education or training	20 (8.7)	92 (40.0)	118 (51.3)
Overcoming stigma
	27. Personal skills or abilities	14 (6.1)	85 (37.0)	131 (57.0)
	26. Making friends don’t use mental health services	12 (5.2)	91 (39.6)	127 (55.2)
Positive treatment
	31. In religious activities	104 (45.2)	3 (1.3)	123 (53.5)
	30. In housing	103 (44.8)	1 (0.4)	126 (54.8)
	32. In employment	99 (43.0)	0 (0.0)	131 (57.0)
	28. By family	97 (42.2)	74 (32.2)	59 (25.7)
	29. In getting welfare benefits	93 (40.4)	0 (0.0)	137 (59.6)

Data are expressed as number (%). Items are arranged in descending order of the proportion of total responses represented by the combined “disadvantage” response categories (a little, moderate, and a lot)

Table 3.

Scores on all of the psychometric scales

	Mean (SD)	Min-max
Discrimination and Stigma Scale-Korean version
Unfair treatment	3.9 (3.0)	0-17
Stopping self	1.3 (2.1)	0-11
Overcoming stigma	3.6 (2.8)	0-6
Positive treatment	13.7 (1.6)	7-15
Internalized Stigma of Mental Illness scale
Alienation	11.4 (3.9)	6-22
Stereotype endorsement	11.9 (3.6)	7-21
Discrimination experience	7.9 (2.7)	5-16
Social withdrawal	10.6 (3.9)	6-24
Stigma resistance	11.4 (2.6)	5-20
Other assessment scales
Hamilton Depression Rating Scale	5.9 (5.3)	0-25
Beck Depression Inventory	8.0 (9.3)	0-45
Social and Occupational Functional Assessment Scale	80.3 (12.1)	45-95
Rosenberg Self-Esteem Scale	19.6 (6.0)	3-30
EuroQol-5D	6.4 (1.4)	5-11

Table 4.

Findings of the reliability analyses for the Discrimination and Stigma Scale-Koran version (DISC 12-K) subscales

	Unfair treatment	Stopping self	Overcoming stigma	Positive treatment
Internal consistency (N=230)
Cronbachs’α	0.987	0.932	0.948	0.917
Correlations (N=230)
Intercorrelation of items	0.861-0.968	0.717-0.959	0.928	0.585-0.656
Corrected item-total correlations	0.917	0.811	0.928	0.793
Inter-rater reliability (N=30)
Item level	0.789-1.000	0.792-0.987	0.891-0.963	0.816-0.972
Subscale level	0.891	0.876	0.936	0.882
Test-retest reliability (N=200)
Item level	0.643-0.936	0.590-0.668	0.694-0.710	0.629-0.713
Subscale level	0.828	0.630	0.711	0.684

The interrater and test-retest reliability were calculated using intraclass correlation coefficients

Table 5.

Validity analyses of the Korean version of the Discrimination and Stigma Scale (DISC 12-K) subscales

	Unfair treatment	Stopping self	Overcoming stigma	Positive treatment
DISC 12-K
Unfair treatment	-	-	-	-
Stopping self	0.341^‡	-	-	-
Overcoming stigma	-0.038	-0.071	-	-
Positive treatment	0.058	0.044	0.005	-
Internalized Stigma of Mental Illness scale
Alienation	0.261^‡	0.337^‡	-0.028	0.050
Stereotype endorsement	0.251^‡	0.375^‡	-0.070	-0.037
Discrimination experience	0.286^‡	0.291^‡	-0.023	-0.055
Social withdrawal	0.315^‡	0.348^‡	-0.012	-0.029
Stigma resistance	0.191^†	0.156^*	-0.143^*	0.110
Other assessment scales
Hamilton Depression Rating Scale	0.249^‡	0.161^*	0.033	0.096
Beck Depression Inventory	0.283^‡	0.255^‡	-0.074	0.127
Social and Occupational Functional Assessment Scale	-0.237^‡	-0.191^†	-0.109	-0.072
Rosenberg Self-Esteem Scale	-0.266^‡	-0.237^‡	-0.230^‡	-0.170^*
EuroQol-5D	0.263^‡	0.151^*	0.070	0.067

Data are Spearman’s correlation coefficients (rho).

^* p<0.05,

^† p<0.01,

^‡ p<0.001

REFERENCES

1. Goffman E. Stigma: Notes on the Management of Spoiled Identity. N.J: Prentice-Hall; 1963.

2. Thornicroft G, Rose D, Kassam A, Sartorius N. Stigma: ignorance, prejudice or discrimination? Br J Psychiatry 2007;190:192-193.

3. Corrigan P. On the Stigma of Mental Illness. Washington, DC: American Psychological Association; 2005.

4. Griffi ths KM, Crisp DA, Jorm AF, Christensen H. Does stigma predict a belief in dealing with depression alone? J Affect Disord 2011;132:413-417.

5. Aromaa E, Tolvanen A, Tuulari J, Wahlbeck K. Personal stigma and use of mental health services among people with depression in a general population in Finland. BMC Psychiatry 2011;11:52

6. Brohan E, Gauci D, Sartorius N, Thornicroft G, GAMIAN-Europe Study Group. Self-stigma, empowerment and perceived discrimination among people with bipolar disorder or depression in 13 European countries: the GAMIAN-Europe study. J Affect Disord 2011;129:56-63.

7. Lasalvia A, Zoppei S, Van Bortel T, Bonetto C, Cristofalo D, Wahlbeck K, et al, ASPEN/INDIGO Study Group. Global pattern of experienced and anticipated discrimination reported by people with major depressive disorder: a cross-sectional survey. Lancet 2013;381:55-62.

8. GBD 2017 Disease and Injury Incidence and Prevalence Collaborators. Global, regional, and national incidence, prevalence, and years lived with disability for 354 diseases and injuries for 195 countries and territories, 1990-2017: a systematic analysis for the Global Burden of Disease Study 2017. Lancet 2018;392:1789-1858.

9. Brohan E, Slade M, Clement S, Thornicroft G. Experiences of mental illness stigma, prejudice and discrimination: a review of measures. BMC Health Serv Res 2010;10:80

10. Link BG. Understanding labeling effects in the area of mental disorders: an assessment of the effect of expectations of rejection. Am J Community Psychol 1987;11:261-273.

11. Corrigan PW, Watson AC, Barr L. The self-stigma of mental illness: implications for self-esteem and self-efficacy. J Soc Clin Psychol 2006;25:875-884.

12. Wahl OF. Mental health consumers experience of stigma. Schizophr Bull 1999;25:467-478.

13. Bjorkman T, Svensson B, Lundberg B. Experiences of stigma among people with severe mental illness. Reliability, acceptability and construct validity of the Swedish versions of two stigma scales measuring devaluation/discrimination and rejection experiences. Nord J Psychiatry 2007;61:332-338.

14. Brohan E, Clement S, Rose D, Sartorius N, Slade M, Thornicroft G. Development and psychometric evaluation of the Discrimination and Stigma Scale (DISC). Psychiatry Res 2013;208:33-40.

15. Thornicroft G, Brohan E, Rose D, Sartorius N, Leese M, INDIGO Study Group. Global pattern of experienced and anticipated discrimination against people with schizophrenia: a cross-sectional survey. Lancet 2009;373:408-415.

16. Jeffery D, Clement S, Corker E, Howard LM, Murray J, Thornicroft G. Discrimination in relation to parenthood reported by community psychiatric service users in the UK: a framework analysis. BMC Psychiatry 2013;13:120

17. Corker EA, Beldie A, Brain C, Jakovljevic M, Jarema M, Karamustafalioglu O, et al. Experience of stigma and discrimination reported by people experiencing the first episode of schizophrenia and those with a first episode of depression: the FEDORA project. Int J Soc Psychiatry 2015;61:438-445.

18. Kang HJ, Kim JW, Kim SY, Kim SW, Shin HY, Shin MG, et al. The MAKE Biomarker Discovery for Enhancing anTidepressant Treatment Effect and Response (MAKE BETTER) Study: Design and Methodology. Psychiatry Investig 2018;15:538-545.

19. Sheehan DV, Lecrubier Y, Sheehan KH, Amorim P, Janavs J, Weiller E, et al. The mini-international neuropsychiatric interview (M.I.N.I): the development and validation of a structured diagnostic psychiatric interview for DSM-IV and ICD-10. J Clin Psychiatry 1998;59(Suppl 20):22-33. quiz 34-57.

20. American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders, 4th Edition. Washington DC: American Psychiatric Press; 1994.

21. Ritsher JB, Otilingam PG, Grajales M. Internalized stigma of mental illness: psychometric properties of a new measure. Psychiatry Res 2003;121:31-49.

22. Hwang TY, Lee WK, Han ES, Kwon EJ. A study on the reliability and validity of the Korean Version of Internalized Stigma of Mental Illness Scale (K-ISMI). J Korean Neuropsychiatr Assoc 2006;45:418-426.

23. Hamilton M. A rating scale for depression. J Neurol Neurosurg Psychiatry 1960;23:56-62.

24. Beck AT, Ward CH, Mendelson M, Mock J, Erbaugh J. An inventory for measuring depression. Arch Gen Psychiatry 1961;4:561-571.

25. Rosenberg M. Conceiving the Self. New York: Basic Books; 1979.

26. Rabin R, de Charro F. EQ-5D: a measure of health status from the EuroQol Group. Ann Med 2001;33:337-343.

27. Yi JS, Bae SO, Ahn YM, Park DB, Noh KS, Shin HK, et al. Validity and reliability of the Korean version of the Hamilton Depression Rating Scale (K-HDRS). J Korean Neuropsychiatr Assoc 2005;44:456-465.

28. Lim SY, Lee EJ, Jeong SW, Kim HC, Jeong CH, Jeon TY, et al. The validation study of Beck Depression Scale 2 in Korean version. Anxiety Mood 2011;7:48-53.

29. Lee JY, Cho MJ, Kwon JS. Global assessment of functioning scale and social and occupational functioning scale. Korean J Psychopharmacol 2006;17:122-127.

30. Bae HN, Choi SW, Yu JC, Lee JS, Choi KS. Reliability and validity of the Korean Version of the Rosenberg Self-Esteem Scale(K-RSES) in adult. Mood Emot 2014;12:43-49.

31. Kim SH, Ahn J, Ock M, Shin S, Park J, Luo N, et al. The EQ-5D-5L valuation study in Korea. Qual Life Res 2016;25:1845-1852.

32. Reneses B, Ochoa S, Vila-Badia R, Lopez-Mico C, Fernandez GarciaAndrade R, Rodriguez R, et al. Validation of the spanish version of the discrimination and stigma scale (DISC 12). Actas Esp Psiquiatr 2019;47:137-148.

33. Thornicroft G, Mehta N, Clement S, Evans-Lacko S, Doherty M, Rose D, et al. Evidence for effective interventions to reduce mental-health-related stigma and discrimination. Lancet 2016;387:1123-1132.

34. Alonso J, Buron A, Rojas-Farreras S, de Graaf R, Haro JM, de Girolamo G, et al. Perceived stigma among individuals with common mental disorders. J Affect Disord 2009;118:180-186.