Reliability and Validity of Chinese Version of the Perceived Competence Scale for Disaster Mental Health Workforce: A Cross-Sectional Study
Article information
Abstract
Objective
This study aimed to translate the English version of the Perceived Competence Scale for Disaster Mental Health Workforce (PCS-DMHW) into Chinese, and to test its reliability and validity in Chinese mental health workers.
Methods
With the consent of Professor Choi, Keimyung University, Korea, and the authorization of the scale, the English version of PCS-DMHW was translated, retranslated and culturally debugged to form the Chinese version of PCS-DMHW. The general information questionnaire and the Chinese version of PCS-DMHW scale were used to investigate 706 mental health workforce from 9 tertiary hospitals in Sichuan province in China from March 24, 2020 to April 14, 2020. The Cronbach’s α coefficient was used to evaluate the internal consistency reliability of the scale, and the test-retest correlation coefficient r was used to evaluate the test-retest reliability of the scale. The content validity indexes (CVI) and exploratory factor analysis (EFA) was used separately for evaluating the content validity and structure validity of the scale.
Results
The Cronbach’s α coefficient of the Chinese version of PCS-DMHW total scale, individual competences and organizational competences subscale was 0.978, 0.956, and 0.964, respectively. The test-retest reliability of the total scale, individual competences and organizational competences subscale was 0.949, 0.932 and 0.927, respectively. The item-level CVI of all scale were ranged from 0.833–1.000, the scale-level CVI (S-CVI)/universal agreement of the total scale, individual competences and organizational competences subscale was 0.833, 0.875, and 0.857, respectively, and the S-CVI/average was 0.972, 0.979, and 0.976, respectively. EFA showed that two principal components were extracted from the subscale of individual competences and organizational competences.
Conclusion
The Chinese version of PCS-DMHW has good reliability and validity, and can be widely used in China.
INTRODUCTION
A disaster is an accidental, large-scale event that causes death, injury, and destruction of property, which occurs on average at least once a day on a global scale. In recent years, due to climate change, crustal activity, social change, and population explosion, the incidence of global disasters such as earthquakes, tsunamis, rainstorms, typhoons, viruses, nuclear radiation, and terrorist attacks has been on the rise [1]. According to the report, the number of disasters worldwide has increased roughly four-fold in the past couple of decades, while the number of persons affected by disasters has increased approximately three-fold over the same period [2].
China is one of the most disaster-prone countries in the world due to its diverse climate, complex geographical environment, and fragile ecological conditions. Every disaster will cause huge casualties and property losses [3,4]. After the disaster, no matter the survivors [5], the people in the disaster area [6,7], the bereaved [8], or the first-line rescuers [9,10], all have different degrees of psychological stress reaction. Disasters may have a negative impact on the mental health of victims in the early and midterm [11,12], with symptoms peaking in the first year after the event [13], and even after decades in which the victims still have significant psychological problems [14-16]. Therefore, it is urgent for mental health workers to carry out psychological intervention to rebuild psychological defense mechanism. However, up to now, there is no professional assessment tool in China to measure mental health workers’ professional knowledge and ability to deal with disasters. So it is very important to establish a core competence scale that can accurately and effectively evaluate Chinese mental health workers’ response to disasters.
In November 2019, Yoon and Choi [17], in the Department of Psychology at Keimyung University, South Korea, developed the Perceived Competence Scale for Disaster Mental Health Workforce (PCS-DMHW) (Supplementary Material in the online-only Data Supplement), which covers both personal and organizational capabilities. This was the first disaster competency scale for mental health workforce. The development process was rigorous and the entry content was reasonably designed, with good reliability and validity. The purpose of this study was to sinicize the scale and evaluate its reliability and validity in Chinese mental health workforce in order to provide a tool for evaluating the disaster competence of mental health workforce in China, so as to provide evidence for improving the disaster competence of mental health workforce.
METHODS
The study was approved by the Biomedical Research Ethics Committee, West China Hospital of Sichuan University (Approval number: 2020254).
Study design
Cross-sectional online self-report survey was conducted in 9 tertiary hospitals in Sichuan province in China. The questionnaire was built on a professional questionnaire survey network platform called “Wenjuan Xing” (www.wjx.cn) and then was shared on social media (WeChat). The online questionnaire had three sections: informed consent, general information questionnaire, and the Chinese version of the PCS-DMHW scale were required. After a brief written informed consent at the beginning of the survey, the questionnaire was answered and it took 10 to 15 minutes for the participants to complete the survey.
Participants
In investigation research, the minimum sample size was 5–10 times the number of items on the scale [18], and there were 48 items in PCS-DMHW. According to 10 times the number of items, 480 samples were needed. Taking into account the 20% invalid questionnaire, the number of sample cases was finally determined to be 576. Using convenience sampling method, we contacted department heads in each hospital and invited them to forward our questionnaire to their WeChat group of staff to recruit participants. 733 mental health workers from 9 tertiary hospitals in Sichuan province were investigated from March 24, 2020 to April 14, 2020, among them, 715 participants completed the anonymous online survey and 18 participants completed the offline survey. Inclusion criteria: participants were aged over 18 years; worked in mental health related units; obtained informed consent and volunteered for the study. Exclusion criteria: mental and psychological health-related interns. Since the questionnaire was mainly collected online, the response rate cannot be calculated. And a total of 706 questionnaires were analyzed excluding 27 questionnaires which included unanswered items or founded it unreliable.
Translation and back translation
We contacted professor Choi, the scale developer, to obtain the authorization of the English version of PCS-DMHW scale, and translated the scale into Chinese through translation, reverse translation, cultural adjustment and evaluation.
The translation and back translation process was divided into the following 3 steps.
1) Forward translation: two chinese-speaking and fluent English-speaking translators were responsible for the translation. The results were translation 1 and translation 2. Translator 1 was a master of Medicine, proficient in medical terminology, mainly from a clinical perspective. Translator 2, with a master’s degree in English and no medical background, was engaged in translation work, mainly debugging the universality of language.
2) Synthesis: a Chinese native speaker, a master of Medicine who was fluent in English and not involved in forward translation, conducted a comparative analysis of the two translations (translation 1 and translation 2). After discussion, the original translation, namely draft translation 3, was formed.
3) Retranslation: four translators proficient in both English and Chinese had retranslated the translation for 4 times, forming retranslation 2, retranslation 3, and retranslation 4 and 4 retranslators were all graduate students, in order to avoid information deviation, we found some hidden translation differences in translation 3. Finally, a medical master who was a native Chinese speaker, fluent in English and not involved in the translation and retranslation process, compared the retranslated English version with the original scale to find out the differences and make appropriate modifications to the Chinese version, they were then compared with the original scale and repeated until the Chinese version was basically similar to the original.
Cultural adjustment
According to the characteristics of Chinese culture, some items in the scale were modified to make them conform to the Chinese cultural background, which was called cultural adjustment. The cultural adjustment of this research was divided into the following two parts.
1) Expert consultation: four chief physicians in medical and psychology related work, one deputy chief physician and one deputy chief nurse, two medical English professors with overseas study experience, one statistics professor and all translators formed an expert committee to review the first draft of the Chinese version of the scale from four aspects: semantic equivalence, idiom equivalence, experience equivalence, and conceptual equivalence according to Chinese actual situation and language habits. After debugging, this research revised item 7 of the organizational capability subscale (“I understand the human and material resources that can be used in the community [e.g., medical facilities, religious institutions, service groups],” to “I understand the human and material resources that can be used in the community [e.g., medical facilities, non-profit organization, service groups]”). In addition, related items were proposed in the English version of the PCS-DMHW scale as a supplementary subscale for future research, including three single items, namely “I am able to adequately respond to stress occurring in disaster sites,” “I am able to be attentive to my emotional and physical reactions to disaster and disaster response activities,” and “I am able to prevent burnout and vicarious traumatization by taking care of myself.” In view of the insufficient statistical power of the correlation among these three single items in disaster competence in the original scale, combined with expert opinions, these 3 items were deleted in the process of sinicization in this study.
2) Preliminary investigation: to ensure the suitability, correctness and acceptability of the scale language, 20 native Chinese mental health workers from West China Hospital of Sichuan University were selected for a preliminary investigation. The scale was filled out after informed consent. Each subject completed the scale, then was interviewed and asked if there were any vague, difficult to understand or unpleasant items. The feedback of the subjects on the project was recorded, and the corresponding modification and proofreading were carried out. Finally, the Chinese version of PCS-DMHW scale was produced.
Measures and instruments
General information questionnaire
The general information questionnaire was designed by the researchers themselves, and the content mainly included age, gender, marital status, working years, medical title, and type of mental health workforce.
Chinese version of the PCS-DMHW scale
The Chinese version of the PCS-DMHW scale was used to measure the core competence of mental health workforce in coping with disasters. The scale consists of two subscales, namely, individual competence and organizational competence subscale. The individual competence subscale included 24 items, which was divided into three dimensions: knowledge, skill, and attitude and the organizational competence subscale included 21 items, which was divided into three dimensions: team cooperation, network system, and human resource management. All items were rated on a 5-point scale (strongly disagree, disagree, neither agree nor disagree, agree, strongly agree). The sum of the two subscales was the total score, which represented the disaster competence of mental health workforce.
Data collection
This survey adopted an online survey method. The researcher introduced the research purpose, significance and precautions for filling out the questionnaire to the survey participants through the “Wenjuan Xing,” and conducted the survey after obtaining the informed consent of the survey participants. After the survey was completed, a sample of 50 participants was randomly selected and repeated measurement was performed two weeks later to evaluate the scale’s retest reliability.
Statistical analysis
Excel was used for data entry and SPSS 21.0 (IBM Corp., Armonk, NY, USA) software was used for data statistics. The internal consistency reliability of Cronbach’s α coefficient rating scale and the retest reliability of Pearson correlation coefficient r rating scale with two test scores were used. The content validity index (CVI) was used to evaluate the content validity of the scale, and the content of the item was evaluated by six experts. A 4-level scoring method was adopted, with 1 being “irrelevant,” 2 being “weakly correlated,” 3 being “moderately correlated,” and 4 being “strongly correlated.” The CVI of the scale was calculated according to the expert evaluation results. Exploratory factor analysis (EFA) was used to evaluate the structural validity of the scale. To test the measurement model of the PCS-DMHW scale and to ensure its construct validity, we first computed the Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy and Bartlett test of sphericity (BTS) to determine whether our data were suitable for EFA. The KMO index ranges between 0 and 1, and values above 0.5 are considered suitable for factor analysis. To ensure the suitability of factor analysis, BTS should be statistically significant (p≤0.05), which indicates that sufficient correlations exist among the variables. EFA was conducted to determine which of the scale’s items should be retained. Factors were extracted using principal axis factoring, which uses estimates of communalities on the diagonal in the extraction process, and direct oblimin rotation was used under the assumption of intercorrelations among factors. The common factors representing the scale structure were extracted to obtain the cumulative variance contribution rate of the common factors. The common factors could explain more than 40% of the variation, and the load value of each item in the corresponding factors was high (≥0.4), which was a relatively ideal factor analysis result. p<0.05 was regarded as the statistically significant difference.
RESULTS
General information
Among the 706 participants, 103 were males (14.6%) and 603 were females (85.4%). Age was 20 to 57 years old. The basic information of mental health workforce was as follows, see Table 1.
General information in mental health workforce (N=706)
Reliability
Internal consistency reliability
This study showed that the Cronbach’s α coefficient of the Chinese version of the PCS-DMHW total scale was 0.978, the Cronbach’s α coefficients of the individual competences and organizational competences subscale was 0.956 and 0.964, respectively, and the Cronbach’s α coefficients in all dimensions of the scale range from 0.863 to 0.928, showing good homogeneity.
Test-retest reliability
The test-retest reliability of the Chinese version of the PCS-DMHW individual competences and organizational competences subscale was 0.932 and 0.927, respectively, and the test-retest reliability of the total scale score was 0.949.
Validity
Content validity
Six experts scored the item content of the scale, and the results showed that the item-level CVI (I-CVI) of the Chinese version of the PCS-DMHW total table was 0.833–1.000, the scale-level CVI/universal agreement (S-CVI/UA) was 0.833, and the S-CVI/average (S-CVI/Ave) was 0.972, the individual and organizational competences subscale I-CVI was 0.833–1.000, S-CVI/UA was 0.875 and 0.857, and S-CVI/Ave was 0.979 and 0.976, respectively.
Structure validity
EFA was used to evaluate the structure validity of the Chinese version of the PCS-DMHW scale. EFA of the individual competences subscale showed that BTS results: χ2=12,685.505, p<0.001, and KMO=0.968, the principal axis decomposition method was used to extract two common factors with eigenvalues greater than 1 in factor analysis. The eigenvalues of the 2 common factors were 12.325 and 2.534, respectively, and the cumulative variance contribution rates were 51.36% and 61.91%, respectively. EFA of the organizational competences subscale showed that the results of BTS results: χ2=12,740.070, KMO=0.970, and two common factors were also extracted, the eigenvalues were 12.448 and 1.875, respectively, and the cumulative variance contribution rate were 59.28% and 68.21%, respectively. The factor load value of each item of the individual competences subscale and the organization competences subscale were greater than 0.4. See Tables 2 and 3.
Common factors of individual competences subscale and factor loading coefficients of items
Common factors of organizational competences subscale and factor loading coefficients of items
DISCUSSION
Significance of the Chinese version of the PCS-DMHW scale
Disaster refers to a crisis event characterized by chaos, suddenness and large-scale trauma, including natural disasters and man-made disasters [19]. China is one of the countries with the most serious natural disasters in the world [20]. In order to better deal with and prevent disasters, the core capabilities of disasters have gradually attracted the attention of scholars [21]. Among them, disaster-related mental health, especially post-traumatic stress disorder and depression have become areas of great concern to the academic community [22,23]. Although disaster medicine in our country has developed rapidly in recent years, it still inevitably suffers from the problems of weak foundation, insufficient attention and insufficient teaching of basic education medicine, the disaster medicine specialized reserve talented person is relatively short [24]. Moreover, in the hospital clinic, most of the medical staff obtain the relevant disaster psychological rescue knowledge through self-study and experience accumulation, and the correctness and pertinence of the knowledge are deficient, it may affect the specialization and comprehensiveness of psychological intervention in disaster rescue [25]. A survey was conducted among medical personnel who had participated in disaster rescue (earthquake, traffic accident, fire, etc.) in four hospitals in Shanghai and Sichuan from October 2013 to January 2014 by using the method of objective sampling, it was found that only 63 (25.5%) of the 247 medical personnel had participated in relevant psychological training, and the mental knowledge of disaster rescue was seriously insufficient [25]. There was another study showed that disaster psychological assistance workers need to receive long-term and systematic training to improve their professional knowledge and skills to ensure their competency [26]. Therefore, how to improve the ability of mental health workers to cope with disasters has gradually become one of the future research directions of the psychology department. But in China, it does not have a good assessment tool to measure the capacity of mental health workers in disasters. So it is of great significance to introduce the English version of the PCS-DMHW into China. First, point out the direction for setting the objectives and contents of psychological rescue courses related to disaster medicine undergraduate or graduate education. It is not only necessary to set up courses to improve personal professional psychological knowledge and skills, but also relevant courses to cultivate team communication, collaboration, resource allocation, and organizational leadership. Second, the Chinese version of the PCS-DMHW scale can be used as an evaluation tool for the effect evaluation of disaster psychological rescue continuing education training project, to evaluate whether the students have mastered relevant professional knowledge and skills, and whether they can complete the team psychological rescue work well. Third, as one of the evaluation criteria for the emergency dispatch of psychological rescuers in sudden disaster events, it evaluates whether the tested person is competent for the psychological rescue work of disaster, so as to ensure the professional psychological rescue ability of the rescue team to the greatest extent.
Reliability and validity analysis of the Chinese version of the PCS-DMHW scale
Reliability is mainly used to examine the internal consistency, accuracy and stability of the scale. The higher the reliability coefficient, the small the variation degree caused by random errors in the measurement process. This study used internal consensus reliability and test-retest reliability to evaluate the reliability of the Chinese version of the PCS-DMHW scale. The internal consensus reliability results showed that the Cronbach’s α coefficient of the Chinese version of the PCS-DMHW scale was 0.978, and the Cronbach’s α coefficients of the individual competences and organizational competences subscale was 0.956 and 0.964, respectively, and the Cronbach’s α coefficients of each field of the scale ranged from 0.863 to 0.928, indicating that the scale had good homogeneity. The test-retest reliability results showed that the Chinese version of PCS-DMHW was 0.949, and the scores of the individual competences and organizational competences subscales was 0.932 and 0.927, respectively, indicating that the scale had better cross-time stability.
Validity refers to the extent to which a research tool truly reflects the concept it is intended to study. The higher the degree of reflecting the research concept, the more accurate and valid the measurement results will be. This study evaluated content validity and construct validity. The results showed that the Chinese version of the PCS-DMHW total table was 0.833–1.000, the S-CVI/UA was 0.833, and the S-CVI/Ave was 0.972, the individual and organizational competences subscale I-CVI was 0.833–1.000, S-CVI/UA was 0.875, 0.857, and S-CVI/Ave was 0.979, 0.976, respectively, which showed that the PCS-DMHW project in China was representative and could reflect the disaster capacity of mental health workers. EFA showed that two common factors were extracted from individual and organizational competences subscale, the cumulative variance contribution rate were 61.91% and 68.21%, respectively, and the factor load value were all greater than 0.40, suggesting that the extracted common factors had strong explanatory power and good construct validity. Among them, Item 7 and Item 11 in organizational competences subscale had double-load phenomenon, which indicated that item differentiation should be improved. Although these two entries should have been removed from a statistical point of view, it was decided to keep them and that they belonged to the original dimension respectively. Common factor 1 in the individual competences subscale contains 18 items, which is named “professional quality” in this study, and are specifically divided into three dimensions: professional knowledge, professional skills, and professional qualifications (items 5, 9, 11), in which professional knowledge covers two aspects, including understanding disaster (items 17, 19, 24) and tailored support (items 8, 12, 15), and professional skills cover three aspects, including problem-solving (items 3, 7, 13), communication (items 2, 6, 18), and information sharing (items 1, 10, 14). Common factor 2 in the individual ability scale contains 6 items, which is named “professional attitude” in this study, including two dimensions: calling (items 20, 21, 22) and ethic (items 4, 16, 23). The common factor 1 in the organizational competences subscale is named “management,” which specifically includes network, conflict management (items 5, 14, 15) and leadership management (items 16, 20, 21). The network includes two aspects, namely disaster management (items 10, 12, 13) and linking local resources (items 2, 7, 18). The common factor 2 in the organizational competences subscale is named “team,” which specifically includes cooperation (items 9, 17, 19), communication (items 6, 8, 11), and followship (items 1, 3, 4).
Compared with the original scale, the Chinese version of PCS-DMHW also has good internal consistency reliability and construct validity in mental health workforce. However, there are two major differences in research results. First, the test-retest reliability coefficient for 2-week interval of the Chinese version of individual ability and organizational competences subscale was both 0.9 above. But the test-retest reliability coefficient for 2-week interval of the Korean version of individual competences and organizational ability subscale was only 0.637 and 0.624, respectively. It can be seen that the test-retest reliability of the Chinese version PCS-DMHW scale in this study is relatively high, and shows good stability and consistency across time in this measurement. Second, we found that the common factors extracted by the team of Yoon and Choi was not completely consistent with the theoretical structure when testing the validity of the English PCS-DMHW scale. The Chinese version of the PCS-DMHW scale was different from the original scale in terms of the number of common factors and items. It may be that the team of Yoon and Choi designed the dimensions and items of the PCS-DMHW scale mainly based on some theoretical ideas, and conducted an exploration analysis to analyze the scale structure from the perspective of statistics. To extract common factors, common factors may be different according to different research objects and sample sizes.
Limitations and prospects
The limitations of this study were: 1) The source of the sample was relatively limited: the sample size of this study was relatively sufficient, but the survey population is mainly concentrated in Sichuan, which has certain regional limitations; 2) The survey method was relatively simple: since the survey time was during the COVID-19 epidemic, it was not suitable to face-to-face gathering of the crowd to carry on the investigation, therefore this research adopted the network investigation method, although this method was not limited by time and space, it was more convenient and fast, but it could not completely guarantee the reliability of the survey results and the accuracy of the samples; and 3) The empirical validity needs to be improved: the limitation of this study was that the scale after the Chinese version only proposed the number of common factors and the items covered, but did not propose clear dimensions. Subsequent research will further explore and verify its structural validity. In addition, this study was the first time to introduce the PCS-DMHW scale in China, and there was a lack of disaster competency related scale for mental health workforce in China. It was suggested that face-to-face investigation should be adopted in future research.
In conclusion, the Chinese version of the PCS-DMHW scale had good reliability and validity, with a concise scale structure, simple testing and scoring methods, and was suitable for evaluating the disaster core competence of mental health workforce in China in the context of Chinese culture. The research could be combined with multiple centers to expand its scale, as well as the sampling breadth and hospital level, and further verify the validity and reliability of the scale through clinical practice.
Supplementary Materials
The online-only Data Supplement is available with this article at https://doi.org/10.30773/pi.2022.0133.
The Perceived Competence Scale for Disaster Mental Health Workforce
Notes
Availability of Data and Material
The dataset supporting the conclusions of this article may be available upon request from the lead author and corresponding author.
Conflicts of Interest
The authors have no potential conflicts of interest to disclose.
Author Contributions
Conceptualization: all authors. Data curation: Qingqing Xiao, Xiaozhen Song. Formal analysis: Qingqing Xiao, Xiaozhen Song. Funding acquisition: Qingqing Xiao, Xuehua Huang, Xiandong Meng. Investigation: all authors. Methodology: Qingqing Xiao, Xiaozhen Song. Project administration: Qingqing Xiao, Xuehua Huang. Supervision: Qingqing Xiao, Xuehua Huang. Writing—original draft: Qingqing Xiao, Xiaozhen Song. Writing—review & editing: Xuehua Huang, Xiandong Meng.
Funding Statement
The study received funding include that: Scientific research project of Sichuan Provincial Health Commission (Popularization and application project) (20PJ014); West China Nursing Discipline Development Special Fund project, Sichuan University (HXHL19026); the Popular Science Training Program of Sichuan Provincial Department of Science and Technology (No. 2022JDKP0068).
Acknowledgements
The authors were grateful to all participants and assessors for their contribution.